Project

General

Profile

Actions

Bug #2386

closed

Interest loop test reports PASSED on failure

Added by Hila Ben Abraham almost 10 years ago. Updated over 9 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Integration Tests
Target version:
Start date:
01/15/2015
Due date:
% Done:

100%

Estimated time:
2.00 h

Description

The test reports PASSED at the end rather than FAILED when fails to connect to the forwarder. It's possible that it reports on the wrong output for other failures as well.


Files

20150704114506.tgz (127 KB) 20150704114506.tgz integ 2150,6 Junxiao Shi, 07/04/2015 09:30 AM
20150704214659.tgz (130 KB) 20150704214659.tgz integ 2150,7 Junxiao Shi, 07/04/2015 05:00 PM
Actions #1

Updated by Hila Ben Abraham over 9 years ago

  • Assignee changed from Hila Ben Abraham to Eric Newberry

I haven't touched it in a while, but @Junxiao confirms that this bug is still reproducible. I've got my hands full so I'm reassigning this task.

Actions #2

Updated by Junxiao Shi over 9 years ago

  • Target version set to v0.4
  • Estimated time set to 2.00 h

A recent instance of this Bug is 1787,3.
In this test run, NFD isn't installed at all (due to compilation error), but Interest loop scenario still reports PASSED.

Actions #3

Updated by Eric Newberry over 9 years ago

  • Status changed from New to In Progress
Actions #4

Updated by Eric Newberry over 9 years ago

  • Status changed from In Progress to Code review
Actions #5

Updated by Junxiao Shi over 9 years ago

@Eric, how did you test the effectiveness of the fix?

Actions #6

Updated by Eric Newberry over 9 years ago

I haven't been able to test it yet. I'm waiting for you to run it and return the results to me.

Actions #7

Updated by Junxiao Shi over 9 years ago

Reply to note-6:

Developer is responsible for testing the patch before submission.
This includes finding a way to test the scenario.

I'm asking for your method because I'm unsure how to perform the test using the automated system I have.

Actions #8

Updated by Eric Newberry over 9 years ago

I can't think of a test method, as it would require there to be a problem with NFD. However, the changes were very minor, command was changed to command || exit and two more exit codes were added.

Actions #9

Updated by Junxiao Shi over 9 years ago

What about: run the scenario without installing the traffic generator. This shall cause the scenario to fail.

Although this is not a comprehensive test of the fix, it's better than not testing at all.

Actions #10

Updated by Eric Newberry over 9 years ago

I don't have the lab environment set up and as a result, when I run the test, I just get SSH connection timed out errors and the like. How do you run the tests? Is it through a virtualized test environment?

Actions #11

Updated by Junxiao Shi over 9 years ago

I have explained to Eric on how to setup an integ environment. Eric agrees to test the patch before it can be approved.

Actions #12

Updated by Eric Newberry over 9 years ago

I tested patch set 6 with the integration testing environment I created for #2436 (without ndn-traffic installed). The tests failed as expected:

======================================================================
FAIL: test_best_route (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 45, in test_best_route
    self.fail(self.errormsg[ret])
AssertionError: nfd-traffic failed

======================================================================
FAIL: test_broadcast (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 38, in test_broadcast
    self.fail(self.errormsg[ret])
AssertionError: nfd-traffic failed

======================================================================
FAIL: test_client_control (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 52, in test_client_control
    self.fail(self.errormsg[ret])
AssertionError: nfd-traffic failed

======================================================================
FAIL: test_ncc (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 59, in test_ncc
    self.fail(self.errormsg[ret])
AssertionError: nfd-traffic failed
Actions #14

Updated by Eric Newberry over 9 years ago

I ran a test on patch set 7 by removing /usr/local/bin/nfd as suggested by Junxiao on Gerrit:

======================================================================
FAIL: test_best_route (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 44, in test_best_route
    self.fail(self.errormsg[ret])
AssertionError: add-nexthop failed

======================================================================
FAIL: test_broadcast (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 37, in test_broadcast
    self.fail(self.errormsg[ret])
AssertionError: add-nexthop failed

======================================================================
FAIL: test_client_control (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 51, in test_client_control
    self.fail(self.errormsg[ret])
AssertionError: add-nexthop failed

======================================================================
FAIL: test_ncc (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 58, in test_ncc
    self.fail(self.errormsg[ret])
AssertionError: add-nexthop failed
Actions #15

Updated by Junxiao Shi over 9 years ago

Actions

Also available in: Atom PDF