Bug #2386
closed
Interest loop test reports PASSED on failure
Added by Hila Ben Abraham over 9 years ago. Updated almost 9 years ago.
100%
Description
The test reports PASSED at the end rather than FAILED when fails to connect to the forwarder. It's possible that it reports on the wrong output for other failures as well.
Files
| 20150704114506.tgz (127 KB) 20150704114506.tgz | integ 2150,6 | Junxiao Shi, 07/04/2015 09:30 AM | |
| 20150704214659.tgz (130 KB) 20150704214659.tgz | integ 2150,7 | Junxiao Shi, 07/04/2015 05:00 PM |
Updated by Hila Ben Abraham almost 9 years ago
- Assignee changed from Hila Ben Abraham to Eric Newberry
I haven't touched it in a while, but @Junxiao confirms that this bug is still reproducible. I've got my hands full so I'm reassigning this task.
Updated by Junxiao Shi almost 9 years ago
- Target version set to v0.4
- Estimated time set to 2.00 h
A recent instance of this Bug is 1787,3.
In this test run, NFD isn't installed at all (due to compilation error), but Interest loop scenario still reports PASSED.
Updated by Eric Newberry almost 9 years ago
- Status changed from New to In Progress
Updated by Eric Newberry almost 9 years ago
- Status changed from In Progress to Code review
Updated by Junxiao Shi almost 9 years ago
@Eric, how did you test the effectiveness of the fix?
Updated by Eric Newberry almost 9 years ago
I haven't been able to test it yet. I'm waiting for you to run it and return the results to me.
Updated by Junxiao Shi almost 9 years ago
Reply to note-6:
Developer is responsible for testing the patch before submission.
This includes finding a way to test the scenario.
I'm asking for your method because I'm unsure how to perform the test using the automated system I have.
Updated by Eric Newberry almost 9 years ago
I can't think of a test method, as it would require there to be a problem with NFD. However, the changes were very minor, command was changed to command || exit and two more exit codes were added.
Updated by Junxiao Shi almost 9 years ago
What about: run the scenario without installing the traffic generator. This shall cause the scenario to fail.
Although this is not a comprehensive test of the fix, it's better than not testing at all.
Updated by Eric Newberry almost 9 years ago
I don't have the lab environment set up and as a result, when I run the test, I just get SSH connection timed out errors and the like. How do you run the tests? Is it through a virtualized test environment?
Updated by Junxiao Shi almost 9 years ago
I have explained to Eric on how to setup an integ environment. Eric agrees to test the patch before it can be approved.
Updated by Eric Newberry almost 9 years ago
I tested patch set 6 with the integration testing environment I created for #2436 (without ndn-traffic installed). The tests failed as expected:
======================================================================
FAIL: test_best_route (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 45, in test_best_route
self.fail(self.errormsg[ret])
AssertionError: nfd-traffic failed
======================================================================
FAIL: test_broadcast (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 38, in test_broadcast
self.fail(self.errormsg[ret])
AssertionError: nfd-traffic failed
======================================================================
FAIL: test_client_control (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 52, in test_client_control
self.fail(self.errormsg[ret])
AssertionError: nfd-traffic failed
======================================================================
FAIL: test_ncc (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 59, in test_ncc
self.fail(self.errormsg[ret])
AssertionError: nfd-traffic failed
Updated by Junxiao Shi almost 9 years ago
- File 20150704114506.tgz 20150704114506.tgz added
Updated by Eric Newberry almost 9 years ago
I ran a test on patch set 7 by removing /usr/local/bin/nfd as suggested by Junxiao on Gerrit:
======================================================================
FAIL: test_best_route (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 44, in test_best_route
self.fail(self.errormsg[ret])
AssertionError: add-nexthop failed
======================================================================
FAIL: test_broadcast (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 37, in test_broadcast
self.fail(self.errormsg[ret])
AssertionError: add-nexthop failed
======================================================================
FAIL: test_client_control (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 51, in test_client_control
self.fail(self.errormsg[ret])
AssertionError: add-nexthop failed
======================================================================
FAIL: test_ncc (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 58, in test_ncc
self.fail(self.errormsg[ret])
AssertionError: add-nexthop failed
Updated by Junxiao Shi almost 9 years ago
- File 20150704214659.tgz 20150704214659.tgz added
- Status changed from Code review to Closed
- % Done changed from 0 to 100