Project

General

Profile

Actions
Copy link

Bug #2386

closed

Interest loop test reports PASSED on failure

Added by Hila Ben Abraham over 9 years ago. Updated almost 9 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Eric Newberry
Category:
Integration Tests
Target version:
v0.4
Start date:
01/15/2015
Due date:
% Done:

100%

Estimated time:
2.00 h

Description

The test reports PASSED at the end rather than FAILED when fails to connect to the forwarder. It's possible that it reports on the wrong output for other failures as well.

Files

Download all files
20150704114506.tgz (127 KB) 20150704114506.tgz integ 2150,6 Junxiao Shi, 07/04/2015 09:30 AM
20150704214659.tgz (130 KB) 20150704214659.tgz integ 2150,7 Junxiao Shi, 07/04/2015 05:00 PM
Actions
Copy link
 #1

Updated by Hila Ben Abraham almost 9 years ago

  • Assignee changed from Hila Ben Abraham to Eric Newberry

I haven't touched it in a while, but @Junxiao confirms that this bug is still reproducible. I've got my hands full so I'm reassigning this task.

Actions
Copy link
 #2

Updated by Junxiao Shi almost 9 years ago

  • Target version set to v0.4
  • Estimated time set to 2.00 h

A recent instance of this Bug is 1787,3.
In this test run, NFD isn't installed at all (due to compilation error), but Interest loop scenario still reports PASSED.

Actions
Copy link
 #3

Updated by Eric Newberry almost 9 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #4

Updated by Eric Newberry almost 9 years ago

  • Status changed from In Progress to Code review
Actions
Copy link
 #5

Updated by Junxiao Shi almost 9 years ago

@Eric, how did you test the effectiveness of the fix?

Actions
Copy link
 #6

Updated by Eric Newberry almost 9 years ago

I haven't been able to test it yet. I'm waiting for you to run it and return the results to me.

Actions
Copy link
 #7

Updated by Junxiao Shi almost 9 years ago

Reply to note-6:

Developer is responsible for testing the patch before submission.
This includes finding a way to test the scenario.

I'm asking for your method because I'm unsure how to perform the test using the automated system I have.

Actions
Copy link
 #8

Updated by Eric Newberry almost 9 years ago

I can't think of a test method, as it would require there to be a problem with NFD. However, the changes were very minor, command was changed to command || exit and two more exit codes were added.

Actions
Copy link
 #9

Updated by Junxiao Shi almost 9 years ago

What about: run the scenario without installing the traffic generator. This shall cause the scenario to fail.

Although this is not a comprehensive test of the fix, it's better than not testing at all.

Actions
Copy link
 #10

Updated by Eric Newberry almost 9 years ago

I don't have the lab environment set up and as a result, when I run the test, I just get SSH connection timed out errors and the like. How do you run the tests? Is it through a virtualized test environment?

Actions
Copy link
 #11

Updated by Junxiao Shi almost 9 years ago

I have explained to Eric on how to setup an integ environment. Eric agrees to test the patch before it can be approved.

Actions
Copy link
 #12

Updated by Eric Newberry almost 9 years ago

I tested patch set 6 with the integration testing environment I created for #2436 (without ndn-traffic installed). The tests failed as expected:

======================================================================
FAIL: test_best_route (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 45, in test_best_route
    self.fail(self.errormsg[ret])
AssertionError: nfd-traffic failed

======================================================================
FAIL: test_broadcast (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 38, in test_broadcast
    self.fail(self.errormsg[ret])
AssertionError: nfd-traffic failed

======================================================================
FAIL: test_client_control (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 52, in test_client_control
    self.fail(self.errormsg[ret])
AssertionError: nfd-traffic failed

======================================================================
FAIL: test_ncc (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 59, in test_ncc
    self.fail(self.errormsg[ret])
AssertionError: nfd-traffic failed
Actions
Copy link
 #14

Updated by Eric Newberry almost 9 years ago

I ran a test on patch set 7 by removing /usr/local/bin/nfd as suggested by Junxiao on Gerrit:

======================================================================
FAIL: test_best_route (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 44, in test_best_route
    self.fail(self.errormsg[ret])
AssertionError: add-nexthop failed

======================================================================
FAIL: test_broadcast (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 37, in test_broadcast
    self.fail(self.errormsg[ret])
AssertionError: add-nexthop failed

======================================================================
FAIL: test_client_control (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 51, in test_client_control
    self.fail(self.errormsg[ret])
AssertionError: add-nexthop failed

======================================================================
FAIL: test_ncc (test_interest_loop.test_interest_loop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/integration-tests/test_interest_loop/test_interest_loop.py", line 58, in test_ncc
    self.fail(self.errormsg[ret])
AssertionError: add-nexthop failed
Actions
Copy link
 #15

Updated by Junxiao Shi almost 9 years ago

Actions
Copy link

Also available in: Atom PDF