Bug #3750
closed
xUnit test reports broken on Ubuntu slaves
100%
Description
It's simply not working. Nothing is published on the build page (e.g. http://jenkins.named-data.net/job/NFD/4165/OS=Ubuntu-14.04-64bit/), and the console output shows xUnit-related errors towards the end.
[xUnit] [INFO] - Starting to record.
[xUnit] [INFO] - Processing BoostTest-1.x (default)
[xUnit] [INFO] - [BoostTest-1.x (default)] - No test report file(s) were found with the pattern 'build/xunit*.xml' relative to '/home/jenkins/workspace/NFD/OS/Ubuntu-14.04-64bit' for the testing framework 'BoostTest-1.x (default)'. Did you enter a pattern relative to the correct directory? Did you generate the result report(s) for 'BoostTest-1.x (default)'?
[xUnit] [WARNING] - No test reports found for the metric 'BoostTest' with the resolved pattern 'build/xunit*.xml'.
[xUnit] [INFO] - Skipping the metric tool processing.
[xUnit] [INFO] - There are errors when processing test results.
[xUnit] [INFO] - Skipping tests recording.
This issue affects all Ubuntu slaves. OSX slaves do not seem to be affected.
Updated by Eric Newberry over 7 years ago
- Status changed from New to In Progress
The current Jenkins scripts run the unit tests with the "--log_sink2" option. However, I could find no documentation of this option and this command produced no output to the reference file when I ran it manually. When I removed the '2', changing the option to "--log_sink" (which is documented), the output appears to be produced correctly.
Updated by Eric Newberry over 7 years ago
It appears that the "--log_format2" option needs to be changed to "--log_format" as well. I'm currently compiling a list of projects affected by this issue.
Updated by Davide Pesavento over 7 years ago
That doesn't explain why OS X slaves are unaffected...
Updated by Davide Pesavento over 7 years ago
--log_format2 and --log_sink2 are custom options, added by us, see https://github.com/named-data/NFD/blob/master/tests/main.cpp
Updated by Davide Pesavento over 7 years ago
Is $XUNIT set in the jenkins environment? (https://github.com/named-data/NFD/blob/master/.jenkins.d/20-tests.sh#L28)
Updated by Eric Newberry over 7 years ago
Davide Pesavento wrote:
Is
$XUNITset in the jenkins environment? (https://github.com/named-data/NFD/blob/master/.jenkins.d/20-tests.sh#L28)
The logging options are only included on the command line if that's the case. It's set in the build shell like so:
XUNIT=1 ./.jenkins
I looked at the help for the unit tests (unit-tests-core --help) and both --log_sink2 and --log_format2 are not listed. As such, I don't believe our customizations are being compiled into the tests on Ubuntu.
@Alex Could you post the help message from one of the OS X nodes for comparison?
Updated by Davide Pesavento over 7 years ago
Eric Newberry wrote:
I looked at the help for the unit tests (
unit-tests-core --help) and both--log_sink2and--log_format2are not listed.
That's expected I think, only Boost.Test built-in options are shown in --help output. However, note the following behavior:
davide@ndnbox:~/NFD$ ./build/unit-tests-core --log_sink2
ERROR: the required argument for option '--log_sink2' is missing
--log_format2 arg Type of second log formatter: HRF or XML
--log_sink2 arg (=-) Second log sink, - for stdout
Test setup error: std::runtime_error: test module initialization failed
davide@ndnbox:~/NFD$ ./build/unit-tests-core -- --log_sink2
Running 63 test cases...
[...]
Updated by Davide Pesavento over 7 years ago
- Assignee changed from Eric Newberry to Alex Afanasyev
Seems to be a regression introduced in NFD commit 3152abac673382b13c39e7df46d830ec9abfc3e0. Other projects are very likely affected as well. Reassigning to Alex, author of the commit.
Updated by Davide Pesavento over 7 years ago
- Related to Bug #3417: Boost 1.60 breaks secondary log output in unit tests added
Updated by Alex Afanasyev over 7 years ago
Huh.. For Boost < 1.60, we need to move -- in the command line :(. How did I not notice it in the original issue.
Should we make a fix as a separate commit(s)?
Updated by Davide Pesavento over 7 years ago
Alex Afanasyev wrote:
Huh.. For Boost < 1.60, we need to move
--in the command line :(. How did I not notice it in the original issue.
meh, do we need yet another conditional in the CI script? is there no way to have a command line compatible with all versions?
Should we make a fix as a separate commit(s)?
I'm confused, separate from what?
Updated by Alex Afanasyev over 7 years ago
Huh.. For Boost < 1.60, we need to move
--in the command line :(. How did I not notice it in the original issue.meh, do we need yet another conditional in the CI script? is there no way to have a command line compatible with all versions?
I thought -- would be a compatible syntax. But from what I see, the old version, simply ignores everything after -- and the new version will complain for any non-standard parameter before --
Should we make a fix as a separate commit(s)?
I'm confused, separate from what?
Separate from https://gerrit.named-data.net/#/c/3079/
Updated by Davide Pesavento over 7 years ago
And boost 1.62.0 (should be released in a few weeks) will change things again...
http://www.boost.org/doc/libs/master/libs/test/doc/html/boost_test/change_log.html
Updated by Alex Afanasyev over 7 years ago
So, the good part of the new change is that we would not need any custom code to have multiple output types. The bad part that we will need 3 customized ways to run the unit tests... One for old boost, one for boost 1.60-1.61, and finally for boost >1.62.
I'll try to make a try for this.
Updated by Alex Afanasyev over 7 years ago
- Status changed from In Progress to Code review
https://gerrit.named-data.net/#/c/3128/, but it needs a test with boost 1.62 as soon as it is released (should be very soon) or at least with release candidates.
Updated by Davide Pesavento over 7 years ago
- Target version set to v0.5
- % Done changed from 0 to 70
Updated by Davide Pesavento over 7 years ago
- Status changed from Code review to Closed
- % Done changed from 70 to 100