Project

General

Profile

Actions

Task #2589

closed

CI: enable AddressSanitizer for unit tests

Added by Davide Pesavento over 9 years ago. Updated about 8 years ago.

Status:
Closed
Priority:
Normal
Category:
Build
Target version:
Start date:
Due date:
% Done:

100%

Estimated time:

Description

Build and run unit tests with -fsanitize=address on the Ubuntu-16.04 slave. This includes adding a configure switch to wscript to make it easier to enable the feature.


Related issues 10 (0 open10 closed)

Related to NLSR - Task #4206: Turn on address sanitizer builds for NLSRClosed08/08/2017

Actions
Blocked by ndn-cxx - Bug #2523: UtilSignal/DisconnectSelfInHandler use-after-freeClosedJunxiao Shi

Actions
Blocked by ndn-cxx - Bug #2313: Undefined behavior in UtilSignal::DestructInHandler test caseClosedJunxiao Shi

Actions
Blocked by ndn-cxx - Bug #2149: Heap-use-after-free in InMemoryStorageClosedJiewen Tan

Actions
Blocked by NFD - Bug #2151: FaceManager/TestFireInterestFilter heap buffer overflowClosed

Actions
Blocked by ndn-cxx - Bug #2307: Global buffer overflow in TestName::ImplictSha256Digest test caseClosedJunxiao Shi

Actions
Blocked by ndn-cxx - Bug #2653: UtilFaceUri/CanonizeEmptyCallback triggers use-after-free in dns::ResolverClosedDavide Pesavento

Actions
Blocked by NFD - Bug #3319: TestFibUpdates/EraseFace/WithInheritedFace use-after-freeClosedVince Lehman

Actions
Blocked by ndn-cxx - Bug #3727: OBufferStream destructor use-after-freeClosedJunxiao Shi

Actions
Blocked by NFD - Bug #3787: Rib::erase use-after-freeClosedDavide Pesavento

Actions
Actions #1

Updated by Davide Pesavento over 9 years ago

  • Blocked by Bug #2523: UtilSignal/DisconnectSelfInHandler use-after-free added
Actions #2

Updated by Davide Pesavento over 9 years ago

  • Blocked by Bug #2313: Undefined behavior in UtilSignal::DestructInHandler test case added
Actions #3

Updated by Davide Pesavento over 9 years ago

  • Blocked by Bug #2148: UtilFaceUri/Canonize{Tcp,Udp} test cases randomly time out on Ubuntu 14.10 added
Actions #4

Updated by Davide Pesavento over 9 years ago

  • Blocked by Bug #2149: Heap-use-after-free in InMemoryStorage added
Actions #5

Updated by Davide Pesavento over 9 years ago

  • Blocked by Bug #2151: FaceManager/TestFireInterestFilter heap buffer overflow added
Actions #6

Updated by Davide Pesavento over 9 years ago

  • Blocked by Bug #2307: Global buffer overflow in TestName::ImplictSha256Digest test case added
Actions #7

Updated by Junxiao Shi over 9 years ago

The wscript option can be added first, so that developers can test with AddressSanitizer easily.

Enabling on Jenkins needs to wait until all blocking issues are closed.

Actions #8

Updated by Davide Pesavento over 9 years ago

  • Blocked by deleted (Bug #2148: UtilFaceUri/Canonize{Tcp,Udp} test cases randomly time out on Ubuntu 14.10)
Actions #9

Updated by Davide Pesavento over 9 years ago

  • Blocked by Bug #2653: UtilFaceUri/CanonizeEmptyCallback triggers use-after-free in dns::Resolver added
Actions #10

Updated by Davide Pesavento over 9 years ago

Notes on some ASAN_OPTIONS to use:

  • we have to disable the LeakSanitizer with detect_leaks=0
  • consider enabling detect_stack_use_after_return and check_initialization_order
Actions #11

Updated by Davide Pesavento about 9 years ago

  • Description updated (diff)
Actions #12

Updated by Davide Pesavento about 9 years ago

  • Blocked by Bug #3319: TestFibUpdates/EraseFace/WithInheritedFace use-after-free added
Actions #13

Updated by Davide Pesavento over 8 years ago

  • Status changed from New to In Progress
  • Assignee set to Davide Pesavento
  • % Done changed from 0 to 50

http://gerrit.named-data.net/2821 implements the first part of this task (configure switch to enable the sanitizer).

I'm unsure about the desired interface though. The current patch has a single option --sanitize=<list of sanitizers to enable>. An alternative would be implementing a separate option for each sanitizer, e.g.: --enable-address-sanitizer, --enable-thread-sanitizer, and so on (or shorthands: --enable-asan, --enable-tsan, ...). I'm also unsure about --enable... vs --with.... We seem to be using both but I don't know if there's a semantic difference.

Actions #14

Updated by Davide Pesavento over 8 years ago

  • Description updated (diff)
Actions #15

Updated by Davide Pesavento over 8 years ago

Davide Pesavento wrote:

I'm unsure about the desired interface though. The current patch has a single option --sanitize=<list of sanitizers to enable>. An alternative would be implementing a separate option for each sanitizer, e.g.: --enable-address-sanitizer, --enable-thread-sanitizer, and so on (or shorthands: --enable-asan, --enable-tsan, ...). I'm also unsure about --enable... vs --with.... We seem to be using both but I don't know if there's a semantic difference.

Any feedback on this design question? I cannot proceed if we don't agree on the "interface".

Actions #16

Updated by Junxiao Shi over 8 years ago

20160607 conference call decides to use --with-sanitizer=sanitizer1,sanitizer2.

Actions #17

Updated by Davide Pesavento about 8 years ago

  • Blocked by Bug #3727: OBufferStream destructor use-after-free added
Actions #18

Updated by Davide Pesavento about 8 years ago

https://gerrit.named-data.net/2821 patch set 3 implements the syntax in note-16

Actions #19

Updated by Junxiao Shi about 8 years ago

I've tested https://gerrit.named-data.net/3053 patchset1 and it's effective.

I added this to the main function of ndnsec:

  char *x = (char*)malloc(10 * sizeof(char*));
  free(x);
  return x[5];

And compile with:

CXX=clang++ ./waf configure --with-sanitizer=address
./waf

Output:

vagrant@m0212:~/clang/ndn-cxx$ build/bin/ndnsec
=================================================================
==13560==ERROR: AddressSanitizer: heap-use-after-free on address 0x60700000dd85 at pc 0x50846b bp 0x7fff90ed86a0 sp 0x7fff90ed8698
READ of size 1 at 0x60700000dd85 thread T0
==13560==WARNING: Trying to symbolize code, but external symbolizer is not initialized!
    #0 0x50846a (/home/vagrant/clang/ndn-cxx/build/bin/ndnsec+0x50846a)
    #1 0x7f4bd0cc5f44 (/lib/x86_64-linux-gnu/libc.so.6+0x21f44)
    #2 0x4a61fc (/home/vagrant/clang/ndn-cxx/build/bin/ndnsec+0x4a61fc)

0x60700000dd85 is located 5 bytes inside of 80-byte region [0x60700000dd80,0x60700000ddd0)
freed by thread T0 here:
    #0 0x48ff99 (/home/vagrant/clang/ndn-cxx/build/bin/ndnsec+0x48ff99)
    #1 0x50843a (/home/vagrant/clang/ndn-cxx/build/bin/ndnsec+0x50843a)
    #2 0x7f4bd0cc5f44 (/lib/x86_64-linux-gnu/libc.so.6+0x21f44)

previously allocated by thread T0 here:
    #0 0x490119 (/home/vagrant/clang/ndn-cxx/build/bin/ndnsec+0x490119)
    #1 0x50842f (/home/vagrant/clang/ndn-cxx/build/bin/ndnsec+0x50842f)
    #2 0x7f4bd0cc5f44 (/lib/x86_64-linux-gnu/libc.so.6+0x21f44)

SUMMARY: AddressSanitizer: heap-use-after-free ??:0 ??
Shadow bytes around the buggy address:
  0x0c0e7fff9b60: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff9b70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff9b80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff9b90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff9ba0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c0e7fff9bb0:[fd]fd fd fd fd fd fd fd fd fd fa fa fa fa 00 00
  0x0c0e7fff9bc0: 00 00 00 00 00 00 00 00 fa fa fa fa 00 00 00 00
  0x0c0e7fff9bd0: 00 00 00 00 00 00 fa fa fa fa 00 00 00 00 00 00
  0x0c0e7fff9be0: 00 00 00 00 fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c0e7fff9bf0: 00 00 fa fa fa fa fd fd fd fd fd fd fd fd fd fa
  0x0c0e7fff9c00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:     fa
  Heap right redzone:    fb
  Freed heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe
==13560==ABORTING
Actions #20

Updated by Junxiao Shi about 8 years ago

I've tested https://gerrit.named-data.net/2821 patchset6 and it's effective.

I added this to the main function of nfd-autoreg:

  char *x = (char*)malloc(10 * sizeof(char*));
  free(x);
  return x[5];

And compile with:

CXX=clang++ ./waf configure --with-sanitizer=address
./waf

Output:

vagrant@m0212:~/clang/NFD$ build/bin/nfd-autoreg 
=================================================================
==14079==ERROR: AddressSanitizer: heap-use-after-free on address 0x60700000dc35 at pc 0x49f76b bp 0x7ffc11852600 sp 0x7ffc118525f8
READ of size 1 at 0x60700000dc35 thread T0
==14079==WARNING: Trying to symbolize code, but external symbolizer is not initialized!
    #0 0x49f76a (/home/vagrant/clang/NFD/build/bin/nfd-autoreg+0x49f76a)
    #1 0x7f8029e9df44 (/lib/x86_64-linux-gnu/libc.so.6+0x21f44)
    #2 0x49f65c (/home/vagrant/clang/NFD/build/bin/nfd-autoreg+0x49f65c)

0x60700000dc35 is located 5 bytes inside of 80-byte region [0x60700000dc30,0x60700000dc80)
freed by thread T0 here:
    #0 0x4893f9 (/home/vagrant/clang/NFD/build/bin/nfd-autoreg+0x4893f9)
    #1 0x49f73a (/home/vagrant/clang/NFD/build/bin/nfd-autoreg+0x49f73a)
    #2 0x7f8029e9df44 (/lib/x86_64-linux-gnu/libc.so.6+0x21f44)

previously allocated by thread T0 here:
    #0 0x489579 (/home/vagrant/clang/NFD/build/bin/nfd-autoreg+0x489579)
    #1 0x49f72f (/home/vagrant/clang/NFD/build/bin/nfd-autoreg+0x49f72f)
    #2 0x7f8029e9df44 (/lib/x86_64-linux-gnu/libc.so.6+0x21f44)

SUMMARY: AddressSanitizer: heap-use-after-free ??:0 ??
Shadow bytes around the buggy address:
  0x0c0e7fff9b30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff9b40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff9b50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff9b60: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff9b70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c0e7fff9b80: fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd fd fd
  0x0c0e7fff9b90: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 fa fa
  0x0c0e7fff9ba0: fa fa 00 00 00 00 00 00 00 00 00 00 fa fa fa fa
  0x0c0e7fff9bb0: 00 00 00 00 00 00 00 00 00 00 fa fa fa fa 00 00
  0x0c0e7fff9bc0: 00 00 00 00 00 00 00 00 fa fa fa fa 00 00 00 00
  0x0c0e7fff9bd0: 00 00 00 00 00 00 fa fa fa fa 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:     fa
  Heap right redzone:    fb
  Freed heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe
==14079==ABORTING
Actions #21

Updated by Junxiao Shi about 8 years ago

But another test fails:

vagrant@m0212:~/clang/NFD$ clang -v
Ubuntu clang version 3.4-1ubuntu3 (tags/RELEASE_34/final) (based on LLVM 3.4)
Target: x86_64-pc-linux-gnu
Thread model: posix
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8.4
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9.3
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8.4
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9.3
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8
vagrant@m0212:~/clang/NFD$ CXX=clang++ ./waf configure --with-sanitizer=address,thread
Setting top to                           : /home/vagrant/clang/NFD 
Setting out to                           : /home/vagrant/clang/NFD/build 
Checking for 'g++' (C++ compiler)        : not found 
Checking for 'clang++' (C++ compiler)    : clang++ 
Checking supported CXXFLAGS              : -std=c++11 
Checking supported CXXFLAGS              : -O2 -g -pedantic -Wall -Wextra -Wno-unused-parameter -fcolor-diagnostics 
Checking for std::is_default_constructible : yes 
Checking for std::is_move_constructible    : yes 
Checking if compiler supports -fsanitize=address : yes 
Checking if compiler supports -fsanitize=thread  : no 
thread sanitizer is not supported by the current compiler
(complete log in /home/vagrant/clang/NFD/build/config.log)

Tail of build.log says:

------------------------------------------------
Checking if compiler supports -fsanitize=thread
==>

int main(int argc, char **argv) {
    (void)argc; (void)argv;
    return 0;
}

<==
[1/2] Compiling build/.conf_check_49f49e323237e0cb97694ad21317f6c3/test.cpp

['clang++', '-O2', '-g', '-pedantic', '-Wall', '-Wextra', '-Wno-unused-parameter', '-fcolor-diagnostics', '-std=c++11', '-fsanitize=address', '-fno-omit-frame-pointer', '-Werror', '-fsanitize=thread', '-fno-omit-frame-pointer', '-DNDEBUG', '-DHAVE_IS_DEFAULT_CONSTRUCTIBLE=1', '-DHAVE_IS_MOVE_CONSTRUCTIBLE=1', '../test.cpp', '-c', '-o', '/home/vagrant/clang/NFD/build/.conf_check_49f49e323237e0cb97694ad21317f6c3/testbuild/test.cpp.1.o']
err: clang: error: invalid argument '-fsanitize=address' not allowed with '-fsanitize=thread'

from /home/vagrant/clang/NFD: Test does not build: Traceback (most recent call last):
  File "/home/vagrant/clang/NFD/.waf-1.8.9-8a9ccbc1c5d3936b0b08e972e4883a9a/waflib/Configure.py", line 343, in run_build
    bld.compile()
  File "/home/vagrant/clang/NFD/.waf-1.8.9-8a9ccbc1c5d3936b0b08e972e4883a9a/waflib/Build.py", line 184, in compile
    raise Errors.BuildError(self.producer.error)
BuildError: Build failed
 -> task in 'testprog' failed (exit status 1): 
    {task 139870019563792: cxx test.cpp -> test.cpp.1.o}
['clang++', '-O2', '-g', '-pedantic', '-Wall', '-Wextra', '-Wno-unused-parameter', '-fcolor-diagnostics', '-std=c++11', '-fsanitize=address', '-fno-omit-frame-pointer', '-Werror', '-fsanitize=thread', '-fno-omit-frame-pointer', '-DNDEBUG', '-DHAVE_IS_DEFAULT_CONSTRUCTIBLE=1', '-DHAVE_IS_MOVE_CONSTRUCTIBLE=1', '../test.cpp', '-c', '-o', '/home/vagrant/clang/NFD/build/.conf_check_49f49e323237e0cb97694ad21317f6c3/testbuild/test.cpp.1.o']

from /home/vagrant/clang/NFD: The configuration failed
no
from /home/vagrant/clang/NFD: thread sanitizer is not supported by the current compiler

This appears to be a compiler limitation rather than problem with the waf tool.

Actions #22

Updated by Davide Pesavento about 8 years ago

Junxiao Shi wrote:

But another test fails:

vagrant@m0212:~/clang/NFD$ clang -v
Ubuntu clang version 3.4-1ubuntu3 (tags/RELEASE_34/final) (based on LLVM 3.4)
[...]
Checking if compiler supports -fsanitize=thread  : no 
thread sanitizer is not supported by the current compiler

[...]
This appears to be a compiler limitation rather than problem with the waf tool.

Yep, this is expected, your clang version is too old.

Actions #23

Updated by Davide Pesavento about 8 years ago

==13560==WARNING: Trying to symbolize code, but external symbolizer is not initialized!

If you get this warning, you should install llvm-symbolizer, see http://clang.llvm.org/docs/AddressSanitizer.html#symbolizing-the-reports

Actions #24

Updated by Davide Pesavento about 8 years ago

  • % Done changed from 50 to 70
Actions #25

Updated by Davide Pesavento about 8 years ago

  • Blocked by Bug #3787: Rib::erase use-after-free added
Actions #26

Updated by Davide Pesavento about 8 years ago

  • Status changed from In Progress to Code review
  • % Done changed from 70 to 100

Patches to enable ASan for unit tests have been uploaded for ndn-cxx, NFD, ndn-tools.

NOTE: ASan has not been enabled on the following platforms:

  • OSX-10.9, because the installed clang doesn't support -fsanitize=address
  • OSX-10.11, because the stack unwinder seems to hang/deadlock while reporting an ASan error
Actions #27

Updated by Davide Pesavento about 8 years ago

  • Status changed from Code review to Closed
  • Target version set to v0.5
Actions #28

Updated by Ashlesh Gawande about 7 years ago

  • Related to Task #4206: Turn on address sanitizer builds for NLSR added
Actions

Also available in: Atom PDF