NFD

The wscript option can be added first, so that developers can test with AddressSanitizer easily.

Enabling on Jenkins needs to wait until all blocking issues are closed.

Notes on some ASAN_OPTIONS to use:

• we have to disable the LeakSanitizer with detect_leaks=0
• consider enabling detect_stack_use_after_return and check_initialization_order

Davide Pesavento wrote:

I'm unsure about the desired interface though. The current patch has a single option --sanitize=<list of sanitizers to enable>. An alternative would be implementing a separate option for each sanitizer, e.g.: --enable-address-sanitizer, --enable-thread-sanitizer, and so on (or shorthands: --enable-asan, --enable-tsan, ...). I'm also unsure about --enable... vs --with.... We seem to be using both but I don't know if there's a semantic difference.

Any feedback on this design question? I cannot proceed if we don't agree on the "interface".

20160607 conference call decides to use --with-sanitizer=sanitizer1,sanitizer2.

https://gerrit.named-data.net/2821 patch set 3 implements the syntax in note-16

I've tested https://gerrit.named-data.net/3053 patchset1 and it's effective.

I added this to the main function of ndnsec:

  char *x = (char*)malloc(10 * sizeof(char*));
  free(x);
  return x[5];

And compile with:

CXX=clang++ ./waf configure --with-sanitizer=address
./waf

Output:

vagrant@m0212:~/clang/ndn-cxx$ build/bin/ndnsec
=================================================================
==13560==ERROR: AddressSanitizer: heap-use-after-free on address 0x60700000dd85 at pc 0x50846b bp 0x7fff90ed86a0 sp 0x7fff90ed8698
READ of size 1 at 0x60700000dd85 thread T0
==13560==WARNING: Trying to symbolize code, but external symbolizer is not initialized!
    #0 0x50846a (/home/vagrant/clang/ndn-cxx/build/bin/ndnsec+0x50846a)
    #1 0x7f4bd0cc5f44 (/lib/x86_64-linux-gnu/libc.so.6+0x21f44)
    #2 0x4a61fc (/home/vagrant/clang/ndn-cxx/build/bin/ndnsec+0x4a61fc)

0x60700000dd85 is located 5 bytes inside of 80-byte region [0x60700000dd80,0x60700000ddd0)
freed by thread T0 here:
    #0 0x48ff99 (/home/vagrant/clang/ndn-cxx/build/bin/ndnsec+0x48ff99)
    #1 0x50843a (/home/vagrant/clang/ndn-cxx/build/bin/ndnsec+0x50843a)
    #2 0x7f4bd0cc5f44 (/lib/x86_64-linux-gnu/libc.so.6+0x21f44)

previously allocated by thread T0 here:
    #0 0x490119 (/home/vagrant/clang/ndn-cxx/build/bin/ndnsec+0x490119)
    #1 0x50842f (/home/vagrant/clang/ndn-cxx/build/bin/ndnsec+0x50842f)
    #2 0x7f4bd0cc5f44 (/lib/x86_64-linux-gnu/libc.so.6+0x21f44)

SUMMARY: AddressSanitizer: heap-use-after-free ??:0 ??
Shadow bytes around the buggy address:
  0x0c0e7fff9b60: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff9b70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff9b80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff9b90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff9ba0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c0e7fff9bb0:[fd]fd fd fd fd fd fd fd fd fd fa fa fa fa 00 00
  0x0c0e7fff9bc0: 00 00 00 00 00 00 00 00 fa fa fa fa 00 00 00 00
  0x0c0e7fff9bd0: 00 00 00 00 00 00 fa fa fa fa 00 00 00 00 00 00
  0x0c0e7fff9be0: 00 00 00 00 fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c0e7fff9bf0: 00 00 fa fa fa fa fd fd fd fd fd fd fd fd fd fa
  0x0c0e7fff9c00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:     fa
  Heap right redzone:    fb
  Freed heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe
==13560==ABORTING

I've tested https://gerrit.named-data.net/2821 patchset6 and it's effective.

I added this to the main function of nfd-autoreg:

  char *x = (char*)malloc(10 * sizeof(char*));
  free(x);
  return x[5];

And compile with:

CXX=clang++ ./waf configure --with-sanitizer=address
./waf

Output:

vagrant@m0212:~/clang/NFD$ build/bin/nfd-autoreg 
=================================================================
==14079==ERROR: AddressSanitizer: heap-use-after-free on address 0x60700000dc35 at pc 0x49f76b bp 0x7ffc11852600 sp 0x7ffc118525f8
READ of size 1 at 0x60700000dc35 thread T0
==14079==WARNING: Trying to symbolize code, but external symbolizer is not initialized!
    #0 0x49f76a (/home/vagrant/clang/NFD/build/bin/nfd-autoreg+0x49f76a)
    #1 0x7f8029e9df44 (/lib/x86_64-linux-gnu/libc.so.6+0x21f44)
    #2 0x49f65c (/home/vagrant/clang/NFD/build/bin/nfd-autoreg+0x49f65c)

0x60700000dc35 is located 5 bytes inside of 80-byte region [0x60700000dc30,0x60700000dc80)
freed by thread T0 here:
    #0 0x4893f9 (/home/vagrant/clang/NFD/build/bin/nfd-autoreg+0x4893f9)
    #1 0x49f73a (/home/vagrant/clang/NFD/build/bin/nfd-autoreg+0x49f73a)
    #2 0x7f8029e9df44 (/lib/x86_64-linux-gnu/libc.so.6+0x21f44)

previously allocated by thread T0 here:
    #0 0x489579 (/home/vagrant/clang/NFD/build/bin/nfd-autoreg+0x489579)
    #1 0x49f72f (/home/vagrant/clang/NFD/build/bin/nfd-autoreg+0x49f72f)
    #2 0x7f8029e9df44 (/lib/x86_64-linux-gnu/libc.so.6+0x21f44)

SUMMARY: AddressSanitizer: heap-use-after-free ??:0 ??
Shadow bytes around the buggy address:
  0x0c0e7fff9b30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff9b40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff9b50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff9b60: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff9b70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c0e7fff9b80: fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd fd fd
  0x0c0e7fff9b90: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 fa fa
  0x0c0e7fff9ba0: fa fa 00 00 00 00 00 00 00 00 00 00 fa fa fa fa
  0x0c0e7fff9bb0: 00 00 00 00 00 00 00 00 00 00 fa fa fa fa 00 00
  0x0c0e7fff9bc0: 00 00 00 00 00 00 00 00 fa fa fa fa 00 00 00 00
  0x0c0e7fff9bd0: 00 00 00 00 00 00 fa fa fa fa 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:     fa
  Heap right redzone:    fb
  Freed heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe
==14079==ABORTING

But another test fails:

vagrant@m0212:~/clang/NFD$ clang -v
Ubuntu clang version 3.4-1ubuntu3 (tags/RELEASE_34/final) (based on LLVM 3.4)
Target: x86_64-pc-linux-gnu
Thread model: posix
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8.4
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9.3
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8.4
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9.3
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8
vagrant@m0212:~/clang/NFD$ CXX=clang++ ./waf configure --with-sanitizer=address,thread
Setting top to                           : /home/vagrant/clang/NFD 
Setting out to                           : /home/vagrant/clang/NFD/build 
Checking for 'g++' (C++ compiler)        : not found 
Checking for 'clang++' (C++ compiler)    : clang++ 
Checking supported CXXFLAGS              : -std=c++11 
Checking supported CXXFLAGS              : -O2 -g -pedantic -Wall -Wextra -Wno-unused-parameter -fcolor-diagnostics 
Checking for std::is_default_constructible : yes 
Checking for std::is_move_constructible    : yes 
Checking if compiler supports -fsanitize=address : yes 
Checking if compiler supports -fsanitize=thread  : no 
thread sanitizer is not supported by the current compiler
(complete log in /home/vagrant/clang/NFD/build/config.log)

Tail of build.log says:

------------------------------------------------
Checking if compiler supports -fsanitize=thread
==>

int main(int argc, char **argv) {
    (void)argc; (void)argv;
    return 0;
}

<==
[1/2] Compiling build/.conf_check_49f49e323237e0cb97694ad21317f6c3/test.cpp

['clang++', '-O2', '-g', '-pedantic', '-Wall', '-Wextra', '-Wno-unused-parameter', '-fcolor-diagnostics', '-std=c++11', '-fsanitize=address', '-fno-omit-frame-pointer', '-Werror', '-fsanitize=thread', '-fno-omit-frame-pointer', '-DNDEBUG', '-DHAVE_IS_DEFAULT_CONSTRUCTIBLE=1', '-DHAVE_IS_MOVE_CONSTRUCTIBLE=1', '../test.cpp', '-c', '-o', '/home/vagrant/clang/NFD/build/.conf_check_49f49e323237e0cb97694ad21317f6c3/testbuild/test.cpp.1.o']
err: clang: error: invalid argument '-fsanitize=address' not allowed with '-fsanitize=thread'

from /home/vagrant/clang/NFD: Test does not build: Traceback (most recent call last):
  File "/home/vagrant/clang/NFD/.waf-1.8.9-8a9ccbc1c5d3936b0b08e972e4883a9a/waflib/Configure.py", line 343, in run_build
    bld.compile()
  File "/home/vagrant/clang/NFD/.waf-1.8.9-8a9ccbc1c5d3936b0b08e972e4883a9a/waflib/Build.py", line 184, in compile
    raise Errors.BuildError(self.producer.error)
BuildError: Build failed
 -> task in 'testprog' failed (exit status 1): 
    {task 139870019563792: cxx test.cpp -> test.cpp.1.o}
['clang++', '-O2', '-g', '-pedantic', '-Wall', '-Wextra', '-Wno-unused-parameter', '-fcolor-diagnostics', '-std=c++11', '-fsanitize=address', '-fno-omit-frame-pointer', '-Werror', '-fsanitize=thread', '-fno-omit-frame-pointer', '-DNDEBUG', '-DHAVE_IS_DEFAULT_CONSTRUCTIBLE=1', '-DHAVE_IS_MOVE_CONSTRUCTIBLE=1', '../test.cpp', '-c', '-o', '/home/vagrant/clang/NFD/build/.conf_check_49f49e323237e0cb97694ad21317f6c3/testbuild/test.cpp.1.o']

from /home/vagrant/clang/NFD: The configuration failed
no
from /home/vagrant/clang/NFD: thread sanitizer is not supported by the current compiler

This appears to be a compiler limitation rather than problem with the waf tool.

Junxiao Shi wrote:

But another test fails:

vagrant@m0212:~/clang/NFD$ clang -v
Ubuntu clang version 3.4-1ubuntu3 (tags/RELEASE_34/final) (based on LLVM 3.4)
[...]
Checking if compiler supports -fsanitize=thread  : no 
thread sanitizer is not supported by the current compiler

[...]
This appears to be a compiler limitation rather than problem with the waf tool.

Yep, this is expected, your clang version is too old.

==13560==WARNING: Trying to symbolize code, but external symbolizer is not initialized!

If you get this warning, you should install llvm-symbolizer, see http://clang.llvm.org/docs/AddressSanitizer.html#symbolizing-the-reports