NFD

I cannot reproduce this problem.

Please paste output of these commands:

• free -m
• uname -a
• cat /etc/lsb-release

so that we can know your environment.

~) cat /etc/lsb-release
LSB_VERSION=1.4
DISTRIB_ID=Arch
DISTRIB_RELEASE=rolling
DISTRIB_DESCRIPTION="Arch Linux"
~) free -m
             total       used       free     shared    buffers     cached
Mem:          3816       2621       1194        156         67        988
-/+ buffers/cache:       1566       2249
Swap:         2811        555       2255
~) uname -a
Linux localhost 3.17.2-1-ARCH #1 SMP PREEMPT Thu Oct 30 20:49:39 CET 2014 x86_64 GNU/Linux

The build time in older version took about 15-20 min.

processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 58
model name  : Intel(R) Core(TM) i5-3317U CPU @ 1.70GHz
stepping    : 9
microcode   : 0x17
cpu MHz     : 1523.691
cache size  : 3072 KB
physical id : 0
siblings    : 4
core id     : 0
cpu cores   : 2
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt
bugs        :
bogomips    : 3393.48
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor   : 1
vendor_id   : GenuineIntel
cpu family  : 6
model       : 58
model name  : Intel(R) Core(TM) i5-3317U CPU @ 1.70GHz
stepping    : 9
microcode   : 0x17
cpu MHz     : 2277.667
cache size  : 3072 KB
physical id : 0
siblings    : 4
core id     : 0
cpu cores   : 2
apicid      : 1
initial apicid  : 1
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt
bugs        :
bogomips    : 3393.48
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor   : 2
vendor_id   : GenuineIntel
cpu family  : 6
model       : 58
model name  : Intel(R) Core(TM) i5-3317U CPU @ 1.70GHz
stepping    : 9
microcode   : 0x17
cpu MHz     : 1929.167
cache size  : 3072 KB
physical id : 0
siblings    : 4
core id     : 1
cpu cores   : 2
apicid      : 2
initial apicid  : 2
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt
bugs        :
bogomips    : 3393.48
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor   : 3
vendor_id   : GenuineIntel
cpu family  : 6
model       : 58
model name  : Intel(R) Core(TM) i5-3317U CPU @ 1.70GHz
stepping    : 9
microcode   : 0x17
cpu MHz     : 1290.804
cache size  : 3072 KB
physical id : 0
siblings    : 4
core id     : 1
cpu cores   : 2
apicid      : 3
initial apicid  : 3
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt
bugs        :
bogomips    : 3393.48
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

Please paste the exact commands you used to build. Don't include the output; just the commands.

Please repeat these commands with each commit, and find out which commit caused a significant increase on build time.

A binary search may be helpful: if commit 1 is good but commit 9 is bad, you can try commit 5 to see whether it's good or bad.

Junxiao Shi wrote:

A binary search may be helpful: if commit 1 is good but commit 9 is bad, you can try commit 5 to see whether it's good or bad.

For this, git bisect is your friend.

What compiler and what version? Are you using custom CXXFLAGS?

I tried building ndn-cxx commit 6f9ec931dbb1b833f7d8d05afe2d0ca21d211163 on OSX 10.8, Mac Mini with 4 cores and 4GB memory.

./waf configure --with-tests --debug --without-pch; ./waf

real    2m32.377s
user    7m51.361s
sys 0m30.928s

./waf configure --with-tests --debug --without-pch; ./waf -j1

real    4m54.479s
user    4m23.150s
sys 0m17.890s

So the problem is probably specific to Arch Linux.

CPPFLAGS="-D_FORTIFY_SOURCE=2"
CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong --param=ssp-buffer-size=4"
CXXFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong --param=ssp-buffer-size=4"
LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro"

What is your flag?

@junxiao
Also, it is not helpful to see your build is faster than mine because this is about a performance regression. Maybe your mac can build it faster before? (also I think mac uses clang now.... you know that is a big difference.)

I suspect it is only Archlinux because arch ships vanilla gcc. Probably ubuntu will have similar problem.

FYI, go-nfd builds in 2 sec. This bug is unfortunate.

What is "vanilla gcc"? Please paste the output of:

• gcc -v
• clang -v

Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-unknown-linux-gnu/4.9.1/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: /build/gcc/src/gcc-4.9-20140903/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.archlinux.org/ --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++ --enable-shared --enable-threads=posix --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-clocale=gnu --disable-libstdcxx-pch --disable-libssp --enable-gnu-unique-object --enable-linker-build-id --enable-cloog-backend=isl --disable-isl-version-check --disable-cloog-version-check --enable-lto --enable-plugin --enable-install-libiberty --with-linker-hash-style=gnu --disable-multilib --disable-werror --enable-checking=release
Thread model: posix
gcc version 4.9.1 20140903 (prerelease) (GCC) 

Please answer the questions in note-5.

./waf configure --prefix=/usr
./waf build
sudo ./waf install

Please answer the second question in note-5:

Please repeat these commands with each commit, and find out which commit caused a significant increase on build time.

A binary search may be helpful: if commit 1 is good but commit 9 is bad, you can try commit 5 to see whether it's good or bad.

For the record, on a Gentoo/Linux amd64 machine, with:

$ gcc --version
gcc (Gentoo 4.9.1 p1.0, pie-0.6.0) 4.9.1

I do not see any noticeable increase in compilation times between ndn-cxx-0.2 and current git. I'm using -j8 and default CXXFLAGS plus -march=native. Build time is always around 45-50 seconds with both versions.

Why is this a "Feature"?

It's not a bug or task either. If you think it is better to call it a bug, then it can be changed.

We don't have any specific compilation deadlines and compilation time is not inherent property of the software, rather something from interactions of the compiler, libraries, and the software itself. On Jenkins bots, compilation time is around 20 minutes and this is not a bug of any kind, it's just the way it is.

In any case. Doesn't matter how it is called. It could be memory issue, though 4gig machine shouldn't really have this issue (unless something else is actively using memory)

Tai-Lin, if you want to experiment, run ./waf build -j1 for both library and nfd. This would force 1-threaded compilation and reduce the amount of memory the compilation uses (one c++ compilation may sometimes use ~ 1gig of memory).

I agree that compilation time is not an important aspect.

But I'd keep this open for a while and wait for the reporter to answer the question in note-15. If no answer is entered within 7 days, this will be Rejected.

It could be memory issue, though 4gig machine shouldn't really have this issue (unless something else is actively using memory)

According to note-2, the 4GB laptop has 2249MB available memory, quite tight for 4 parallel gcc processes.

Is there a way to restrict number of parallel compilations in wscript?

The restriction could be MIN(nCpuCores, FLOOR(memoryGbs / 768)).

Most likely, it is possible, but no idea how exactly.