Task #1819
closed
Forwarding benchmark
100%
Description
Run a benchmark of forwarding performance on ONL.
The report should contain the following numbers: throughput (packets per seconds), loss rate (%), and processing delay (ms) of Interest and Data forwarding.
The benchmark should use optimized library and NFD code, with logging turned off.
This is not profiling. The report doesn't need to point out which area in NFD causes the slowdown.
Files
Updated by Junxiao Shi over 9 years ago
This is John DeHart's benchmark report for NFD 0.1, as a reference for this Task.
- Optimized nfd tests For these tests nfd was built with the standard default compilation options, defaultFlags = ['-O2', '-g', '-Wall']
The following tests were run with 128 client all routing through one central nfd router. There are no
servers to provide content for the supplied interests. There are hosts running nfd that the interests are routed to.16 client hosts running 8 ndn-traffic processes and one nfd.
16 server hosts running one nfd.
1 router host running 1 nfd as the central router.Three test cases for name length:
short: /example/000
medium: /example/ABCDE/FGHIJ/KLMNO/PQRST/UVWXY/Z/000
long: /example/ABCDE/FGHIJ/KLMNO/PQRST/UVWXY/Z/ABCDE/FGHIJ/KLMNO/PQRST/UVWXY/Z/ABCDE/FGHIJ/KLMNO/PQRST/UVWXY/Z/ABCDE/FGHIJ/KLMNO/PQRST/UVWXY/Z/000128 different base names: end of name ranged from 000 to 127.
A sequence number is appended by ndn-traffic to each Interest to force every name to be unique.Applied load was approximately 42000 interests/sec.
Approximate rate nfd was able to handle:
short: 37500 Interests/sec
medium: 22400 Interests/sec
long: 7100 Interests/sec
This note is informative. See Task Description on the exact requirements.
Updated by Chengyu Fan over 9 years ago
- File forwarding-benchmark-testing-steps.txt forwarding-benchmark-testing-steps.txt added
- Status changed from New to In Progress
- % Done changed from 0 to 100
This is the report for the NFD forwarding benchmark.
The tests were running over ONL (https://onl.wustl.edu/). John's script (https://github.com/WU-ARL/NFD_Performance_Testing_on_ONL.git) is modified to manage the tests.
The latest NFD and NDN-cxx library were used in the tests. They were built with the standard default compilation options.
I. Experiments:
The tests were run with 128 client all routing through one central nfd router. Two scenarios are provided: content provided, and no content provided.
In each of the scenarios, three different name length are used. Note that 128 different base names were given: end of name ranged from 000 to 127.
Short: /abcde/000
Medium: /abcde/fghij/klmno/pqrst/125
Long: /abcde/fghij/klmno/pqrst/uvwxy/zabcd/efghi/jklmn/opqrs/tuvwx/yzabc/defgh/ijklm/nopqr/stuvw/xyzab/cdefg/hijkl/mnopq/rstuv/wxyza/bcdef/ghijk/lmnop/qrstu/127
A sequence number is appended by ndn-traffic to each Interest to force every name to be unique.
II. Results:
A. Approximate rate nfd was able to handle:
Short:
With content provided: 4800 Interests/s
W/o content provided: 14800 Interests/sMedium:
With content provided: 3500 Interests/s
W/o content provided: 10000 Interests/sLong:
With content provided: 1100 Interests/s
W/o content provided: 2000 Interests/s
B. When the Interest send rate is below the threshold above, the loss rate is around 0.02%, which may be introduced when the test is killed.
C. Approximate latency for nfd:
Short: around 0.20ms
Medium: around 0.25ms
Long: > 0.6ms, sometimes around 1ms.
Updated by Lan Wang over 9 years ago
CPU and RAM of the machine? Do these matter?
Updated by Chengyu Fan over 9 years ago
As is set in the NDN traffic generator configuration file, Content=AAAAAAAAAA.
For different tests, the packet size is a little different:
Short: UDP packet size for Interest is 30 bytes, for Content is 352 bytes.
Medium: UDP packet size for Interest is 51 bytes, for Content is 373 bytes.
Long: UDP packet size for Interest is 197 bytes, for Content is 519 bytes.
All the experiment machines are PC2core on ONL. Currently, I don't know the exact machine specifications.
Updated by Chengyu Fan over 9 years ago
To compare with John's results, I redo the tests using nfd v0.1. over ONL.
Two ndn-cxx library are used separately:
the one installed on ONL: this one gave me the results pretty much John's if the load is below a threshold (for the medium name, load is about 9000 Interests/s). If it is above that, the content serving will soon drop to 0.
the ndn-cxx v0.1.0: this one gave me the same results as I got in the v0.2.0. But it is more stable: with a heavy load (>12000Interests/s), although the nfd cannot forward all of them, the content serving goes well (#Interests forwarded = #content served)
Updated by Junxiao Shi over 9 years ago
New benchmark is needed:
- Name length: medium
- contents are served
- Data payload size is set such that UDP packets are close to 1500 octets
Updated by Chengyu Fan over 9 years ago
- File 128pairs-latestNFD-latestCxx-mediumName-udp1435.png 128pairs-latestNFD-latestCxx-mediumName-udp1435.png added
The tests parameters:
- Name length: medium;
- Contents are served;
- Data payload size: 1435;
- NFD version: latest
- NDN-cxx version: latest
128 client-server pairs are used. 16 machines are for clients, and 16 machines are for servers (so each machine host 8 clients or 8 servers). Those machines are connected through a single NFD router. All the experiment machines are PC2core on ONL.
The results are the same:
- Approximate latency for nfd is around 0.25ms;
- Approximate rate nfd was able to handle is around 3500 Interests/s
Updated by Junxiao Shi over 9 years ago
- Status changed from In Progress to Closed
Updated by Chengyu Fan over 9 years ago
This report describes the forwarding performance for the latest NFD released (v0.2).
The performance tests is running over ONL using the installed NFD.
The old reports have issues because the default compilation options are overridden when specifying the local ndn-cxx library directory using CXXFLAGS, which means there is no optimization.
The scripts for the tests is improved, so the manual to run the tests is uploaded again.
I. Experiments:
The tests were run with 64 client-server pairs all routing through one central nfd router. Two scenarios are provided: content provided, and no content provided.
In each of the scenarios, three different name length are used. Note that 128 different base names were given: end of name ranged from 000 to 127.
Short: /abcde/000
Medium: /abcde/fghij/klmno/pqrst/uvwxy/125
Long: /abcde/fghij/klmno/pqrst/uvwxy/zabcd/efghi/jklmn/opqrs/tuvwx/yzabc/defgh/ijklm/nopqr/stuvw/xyzab/cdefg/hijkl/mnopq/rstuv/wxyza/bcdef/ghijk/lmnop/qrstu/127
A sequence number is appended by ndn-traffic to each Interest to force every name to be unique.
II. Results:
A. Approximate rate nfd was able to handle:
Short: With content provided: 12000 Interests/s; W/o content provided: 40000 Interests/s
Medium: With content provided: 10000 Interests/s; W/o content provided: 30000 Interests/s
Long: With content provided: 4200 Interests/s; W/o content provided: 9600 Interests/s
Updated by Chengyu Fan over 9 years ago
- File forwarding-benchmark-report-v0.3.txt forwarding-benchmark-report-v0.3.txt added
- File forwarding-benchmark-testing-steps-v0.3.txt forwarding-benchmark-testing-steps-v0.3.txt added
The forwarding benchmark testing steps and report version 0.3 are uploaded. (They are conducted on the NFD v0.2.)
Please ignore the old reports.