NFD

Given my total lack of knowledge of the integrated tests that we have or how to run them, I can recommend digging deep into them and creating documentation / environment to re-run such tests. Currently, Junxiao is using emulab. I would like to see something that is less dependent on infrastructure (personally, I prefer thing based on Vagrant system, but something else could be fine too).

The early developers of integ was using five connected VirtualBox boxes.

I created Emulab topology for use in code reviews, in order to verify whether a patchset is correct. It's not designed to be a development environment.

While I won't disagree with someone creating a integ devbox with vagrant, I don't think it's better than Emulab:
Vagrant can only be deployed on a physical machine.

I'm thinking about making an integ devbox with OpenVZ.
The whole system is contained within one VirtualBox VM, and no installation is needed on the physical machine.

One extra benefit is that install_apps.py is performed only once: programs are installed in one container which is cloned into four identical containers, and then a topology is created and tests are executed.

Why would you run virtualization environment inside the virtual environment? It is possible to do with vagrant too, though more heavy weight. There are many things that can be done with vagrant about sharing things between VMs. This a big part of vagrant's provisioning.

Developers should be able to run integration tests in some convenient form. I feel vagrant provides a simple, universal (and I would say "lightweight") approach to do that. By lightweight I mean that the only thing one needs to install on the host machine (be it windows, mac, linux) is vagrant itself and virtualbox. The rest will be done automatically by vagrant. No need to manually install any OSes in any of the virtualboxes.

I've gotten all of the tests working on the Vagrant setup except for one: test_traffic.
It's not failing a test case, instead it's getting an error when trying to terminate NFD.
Have you ever gotten an "operation not permitted" error with killNfd()?

I didn't see this message in the log in any recent integration-tests log.

"operation not permitted" is the error message for EPERM.

kill(2) returns EPERM when the caller has insufficient privilege to send a signal to the destination process.

nfd process is started with sudo, so subprocess.Popen.terminate may not be able to terminate the process.

Junxiao Shi wrote:

nfd process is started with sudo, so subprocess.Popen.terminate may not be able to terminate the process.

Perhaps this should be changed to terminate the process as root?

By changing line 74 of library_helpers/process from:

self.subprocesses[processKey].terminate()

to

subprocess.call(['sudo', 'kill', '-s', 'SIGTERM', str(self.subprocesses[processKey].pid)])

the EPERM error was resolved. I believe this change should be made in the integration-tests repository.

I've finished the Vagrantfile and the shell script that creates the environment, runs the tests, and exports the results. I'm waiting on some blocking issues (#3002 and #3012) before submitting what I have.

Currently, the script just pulls the integration-tests master and runs it. Should it also be able to use different commits, like those from Gerrit?

Currently, the script just pulls the integration-tests master and runs it.

Vagrantfile should exist within the repository ("where" is to be answered in note-9), and run whatever code is in the same repository. It shouldn't cause the guests to download code from somewhere else.

Alex Afanasyev wrote:

Developers should be able to run integration tests in some convenient form. I feel vagrant provides a simple, universal (and I would say "lightweight") approach to do that.

A really lightweight way of doing integration testing would be using Linux namespaces (containers) instead of full-blown virtual machines. I was actually thinking about this a few days ago: why don't we use some container technology (systemd-nspawn + some bash, docker, libvirt, even some minimal python library, whatever...) in order to conveniently run integration tests in an automated way? It would be vastly more efficient too.

Davide Pesavento wrote:

A really lightweight way of doing integration testing would be using Linux namespaces (containers) instead of full-blown virtual machines. I was actually thinking about this a few days ago: why don't we use some container technology (systemd-nspawn + some bash, docker, libvirt, even some minimal python library, whatever...) in order to conveniently run integration tests in an automated way? It would be vastly more efficient too.

This seems to be pretty close to one of the goals we have for Mini-NDN (Task #2941). We are working towards using Mini-NDN, a Linux container based emulator forked from Mininet, to run NFD's integration tests.

Most of our Jenkins system is run with Vagrant, so it was simple to use Vagrant for this.

We also needed to simulate multiple networks connecting the VMs in the setup. I'm uncertain if this is possible with LXC.

Eric Newberry wrote:

We also needed to simulate multiple networks connecting the VMs in the setup. I'm uncertain if this is possible with LXC.

Yes, in fact that part is almost identical between a VM-based setup and a container-based setup. You just have different network namespaces instead of different VMs.

Davide Pesavento wrote:

A really lightweight way of doing integration testing would be using Linux namespaces (containers) instead of full-blown virtual machines. I was actually thinking about this a few days ago: why don't we use some container technology (systemd-nspawn + some bash, docker, libvirt, even some minimal python library, whatever...) in order to conveniently run integration tests in an automated way? It would be vastly more efficient too.

This is another good way of doing tests. However, we wouldn't be able to test some of face types (ethernet) using containers.

I think the main point is not lighweightiness, but simplicity of running stuff and functionality. Spawning a few linux VMs with vagrant is a relatively quick operation (though I agreee that it is still much heavier than docker-like thing).