Task #2436
closed
Design vagrant provisioning profile to run integration tests
100%
Description
Vagrant system is specifically designed to recreate necessary environments.
One of the use cases is multi-machine scenario which is exactly what is needed to run IntegrationTests.
This Task is to design a Vagrantfile that recreates the network topology specified in multi-hosts.conf.
Files
Updated by Junxiao Shi almost 10 years ago
Given my total lack of knowledge of the integrated tests that we have or how to run them, I can recommend digging deep into them and creating documentation / environment to re-run such tests. Currently, Junxiao is using emulab. I would like to see something that is less dependent on infrastructure (personally, I prefer thing based on Vagrant system, but something else could be fine too).
The early developers of integ was using five connected VirtualBox boxes.
I created Emulab topology for use in code reviews, in order to verify whether a patchset is correct. It's not designed to be a development environment.
While I won't disagree with someone creating a integ devbox with vagrant, I don't think it's better than Emulab:
Vagrant can only be deployed on a physical machine.
I'm thinking about making an integ devbox with OpenVZ.
The whole system is contained within one VirtualBox VM, and no installation is needed on the physical machine.
One extra benefit is that install_apps.py is performed only once: programs are installed in one container which is cloned into four identical containers, and then a topology is created and tests are executed.
Updated by Alex Afanasyev almost 10 years ago
Why would you run virtualization environment inside the virtual environment? It is possible to do with vagrant too, though more heavy weight. There are many things that can be done with vagrant about sharing things between VMs. This a big part of vagrant's provisioning.
Developers should be able to run integration tests in some convenient form. I feel vagrant provides a simple, universal (and I would say "lightweight") approach to do that. By lightweight I mean that the only thing one needs to install on the host machine (be it windows, mac, linux) is vagrant itself and virtualbox. The rest will be done automatically by vagrant. No need to manually install any OSes in any of the virtualboxes.
Updated by Eric Newberry over 9 years ago
- Status changed from New to In Progress
- Assignee set to Eric Newberry
Updated by Junxiao Shi over 9 years ago
I've gotten all of the tests working on the Vagrant setup except for one: test_traffic.
It's not failing a test case, instead it's getting an error when trying to terminate NFD.
Have you ever gotten an "operation not permitted" error with killNfd()?
I didn't see this message in the log in any recent integration-tests log.
"operation not permitted" is the error message for EPERM.
kill(2) returns EPERM when the caller has insufficient privilege to send a signal to the destination process.
nfd process is started with sudo, so subprocess.Popen.terminate may not be able to terminate the process.
Updated by Eric Newberry over 9 years ago
Junxiao Shi wrote:
nfdprocess is started withsudo, so subprocess.Popen.terminate may not be able to terminate the process.
Perhaps this should be changed to terminate the process as root?
Updated by Eric Newberry over 9 years ago
By changing line 74 of library_helpers/process from:
self.subprocesses[processKey].terminate()
to
subprocess.call(['sudo', 'kill', '-s', 'SIGTERM', str(self.subprocesses[processKey].pid)])
the EPERM error was resolved. I believe this change should be made in the integration-tests repository.
Updated by Junxiao Shi over 9 years ago
- Description updated (diff)
- Target version set to v0.4
Moved from original description:
So far, the following script allow to instantiate two VMs and install prerequisites for ndn-cxx
Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"
$script = <<SCRIPT
sudo sudo apt-get -y install software-properties-common
sudo add-apt-repository -y ppa:named-data/ppa
sudo apt-get update -qq
sudo apt-get install -y build-essential git
sudo apt-get install -y libsqlite3-dev libcrypto++-dev libboost-all-dev
sudo apt-get install -y pkg-config libpcap-dev
SCRIPT
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = "ubuntu/trusty64"
config.vm.provider "virtualbox" do |vb|
vb.gui = false
vb.customize ["modifyvm", :id, "--memory", "1024", "--cpus", "1"]
end
config.vm.define "hub" do |hub|
hub.vm.network "private_network", ip: "192.168.1.2"
hub.vm.provision "shell", inline: $script, privileged: false
hub.vm.provision "shell", privileged: true, inline: <<SCRIPT
apt-get install -y nfd ndn-autoconfig-server
echo "FACE_URI=\"udp4://192.168.1.2:6363\"" > /etc/default/ndn-autoconfig-server
echo "ROUTABLE_PREFIXES=\"-p /test/prefix1 -p /test/prefix2\"" >> /etc/default/ndn-autoconfig-server
restart ndn-autoconfig-server
SCRIPT
end
config.vm.define "client" do |client|
client.vm.network "private_network", ip: "192.168.1.100"
client.vm.provision "shell", inline: $script, privileged: false
client.vm.provider "virtualbox" do |vb|
vb.gui = false
vb.customize ["modifyvm", :id, "--memory", "4024", "--cpus", "4"]
end
end
end
Updated by Junxiao Shi over 9 years ago
- Description updated (diff)
The Vagrantfile should be committed into the integration-tests repository.
One question is: where in the repository should this file be placed?
Vagrantfileis just a single file, so it can be placed in the root directory along withmulti-hosts.conf- NFD-android repository places it in
.vagrant/Vagrantfile, but I can't find any reference that recommends putting it into a hidden directory
Updated by Eric Newberry over 9 years ago
I've finished the Vagrantfile and the shell script that creates the environment, runs the tests, and exports the results. I'm waiting on some blocking issues (#3002 and #3012) before submitting what I have.
Currently, the script just pulls the integration-tests master and runs it. Should it also be able to use different commits, like those from Gerrit?
Updated by Junxiao Shi over 9 years ago
Currently, the script just pulls the integration-tests master and runs it.
Vagrantfile should exist within the repository ("where" is to be answered in note-9), and run whatever code is in the same repository. It shouldn't cause the guests to download code from somewhere else.
Updated by Eric Newberry over 9 years ago
- Blocked by Bug #3002: test_localhost_scope: fails when run multiple times added
Updated by Eric Newberry over 9 years ago
- Blocks Task #3012: library_helpers: Send SIGTERM as root added
Updated by Eric Newberry over 9 years ago
- Blocks deleted (Task #3012: library_helpers: Send SIGTERM as root)
Updated by Eric Newberry over 9 years ago
- Blocked by Task #3012: library_helpers: Send SIGTERM as root added
Updated by Eric Newberry over 9 years ago
- Status changed from In Progress to Code review
- % Done changed from 0 to 100
Updated by Davide Pesavento over 9 years ago
Alex Afanasyev wrote:
Developers should be able to run integration tests in some convenient form. I feel vagrant provides a simple, universal (and I would say "lightweight") approach to do that.
A really lightweight way of doing integration testing would be using Linux namespaces (containers) instead of full-blown virtual machines. I was actually thinking about this a few days ago: why don't we use some container technology (systemd-nspawn + some bash, docker, libvirt, even some minimal python library, whatever...) in order to conveniently run integration tests in an automated way? It would be vastly more efficient too.
Updated by Vince Lehman over 9 years ago
Davide Pesavento wrote:
A really lightweight way of doing integration testing would be using Linux namespaces (containers) instead of full-blown virtual machines. I was actually thinking about this a few days ago: why don't we use some container technology (systemd-nspawn + some bash, docker, libvirt, even some minimal python library, whatever...) in order to conveniently run integration tests in an automated way? It would be vastly more efficient too.
This seems to be pretty close to one of the goals we have for Mini-NDN (Task #2941). We are working towards using Mini-NDN, a Linux container based emulator forked from Mininet, to run NFD's integration tests.
Updated by Eric Newberry over 9 years ago
Most of our Jenkins system is run with Vagrant, so it was simple to use Vagrant for this.
We also needed to simulate multiple networks connecting the VMs in the setup. I'm uncertain if this is possible with LXC.
Updated by Davide Pesavento over 9 years ago
Eric Newberry wrote:
We also needed to simulate multiple networks connecting the VMs in the setup. I'm uncertain if this is possible with LXC.
Yes, in fact that part is almost identical between a VM-based setup and a container-based setup. You just have different network namespaces instead of different VMs.
Updated by Junxiao Shi over 9 years ago
- File 2436,6_nohup.out 2436,6_nohup.out added
Updated by Alex Afanasyev over 9 years ago
Davide Pesavento wrote:
A really lightweight way of doing integration testing would be using Linux namespaces (containers) instead of full-blown virtual machines. I was actually thinking about this a few days ago: why don't we use some container technology (systemd-nspawn + some bash, docker, libvirt, even some minimal python library, whatever...) in order to conveniently run integration tests in an automated way? It would be vastly more efficient too.
This is another good way of doing tests. However, we wouldn't be able to test some of face types (ethernet) using containers.
I think the main point is not lighweightiness, but simplicity of running stuff and functionality. Spawning a few linux VMs with vagrant is a relatively quick operation (though I agreee that it is still much heavier than docker-like thing).
Updated by Junxiao Shi over 9 years ago
- File 2436,7_nohup.out 2436,7_nohup.out added
Updated by Junxiao Shi over 9 years ago
- Status changed from Code review to Closed