Project

General

Profile

Actions

Task #2436

closed

Design vagrant provisioning profile to run integration tests

Added by Alex Afanasyev about 9 years ago. Updated over 8 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Integration Tests
Target version:
Start date:
Due date:
% Done:

100%

Estimated time:

Description

Vagrant system is specifically designed to recreate necessary environments.

One of the use cases is multi-machine scenario which is exactly what is needed to run IntegrationTests.

This Task is to design a Vagrantfile that recreates the network topology specified in multi-hosts.conf.


Files

2436,6_nohup.out (321 KB) 2436,6_nohup.out Junxiao Shi, 07/16/2015 01:11 PM
2436,7_nohup.out (12 KB) 2436,7_nohup.out Junxiao Shi, 07/18/2015 12:53 PM

Related issues 2 (0 open2 closed)

Blocked by NFD - Bug #3002: test_localhost_scope: fails when run multiple timesClosedEric Newberry07/01/2015

Actions
Blocked by NFD - Task #3012: library_helpers: Send SIGTERM as rootClosedEric Newberry07/05/2015

Actions
Actions #1

Updated by Alex Afanasyev about 9 years ago

  • Description updated (diff)
Actions #2

Updated by Junxiao Shi about 9 years ago

Given my total lack of knowledge of the integrated tests that we have or how to run them, I can recommend digging deep into them and creating documentation / environment to re-run such tests. Currently, Junxiao is using emulab. I would like to see something that is less dependent on infrastructure (personally, I prefer thing based on Vagrant system, but something else could be fine too).

The early developers of integ was using five connected VirtualBox boxes.

I created Emulab topology for use in code reviews, in order to verify whether a patchset is correct. It's not designed to be a development environment.

While I won't disagree with someone creating a integ devbox with vagrant, I don't think it's better than Emulab:
Vagrant can only be deployed on a physical machine.

I'm thinking about making an integ devbox with OpenVZ.
The whole system is contained within one VirtualBox VM, and no installation is needed on the physical machine.

One extra benefit is that install_apps.py is performed only once: programs are installed in one container which is cloned into four identical containers, and then a topology is created and tests are executed.

Actions #3

Updated by Alex Afanasyev about 9 years ago

Why would you run virtualization environment inside the virtual environment? It is possible to do with vagrant too, though more heavy weight. There are many things that can be done with vagrant about sharing things between VMs. This a big part of vagrant's provisioning.

Developers should be able to run integration tests in some convenient form. I feel vagrant provides a simple, universal (and I would say "lightweight") approach to do that. By lightweight I mean that the only thing one needs to install on the host machine (be it windows, mac, linux) is vagrant itself and virtualbox. The rest will be done automatically by vagrant. No need to manually install any OSes in any of the virtualboxes.

Actions #4

Updated by Eric Newberry almost 9 years ago

  • Status changed from New to In Progress
  • Assignee set to Eric Newberry
Actions #5

Updated by Junxiao Shi over 8 years ago

I've gotten all of the tests working on the Vagrant setup except for one: test_traffic.
It's not failing a test case, instead it's getting an error when trying to terminate NFD.
Have you ever gotten an "operation not permitted" error with killNfd()?

I didn't see this message in the log in any recent integration-tests log.

"operation not permitted" is the error message for EPERM.

kill(2) returns EPERM when the caller has insufficient privilege to send a signal to the destination process.

nfd process is started with sudo, so subprocess.Popen.terminate may not be able to terminate the process.

Actions #6

Updated by Eric Newberry over 8 years ago

Junxiao Shi wrote:

nfd process is started with sudo, so subprocess.Popen.terminate may not be able to terminate the process.

Perhaps this should be changed to terminate the process as root?

Actions #7

Updated by Eric Newberry over 8 years ago

By changing line 74 of library_helpers/process from:

self.subprocesses[processKey].terminate()

to

subprocess.call(['sudo', 'kill', '-s', 'SIGTERM', str(self.subprocesses[processKey].pid)])

the EPERM error was resolved. I believe this change should be made in the integration-tests repository.

Actions #8

Updated by Junxiao Shi over 8 years ago

  • Description updated (diff)
  • Target version set to v0.4

Moved from original description:

So far, the following script allow to instantiate two VMs and install prerequisites for ndn-cxx

Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

$script = <<SCRIPT
sudo sudo apt-get -y install software-properties-common
sudo add-apt-repository -y ppa:named-data/ppa
sudo apt-get update -qq

sudo apt-get install -y build-essential git
sudo apt-get install -y libsqlite3-dev libcrypto++-dev libboost-all-dev
sudo apt-get install -y pkg-config libpcap-dev
SCRIPT

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "ubuntu/trusty64"

  config.vm.provider "virtualbox" do |vb|
    vb.gui = false
    vb.customize ["modifyvm", :id, "--memory", "1024", "--cpus", "1"]
  end

  config.vm.define "hub" do |hub|
    hub.vm.network "private_network", ip: "192.168.1.2"
    hub.vm.provision "shell", inline: $script, privileged: false
    hub.vm.provision "shell", privileged: true, inline: <<SCRIPT
apt-get install -y nfd ndn-autoconfig-server
echo "FACE_URI=\"udp4://192.168.1.2:6363\"" > /etc/default/ndn-autoconfig-server
echo "ROUTABLE_PREFIXES=\"-p /test/prefix1 -p /test/prefix2\"" >> /etc/default/ndn-autoconfig-server
restart ndn-autoconfig-server
SCRIPT
  end

  config.vm.define "client" do |client|
    client.vm.network "private_network", ip: "192.168.1.100"
    client.vm.provision "shell", inline: $script, privileged: false
    client.vm.provider "virtualbox" do |vb|
      vb.gui = false
      vb.customize ["modifyvm", :id, "--memory", "4024", "--cpus", "4"]
    end
  end

end
Actions #9

Updated by Junxiao Shi over 8 years ago

  • Description updated (diff)

The Vagrantfile should be committed into the integration-tests repository.

One question is: where in the repository should this file be placed?

  • Vagrantfile is just a single file, so it can be placed in the root directory along with multi-hosts.conf
  • NFD-android repository places it in .vagrant/Vagrantfile, but I can't find any reference that recommends putting it into a hidden directory
Actions #10

Updated by Eric Newberry over 8 years ago

I've finished the Vagrantfile and the shell script that creates the environment, runs the tests, and exports the results. I'm waiting on some blocking issues (#3002 and #3012) before submitting what I have.

Currently, the script just pulls the integration-tests master and runs it. Should it also be able to use different commits, like those from Gerrit?

Actions #11

Updated by Junxiao Shi over 8 years ago

Currently, the script just pulls the integration-tests master and runs it.

Vagrantfile should exist within the repository ("where" is to be answered in note-9), and run whatever code is in the same repository. It shouldn't cause the guests to download code from somewhere else.

Actions #12

Updated by Eric Newberry over 8 years ago

  • Blocked by Bug #3002: test_localhost_scope: fails when run multiple times added
Actions #13

Updated by Eric Newberry over 8 years ago

  • Blocks Task #3012: library_helpers: Send SIGTERM as root added
Actions #14

Updated by Eric Newberry over 8 years ago

  • Blocks deleted (Task #3012: library_helpers: Send SIGTERM as root)
Actions #15

Updated by Eric Newberry over 8 years ago

  • Blocked by Task #3012: library_helpers: Send SIGTERM as root added
Actions #16

Updated by Eric Newberry over 8 years ago

  • Status changed from In Progress to Code review
  • % Done changed from 0 to 100
Actions #17

Updated by Davide Pesavento over 8 years ago

Alex Afanasyev wrote:

Developers should be able to run integration tests in some convenient form. I feel vagrant provides a simple, universal (and I would say "lightweight") approach to do that.

A really lightweight way of doing integration testing would be using Linux namespaces (containers) instead of full-blown virtual machines. I was actually thinking about this a few days ago: why don't we use some container technology (systemd-nspawn + some bash, docker, libvirt, even some minimal python library, whatever...) in order to conveniently run integration tests in an automated way? It would be vastly more efficient too.

Actions #18

Updated by Vince Lehman over 8 years ago

Davide Pesavento wrote:

A really lightweight way of doing integration testing would be using Linux namespaces (containers) instead of full-blown virtual machines. I was actually thinking about this a few days ago: why don't we use some container technology (systemd-nspawn + some bash, docker, libvirt, even some minimal python library, whatever...) in order to conveniently run integration tests in an automated way? It would be vastly more efficient too.

This seems to be pretty close to one of the goals we have for Mini-NDN (Task #2941). We are working towards using Mini-NDN, a Linux container based emulator forked from Mininet, to run NFD's integration tests.

Actions #19

Updated by Eric Newberry over 8 years ago

Most of our Jenkins system is run with Vagrant, so it was simple to use Vagrant for this.

We also needed to simulate multiple networks connecting the VMs in the setup. I'm uncertain if this is possible with LXC.

Actions #20

Updated by Davide Pesavento over 8 years ago

Eric Newberry wrote:

We also needed to simulate multiple networks connecting the VMs in the setup. I'm uncertain if this is possible with LXC.

Yes, in fact that part is almost identical between a VM-based setup and a container-based setup. You just have different network namespaces instead of different VMs.

Actions #22

Updated by Alex Afanasyev over 8 years ago

Davide Pesavento wrote:

A really lightweight way of doing integration testing would be using Linux namespaces (containers) instead of full-blown virtual machines. I was actually thinking about this a few days ago: why don't we use some container technology (systemd-nspawn + some bash, docker, libvirt, even some minimal python library, whatever...) in order to conveniently run integration tests in an automated way? It would be vastly more efficient too.

This is another good way of doing tests. However, we wouldn't be able to test some of face types (ethernet) using containers.

I think the main point is not lighweightiness, but simplicity of running stuff and functionality. Spawning a few linux VMs with vagrant is a relatively quick operation (though I agreee that it is still much heavier than docker-like thing).

Actions #24

Updated by Junxiao Shi over 8 years ago

  • Status changed from Code review to Closed
Actions

Also available in: Atom PDF