Bug #3676
closed
WebSocketTransport/StaticPropertiesNonLocalIpv4 RemoteUri mismatch
Added by Junxiao Shi almost 8 years ago. Updated over 7 years ago.
100%
Description
Face/TestWebSocketTransport/StaticPropertiesNonLocalIpv4 test case picks an IP address from an available NIC, creates a WebSocket server to listen on an endpoint with this IP address, creates a WebSocket client to connect to the server endpoint, and then constructs a WebSocketTransport from a connect handle accepted by the WebSocket server.
The test case then expects the WebSocketTransport to report the initially picked IP address on both LocalUri and RemoteUri fields.
However, this fails in one instance https://travis-ci.org/yoursunny/ndn-cxx-breaks/jobs/145665660#L2147-L2150:
1468878712.221268 INFO: [WebSocketTransport] [id=0,local=ws://172.17.0.1:20070,remote=wsclient://10.128.0.98:36287] Creating transport
../tests/daemon/face/websocket-transport.t.cpp(243): error in "StaticPropertiesNonLocalIpv4": check transport->getRemoteUri().getHost() == address.to_string() failed [10.128.0.98 != 172.17.0.1]
The test case assumes the underlying socket of WebSocket client would choose the same IP address, given that IP address belongs to an available NIC.
However, this assumption isn't always true. Certain configuration of the IP routing table could cause a different IP address being chosen for the WebSocket client, so that RemoteUri seen on server-side connect handle would have an unexpected IP address.
Updated by Junxiao Shi almost 8 years ago
Only a subset of TravisCS builds have this problem.
Failed builds:
- https://travis-ci.org/named-data/NFD/jobs/145205069#L2176-L2179
- https://travis-ci.org/named-data/NFD/jobs/133409154#L2174-L2177
Successful builds:
- https://travis-ci.org/named-data/NFD/jobs/144870416
- https://travis-ci.org/named-data/NFD/jobs/144814443
- https://travis-ci.org/named-data/NFD/jobs/144523320
I haven't found how to reproduce this IPv4 routing configuration in Vagrant.
Updated by Junxiao Shi over 7 years ago
I made TravisCI reveal its IP settings with this .travis.yml:
sudo: required
language: generic
script:
- ip link
- ip addr
- ip route
and triggered builds several times:
$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc mq state UP qlen 1000
link/ether 42:01:0a:f0:00:a0 brd ff:ff:ff:ff:ff:ff
$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc mq state UP qlen 1000
link/ether 42:01:0a:f0:00:a0 brd ff:ff:ff:ff:ff:ff
inet 10.240.0.160/32 brd 10.240.0.160 scope global eth0
valid_lft forever preferred_lft forever
$ ip route
default via 10.240.0.1 dev eth0
default via 10.240.0.1 dev eth0 metric 100
10.240.0.1 dev eth0 scope link
$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc mq state UP qlen 1000
link/ether 42:01:0a:80:00:1a brd ff:ff:ff:ff:ff:ff
$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc mq state UP qlen 1000
link/ether 42:01:0a:80:00:1a brd ff:ff:ff:ff:ff:ff
inet 10.128.0.26/32 brd 10.128.0.26 scope global eth0
valid_lft forever preferred_lft forever
$ ip route
default via 10.128.0.1 dev eth0
default via 10.128.0.1 dev eth0 metric 100
10.128.0.1 dev eth0 scope link
$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc mq state UP qlen 1000
link/ether 42:01:0a:80:00:26 brd ff:ff:ff:ff:ff:ff
$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc mq state UP qlen 1000
link/ether 42:01:0a:80:00:26 brd ff:ff:ff:ff:ff:ff
inet 10.128.0.38/32 brd 10.128.0.38 scope global eth0
valid_lft forever preferred_lft forever
$ ip route
default via 10.128.0.1 dev eth0
default via 10.128.0.1 dev eth0 metric 100
10.128.0.1 dev eth0 scope link
Although I can easily reproduce these settings in a Vagrant box, I still don't know where 172.17.0.1 comes from.
Updated by Junxiao Shi over 7 years ago
note-2 .travis.yml is missing two lines to select trusty environment:
os: linux
dist: trusty
After adding them, some of the outputs are:
$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 42:01:0a:f0:00:f9 brd ff:ff:ff:ff:ff:ff
3: lxcbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether 3a:d3:65:da:ae:7c brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
link/ether 02:42:ac:ab:b8:69 brd ff:ff:ff:ff:ff:ff
$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc mq state UP group default qlen 1000
link/ether 42:01:0a:f0:00:f9 brd ff:ff:ff:ff:ff:ff
inet 10.240.0.249/32 brd 10.240.0.249 scope global eth0
valid_lft forever preferred_lft forever
3: lxcbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether 3a:d3:65:da:ae:7c brd ff:ff:ff:ff:ff:ff
inet 10.0.3.1/24 brd 10.0.3.255 scope global lxcbr0
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:ac:ab:b8:69 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 scope global docker0
valid_lft forever preferred_lft forever
$ ifconfig
docker0 Link encap:Ethernet HWaddr 02:42:ac:ab:b8:69
inet addr:172.17.0.1 Bcast:0.0.0.0 Mask:255.255.0.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth0 Link encap:Ethernet HWaddr 42:01:0a:f0:00:f9
inet addr:10.240.0.249 Bcast:10.240.0.249 Mask:255.255.255.255
UP BROADCAST RUNNING MULTICAST MTU:1460 Metric:1
RX packets:2071 errors:0 dropped:0 overruns:0 frame:0
TX packets:2012 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:15774068 (15.7 MB) TX bytes:271093 (271.0 KB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
lxcbr0 Link encap:Ethernet HWaddr 3a:d3:65:da:ae:7c
inet addr:10.0.3.1 Bcast:10.0.3.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
$ ip route
default via 10.240.0.1 dev eth0
10.0.3.0/24 dev lxcbr0 proto kernel scope link src 10.0.3.1
10.240.0.1 dev eth0 scope link
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.240.0.1 0.0.0.0 UG 0 0 0 eth0
10.0.3.0 * 255.255.255.0 U 0 0 0 lxcbr0
10.240.0.1 * 255.255.255.255 UH 0 0 0 eth0
172.17.0.0 * 255.255.0.0 U 0 0 0 docker0
$ route -6
Kernel IPv6 routing table
Destination Next Hop Flag Met Ref Use If
::/0 :: !n -1 1 5 lo
::/0 :: !n -1 1 5 lo
I haven't figured out how to replicate this settings, but "docker" is a definite clue.
Updated by Junxiao Shi over 7 years ago
It's worth noting that TcpTransport/StaticPropertiesNonLocalIpv4 test case has the same logic, and the RemoteUri field is correct.
Thus, this might as well be a bug in websocket++ 0.7.
Updated by Davide Pesavento over 7 years ago
I can reproduce locally:
$ sudo modprobe dummy
$ sudo ifconfig dummy0 192.168.100.100
$ # still works
$ ./build/unit-tests-daemon -t Face/TestWebSocketTransport/StaticPropertiesNonLocalIpv4
Running 1 test case...
1474225798.881305 INFO: [WebSocketTransport] [id=0,local=ws://192.168.100.100:20070,remote=wsclient://192.168.100.100:49338] Creating transport
*** No errors detected
$ # add rule
$ sudo iptables -t nat -A POSTROUTING -s 192.168.100.0/24 ! -o dummy0 -j MASQUERADE
$ # now it fails
$ ./build/unit-tests-daemon -t Face/TestWebSocketTransport/StaticPropertiesNonLocalIpv4
Running 1 test case...
1474226205.818094 INFO: [WebSocketTransport] [id=0,local=ws://192.168.100.100:20070,remote=wsclient://10.0.2.7:49460] Creating transport
../tests/daemon/face/websocket-transport.t.cpp(249): error in "StaticPropertiesNonLocalIpv4": check transport->getRemoteUri().getHost() == address.to_string() failed [10.0.2.7 != 192.168.100.100]
*** 1 failure detected (1 failure expected) in test suite "Daemon Tests"
$ # flush chain
$ sudo iptables -t nat -F POSTROUTING
$ # works again
$ ./build/unit-tests-daemon -t Face/TestWebSocketTransport/StaticPropertiesNonLocalIpv4
Running 1 test case...
1474226294.476767 INFO: [WebSocketTransport] [id=0,local=ws://192.168.100.100:20070,remote=wsclient://192.168.100.100:49462] Creating transport
*** No errors detected
I have no idea why TcpTransport is not affected though.
$ sudo iptables -t nat -A POSTROUTING -s 192.168.100.0/24 ! -o dummy0 -j MASQUERADE
$ ./build/unit-tests-daemon -t Face/TestTcpTransport/StaticPropertiesNonLocalIpv4
Running 1 test case...
1474226629.201698 INFO: [TcpTransport] [id=0,local=tcp4://192.168.100.100:53778,remote=tcp4://192.168.100.100:7070] Creating transport
*** No errors detected
Updated by Davide Pesavento over 7 years ago
FTR, the output of iptables-save on Travis:
# Generated by iptables-save v1.4.21 on Sun Sep 18 18:05:21 2016
*mangle
:PREROUTING ACCEPT [2601:15816017]
:INPUT ACCEPT [2601:15816017]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2234:295890]
:POSTROUTING ACCEPT [2234:295890]
-A POSTROUTING -o lxcbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Sun Sep 18 18:05:22 2016
# Generated by iptables-save v1.4.21 on Sun Sep 18 18:05:22 2016
*nat
:PREROUTING ACCEPT [33:1980]
:INPUT ACCEPT [33:1980]
:OUTPUT ACCEPT [167:12678]
:POSTROUTING ACCEPT [167:12678]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 10.0.3.0/24 ! -d 10.0.3.0/24 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Sun Sep 18 18:05:22 2016
# Generated by iptables-save v1.4.21 on Sun Sep 18 18:05:22 2016
*filter
:INPUT ACCEPT [2558:15807616]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2200:294345]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:sshguard - [0:0]
-A INPUT -j sshguard
-A INPUT -i lxcbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i lxcbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i lxcbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A INPUT -i lxcbr0 -p udp -m udp --dport 67 -j ACCEPT
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o lxcbr0 -j ACCEPT
-A FORWARD -i lxcbr0 -j ACCEPT
-A DOCKER-ISOLATION -j RETURN
COMMIT
# Completed on Sun Sep 18 18:05:22 2016
Updated by Davide Pesavento over 7 years ago
I have no idea why TcpTransport is not affected though.
I figured it out.
TcpTransport tests are done from the client side (the transport is created from the socket that performs the connect), so the transport sees the "right" addresses (i.e. non-NATed) and puts 192.168.100.100 in both local and remote URIs. On the other hand, our WebSocketTransport models the server side only, so its tests are done from the server side, which only sees the NATed IP of the client, and that's what it uses to configure the remote URI of the transport.
As you can see below, in both cases the server sees the NATed IP address of the client (10.0.2.7), so the behavior is actually consistent.
# TcpTransport
$ ss -n4t
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.100.100:7070 10.0.2.7:53830
ESTAB 0 0 192.168.100.100:53830 192.168.100.100:7070
# WebSocketTransport
$ ss -n4t
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.100.100:49520 192.168.100.100:20070
ESTAB 0 0 192.168.100.100:20070 10.0.2.7:49520
Updated by Junxiao Shi over 7 years ago
- Assignee set to Weiwei Liu
- Target version set to v0.5
According to note-5 analysis, it's wrong to assume RemoteUri has the same IP address.
Thus, this test case can just check RemoteUri has the proper scheme and port number, and should not check IP address portion.
This issue is assigned to Weiwei who authored this test case in nfd:commit:93606238531f33ce8a4daf1279741731e595f57f.
Updated by Davide Pesavento over 7 years ago
Strictly speaking, you cannot check the port number either, because some forms of NAT do not always preserve port numbers.
Updated by Davide Pesavento over 7 years ago
- Status changed from New to In Progress
- Assignee changed from Weiwei Liu to Davide Pesavento
- % Done changed from 0 to 50
how about something like this? https://gerrit.named-data.net/3231
Updated by Davide Pesavento over 7 years ago
- Status changed from In Progress to Code review
- % Done changed from 50 to 100
Updated by Davide Pesavento over 7 years ago
- Status changed from Code review to Closed