How to configure BPi-R2 as one-armed "router on a stick"

Banana Pi Router design
1

I’ve been trying to configure the BPi-R2 as a one-armed router (of a fashion), and have hit some problems I don’t understand. I’m conversant enough with Linux networking, including software bridging with the old brctl tool, but it’s my first experience with DSA and the ip-bridge utility, and I’ve probably made a mistake.

I wonder if the below approach of adding a wan VLAN interface to the br-lan bridge is the right way to do it and I’m just missing some detail, or if I should switch to the (unfamiliar) bridge vlan tool in a completely different approach, or what. Ideas?

Facility network environment: We have a router (MikroTik PoE running RouterOS) with all active ports configured as “hybrid” trunk ports, i.e. untagged traffic is VLAN 1, and tagged VLANs 100-103. A couple of WiFi APs (Ubiquiti AC running OpenWRT) broadcast several ESSIDs, each bridged to one of the VLANs 100-103.

Goal configuration: The goal is to set up the BPi-R2 in a lab as a router. It will have a PXE boot environment on the br-lan bridge for provisioning machines. The wan interface forwards Internet traffic through the MikroTik, as you’d expect. Also, the wan interface should forward VLAN 103 traffic between the BPi’s br-lan bridge and the rest of the network to provide access to lab machines via the lab ESSID from other parts of the facility.

Current config (failing): The wan port is connected to one of the hybrid trunk ports on the Mikrotik, untagged, carrying masqueraded traffic from the br-lan bridge to the Internet. The VLAN 103 traffic goes over the wan.103 port, which is slaved to the br-lan bridge together with the four lan0-lan3 ports. This configuration partly works: hosts wired to the lan0-lan3 ports can access the Internet and hosts on the lab ESSID can, too. The BPi-R2 can ping both groups of hosts. However, the two groups of hosts cannot talk with each other.

OS and kernel: I actually started this experiment with Debian Buster with @frank-w’s 5.4 kernel, and when I hit the wall, switched over to OpenWRT (built from master branch, kernel 5.4.101), and seeing the exact same problem.

Output of `ip`, `bridge`, etc. utilities 
root@OpenWrt:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1504 qdisc fq_codel state UP qlen 1000
    link/ether aa:b6:32:5c:a2:33 brd ff:ff:ff:ff:ff:ff
3: wan@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether aa:b6:32:5c:a2:33 brd ff:ff:ff:ff:ff:ff
4: lan0@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN qlen 1000
    link/ether aa:b6:32:5c:a2:33 brd ff:ff:ff:ff:ff:ff
5: lan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether aa:b6:32:5c:a2:33 brd ff:ff:ff:ff:ff:ff
6: lan2@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN qlen 1000
    link/ether aa:b6:32:5c:a2:33 brd ff:ff:ff:ff:ff:ff
7: lan3@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN qlen 1000
    link/ether aa:b6:32:5c:a2:33 brd ff:ff:ff:ff:ff:ff
11: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether aa:b6:32:5c:a2:33 brd ff:ff:ff:ff:ff:ff
12: wan.103@wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether aa:b6:32:5c:a2:33 brd ff:ff:ff:ff:ff:ff
root@OpenWrt:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1504 qdisc fq_codel state UP qlen 1000
    link/ether aa:b6:32:5c:a2:33 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a8b6:32ff:fe5c:a233/64 scope link 
       valid_lft forever preferred_lft forever
3: wan@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether aa:b6:32:5c:a2:33 brd ff:ff:ff:ff:ff:ff
    inet 10.88.0.222/24 brd 10.88.0.255 scope global wan
       valid_lft forever preferred_lft forever
    inet6 2605:a601:ab40:4000:a8b6:32ff:fe5c:a233/64 scope global dynamic noprefixroute 
       valid_lft 14274sec preferred_lft 14274sec
    inet6 fe80::a8b6:32ff:fe5c:a233/64 scope link 
       valid_lft forever preferred_lft forever
4: lan0@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN qlen 1000
    link/ether aa:b6:32:5c:a2:33 brd ff:ff:ff:ff:ff:ff
5: lan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether aa:b6:32:5c:a2:33 brd ff:ff:ff:ff:ff:ff
6: lan2@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN qlen 1000
    link/ether aa:b6:32:5c:a2:33 brd ff:ff:ff:ff:ff:ff
7: lan3@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN qlen 1000
    link/ether aa:b6:32:5c:a2:33 brd ff:ff:ff:ff:ff:ff
11: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether aa:b6:32:5c:a2:33 brd ff:ff:ff:ff:ff:ff
    inet 10.64.0.1/24 brd 10.64.0.255 scope global br-lan
       valid_lft forever preferred_lft forever
    inet6 fd9b:4875:7d7::1/60 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::a8b6:32ff:fe5c:a233/64 scope link 
       valid_lft forever preferred_lft forever
12: wan.103@wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether aa:b6:32:5c:a2:33 brd ff:ff:ff:ff:ff:ff
root@OpenWrt:~# bridge link
4: lan0@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br-lan state disabled priority 32 cost 100 
5: lan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br-lan state forwarding priority 32 cost 4 
6: lan2@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br-lan state disabled priority 32 cost 100 
7: lan3@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br-lan state disabled priority 32 cost 4 
12: wan.103@wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br-lan state forwarding priority 32 cost 4 
root@OpenWrt:~# brctl show
bridge name     bridge id               STP enabled     interfaces
br-lan          7fff.aab6325ca233       yes             lan0
                                                        lan1
                                                        lan2
                                                        lan3
                                                        wan.103
root@OpenWrt:~# bridge vlan
port    vlan ids
lan0     1 PVID Egress Untagged

lan1     1 PVID Egress Untagged

lan2     1 PVID Egress Untagged

lan3     1 PVID Egress Untagged

br-lan   1 PVID Egress Untagged

wan.103  1 PVID Egress Untagged
2

I got the one-armed “router on a stick” configuration working perfectly. Instructions for doing it manually and doing it with Debian/Ubuntu’s /etc/network/interfaces follow.

My intuition that the new ip-bridge utility is the way forward was correct. The old brctl utility is apparently obsolete, and can’t configure a VLAN-aware bridge. Because of this and no direct support for veth virtual network interfaces, the Debian config relies heavily on inelegant pre-up/post-up commands.

This configuration could also be done on OpenWRT, but will require installing the ip-bridge and (IIRC) ip-full packages, and I believe can’t be done in LuCI.

On OSs with systemd-networkd, this configuration is supported natively, which is more elegant, but may be unfamiliar to many and may not be nicely integrated into user-friendly configuration interfaces. (I’d be interested to hear otherwise.)

Command line procedure

Here’s the manual procedure to set up the one-armed router. The ip and bridge utilities must be installed.

`bash` commands for manual configuration 
# Create the "internal" network bridge; must be down if already exists
ip link add name br_int type bridge
ip link set dev br_int down
# Set vlans:  br_int and lan0-lan3 all on VLAN 103, untagged
ip link set dev br_int type bridge vlan_filtering 1
for i in 0 1 2 3; do
    bridge vlan del dev lan${i} vid 1
    bridge vlan add dev lan${i} vid 103 pvid untagged
done
bridge vlan del dev br_int vid 1 self
bridge vlan add dev br_int vid 103 pvid untagged self
# Bring up the bridge interface
ip set dev br_int up

# At this point, start the DHCP server; hosts should be able to
# communicate on local network

# Add wan interface to bridge and set VLANs: 103 tagged (PVID 1 set by default)
ip link set dev wan master br_int
bridge vlan add dev wan vid 103
# Add "external" network VLAN interface to bridge and configure
ip link add link br_int name br_ext type vlan id 1
bridge vlan add dev br_int vid 1 self
# Now configure external IP and default route or run dhclient.  Done!

Using /etc/network/interfaces:

In OpenWRT I couldn’t quickly figure out how to make settings persist, or make built .ipks available, or recognize the SATA disk, and needed to rebuild the kernel with CONFIG_VETH, and lots of other things were missing to even get to where I could start getting Docker Engine to work, and on top of that, OpenWRT is still using these obsolete tools, so I punted. The Docker network appliance would’ve been pretty cool, though.

Switching back to Debian, the following /etc/network/interfaces file configures the same network. Debian also seems to use the obsolete brctl tool and needs hackery to add the full configuration, but it was easy, even if ugly.

`/etc/network/interfaces` file on Debian/Ubuntu 
auto eth0
iface eth0 inet manual
  pre-up ip link set $IFACE up
  post-down ip link set $IFACE down

auto wan
iface wan inet manual
    pre-up ip link set $IFACE up
    post-down ip link set $IFACE down

auto lan0
iface lan0 inet manual
    pre-up ip link set $IFACE up
    post-down ip link set $IFACE down

auto lan1
iface lan1 inet manual
    pre-up ip link set $IFACE up
    post-down ip link set $IFACE down

auto lan2
iface lan2 inet manual
    pre-up ip link set $IFACE up
    post-down ip link set $IFACE down

auto lan3
iface lan3 inet manual
    pre-up ip link set $IFACE up
    post-down ip link set $IFACE down

auto br_int
iface br_int inet static
    # Create bridge interface
    pre-up ip link add name br_int type bridge
    # Enable VLAN filtering on bridge
    pre-up ip link set dev br_int type bridge vlan_filtering 1
    # Set VLANs:  PVID 103; untagged 1
    pre-up bridge vlan add dev $IFACE vid 103 pvid untagged self
    pre-up bridge vlan del dev $IFACE vid 1 self
    pre-up bridge vlan add dev $IFACE vid 1 self
    address 10.64.0.1
    netmask 255.255.255.0
    bridge_ports lan0 lan1 lan2 lan3 wan
    bridge_stp yes
    bridge_maxwait 0
    # Configure port VLANs
    post-up bridge vlan add dev wan vid 103
    post-up bridge vlan del dev lan0 vid 1
    post-up bridge vlan add dev lan0 vid 103 pvid untagged
    post-up bridge vlan del dev lan1 vid 1
    post-up bridge vlan add dev lan1 vid 103 pvid untagged
    post-up bridge vlan del dev lan2 vid 1
    post-up bridge vlan add dev lan2 vid 103 pvid untagged
    post-up bridge vlan del dev lan3 vid 1
    post-up bridge vlan add dev lan3 vid 103 pvid untagged

iface br_int inet6 static
    # fd = private; 22:c8b4:d740 = random global id; 0064 = subnet id; addr 1
    address fd22:c8b4:d740:0064::1
    netmask 64
    dad-attempts 0

auto br_ext
iface br_ext inet dhcp
    pre-up ip link add link br_int name br_ext type vlan id 1
    vlan-raw-device br_int
    post-down ip link set $IFACE down

Results

This configuration satisfies the OP “goal configuration” perfectly.

Now, both wired clients (on the BPi-R2’s LAN ports) and wireless clients (bridged in over VLAN 103 on the BPi-R2’s WAN port) can get DHCP addresses from the BPi-R2 and can talk to each other on the “internal” network. They can also all connect to the Internet with traffic masqueraded over the BPi-R2’s WAN port (untagged VLAN 1).

Diagnostic shell commands showing bridge and VLAN configuration 
root@bpi-r2:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether a2:2a:4d:2a:d4:7e brd ff:ff:ff:ff:ff:ff
3: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
4: wan@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br_int state UP mode DEFAULT group default qlen 1000
    link/ether a2:2a:4d:2a:d4:7e brd ff:ff:ff:ff:ff:ff
5: lan0@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br_int state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether a2:2a:4d:2a:d4:7e brd ff:ff:ff:ff:ff:ff
6: lan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br_int state UP mode DEFAULT group default qlen 1000
    link/ether a2:2a:4d:2a:d4:7e brd ff:ff:ff:ff:ff:ff
7: lan2@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br_int state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether a2:2a:4d:2a:d4:7e brd ff:ff:ff:ff:ff:ff
8: lan3@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br_int state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether a2:2a:4d:2a:d4:7e brd ff:ff:ff:ff:ff:ff
9: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:08:22:2a:c1:fb brd ff:ff:ff:ff:ff:ff
10: br_int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether a2:2a:4d:2a:d4:7e brd ff:ff:ff:ff:ff:ff
11: br_ext@br_int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether a2:2a:4d:2a:d4:7e brd ff:ff:ff:ff:ff:ff
12: br-7ac2e4ace930: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:58:f1:46:2a brd ff:ff:ff:ff:ff:ff
13: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:1f:f4:b4:70 brd ff:ff:ff:ff:ff:ff
15: veth6d93e8e@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-7ac2e4ace930 state UP mode DEFAULT group default 
    link/ether 3a:14:e8:48:4a:61 brd ff:ff:ff:ff:ff:ff link-netnsid 0
root@bpi-r2:~# bridge link
4: wan state UP @eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br_int state forwarding priority 32 cost 4 
5: lan0 state LOWERLAYERDOWN @eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br_int state disabled priority 32 cost 100 
6: lan1 state UP @eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br_int state forwarding priority 32 cost 4 
7: lan2 state LOWERLAYERDOWN @eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br_int state disabled priority 32 cost 100 
8: lan3 state LOWERLAYERDOWN @eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br_int state disabled priority 32 cost 100 
15: veth6d93e8e state UP @(null): <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br-7ac2e4ace930 state forwarding priority 32 cost 2 
root@bpi-r2:~# bridge vlan
port    vlan ids
wan      1 PVID Egress Untagged
         103
lan0     103 PVID Egress Untagged
lan1     103 PVID Egress Untagged
lan2     103 PVID Egress Untagged
lan3     103 PVID Egress Untagged
br_int   1
         103 PVID Egress Untagged
br-7ac2e4ace930  1 PVID Egress Untagged
docker0  1 PVID Egress Untagged
veth6d93e8e      1 PVID Egress Untagged
3

So all is working now? What was the mistake? For ugly interfaces-file you can create a script containing commands for br_int and call this in pre-up/post-up (passing mode and iface as param)…maybe use it for other interfaces

Afaik debian uses pre-up/post-up hooks /etc/network/ip-*.d/bridge,which is a link to script which calls brctl. Maybe newer network-daemon (systemd-networkd, netplan) support bridge command natively,but then you need ro configure full network with this

4

Did you see the complete instructions I posted for both manual and /etc/network/interfaces, plus debug commands? I hid them in expandable sections in order to keep the post tidy, but maybe that’s not helpful if folks who might benefit won’t notice them.

5

I saw them,but i wanted to know if it now works…

6

Oh, yes, works perfectly! I’ll edit to make that more clear. Thanks, and I’m happy for further suggestions to make this more useful.

7

Only the one i posted above to move code (params interface and state) to a script to have only 2 lines per interface in interfaces file. And this is then also portable to other network daemons