Low CPU Utilization Bufferbloat Mitigation with Hardware Offloading Enabled

Banana Pi Router design OpenWRT One(MT7981)/BPI-R2 Mini
1

NOTE: SEE NEXT POST BELOW. THERE IS A MUCH BETTER SETUP.

Target Use Case:

You want to mitigate bufferbloat by capping upload speed to just below the ISP’s limit, but for whatever reason, you need to limit CPU utilization.

Standard solutions like SQM (Smart Queue Management) add too much CPU overhead, and standard Firewall4 (fw4 / nftables) rules get completely bypassed (for already-established connections) if Software or Hardware Flow Offloading is enabled.

But there is a way to mitigate bufferbloat, while using hardware and/or software offloading, while minimizing CPU utilization.

The Solution:

This method hooks directly into the network interface queue using Linux Traffic Control (tc) via a Token Bucket Filter (tbf) script. Because it configures the physical card’s egress queue, it works cooperatively alongside Hardware / Software Flow Offloading, keeps CPU usage near 0%, and perfectly caps upload speed to just below the ISP’s limit.

Step-by-Step Implementation:

Configure the Custom Startup Script

Create /etc/firewall.user:

touch /etc/firewall.user

Write the /etc/firewall.user script which defines the Traffic Control queue on the WAN interface (replace eth0 with your actual WAN interface name if it differs). You can choose between fq_codel and cake. fq_codel worked better for me.

FOR FQ_CODEL:

cat << 'EOF' > /etc/firewall.user
# 1. Clear any legacy queue configurations
tc qdisc del dev eth0 root 2>/dev/null

# 2. Add an HTB (Hierarchical Token Bucket) shaper capped at 33Mbit
# Explicitly setting quantum to 1500 prevents kernel scheduler warnings on lower speeds
tc qdisc add dev eth0 root handle 1: htb default 11
tc class add dev eth0 parent 1: classid 1:11 htb rate 33mbit ceil 33mbit quantum 3000

# 3. Attach FQ-CoDel directly inside that class to destroy upload bufferbloat
tc qdisc add dev eth0 parent 1:11 handle 11: fq_codel
EOF

You should keep the quantum at a multiple of 1500 (ie: 1500, 3000, 4500, 6000, etc.), unless you’ve set your MTU to a different size (in which case, use the multiple of your MTU size). As you increase the quantum, your jitter will increase. When it reaches a point where you no longer have an A+ Bufferbloat grade, back off a bit.

Remember, if you’re experimenting, after everything is set up, all you should have to do is make changes to the /etc/firewall.user script above (make your changes, then copy the entire script from cat to EOF into a terminal, then press Enter), then you have to restart the firewall each time you make changes, via:

/etc/init.d/firewall restart

FOR CAKE:

cat << 'EOF' > /etc/firewall.user
# 1. Clear any legacy queue configurations safely
tc qdisc del dev eth0 root 2>/dev/null

# 2. Apply Cake directly with exact Docsis framing overhead rules
# 'triple-isolate' ensures per-host fairness to protect gaming/VoIP devices
# 'docsis' handles the packet framing overhead calculations automatically
# 'wash' safely cleans up incoming diffserv marks
tc qdisc add dev eth0 root cake bandwidth 33mbit docsis triple-isolate wash
EOF

If you want to use cake, you’ll have to install two packages:

apk update
apk add kmod-sched-cake tc-full

We only target upload speed, because on fast connections, download is not often saturated. But when upload speed is saturated, that affects downloads, so we want the slower upload speed to be as stable as possible.

Setting the raw rate slightly higher (~10% higher) than your target rate yields a true payload throughput at your target rate after accounting for Ethernet, IP, and TCP header encapsulation overhead.

Register the Script in Firewall4 (fw4):

To ensure the script runs automatically every time the firewall reloads or the router reboots, register it as a script-type include block in /etc/config/firewall:

uci add firewall include
uci set firewall.@include[-1].name='Bufferbloat mitigation (USER SET)'
uci set firewall.@include[-1].type='script'
uci set firewall.@include[-1].path='/etc/firewall.user'
uci set firewall.@include[-1].fw4_compatible='1'

Save changes and restart the firewall:

uci commit firewall
/etc/init.d/firewall restart

Add:

/etc/firewall.user

… to the list at System >> Backup / Flash Firmware >> Configuration, to be sure it survives a firmware update.

And that is it.

--------------------

Monitoring dropped packets and stats:

Because this operates within the tc subsystem rather than netfilter, you monitor it directly from the network queue. You can check statistics and bufferbloat drops by running:

tc -s qdisc show dev eth0

To monitor your stats dynamically in real-time during an active speed test, use this terminal loop:

while true; do clear; tc -s qdisc show dev eth0; sleep 1; done

Hit Ctrl-C to exit the loop in the code above when you’re done testing.

To run the BufferBloat test:

https://www.waveform.com/tools/bufferbloat

2

Let’s update the procedure above to do away with the legacy iptables and user-space handling used above.

We’ll initialize this as a service, instead.

First, for those who’ve already implemented the setup above, let’s strip that all out.

Issue the command:

uci show firewall

Which should show:

firewall.wan_throttle_include=include
firewall.wan_throttle_include.type='script'
firewall.wan_throttle_include.path='/etc/firewall.user'
firewall.wan_throttle_include.fw4_compatible='1'

Delete the /etc/config/firewall include settings:

uci delete firewall.wan_throttle_include
uci commit firewall
fw4 reload

Issue the command again:

uci show firewall

The wan_throttle_include entries should all be gone.

Delete the old file:

rm /etc/firewall.user

Remove the old file from the /etc/sysupgrade.conf firmware update protection list:

sed -i '\#/etc/firewall.user#d' /etc/sysupgrade.conf

--------------------

Create the traffic-shaping file:

touch /etc/init.d/wan_throttle

Populate the traffic-shaping file:

FOR FQ_CODEL:

cat << 'EOF' > /etc/init.d/wan_throttle
#!/bin/sh /etc/rc.common

# Boots late in the system sequence to ensure the eth0 interface is fully active
START=95

# Explicitly register 'status' as an authorized custom command
EXTRA_COMMANDS="status"
EXTRA_HELP="	status          Show active kernel traffic control statistics"

# USER SETTINGS - DO NOT CHANGE ANYTHING ELSE EXCEPT THESE 3 SETTINGS!
# QUANTUM SHOULD BE A MULTIPLE OF MTU + ETHERNET HEADER SIZE
# DEFAULT: 1500 + 14 = 1514
QUANTUM=3028
UPLIMIT_MB="33"
INTERFACE='eth0'

start() {
    # Clear any legacy queue configurations on $INTERFACE
    tc qdisc del dev $INTERFACE root 2>/dev/null

    # Calculate optimum burst
    BURST_KB=$(( ((UPLIMIT_MB * 9 * 15) + 50) / 100 ))
    [ "$BURST_KB" -lt 15 ] && BURST_KB=15

    # Add HTB shaper
    tc qdisc add dev $INTERFACE root handle 1: htb default 11
    tc class add dev $INTERFACE parent 1: classid 1:11 htb rate "${UPLIMIT_MB}mbit" ceil "${UPLIMIT_MB}mbit" burst "${BURST_KB}k" quantum $QUANTUM

    # Attach FQ-CoDel inside the HTB class and explicitly bind its leaf quantum
    tc qdisc add dev $INTERFACE parent 1:11 handle 11: fq_codel quantum 512 target 15ms interval 100ms
    tc filter add dev $INTERFACE parent 1: protocol ip prio 1 u32 match ip dst 0.0.0.0/0 flowid 1:11
}

stop() {
    # Fail-safe cleanup if the service is stopped or manually restarted
    tc qdisc del dev $INTERFACE root 2>/dev/null
}

status() {
    # Automatically output active kernel traffic control statistics
    tc -s qdisc show dev $INTERFACE
}
EOF

You should keep the quantum at a multiple of 1514 (ie: 1514, 3028, 4542, 6056, etc.), unless you’ve set your MTU to a different size (in which case, use the multiple of your MTU size + Ethernet header size). As you increase the quantum , your jitter will increase. When it reaches a point where you no longer have an A+ Bufferbloat grade, back off a bit.

FOR CAKE:

cat << 'EOF' > /etc/init.d/wan_throttle
#!/bin/sh /etc/rc.common

# Boots late in the system sequence to ensure the eth0 interface is fully active
START=95

# Explicitly tell OpenWRT that 'status' is an authorized custom command
EXTRA_COMMANDS="status"
EXTRA_HELP="	status          Show active kernel traffic control statistics"

# USER SETTINGS - DO NOT CHANGE ANYTHING ELSE EXCEPT THESE 2 SETTINGS!
UPLIMIT_MB="33"
INTERFACE='eth0'

start() {
    # 1. Clear any legacy queue configurations safely
    tc qdisc del dev $INTERFACE root 2>/dev/null

    # 2. Apply CAKE directly with DOCSIS overhead and fairness rules
    tc qdisc add dev $INTERFACE root cake bandwidth "${UPLIMIT_MB}mbit" docsis triple-isolate wash
}

stop() {
    # Fail-safe cleanup if the service is stopped or manually restarted
    tc qdisc del dev $INTERFACE root 2>/dev/null
}

status() {
    # Automatically output active kernel traffic control statistics
    tc -s qdisc show dev $INTERFACE
}
EOF

Make the traffic-shaping file executable:

chmod +x /etc/init.d/wan_throttle

Enable and start the wan_throttle service:

/etc/init.d/wan_throttle enable
/etc/init.d/wan_throttle start

Ensure the wan_throttle script is working properly:

tc -s qdisc show dev eth0

If you see “active with no instances”, something has gone wrong. You should see the connection stats.

Enforce firmware update persistence:

echo "/etc/init.d/wan_throttle" >> /etc/sysupgrade.conf

Now test for bufferbloat: https://www.waveform.com/tools/bufferbloat

--------------------

Add an alias (so you don’t have to type the /etc/init.d/ path when calling wan_throttle):

echo "alias wan_throttle='/etc/init.d/wan_throttle'" >> /etc/profile

Reload the profile:

source /etc/profile

Make it survive firmware updates:

echo "/etc/profile" >> /etc/sysupgrade.conf

When experimenting:

  1. Change the settings in the /etc/init.d/wan_throttle file above (whichever of the 2 scripts you chose to use), then copy the entire thing from ‘cat’ to ‘EOF’ and paste it into a Terminal window, then press Enter.

  2. To initialize the changes, you have to do:

wan_throttle restart
  1. Test for bufferbloat again: https://www.waveform.com/tools/bufferbloat

You can now monitor the status of the wan_throttle service via:

wan_throttle status
3

Can this be applied when wan is a switch port (e.g. wan@eth0)? or a bridge (e.g., br-wan)?

4

The above works well if you have a single, fixed-bandwidth connection… but if you’re on a dynamic-bandwidth connection, it won’t work.

Further, if you’re multi-WAN’d (to get a backup data path if your main ISP goes down) and the different WANs have different bandwidths, your bufferbloat will return on the slower connection… so let’s make the setting of the upload limit dynamic.

In a terminal window logged into the OpenWRT One router via ssh (issue: ssh [email protected] then enter the root password), copy and paste the below all-in-one-go, then press Enter.

cat << 'EOF' > /etc/init.d/wan_throttle_autorate
#!/bin/sh /etc/rc.common

START=96

EXTRA_COMMANDS="status"
EXTRA_HELP="	status          Show traffic shaping statistics"

# ===v===v=== USER SETTINGS===v===v===
INTERFACE='eth0'
SCHEDULER="FQ_CODEL"       # Options: FQ_CODEL or CAKE

# Make this a multiple of MTU size (usually 1500) + Ethernet header size
QUANTUM=3028

# Set MAX_KBPS to your target maximum+10%, to account for protocol overhead.
MAX_KBPS=33000             # Maximum upload ceiling (kbps)
MIN_KBPS=100               # Minimum upload floor (kbps)

DECAY_RATE_PERCENT=10       # Percent to decay during idle periods (1-25%)
DECAY_TRIGGER=50          # Data rate below which decay starts (kbps)

ENABLE_LOGGING=0          # Options: 1 or 0
# ===^===^=== USER SETTINGS===^===^===

PID_FILE="/tmp/wan_throttle_autorate.pid"
STATE_FILE="/tmp/wan_throttle_autorate.state"

get_tx_bytes() {
    grep "$INTERFACE:" /proc/net/dev | awk '{print $10}'
}

run_daemon() {
    LAST_SET_SHAPER=0
    POLL_INTERVAL=5

    # Initialize at full speed
    CURRENT_PEAK_KBPS=$MAX_KBPS

        # Write state to file in RAM
        echo "Interval: 5s | Instant: 1000kbps | Tracked Peak: 1000kbps | Burst: 15k | Shaper injected: ${SHAPER_TARGET_KBPS}kbit" > "$STATE_FILE"

    TX_PREV=$(get_tx_bytes)

    while true; do
        sleep $POLL_INTERVAL

        TX_CURR=$(get_tx_bytes)
        if [ "$TX_CURR" -lt "$TX_PREV" ]; then
            TX_PREV=$TX_CURR
            continue
        fi

        BYTES_DELTA=$((TX_CURR - TX_PREV))
        INSTANT_BYTES_PER_SEC=$((BYTES_DELTA / POLL_INTERVAL))
        TX_PREV=$TX_CURR

        # Calculate instant throughput in kbps
        INSTANT_KBPS=$(( (INSTANT_BYTES_PER_SEC * 8) / 1000 ))

        # --- AUTO-SCALE LOGIC ---
        if [ "$LAST_SET_SHAPER" -gt 0 ]; then
            UP_THRESHOLD_KBPS=$(( (LAST_SET_SHAPER * 3) / 4 ))
        else
            UP_THRESHOLD_KBPS=$(( (MAX_KBPS * 3) / 4 ))
        fi

        if [ "$INSTANT_KBPS" -ge "$UP_THRESHOLD_KBPS" ]; then
            CURRENT_PEAK_KBPS=$(( (INSTANT_KBPS * 3) / 2 ))
            [ "$CURRENT_PEAK_KBPS" -gt "$MAX_KBPS" ] && CURRENT_PEAK_KBPS=$MAX_KBPS
            POLL_INTERVAL=1

        elif [ "$INSTANT_KBPS" -ge "$DECAY_TRIGGER" ]; then
            POLL_INTERVAL=1

        else
            DECAY_AMOUNT=$(( (CURRENT_PEAK_KBPS * DECAY_RATE_PERCENT) / 100 ))
            [ "$DECAY_AMOUNT" -lt 1 ] && DECAY_AMOUNT=1

            CURRENT_PEAK_KBPS=$(( CURRENT_PEAK_KBPS - DECAY_AMOUNT ))
            POLL_INTERVAL=5
        fi

        # Enforce minimum data rate
        if [ "$CURRENT_PEAK_KBPS" -lt "$MIN_KBPS" ]; then
            CURRENT_PEAK_KBPS=$MIN_KBPS
        fi

        # Set shaper to 90% of tracked peak for protocol overhead margin
        SHAPER_TARGET_KBPS=$(( (CURRENT_PEAK_KBPS * 90) / 100 ))
        [ "$SHAPER_TARGET_KBPS" -lt "$MIN_KBPS" ] && SHAPER_TARGET_KBPS=$MIN_KBPS

        # Set new limit only upon actual change
        if [ "$LAST_SET_SHAPER" != "$SHAPER_TARGET_KBPS" ]; then
            if [ "$SCHEDULER" = "CAKE" ]; then
                tc qdisc change dev "$INTERFACE" root cake bandwidth "${SHAPER_TARGET_KBPS}kbit" docsis triple-isolate wash 2>/dev/null

            elif [ "$SCHEDULER" = "FQ_CODEL" ]; then
                BURST_KB=$(( ((SHAPER_TARGET_KBPS * 9 * 15) + 50000) / 100000 ))
                [ "$BURST_KB" -lt 15 ] && BURST_KB=15

               tc class change dev "$INTERFACE" parent 1: classid 1:11 htb rate "${SHAPER_TARGET_KBPS}kbit" ceil "${SHAPER_TARGET_KBPS}kbit" burst "${BURST_KB}k" quantum "$QUANTUM"  2>/dev/null

            fi

        # Write state to file in RAM
        echo "Interval: ${POLL_INTERVAL}s | Instant: ${INSTANT_KBPS}kbps | Tracked Peak: ${CURRENT_PEAK_KBPS}kbps | Burst: ${BURST_KB}k | Shaper injected: ${SHAPER_TARGET_KBPS}kbit" > "$STATE_FILE"

            [ "$ENABLE_LOGGING" -eq 1 ] && logger -t wan_throttle_autorate "Scheduler: ${SCHEDULER} | Interval: ${POLL_INTERVAL}s | Instant: ${INSTANT_KBPS}k | Tracked Peak: ${CURRENT_PEAK_KBPS}k | Shaper: ${SHAPER_TARGET_KBPS}kbit | Burst: "${BURST_KB}k" | QUANTUM: ${QUANTUM}"
            LAST_SET_SHAPER=$SHAPER_TARGET_KBPS
        fi
    done
}

start() {
    # Prevent duplicate daemon instances from spawning
    if [ -f "$PID_FILE" ] && kill -0 $(cat "$PID_FILE") 2>/dev/null; then
        echo "wan_throttle_autorate is already running."
        exit 0
    fi

    # Clear any legacy queue configurations on $INTERFACE
    tc qdisc del dev "$INTERFACE" root 2>/dev/null

    # Create the queue architecture based on user selection
    if [ "$SCHEDULER" = "CAKE" ]; then
        # CAKE attaches directly to the root interface. Seed it at the max set speed.
        tc qdisc add dev "$INTERFACE" root cake bandwidth "${MAX_KBPS}kbit" docsis triple-isolate wash 2>/dev/null

    elif [ "$SCHEDULER" = "FQ_CODEL" ]; then
        # Calculate an initial burst limit for the maximum ceiling speed
        BURST_KB=$(( ((MAX_KBPS * 9 * 15) + 50000) / 100000 ))
        [ "$BURST_KB" -lt 15 ] && BURST_KB=15

        # Add HTB shaper
        tc qdisc add dev "$INTERFACE" root handle 1: htb default 11 2>/dev/null
        tc class add dev "$INTERFACE" parent 1: classid 1:11 htb rate "${MAX_KBPS}kbit" ceil "${MAX_KBPS}kbit" burst "${BURST_KB}k" quantum "$QUANTUM"  2>/dev/null

        # Attach FQ-CoDel inside the HTB class and explicitly bind its leaf quantum
        tc qdisc add dev "$INTERFACE" parent 1:11 handle 11: fq_codel quantum 512 target 15ms interval 100ms  2>/dev/null
        tc filter add dev "$INTERFACE" parent 1: protocol ip prio 1 u32 match ip dst 0.0.0.0/0 flowid 1:11

    else
        logger -t wan_throttle_autorate "SCHEDULER: $SCHEDULER should be either CAKE or FQ-CODEL"
        exit 1
    fi

    # Launch the background autorate daemon
    run_daemon &
    echo $! > "$PID_FILE"
    logger -t wan_throttle_autorate "wan_throttle_autorate started."
}

stop() {
    if [ -f "$PID_FILE" ] && kill -0 $(cat "$PID_FILE") 2>/dev/null; then
        kill $(cat "$PID_FILE") 2>/dev/null
        rm -f "$PID_FILE"
        rm -f "$STATE_FILE"
    fi
    logger -t wan_throttle_autorate "wan_throttle_autorate stopped."

    if [ "$SCHEDULER" = "CAKE" ]; then
        # Cake attaches directly to the root. Delete the root qdisc structure.
        tc qdisc del dev "$INTERFACE" root 2>/dev/null
    elif [ "$SCHEDULER" = "FQ_CODEL" ]; then
        # HTB creates a parent structure at 1:. Deleting it completely un-throttles the connection.
        tc qdisc del dev "$INTERFACE" root handle 1: 2>/dev/null
        # Fallback flush just in case the handles are nested differently
        tc qdisc del dev "$INTERFACE" root 2>/dev/null
    fi
}

status() {
    if [ -f "$PID_FILE" ] && kill -0 $(cat "$PID_FILE") 2>/dev/null; then
        echo "wan_throttle_autorate status: RUNNING (PID: $(cat "$PID_FILE")) [Mode: "$SCHEDULER"]"

        if [ -f "$STATE_FILE" ]; then
            printf "\n%s\n\n" "$(cat "$STATE_FILE")"

            logger -t wan_throttle_autorate "Status: RUNNING | $(cat "$STATE_FILE" | tr '\n' ' ')"
        fi
    else
        echo "wan_throttle_autorate stopped."
        logger -t wan_throttle_autorate "wan_throttle_autorate stopped."
    fi
}
EOF

Make it executable:

chmod +x /etc/init.d/wan_throttle_autorate

In a terminal window connected to the router via ssh, set up a log monitor:

logread -f | grep wan_throttle_autorate

Leave that running.

In another terminal window connected to the router via ssh:

Start it manually to test it:

/etc/init.d/wan_throttle_autorate start

In the log monitor window, you should see log messages for the service.

Check the status:

/etc/init.d/wan_throttle_autorate status

Stop it:

/etc/init.d/wan_throttle_autorate stop

In the LuCI interface, go to System >> Startup, scroll down the list until you find wan_throttle_autorate. It should have a red button that says “Disabled”. Click that to enable it, then click the “Start” button.

Or you can issue:

/etc/init.d/wan_throttle_autorate enable
/etc/init.d/wan_throttle_autorate start

wan_throttle_autorate DOES NOT require wan_throttle (in prior post above). It replaces it. Both can be run… wan_throttle will start up first, wan_throttle_autorate will start up next, there should be no conflicts between the two.

Add an alias (so you don’t have to type the /etc/init.d/ path when calling wan_throttle_autorate):

echo "alias wan_throttle_autorate='/etc/init.d/wan_throttle_autorate'" >> /etc/profile

Reload the profile:

source /etc/profile

Make it survive firmware updates: Go to System >> Backup / Flash Firmware >> Configuration tab. Ensure that:

/etc/init.d/wan_throttle_autorate
/etc/profile

… are in that list.

Commands:

Start it:

wan_throttle_autorate start

Stop it:

wan_throttle_autorate stop

Restart it:

wan_throttle_autorate restart

Display status:

wan_throttle_autorate status

The script logs when ‘status’ is explicitly called… even if you’ve got logging disabled, you can get it to log by issuing:

while true; do wan_throttle_autorate status && sleep 5; done