Look where crypto_givcipher_type and sys_close are defined and make sure file with implementation is linked by makefile.
UPDATE: I have removed version 1.9 and pulled the latest master from github and it works fine. It appears 1.9 release does not work with 5.x kernels and it was fixed after 1.10 release. (https://github.com/cryptodev-linux/cryptodev-linux/commit/f971e0cd4a0ebe59fb2e8e17240399bf6901b09b)
1 Like
i have added cryptodev 1.10 + the Patch you’ve sorted out to my 5.4-main tree
thanks for checking
5.4.27-bpi-r2-main
I could not get cryptodev working. I spent nearly two whole days trying to figure out how to enable it for OpenSSL.
It turns out, that devcryptoeng is not enabled by default and the defines HAVE_CRYPTODEV and USE_CRYPTODEV_DIGESTS don’t change a thing about that. You need to add one more argument - enable-devcryptoeng.
Working commands for me were
$ export CROSS=arm-linux-gnueabihf
$ export CC=${CROSS}-gcc
$ export LD=${CROSS}-ld
$ export AS=${CROSS}-as
$ export AR=${CROSS}-ar
$ export DEB_HOST_ARCH=armhf
$ export DEB_BUILD_OPTIONS=nocheck
$ git clone https://github.com/cryptodev-linux/cryptodev-linux.git
$ apt-get source openssl
# You must match your version
$ cd openssl-1.1.1.d
$ sed -i -e "s/CONFARGS =/CONFARGS = enable-devcryptoeng -DHAVE_CRYPTODEV -DUSE_CRYPTODEV_DIGESTS -I..\/..\/cryptodev-linux/" debian/rules
# I also had to disable shlibdeps by commenting `dh_shlibdeps` at the end of the `debian/rules`
$dpkg-buildpackage -us -uc -aarmhf -b
You need to have cryptodev already compiled. Then just install created deb packages.
Hope this helps. If you found anything wrong, let me know and I will edit this post.
1 Like
Can you post a patch for cryptodev i included in my kernelrepo (5.4-main)?
The soonest I will have time to do it is maybe next weekend.
Anyway… I had problem with SSH. I couldn’t establish a connection. devcrypto was obviously on fault.
I found this article where it says where is the problem…
Disabling digests
Please, don’t enable digests unless you know what you’re doing. They are usually slower than software, >except for large (> 10k) blocks. Some applications–openssh, for example–will not work with /dev/crypto >digests. This is a limitation of how the engine works. Openssh will save a partial digest, and then fork, >duplicating that context, and working with successive copies of it, which is useful for HMAC, where the >hash of the key remains constant. In the kernel, however, those contexts are still linked to the same >session, so when one process calls another update, or closes that digest context, the kernel session is >changed/closed for all of the instances, and you’ll get a libcrypto failure. For well-behaved applications >using large update blocks, you may enable digests. Use a separate copy of the
openssl.cnf>configuration file, and setOPENSSL_CONF=_path_to_filein the environment before running it (add it >to the respective file in /etc/init.d/). Again, benchmarking the actual application you’re using is the best >way to gauge the impact of hardware crypto.
I tried to disable using DIGESTS in OpenSSL through configuration, but it seems that it ignores default_algorithms for cryptodev, even with USE_SOFTDRIVERS=1 I also removed defines for compilation, so that only enable-devcryptoeng was left, but it didn’t help. At least I found out, that cryptodev can be enabled just with enable-devcryptoeng and no other defines are needed.
Anyway… that lead me to the only possible solution left - removing it from source. Source file where you can disable it is in openssl/crypto/engines
After I compiled it, I got this (missing digests)
root@claudius:~# openssl engine -c
(devcrypto) /dev/crypto engine
[DES-CBC, DES-EDE3-CBC, AES-128-CBC, AES-192-CBC, AES-256-CBC, AES-128-CTR, AES-192-CTR, AES-256-CTR, AES-128-ECB, AES-192-ECB, AES-256-ECB]
(dynamic) Dynamic engine loading support
I am planning to disable all ciphers except AES-CBC. I need to use cryptodev only for VPN and more ciphers would just add more interrupts to core0.
If somebody could figure out how to disable cryptodev for DIGESTS without making modifications to source, that would be great. Here is a “documentation” https://www.openssl.org/docs/man1.1.1/man5/config.html#Engine-Configuration-Module
I couldn’t find any usage of engine config, except for some sample
1 Like
I had also that problem with SSH when I enabled devcrypto for OpenSSL. Initially I did not know why I cannot login and then I found some page that stated you cannot accelerate OpenSSH connection because latest version of OpenSSH enforce usage of seccomp sandbox which forbids some syscalls required to use AF_ALG. I think the same applied to devcrypto engine also.
Refer to Debian bug #931271.
I have recompiled openssh from source and disabled seccomp sandbox to allow me to connect remotely to the box
#Edit debian/rules and add it to common build options:
confflags += --with-sandbox=no
1 Like
Hi everyone!
I’ve tested my R2 with AF_ALG acceleration - work good for me. All i needed to do is to add following kernel config options:
CONFIG_CRYPTO_USER=m
CONFIG_CRYPTO_USER_API=m # added automatically after one options below was added
CONFIG_CRYPTO_USER_API_HASH=m # probably usable, but was not used this time
CONFIG_CRYPTO_USER_API_SKCIPHER=m # this one was really used
CONFIG_CRYPTO_USER_API_RNG=m # - theese two are kikely useless for R2
CONFIG_CRYPTO_USER_API_AEAD=m # /
After recompiling kernel and reboot, ssl conf needed to be changed:
cat /etc/ssl/openssl.cnf:
openssl_conf = openssl_def
[openssl_def]
engines = openssl_engines
[openssl_engines]
afalg = afalg_engine
[afalg_engine]
init=1
openssl engine list:
r2-gentoo ~ # openssl engine -t -c -v
(dynamic) Dynamic engine loading support
[ unavailable ]
SO_PATH, NO_VCHECK, ID, LIST_ADD, DIR_LOAD, DIR_ADD, LOAD
(afalg) AFALG engine support
[AES-128-CBC, AES-192-CBC, AES-256-CBC]
[ available ]
and finnaly perfomance test:
no acceleration:
bpi-r2-gentoo ~ # time openssl speed -evp aes-128-cbc -elapsed
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128-cbc for 3s on 16 size blocks: 4551769 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 64 size blocks: 1420522 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 357557 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 79175 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 9870 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 16384 size blocks: 4496 aes-128-cbc's in 3.00s
OpenSSL 1.1.1j 16 Feb 2021
built on: Fri Feb 19 22:05:09 2021 UTC
options:bn(64,32) rc4(char) des(long) aes(partial) idea(int) blowfish(ptr)
compiler: armv7a-hardfloat-linux-gnueabi-gcc -fPIC -pthread -Wa,--noexecstack -Wall -O2 -pipe -march=armv7-a -mfpu=neon-vfpv4 -mfloat-abi=hard -fno-strict-aliasing -Wa,--noexecstack -DOPENSSL_USE_NODELETE -DOPEN
SSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DZLIB -DNDEBUG -
DL_ENDIAN -DOPENSSL_NO_BUF_FREELISTS
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
aes-128-cbc 24276.10k 30304.47k 30511.53k 27025.07k 26951.68k 24554.15k
real 0m18,025s
user 0m15,880s
sys 0m0,060s
AF_ALG:
r2-gentoo ~ # time openssl speed -evp aes-128-cbc -elapsed
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128-cbc for 3s on 16 size blocks: 63195 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 64 size blocks: 62880 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 61864 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 59010 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 29029 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 16384 size blocks: 16145 aes-128-cbc's in 3.00s
OpenSSL 1.1.1j 16 Feb 2021
built on: Mon Feb 22 11:56:18 2021 UTC
options:bn(64,32) rc4(char) des(long) aes(partial) idea(int) blowfish(ptr)
compiler: armv7a-hardfloat-linux-gnueabi-gcc -fPIC -pthread -Wa,--noexecstack -Wall -O2 -pipe -march=armv7-a -mfpu=neon-vfpv4 -mfloat-abi=hard -fno-strict-aliasing -Wa,--noexecstack -DOPENSSL_USE_NODELETE -DOPEN
SSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DZLIB -DNDEBUG -
DL_ENDIAN -DOPENSSL_NO_BUF_FREELISTS
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
aes-128-cbc 337.04k 1341.44k 5279.06k 20142.08k 79268.52k 88173.23k
real 0m18,543s
user 0m0,824s
sys 0m12,075s
user 0m15,880s - non-accelerated vs user 0m0,824s - accelerated
~2.94 times faster on 8192 bytes blocks and ~3.6 times faster on 16384 bytes blocks (single run)
similar difference whet testing aes-192-cbc and aes-256-cbc
r2-gentoo ~ # cat /proc/interrupts | grep aes
51: 339897 0 0 0 MT_SYSIRQ 82 Level mtk-aes
52: 0 0 0 0 MT_SYSIRQ 83 Level mtk-aes
As for me - a good way to accelerate out of box: on mainline kernel w/o additional modules and mainline openssl w/o patches/additional engines.
Openvpn/openssh(sandbox=no required) are not yet tested,
Also testing on standard @frank-w’s kernel config is probably needed.
P.S. tested on 5.9.0-rc8 kernel
so you need no recompile of openssl and no change in dts (for using eip97 engine imho now using different compatible/driver), right?
you can create a pull request (for LTS main branch), then i add the config-options
Yes, at least for my kernel config. Only openssl configuration is neeed to be changed.
Sure, what is the LTS-main branch, 5.4 or 5.10? I’ll can do for both, but 5.4 is low-priority.
I’ll appreciate for any additional tests by community.
5.10 is enough…can cherry-pick it to 5.4.
Done.
Compiled and tested with importconfig. Works with openssl speed -evp aes-256-cbc -elapsed -engine afalg (no engine specified required after modifying openssl conf).
Still need testing with openvpn/openssh if possible,
Tried to test openvpn:
time openvpn --engine afalg --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc
Got cipher error on 1st run:
Mon Mar 1 19:39:07 2021 cipher_ctx_update: EVP_CipherUpdate() failed
dmesg:
[ 2278.586698] ------------[ cut here ]------------
[ 2278.586737] WARNING: CPU: 2 PID: 4612 at lib/refcount.c:28 refcount_warn_saturate+0x13c/0x174
[ 2278.586742] refcount_t: underflow; use-after-free.
[ 2278.586747] Modules linked in: algif_skcipher af_alg i2c_dev lima gpu_sched spi_mt65xx pwm_mediatek mtk_pmic_keys [last unloaded: cryptodev]
[ 2278.586798] CPU: 2 PID: 4612 Comm: openvpn Tainted: G O 5.10.18-bpi-r2-main #1
[ 2278.586803] Hardware name: Mediatek Cortex-A7 (Device Tree)
[ 2278.586809] Backtrace:
[ 2278.586827] [<c0dc9cf8>] (dump_backtrace) from [<c0dca0a4>] (show_stack+0x20/0x24)
[ 2278.586837] r7:0000001c r6:600d0013 r5:00000000 r4:c14ef938
[ 2278.586850] [<c0dca084>] (show_stack) from [<c0dcdd48>] (dump_stack+0xcc/0xe0)
[ 2278.586862] [<c0dcdc7c>] (dump_stack) from [<c0126840>] (__warn+0xfc/0x114)
[ 2278.586871] r7:0000001c r6:c057a580 r5:00000009 r4:c1181000
[ 2278.586881] [<c0126744>] (__warn) from [<c0dca7b4>] (warn_slowpath_fmt+0xa8/0xc4)
[ 2278.586890] r7:0000001c r6:c1181000 r5:c118103c r4:d3640000
[ 2278.586902] [<c0dca710>] (warn_slowpath_fmt) from [<c057a580>] (refcount_warn_saturate+0x13c/0x174)
[ 2278.586913] r9:c75d61c0 r8:00000000 r7:000000f6 r6:d3640000 r5:0051d3f0 r4:c5b7fa80
[ 2278.586927] [<c057a444>] (refcount_warn_saturate) from [<c037dfa4>] (sys_io_submit+0x6d4/0x8d4)
[ 2278.586939] [<c037d8d0>] (sys_io_submit) from [<c0100060>] (ret_fast_syscall+0x0/0x54)
[ 2278.586945] Exception stack(0xd3641fa8 to 0xd3641ff0)
[ 2278.586955] 1fa0: 00000000 00000000 b6fc7000 00000001 bed958e4 00000020
[ 2278.586965] 1fc0: 00000000 00000000 bed958e4 000000f6 b6f9a000 bed958e8 00000000 bed958f4
[ 2278.586972] 1fe0: bed958c0 bed958b0 b6f88698 b6c08e30
[ 2278.586982] r10:000000f6 r9:d3640000 r8:c0100264 r7:000000f6 r6:bed958e4 r5:00000000
[ 2278.586988] r4:00000000
[ 2278.586994] ---[ end trace 54f7a3c195c24c86 ]---
and kernel panic on 2nd run:
serial output:
r2-gentoo login: af_algaf_alg[ 2357.439515] Internal error: Oops: 5 [#1] SMP ARM
[ 2357.444256] Modules linked in: algif_skcipher afdev lima gpu_sched spi_mt65xx pwm_mediatek mtk_pmic_keys [last unloaded: af_alg]
[ 2357.456905] CPU: 2 PID: 4629 Comm: openvpn Tainted: G W O 5.10.18-bpi-r2-main #1
[ 2357.465595] Hardware name: Mediatek Cortex-A7 (Device Tree)
[ 2357.471271] PC is at release_sock+0x3c/0x90
[ 2357.475524] LR is at release_sock+0x24/0x90
[ 2357.479779] pc : [<c0a7f440>] lr : [<c0a7f428>] psr: 600d0013
[ 2357.486148] sp : d54cdd80 ip : d54cdd80 fp : d54cdd94
[ 2357.491456] r10: c47ad6c0 r9 : d54cc000 r8 : 00000000
[ 2357.496765] r7 : 00000030 r6 : c5f66c00 r5 : c5f66c70 r4 : c5f66c00
[ 2357.503390] r3 : 00000000 r2 : 00000007 r1 : 00000007 r0 : c5f66c70
[ 2357.510018] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
[ 2357.517268] Control: 10c5387d Table: 954bc06a DAC: 00000051
[ 2357.523116] Process openvpn (pid: 4629, stack limit = 0x777cd319)
[ 2357.529314] Stack: (0xd54cdd80 to 0xd54ce000)
[ 2357.533748] dd80: 00000000 c5f65000 d54cddb4 d54cdd98 bf04208c c0a7f410 c81aea80 d54cdf40
[ 2357.542059] dda0: 00000010 00000030 d54cddd4 d54cddb8 bf042300 bf04200c d54cdf40 00000010
[ 2357.550373] ddc0: c81aea80 00000030 d54cde34 d54cddd8 c0a787c0 bf0422e8 00000000 00000051
[ 2357.558685] dde0: d54cde34 d54cddf0 c0a7a40c c0576960 d54cde40 d54cdf48 e06fc5c4 00000000
[ 2357.566999] de00: beb028e8 92899668 e06fc5c4 00000000 d54cdf40 d54cc000 c81aea80 00000000
[ 2357.575304] de20: 00000000 00000128 d54cdf2c d54cde38 c0a7a4ac c0a78604 00000000 92899668
[ 2357.583613] de40: 00000000 00597038 00000010 c15bb6c0 c157dd08 c75d62c0 00000051 beb02998
[ 2357.591926] de60: 00000001 00000001 c0378f10 d54cded8 c75d61c0 00000001 d54cc000 beb02998
[ 2357.600240] de80: d54cc000 000000f5 d54cdeb4 d54cde98 c037b15c c037ad78 00000001 c75d61c0
[ 2357.608554] dea0: 00000001 c037b3d0 d54cdf5c d54cdeb8 c037b3d0 c01a2a9c d54cded8 c036fcf8
[ 2357.616868] dec0: 00000000 c2ee7db0 00000000 beb02998 00000000 00000000 00000001 00000008
[ 2357.625183] dee0: 00000004 00000000 00000000 d54cdee0 00000000 c033dbd0 d54cdf14 d54cdf08
[ 2357.633495] df00: c033dbd0 92899668 d54cc000 beb028f4 00000000 c81aea80 c0100264 d54cc000
[ 2357.641806] df20: d54cdf94 d54cdf30 c0a7a928 c0a7a438 00000000 00000000 00000000 fffffff7
[ 2357.650116] df40: 00000000 00000000 00000005 00000000 00000010 d54cde44 00000001 c037b2d0
[ 2357.658430] df60: c47ad6c0 92899668 00000030 00000000 00000000 92899668 0058d0a0 0059bebc
[ 2357.666743] df80: b6f12840 00000128 d54cdfa4 d54cdf98 c0a7a980 c0a7a8d0 00000000 d54cdfa8
[ 2357.675053] dfa0: c0100060 c0a7a970 0058d0a0 0059bebc 00000006 beb028f4 00000000 00000000
[ 2357.683364] dfc0: 0058d0a0 0059bebc b6f12840 00000128 b6ee3000 beb028e8 00000001 beb028f4
[ 2357.691676] dfe0: b6ee3054 beb028a8 b6ed1598 b6b57458 600d0010 00000006 000000000
[ 2357.699983] Backtrace:
[ 2357.702512] [<c0a7f404>] (release_sock) from [<bf04208c>] (skcipher_check_key+0x8c/0x90 [algif_skcipher])
[ 2357.712241] r5:c5f65000 r4:00000000
[ 2357.715889] [<bf042000>] (skcipher_check_key [algif_skcipher]) from [<bf042300>] (skcipher_sendmsg_nokey+0x24/0x54 [algif_skcipher])
[ 2357.727991] r7:00000030 r6:00000010 r5:d54cdf40 r4:c81aea80
[ 2357.733760] [<bf0422dc>] (skcipher_sendmsg_nokey [algif_skcipher]) from [<c0a787c0>] (____sys_sendmsg+0x1c8/0x260)
[ 2357.744282] r7:00000030 r6:c81aea80 r5:00000010 r4:d54cdf40
[ 2357.750053] [<c0a785f8>] (____sys_sendmsg) from [<c0a7a4ac>] (___sys_sendmsg+0x80/0xac)
[ 2357.758194] r10:00000128 r9:00000000 r8:00000000 r7:c81aea80 r6:d54cc000 r5:d54cdf40
[ 2357.766147] r4:00000000
[ 2357.768746] [<c0a7a42c>] (___sys_sendmsg) from [<c0a7a928>] (__sys_sendmsg+0x64/0xa0)
[ 2357.776713] r9:d54cc000 r8:c0100264 r7:c81aea80 r6:00000000 r5:beb028f4 r4:d54cc000
[ 2357.784585] [<c0a7a8c4>] (__sys_sendmsg) from [<c0a7a980>] (sys_sendmsg+0x1c/0x20)
[ 2357.792284] r7:00000128 r6:b6f12840 r5:0059bebc r4:0058d0a0
[ 2357.798050] [<c0a7a964>] (sys_sendmsg) from [<c0100060>] (ret_fast_syscall+0x0/0x54)
[ 2357.805921] Exception stack(0xd54cdfa8 to 0xd54cdff0)
[ 2357.811064] dfa0: 0058d0a0 0059bebc 00000006 beb028f4 00000000 00000000
[ 2357.819378] dfc0: 0058d0a0 0059bebc b6f12840 00000128 b6ee3000 beb028e8 00000001 beb028f4
[ 2357.827682] dfe0: b6ee3054 beb028a8 b6ed1598 b6b57458
[ 2357.832821] Code: 0a000001 e1a00004 ebffffaf e5943020 (e5933048)
[ 2357.839071] ---[ end trace 54f7a3c195c24c87 ]---
[ 2357.843798] Kernel panic - not syncing: Fatal exception in interrupt
[ 2357.843820] CPU0: stopping
[ 2357.843833] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G D W O 5.10.18-bpi-r2-main #1
[ 2357.843836] Hardware name: Mediatek Cortex-A7 (Device Tree)
[ 2357.843839] Backtrace:
[ 2357.843862] [<c0dc9cf8>] (dump_backtrace) from [<c0dca0a4>] (show_stack+0x20/0x24)
[ 2357.843870] r7:00000000 r6:60030193 r5:00000000 r4:c14ef938
[ 2357.843880] [<c0dca084>] (show_stack) from [<c0dcdd48>] (dump_stack+0xcc/0xe0)
[ 2357.843890] [<c0dcdc7c>] (dump_stack) from [<c0110290>] (do_handle_IPI+0x328/0x35c)
[ 2357.843897] r7:00000000 r6:00000000 r5:c1572e40 r4:c15bb640
[ 2357.843905] [<c010ff68>] (do_handle_IPI) from [<c01102ec>] (ipi_handler+0x28/0x30)
[ 2357.843913] r9:c1401ed0 r8:c2029000 r7:00000000 r6:1e1f2000 r5:c2030e00 r4:00000014
[ 2357.843923] [<c01102c4>] (ipi_handler) from [<c01946d4>] (handle_percpu_devid_fasteoi_ipi+0x90/0x160)
[ 2357.843933] [<c0194644>] (handle_percpu_devid_fasteoi_ipi) from [<c018d76c>] (generic_handle_irq+0x44/0x54)
[ 2357.843939] r7:00000000 r6:00000001 r5:00000000 r4:c13a6c24
[ 2357.843948] [<c018d728>] (generic_handle_irq) from [<c018de7c>] (__handle_domain_irq+0x6c/0xc0)
[ 2357.843957] [<c018de10>] (__handle_domain_irq) from [<c010135c>] (gic_handle_irq+0x8c/0xa0)
[ 2357.843965] r9:c1401ed0 r8:e100200c r7:c13a6c30 r6:e1002000 r5:c14ef9f8 r4:c14057cc
[ 2357.843973] [<c01012d0>] (gic_handle_irq) from [<c0100b0c>] (__irq_svc+0x6c/0x90)
[ 2357.843977] Exception stack(0xc1401ed0 to 0xc1401f18)
[ 2357.843983] 1ec0: 00000000 01779ccc df59a304 c011ee40
[ 2357.843991] 1ee0: c1573ba0 00000000 c1404f10 c1404f58 c1572634 c114496c e07fcd00 c1401f2c
[ 2357.843997] 1f00: c1401f30 c1401f20 c01096e4 c01096e8 60030013 ffffffff
[ 2357.844005] r9:c1400000 r8:c1572634 r7:c1401f04 r6:ffffffff r5:60030013 r4:c01096e8
[ 2357.844018] [<c01096a0>] (arch_cpu_idle) from [<c0de1148>] (default_idle_call+0x48/0x10c)
[ 2357.844027] [<c0de1100>] (default_idle_call) from [<c015f500>] (do_idle+0xe4/0x150)
[ 2357.844032] r5:00000000 r4:c1400000
[ 2357.844039] [<c015f41c>] (do_idle) from [<c015f83c>] (cpu_startup_entry+0x28/0x2c)
[ 2357.844047] r9:c1357a54 r8:00000000 r7:c1357a54 r6:c1404ec0 r5:c1400000 r4:000000d8
[ 2357.844055] [<c015f814>] (cpu_startup_entry) from [<c0dda448>] (rest_init+0xbc/0xc4)
[ 2357.844067] [<c0dda38c>] (rest_init) from [<c1300ad8>] (arch_call_rest_init+0x18/0x1c)
[ 2357.844072] r5:c1400000 r4:c15bb040
[ 2357.844081] [<c1300ac0>] (arch_call_rest_init) from [<c1301078>] (start_kernel+0x524/0x55c)
[ 2357.844090] [<c1300b54>] (start_kernel) from [<00000000>] (0x0)
[ 2357.844096] CPU3: stopping
[ 2357.844104] CPU: 3 PID: 4536 Comm: emerge Tainted: G D W O 5.10.18-bpi-r2-main #1
[ 2357.844107] Hardware name: Mediatek Cortex-A7 (Device Tree)
[ 2357.844109] Backtrace:
[ 2357.844121] [<c0dc9cf8>] (dump_backtrace) from [<c0dca0a4>] (show_stack+0x20/0x24)
[ 2357.844128] r7:00000000 r6:60030193 r5:00000000 r4:c14ef938
[ 2357.844136] [<c0dca084>] (show_stack) from [<c0dcdd48>] (dump_stack+0xcc/0xe0)
[ 2357.844145] [<c0dcdc7c>] (dump_stack) from [<c0110290>] (do_handle_IPI+0x328/0x35c)
[ 2357.844151] r7:00000000 r6:00000003 r5:c1572e40 r4:c15bb640
[ 2357.844159] [<c010ff68>] (do_handle_IPI) from [<c01102ec>] (ipi_handler+0x28/0x30)
[ 2357.844167] r9:ca16dfb0 r8:c2029000 r7:00000000 r6:1e22e000 r5:c2030e00 r4:00000014
[ 2357.844175] [<c01102c4>] (ipi_handler) from [<c01946d4>] (handle_percpu_devid_fasteoi_ipi+0x90/0x160)
[ 2357.844184] [<c0194644>] (handle_percpu_devid_fasteoi_ipi) from [<c018d76c>] (generic_handle_irq+0x44/0x54)
[ 2357.844190] r7:00000000 r6:00000001 r5:00000000 r4:c13a6c24
[ 2357.844200] [<c018d728>] (generic_handle_irq) from [<c018de7c>] (__handle_domain_irq+0x6c/0xc0)
[ 2357.844208] [<c018de10>] (__handle_domain_irq) from [<c010135c>] (gic_handle_irq+0x8c/0xa0)
[ 2357.844216] r9:ca16dfb0 r8:e100200c r7:c13a6c30 r6:e1002000 r5:c14ef9f8 r4:c14057cc
[ 2357.844223] [<c01012d0>] (gic_handle_irq) from [<c0100eb4>] (__irq_usr+0x54/0x80)
[ 2357.844227] Exception stack(0xca16dfb0 to 0xca16dff8)
[ 2357.844232] dfa0: b6569e60 b660acc4 00000040 0000003f
[ 2357.844240] dfc0: 00000030 b660aca0 7e486a30 00517f64 b660aca0 b6f22000 7e486a30 00000040
[ 2357.844246] dfe0: b6f30798 beacb800 00001e2c b6cd8180 80030010 ffffffff
[ 2357.844254] r9:b6f22000 r8:10c5387d r7:10c5387d r6:ffffffff r5:80030010 r4:b6cd8180
[ 2357.844259] CPU1: stopping
[ 2357.844266] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G D W O 5.10.18-bpi-r2-main #1
[ 2357.844269] Hardware name: Mediatek Cortex-A7 (Device Tree)
[ 2357.844271] Backtrace:
[ 2357.844283] [<c0dc9cf8>] (dump_backtrace) from [<c0dca0a4>] (show_stack+0x20/0x24)
[ 2357.844290] r7:00000000 r6:60030193 r5:00000000 r4:c14ef938
[ 2357.844298] [<c0dca084>] (show_stack) from [<c0dcdd48>] (dump_stack+0xcc/0xe0)
[ 2357.844306] [<c0dcdc7c>] (dump_stack) from [<c0110290>] (do_handle_IPI+0x328/0x35c)
[ 2357.844313] r7:00000000 r6:00000001 r5:c1572e40 r4:c15bb640
[ 2357.844320] [<c010ff68>] (do_handle_IPI) from [<c01102ec>] (ipi_handler+0x28/0x30)
[ 2357.844328] r9:c2171f30 r8:c2029000 r7:00000000 r6:1e206000 r5:c2030e00 r4:00000014 357.844376] r9:c2171f30 r8:e100200c r7:c13a6c30 r6:e1002000 r5:c14ef9f8 r4:c14057cc
[ 2357.844383] [<c01012d0>] (gic_handle_irq) from [<c0100b0c>] (__irq_svc+0x6c/0x90)
[ 2357.844387] Exception stack(0xc2171f30 to 0xc2171f78)
[ 2357.844393] 1f20: 00000000 00d5c648 df5ae304 c011ee40
[ 2357.844400] 1f40: c1573ba0 00000001 c1404f10 c1404f58 c1572634 c114496c 00000000 c2171f8c
[ 2357.844407] 1f60: c2171f90 c2171f80 c01096e4 c01096e8 60030013 ffffffff
[ 2357.844414] r9:c2170000 r8:c1572634 r7:c2171f64 r6:ffffffff r5:60030013 r4:c01096e8
[ 2357.844424] [<c01096a0>] (arch_cpu_idle) from [<c0de1148>] (default_idle_call+0x48/0x10c)
[ 2357.844433] [<c0de1100>] (default_idle_call) from [<c015f500>] (do_idle+0xe4/0x150)
[ 2357.844437] r5:00000001 r4:c2170000
[ 2357.844444] [<c015f41c>] (do_idle) from [<c015f83c>] (cpu_startup_entry+0x28/0x2c)
[ 2357.844452] r9:410fc073 r8:8000406a r7:c15bb650 r6:10c0387d r5:00000001 r4:00000091
[ 2357.844460] [<c015f814>] (cpu_startup_entry) from [<c0110bcc>] (secondary_start_kernel+0x170/0x194)
[ 2357.844468] [<c0110a5c>] (secondary_start_kernel) from [<801017cc>] (0x801017cc)
[ 2357.844472] r5:00000051 r4:8216806a
[ 2358.971480] Rebooting in 3 seconds..357.844337] [<c01102c4>] (ipi_handler) from [<c01946d4>] (handle_percpu_devid_fasteoi_ipi+0x90/0x160)
[ 2357.844345] [<c0194644>] (handle_percpu_devid_fasteoi_ipi) from [<c018d76c>] (generic_handle_irq+0x44/0x54)
[ 2357.844351] r7:00000000 r6:00000001 r5:00000000 r4:c13a6c24
[ 2357.844360] [<c018d728>] (generic_handle_irq) from [<c018de7c>] (__handle_domain_irq+0x6c/0xc0)
[ 2357.844368] [<c018de10>] (__handle_domain_irq) from [<c010135c>] (gic_handle_irq+0x8c/0xa0)
[ 2357.844376] r9:c2171f30 r8:e100200c r7:c13a6c30 r6:e1002000 r5:c14ef9f8 r4:c14057cc
[ 2357.844383] [<c01012d0>] (gic_handle_irq) from [<c0100b0c>] (__irq_svc+0x6c/0x90)
[ 2357.844387] Exception stack(0xc2171f30 to 0xc2171f78)
[ 2357.844393] 1f20: 00000000 00d5c648 df5ae304 c011ee40
[ 2357.844400] 1f40: c1573ba0 00000001 c1404f10 c1404f58 c1572634 c114496c 00000000 c2171f8c
[ 2357.844407] 1f60: c2171f90 c2171f80 c01096e4 c01096e8 60030013 ffffffff
[ 2357.844414] r9:c2170000 r8:c1572634 r7:c2171f64 r6:ffffffff r5:60030013 r4:c01096e8
[ 2357.844424] [<c01096a0>] (arch_cpu_idle) from [<c0de1148>] (default_idle_call+0x48/0x10c)
[ 2357.844433] [<c0de1100>] (default_idle_call) from [<c015f500>] (do_idle+0xe4/0x150)
[ 2357.844437] r5:00000001 r4:c2170000
[ 2357.844444] [<c015f41c>] (do_idle) from [<c015f83c>] (cpu_startup_entry+0x28/0x2c)
[ 2357.844452] r9:410fc073 r8:8000406a r7:c15bb650 r6:10c0387d r5:00000001 r4:00000091
[ 2357.844460] [<c015f814>] (cpu_startup_entry) from [<c0110bcc>] (secondary_start_kernel+0x170/0x194)
[ 2357.844468] [<c0110a5c>] (secondary_start_kernel) from [<801017cc>] (0x801017cc)
[ 2357.844472] r5:00000051 r4:8216806a
[ 2358.971480] Rebooting in 3 seconds..
UPD:
openvpn test success depends on --tun-mtu in works great with --tun-mtu 8192, and fails with --tun-mtu 16384
Some openvpn perfomance tests:
setup:
server:
AF_ALG:
openvpn --engine afalg --dev tun --proto udp --port 11000 --secret ./static.key --tun-mtu 8192 --ifconfig 192.168.222.11 192.168.222.10 --cipher aes-256-cbc
SW:
openvpn --dev tun --proto udp --port 11000 --secret ./static.key --tun-mtu 8192 --ifconfig 192.168.222.11 192.168.222.10 --cipher aes-256-cbc
client:
openvpn --dev tun --proto udp --port 11000 --secret ./static.key --tun-mtu 8192 --ifconfig 192.168.222.10 192.168.222.11 --cipher aes-256-cbc --remote <server_ip>
results:
upload/download
network w/o openvpn - using wifi:
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.0.1.150, port 59184
[ 5] local 10.0.1.2 port 5201 connected to 10.0.1.150 port 59186
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 10.8 MBytes 90.7 Mbits/sec
[ 5] 1.00-2.00 sec 13.3 MBytes 112 Mbits/sec
[ 5] 2.00-3.00 sec 11.7 MBytes 98.4 Mbits/sec
[ 5] 3.00-4.00 sec 10.8 MBytes 90.2 Mbits/sec
[ 5] 4.00-5.00 sec 9.66 MBytes 81.1 Mbits/sec
[ 5] 5.00-6.00 sec 10.6 MBytes 88.8 Mbits/sec
[ 5] 6.00-7.00 sec 10.4 MBytes 87.6 Mbits/sec
[ 5] 7.00-8.00 sec 9.96 MBytes 83.5 Mbits/sec
[ 5] 8.00-9.00 sec 9.92 MBytes 83.2 Mbits/sec
[ 5] 9.00-10.00 sec 10.2 MBytes 85.8 Mbits/sec
[ 5] 10.00-10.02 sec 214 KBytes 78.6 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.02 sec 108 MBytes 90.1 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.0.1.150, port 59196
[ 5] local 10.0.1.2 port 5201 connected to 10.0.1.150 port 59198
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 31.2 MBytes 262 Mbits/sec 22 498 KBytes
[ 5] 1.00-2.00 sec 18.8 MBytes 157 Mbits/sec 0 564 KBytes
[ 5] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 609 KBytes
[ 5] 3.00-4.00 sec 22.5 MBytes 189 Mbits/sec 0 641 KBytes
[ 5] 4.00-5.00 sec 20.0 MBytes 168 Mbits/sec 0 659 KBytes
[ 5] 5.00-6.00 sec 21.2 MBytes 178 Mbits/sec 0 667 KBytes
[ 5] 6.00-7.00 sec 21.2 MBytes 178 Mbits/sec 4 499 KBytes
[ 5] 7.00-8.00 sec 18.8 MBytes 157 Mbits/sec 0 533 KBytes
[ 5] 8.00-9.00 sec 21.2 MBytes 178 Mbits/sec 0 553 KBytes
[ 5] 9.00-10.00 sec 21.2 MBytes 178 Mbits/sec 0 567 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 220 MBytes 184 Mbits/sec 26 sender
AF_ALG:
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.222.10, port 52138
[ 5] local 192.168.222.11 port 5201 connected to 192.168.222.10 port 52140
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 2.74 MBytes 23.0 Mbits/sec
[ 5] 1.00-2.00 sec 2.99 MBytes 25.1 Mbits/sec
[ 5] 2.00-3.00 sec 3.22 MBytes 27.0 Mbits/sec
[ 5] 3.00-4.00 sec 3.09 MBytes 25.9 Mbits/sec
[ 5] 4.00-5.00 sec 2.95 MBytes 24.7 Mbits/sec
[ 5] 5.00-6.00 sec 3.27 MBytes 27.4 Mbits/sec
[ 5] 6.00-7.00 sec 3.20 MBytes 26.8 Mbits/sec
[ 5] 7.00-8.00 sec 3.22 MBytes 27.0 Mbits/sec
[ 5] 8.00-9.00 sec 3.24 MBytes 27.1 Mbits/sec
[ 5] 9.00-10.00 sec 3.13 MBytes 26.3 Mbits/sec
[ 5] 10.00-10.00 sec 5.23 KBytes 14.1 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 31.0 MBytes 26.0 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.222.10, port 52152
[ 5] local 192.168.222.11 port 5201 connected to 192.168.222.10 port 52154
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 13.8 MBytes 115 Mbits/sec 2 92.8 KBytes
[ 5] 1.00-2.00 sec 2.50 MBytes 21.0 Mbits/sec 4 91.5 KBytes
[ 5] 2.00-3.00 sec 2.50 MBytes 21.0 Mbits/sec 4 79.7 KBytes
[ 5] 3.00-4.00 sec 2.50 MBytes 21.0 Mbits/sec 2 71.9 KBytes
[ 5] 4.00-5.00 sec 2.50 MBytes 21.0 Mbits/sec 0 92.8 KBytes
[ 5] 5.00-6.00 sec 2.50 MBytes 21.0 Mbits/sec 2 82.3 KBytes
[ 5] 6.00-7.00 sec 2.50 MBytes 21.0 Mbits/sec 4 73.2 KBytes
[ 5] 7.00-8.00 sec 2.50 MBytes 21.0 Mbits/sec 0 94.1 KBytes
[ 5] 8.00-9.00 sec 2.50 MBytes 21.0 Mbits/sec 2 84.9 KBytes
[ 5] 9.00-10.00 sec 2.50 MBytes 21.0 Mbits/sec 2 74.5 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 36.2 MBytes 30.4 Mbits/sec 22 sender
SW:
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.222.10, port 52204
[ 5] local 192.168.222.11 port 5201 connected to 192.168.222.10 port 52206
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 4.52 MBytes 37.9 Mbits/sec
[ 5] 1.00-2.00 sec 5.03 MBytes 42.2 Mbits/sec
[ 5] 2.00-3.00 sec 6.01 MBytes 50.4 Mbits/sec
[ 5] 3.00-4.00 sec 5.22 MBytes 43.8 Mbits/sec
[ 5] 4.00-5.00 sec 5.83 MBytes 48.9 Mbits/sec
[ 5] 5.00-6.00 sec 4.56 MBytes 38.2 Mbits/sec
[ 5] 6.00-7.00 sec 4.36 MBytes 36.6 Mbits/sec
[ 5] 7.00-8.00 sec 3.47 MBytes 29.1 Mbits/sec
[ 5] 8.00-9.00 sec 4.66 MBytes 39.1 Mbits/sec
[ 5] 9.00-10.00 sec 4.81 MBytes 40.3 Mbits/sec
[ 5] 10.00-10.00 sec 24.8 KBytes 46.5 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 48.5 MBytes 40.7 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.222.10, port 52212
[ 5] local 192.168.222.11 port 5201 connected to 192.168.222.10 port 52214
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 15.0 MBytes 126 Mbits/sec 11 95.4 KBytes
[ 5] 1.00-2.00 sec 3.75 MBytes 31.5 Mbits/sec 1 98.0 KBytes
[ 5] 2.00-3.00 sec 5.00 MBytes 41.9 Mbits/sec 6 67.9 KBytes
[ 5] 3.00-4.00 sec 3.75 MBytes 31.5 Mbits/sec 8 84.9 KBytes
[ 5] 4.00-5.00 sec 3.75 MBytes 31.5 Mbits/sec 15 88.9 KBytes
[ 5] 5.00-6.00 sec 3.75 MBytes 31.5 Mbits/sec 4 92.8 KBytes
[ 5] 6.00-7.00 sec 5.00 MBytes 41.9 Mbits/sec 1 95.4 KBytes
[ 5] 7.00-8.00 sec 3.75 MBytes 31.5 Mbits/sec 20 52.3 KBytes
[ 5] 8.00-9.00 sec 3.75 MBytes 31.5 Mbits/sec 19 47.0 KBytes
[ 5] 9.00-10.00 sec 5.00 MBytes 42.0 Mbits/sec 6 67.9 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 52.5 MBytes 44.0 Mbits/sec 91 sender
Looks strange, bun in this case HW accelerated openvpn seems to be slower than software-only. Probably it depends on small buffer size (i might be wrong - i think it depends on mtu size)
I couldn’t to force openssh to use afalg engine after recompiling with --with-sandbox=no so for now i’m unable to benchmark ssh.
P.S. openvpn is clearly uses aes_mtk - it’s visible from/proc/interrupts
UPD2:
after openvpn tuning i finnaly got some nicer results 
added --fragment 0 --mssfix 0 to both client and server configs
HW:
Accepted connection from 192.168.222.10, port 53218
[ 5] local 192.168.222.11 port 5201 connected to 192.168.222.10 port 53220
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 9.33 MBytes 78.3 Mbits/sec
[ 5] 1.00-2.00 sec 10.6 MBytes 88.7 Mbits/sec
[ 5] 2.00-3.00 sec 9.66 MBytes 81.1 Mbits/sec
[ 5] 3.00-4.00 sec 11.9 MBytes 99.5 Mbits/sec
[ 5] 4.00-5.00 sec 11.0 MBytes 92.6 Mbits/sec
[ 5] 5.00-6.00 sec 11.4 MBytes 95.7 Mbits/sec
[ 5] 6.00-7.00 sec 10.8 MBytes 90.2 Mbits/sec
[ 5] 7.00-8.00 sec 11.1 MBytes 93.0 Mbits/sec
[ 5] 8.00-9.00 sec 10.9 MBytes 91.1 Mbits/sec
[ 5] 9.00-10.00 sec 10.8 MBytes 90.7 Mbits/sec
[ 5] 10.00-10.06 sec 707 KBytes 97.7 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.06 sec 108 MBytes 90.1 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.222.10, port 53222
[ 5] local 192.168.222.11 port 5201 connected to 192.168.222.10 port 53224
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 20.0 MBytes 168 Mbits/sec 1 294 KBytes
[ 5] 1.00-2.00 sec 8.75 MBytes 73.4 Mbits/sec 0 413 KBytes
[ 5] 2.00-3.00 sec 10.0 MBytes 83.9 Mbits/sec 0 501 KBytes
[ 5] 3.00-4.00 sec 8.75 MBytes 73.4 Mbits/sec 0 572 KBytes
[ 5] 4.00-5.00 sec 8.75 MBytes 73.4 Mbits/sec 4 350 KBytes
[ 5] 5.00-6.00 sec 10.0 MBytes 83.9 Mbits/sec 0 453 KBytes
[ 5] 6.00-7.00 sec 8.75 MBytes 73.4 Mbits/sec 0 533 KBytes
[ 5] 7.00-8.00 sec 10.0 MBytes 83.9 Mbits/sec 10 318 KBytes
[ 5] 8.00-9.00 sec 8.75 MBytes 73.4 Mbits/sec 0 421 KBytes
[ 5] 9.00-10.00 sec 8.75 MBytes 73.4 Mbits/sec 0 509 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 102 MBytes 85.9 Mbits/sec 15 sender
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
SW:
Accepted connection from 192.168.222.10, port 53132
[ 5] local 192.168.222.11 port 5201 connected to 192.168.222.10 port 53134
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 8.75 MBytes 73.4 Mbits/sec
[ 5] 1.00-2.00 sec 8.86 MBytes 74.3 Mbits/sec
[ 5] 2.00-3.00 sec 8.30 MBytes 69.6 Mbits/sec
[ 5] 3.00-4.00 sec 9.44 MBytes 79.2 Mbits/sec
[ 5] 4.00-5.00 sec 8.43 MBytes 70.7 Mbits/sec
[ 5] 5.00-6.00 sec 8.36 MBytes 70.1 Mbits/sec
[ 5] 6.00-7.00 sec 8.42 MBytes 70.7 Mbits/sec
[ 5] 7.00-8.00 sec 8.02 MBytes 67.3 Mbits/sec
[ 5] 8.00-9.00 sec 9.18 MBytes 77.0 Mbits/sec
[ 5] 9.00-10.00 sec 8.38 MBytes 70.3 Mbits/sec
[ 5] 10.00-10.03 sec 358 KBytes 85.7 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.03 sec 86.5 MBytes 72.3 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.222.10, port 53140
[ 5] local 192.168.222.11 port 5201 connected to 192.168.222.10 port 53142
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 18.8 MBytes 157 Mbits/sec 4 278 KBytes
[ 5] 1.00-2.00 sec 10.0 MBytes 83.9 Mbits/sec 0 397 KBytes
[ 5] 2.00-3.00 sec 8.75 MBytes 73.4 Mbits/sec 4 270 KBytes
[ 5] 3.00-4.00 sec 10.0 MBytes 83.9 Mbits/sec 0 390 KBytes
[ 5] 4.00-5.00 sec 10.0 MBytes 83.9 Mbits/sec 2 350 KBytes
[ 5] 5.00-6.00 sec 8.75 MBytes 73.4 Mbits/sec 0 445 KBytes
[ 5] 6.00-7.00 sec 8.75 MBytes 73.4 Mbits/sec 9 175 KBytes
[ 5] 7.00-8.00 sec 10.0 MBytes 83.9 Mbits/sec 0 334 KBytes
[ 5] 8.00-9.00 sec 8.75 MBytes 73.4 Mbits/sec 33 231 KBytes
[ 5] 9.00-10.00 sec 8.75 MBytes 73.4 Mbits/sec 4 246 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 102 MBytes 85.9 Mbits/sec 56 sender
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
UPD2:
When using --fragment 0 --mssfix 0, the speed is highly depends on --tun-mtu
–tun-mtu 1500:
upload
[ 5] 0.00-10.00 sec 35.9 MBytes 30.1 Mbits/sec receiver
download
[ 5] 0.00-10.00 sec 36.2 MBytes 30.4 Mbits/sec 43 sender
–tun-mtu 16000:
upload
[ 5] 0.00-10.11 sec 121 MBytes 101 Mbits/sec receiver
download
[ 5] 0.00-10.01 sec 144 MBytes 120 Mbits/sec 30 sender
–tun-mtu 16000 - no acceleration:
upload
[ 5] 0.00-10.05 sec 112 MBytes 93.4 Mbits/sec receiver
download
[ 5] 0.00-10.01 sec 126 MBytes 106 Mbits/sec 11 sender
full server/client command:
openvpn --dev tun --proto udp --port 11000 --secret ./static.key --tun-mtu 16000 --ifconfig 192.168.222.11 192.168.222.10 --cipher aes-256-cbc --fragment 0 --mssfix 0
openvpn --dev tun --proto udp --port 11000 --secret ./static.key --tun-mtu 16000 --ifconfig 192.168.222.10 192.168.222.11 --cipher aes-256-cbc --fragment 0 --mssfix 0 --remote 10.0.1.2
UPD3:
testing with --tun-mtu 16384 crashes server with same dmesg as openvpn test above, so --tun-mtu 16000 seems to be working configuration near it’s hw limit
setting ‘–tun-mtu 32768’ in sw mode didn’t gave any speed advantages compairing to --tun-mtu 16000 in sw mode
same test performed using LAN
SW mtu 16000:
vim3 ~ # iperf3 -c 192.168.222.11
Connecting to host 192.168.222.11, port 5201
[ 5] local 192.168.222.10 port 47436 connected to 192.168.222.11 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 14.1 MBytes 118 Mbits/sec 0 934 KBytes
[ 5] 1.00-2.00 sec 11.2 MBytes 94.4 Mbits/sec 2 857 KBytes
[ 5] 2.00-3.00 sec 12.5 MBytes 105 Mbits/sec 0 950 KBytes
[ 5] 3.00-4.00 sec 13.8 MBytes 115 Mbits/sec 2 763 KBytes
[ 5] 4.00-5.00 sec 12.5 MBytes 105 Mbits/sec 1 623 KBytes
[ 5] 5.00-6.00 sec 12.5 MBytes 105 Mbits/sec 0 763 KBytes
[ 5] 6.00-7.00 sec 12.5 MBytes 105 Mbits/sec 0 888 KBytes
[ 5] 7.00-8.00 sec 12.5 MBytes 105 Mbits/sec 1 763 KBytes
[ 5] 8.00-9.00 sec 11.2 MBytes 94.4 Mbits/sec 0 888 KBytes
[ 5] 9.00-10.00 sec 12.5 MBytes 105 Mbits/sec 3 716 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 125 MBytes 105 Mbits/sec 9 sender
[ 5] 0.00-10.02 sec 122 MBytes 102 Mbits/sec receiver
iperf Done.
vim3 ~ # iperf3 -c 192.168.222.11 -R
Connecting to host 192.168.222.11, port 5201
Reverse mode, remote host 192.168.222.11 is sending
[ 5] local 192.168.222.10 port 47440 connected to 192.168.222.11 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 11.1 MBytes 93.1 Mbits/sec
[ 5] 1.00-2.00 sec 11.5 MBytes 96.7 Mbits/sec
[ 5] 2.00-3.00 sec 11.6 MBytes 97.3 Mbits/sec
[ 5] 3.00-4.00 sec 11.6 MBytes 97.0 Mbits/sec
[ 5] 4.00-5.00 sec 11.6 MBytes 97.6 Mbits/sec
[ 5] 5.00-6.00 sec 11.7 MBytes 97.7 Mbits/sec
[ 5] 6.00-7.00 sec 11.6 MBytes 97.6 Mbits/sec
[ 5] 7.00-8.00 sec 11.7 MBytes 98.1 Mbits/sec
[ 5] 8.00-9.00 sec 11.7 MBytes 98.1 Mbits/sec
[ 5] 9.00-10.00 sec 11.6 MBytes 97.6 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 126 MBytes 106 Mbits/sec 6 sender
[ 5] 0.00-10.00 sec 116 MBytes 97.1 Mbits/sec receiver MBytes 98.1 Mbits/sec
[ 5] 8.00-9.00 sec 11.7 MBytes 98.1 Mbits/sec
[ 5] 9.00-10.00 sec 11.6 MBytes 97.6 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 126 MBytes 106 Mbits/sec 6 sender
[ 5] 0.00-10.00 sec 116 MBytes 97.1 Mbits/sec receiver
iperf Done.
HW, mtu 16000:
vim3 ~ # iperf3 -c 192.168.222.11
Connecting to host 192.168.222.11, port 5201
[ 5] local 192.168.222.10 port 47444 connected to 192.168.222.11 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 17.7 MBytes 148 Mbits/sec 0 1.11 MBytes
[ 5] 1.00-2.00 sec 16.2 MBytes 136 Mbits/sec 0 1.96 MBytes
[ 5] 2.00-3.00 sec 17.5 MBytes 147 Mbits/sec 4 2.48 MBytes
[ 5] 3.00-4.00 sec 17.5 MBytes 147 Mbits/sec 6 1.51 MBytes
[ 5] 4.00-5.00 sec 17.5 MBytes 147 Mbits/sec 0 1.63 MBytes
[ 5] 5.00-6.00 sec 15.0 MBytes 126 Mbits/sec 50 1.25 MBytes
[ 5] 6.00-7.00 sec 16.2 MBytes 136 Mbits/sec 0 1.35 MBytes
[ 5] 7.00-8.00 sec 16.2 MBytes 136 Mbits/sec 0 1.38 MBytes
[ 5] 8.00-9.00 sec 15.0 MBytes 126 Mbits/sec 0 1.48 MBytes
[ 5] 9.00-10.00 sec 15.0 MBytes 126 Mbits/sec 0 1.55 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 164 MBytes 138 Mbits/sec 60 sender
[ 5] 0.00-10.00 sec 162 MBytes 136 Mbits/sec receiver
iperf Done.
vim3 ~ # iperf3 -c 192.168.222.11 -R
Connecting to host 192.168.222.11, port 5201
Reverse mode, remote host 192.168.222.11 is sending
[ 5] local 192.168.222.10 port 47448 connected to 192.168.222.11 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 13.1 MBytes 110 Mbits/sec
[ 5] 1.00-2.00 sec 13.6 MBytes 114 Mbits/sec
[ 5] 2.00-3.00 sec 13.6 MBytes 114 Mbits/sec
[ 5] 3.00-4.00 sec 13.6 MBytes 114 Mbits/sec
[ 5] 4.00-5.00 sec 13.2 MBytes 110 Mbits/sec
[ 5] 5.00-6.00 sec 14.0 MBytes 118 Mbits/sec
[ 5] 6.00-7.00 sec 13.4 MBytes 113 Mbits/sec
[ 5] 7.00-8.00 sec 13.6 MBytes 114 Mbits/sec
[ 5] 8.00-9.00 sec 13.7 MBytes 115 Mbits/sec
[ 5] 9.00-10.00 sec 13.7 MBytes 115 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 146 MBytes 123 Mbits/sec 8 sender
[ 5] 0.00-10.00 sec 136 MBytes 114 Mbits/sec receiver
iperf Done.
In this case HW acceleration definitely gives ~30% boost for upload and ~15% for download.
Raw lan speed 870 to 920 mbits
So for mtu 1500 (or smaller like used in internet,my 1492,or manually reduced against connection issues) there is no benefit,right?
Not exactly: encrypted traffic still goes through LAN/WAN/WiFi with mtu 1492/1500, 16000 is only for virtual tun/tap interface. The only hint i see - the firewall needs to apply mss-fix in case when traffic should be routed to some real interfaces (any home router always do it when uses pppoe/pptp ,and mtu is 1492 for example). It still needs to be tested on some real cases ;).
P.S. Also in case 16000 in not a hw limit, but only some driver bug and it can be increased in some future - it may give some more improvements.
devcrypto testing:
I’ve tried to use cryptodev driver. After compiling kernel module and rebuilding openssl with devcrypto engine, i got:
r2-gentoo ~ # openssl engine -t -c
(devcrypto) /dev/crypto engine
[AES-128-CBC, AES-192-CBC, AES-256-CBC, AES-128-CTR, AES-192-CTR, AES-256-CTR, AES-128-ECB, AES-192-ECB, AES-256-ECB, MD5, SHA1, SHA224, SHA256, SHA384, SHA512]
[ available ]
(dynamic) Dynamic engine loading support
[ unavailable ]
(afalg) AFALG engine support
[AES-128-CBC, AES-192-CBC, AES-256-CBC]
[ available ]
quick benchmark showed that if faster than AF_ALG in singlethread and multithread mode:
r2-gentoo ~ # openssl speed -elapsed -evp aes-256-cbc -engine devcrypto
engine "devcrypto" set.
...
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
aes-256-cbc 524.01k 2119.77k 8263.25k 30785.19k 83902.46k 86731.43k
r2-gentoo ~ # openssl speed -elapsed -evp aes-256-cbc -engine devcrypto -multi 2
Forked child 0
Forked child 1
...
evp 1037.79k 4108.90k 16176.68k 60337.15k 140787.71k 140170.58k
r2-gentoo ~ # openssl speed -elapsed -evp aes-256-cbc --engine afalg
engine "afalg" set.
...
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
aes-256-cbc 486.05k 1948.84k 7123.29k 18607.79k 64984.41k 79779.16k
r2-gentoo ~ # openssl speed -elapsed -evp aes-256-cbc --engine afalg -multi 2
Forked child 0
Forked child 1
engine "afalg" set.
evp 609.21k 2646.38k 10122.07k 34820.10k 119229.10k 142447.96k
but i failed to run openvpn, and self test showed:
openvpn --test-crypto --secret ./static.key --cipher AES-256-CBC --tun-mtu 100 --engine devcrypto
Tue Mar 2 21:25:04 2021 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
Tue Mar 2 21:25:04 2021 OpenVPN 2.4.9 armv7a-hardfloat-linux-gnueabi [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Apr 29 2020
Tue Mar 2 21:25:04 2021 library versions: OpenSSL 1.1.1j 16 Feb 2021, LZO 2.10
Tue Mar 2 21:25:04 2021 OpenVPN 2.4.9 armv7a-hardfloat-linux-gnueabi [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Apr 29 2020
Tue Mar 2 21:25:04 2021 Initializing OpenSSL support for engine 'devcrypto'
Tue Mar 2 21:25:04 2021 Entering OpenVPN crypto self-test mode.
Tue Mar 2 21:25:04 2021 TESTING ENCRYPT/DECRYPT of packet length=1
Tue Mar 2 21:25:04 2021 TESTING ENCRYPT/DECRYPT of packet length=2
Tue Mar 2 21:25:04 2021 TESTING ENCRYPT/DECRYPT of packet length=3
Tue Mar 2 21:25:04 2021 TESTING ENCRYPT/DECRYPT of packet length=4
Tue Mar 2 21:25:04 2021 TESTING ENCRYPT/DECRYPT of packet length=5
Tue Mar 2 21:25:04 2021 TESTING ENCRYPT/DECRYPT of packet length=6
Tue Mar 2 21:25:04 2021 TESTING ENCRYPT/DECRYPT of packet length=7
Tue Mar 2 21:25:04 2021 TESTING ENCRYPT/DECRYPT of packet length=8
Tue Mar 2 21:25:04 2021 Authenticate/Decrypt packet error: cipher final failed
Tue Mar 2 21:25:04 2021 SELF TEST FAILED, src.len=8 buf.len=0
Tue Mar 2 21:25:04 2021 Exiting due to fatal error
Taking into account good results with AF_ALG, i think it’s a devcrypto issue(?)
BTW maximum working mtu with AF_ALG is 16375
Have you used latest cryptodev?
Yep, i used github version
Maybe file an issue on github…have nor found solution for this error in openvpn with cryptodev