Is it possible to have the crypto extensions working?

Banana Pi Router design
52

@frank-w I have copied the utils folder from 4.14-main repository to 5.4-main branch and I tried to compile cryptodev (I saw you updated it to version 1.9) but it fails. When I ran ./build.sh cryptodev it throws the below compilation error

Building modules, stage 2.
  MODPOST 1 modules
ERROR: "crypto_givcipher_type" [/home/sysadmin/bpi-r2_5.4/utils/cryptodev/cryptodev-linux/cryptodev.ko] undefined!
ERROR: "sys_close" [/home/sysadmin/bpi-r2_5.4/utils/cryptodev/cryptodev-linux/cryptodev.ko] undefined!
make[2]: *** [scripts/Makefile.modpost:94: __modpost] Error 1
make[1]: *** [Makefile:1609: modules] Error 2
make[1]: Leaving directory '/home/sysadmin/bpi-r2_5.4'
make: *** [Makefile:27: build] Error 2
53

Look where crypto_givcipher_type and sys_close are defined and make sure file with implementation is linked by makefile.

54

UPDATE: I have removed version 1.9 and pulled the latest master from github and it works fine. It appears 1.9 release does not work with 5.x kernels and it was fixed after 1.10 release. (https://github.com/cryptodev-linux/cryptodev-linux/commit/f971e0cd4a0ebe59fb2e8e17240399bf6901b09b)

1 Like
55

i have added cryptodev 1.10 + the Patch you’ve sorted out to my 5.4-main tree

thanks for checking

56

5.4.27-bpi-r2-main

I could not get cryptodev working. I spent nearly two whole days trying to figure out how to enable it for OpenSSL.

It turns out, that devcryptoeng is not enabled by default and the defines HAVE_CRYPTODEV and USE_CRYPTODEV_DIGESTS don’t change a thing about that. You need to add one more argument - enable-devcryptoeng.

Working commands for me were

$ export CROSS=arm-linux-gnueabihf
$ export CC=${CROSS}-gcc
$ export LD=${CROSS}-ld
$ export AS=${CROSS}-as
$ export AR=${CROSS}-ar
$ export DEB_HOST_ARCH=armhf
$ export DEB_BUILD_OPTIONS=nocheck

$ git clone https://github.com/cryptodev-linux/cryptodev-linux.git
$ apt-get source openssl
# You must match your version
$ cd openssl-1.1.1.d
$ sed -i -e "s/CONFARGS  =/CONFARGS = enable-devcryptoeng -DHAVE_CRYPTODEV -DUSE_CRYPTODEV_DIGESTS -I..\/..\/cryptodev-linux/" debian/rules
# I also had to disable shlibdeps by commenting `dh_shlibdeps` at the end of the `debian/rules`
$dpkg-buildpackage -us -uc -aarmhf -b

You need to have cryptodev already compiled. Then just install created deb packages.

Hope this helps. If you found anything wrong, let me know and I will edit this post.

1 Like
57

Can you post a patch for cryptodev i included in my kernelrepo (5.4-main)?

58

The soonest I will have time to do it is maybe next weekend.

Anyway… I had problem with SSH. I couldn’t establish a connection. devcrypto was obviously on fault.

I found this article where it says where is the problem…

Disabling digests

Please, don’t enable digests unless you know what you’re doing. They are usually slower than software, >except for large (> 10k) blocks. Some applications–openssh, for example–will not work with /dev/crypto >digests. This is a limitation of how the engine works. Openssh will save a partial digest, and then fork, >duplicating that context, and working with successive copies of it, which is useful for HMAC, where the >hash of the key remains constant. In the kernel, however, those contexts are still linked to the same >session, so when one process calls another update, or closes that digest context, the kernel session is >changed/closed for all of the instances, and you’ll get a libcrypto failure. For well-behaved applications >using large update blocks, you may enable digests. Use a separate copy of the openssl.cnf >configuration file, and set OPENSSL_CONF=_path_to_file in the environment before running it (add it >to the respective file in /etc/init.d/). Again, benchmarking the actual application you’re using is the best >way to gauge the impact of hardware crypto.

I tried to disable using DIGESTS in OpenSSL through configuration, but it seems that it ignores default_algorithms for cryptodev, even with USE_SOFTDRIVERS=1 I also removed defines for compilation, so that only enable-devcryptoeng was left, but it didn’t help. At least I found out, that cryptodev can be enabled just with enable-devcryptoeng and no other defines are needed.

Anyway… that lead me to the only possible solution left - removing it from source. Source file where you can disable it is in openssl/crypto/engines

After I compiled it, I got this (missing digests)

root@claudius:~# openssl engine -c
(devcrypto) /dev/crypto engine
 [DES-CBC, DES-EDE3-CBC, AES-128-CBC, AES-192-CBC, AES-256-CBC, AES-128-CTR, AES-192-CTR, AES-256-CTR, AES-128-ECB, AES-192-ECB, AES-256-ECB]
(dynamic) Dynamic engine loading support

I am planning to disable all ciphers except AES-CBC. I need to use cryptodev only for VPN and more ciphers would just add more interrupts to core0.

If somebody could figure out how to disable cryptodev for DIGESTS without making modifications to source, that would be great. Here is a “documentation” https://www.openssl.org/docs/man1.1.1/man5/config.html#Engine-Configuration-Module

I couldn’t find any usage of engine config, except for some sample

1 Like
59

I had also that problem with SSH when I enabled devcrypto for OpenSSL. Initially I did not know why I cannot login and then I found some page that stated you cannot accelerate OpenSSH connection because latest version of OpenSSH enforce usage of seccomp sandbox which forbids some syscalls required to use AF_ALG. I think the same applied to devcrypto engine also.

Refer to Debian bug #931271.

I have recompiled openssh from source and disabled seccomp sandbox to allow me to connect remotely to the box

#Edit debian/rules and add it to common build options:

confflags += --with-sandbox=no

1 Like
60

Hi everyone!

I’ve tested my R2 with AF_ALG acceleration - work good for me. All i needed to do is to add following kernel config options:

CONFIG_CRYPTO_USER=m
CONFIG_CRYPTO_USER_API=m              # added automatically after one options below was added
CONFIG_CRYPTO_USER_API_HASH=m         # probably usable, but was not used this time
CONFIG_CRYPTO_USER_API_SKCIPHER=m # this one was really used
CONFIG_CRYPTO_USER_API_RNG=m      # - theese two are kikely useless for R2
CONFIG_CRYPTO_USER_API_AEAD=m    # /

After recompiling kernel and reboot, ssl conf needed to be changed:

cat /etc/ssl/openssl.cnf:

openssl_conf = openssl_def

[openssl_def]
engines = openssl_engines

[openssl_engines]
afalg = afalg_engine

[afalg_engine]
init=1

openssl engine list:

r2-gentoo ~ # openssl engine -t -c -v
(dynamic) Dynamic engine loading support
     [ unavailable ]
     SO_PATH, NO_VCHECK, ID, LIST_ADD, DIR_LOAD, DIR_ADD, LOAD
(afalg) AFALG engine support
 [AES-128-CBC, AES-192-CBC, AES-256-CBC]
     [ available ]

and finnaly perfomance test:

no acceleration:

bpi-r2-gentoo ~ # time openssl speed -evp aes-128-cbc -elapsed
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128-cbc for 3s on 16 size blocks: 4551769 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 64 size blocks: 1420522 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 357557 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 79175 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 9870 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 16384 size blocks: 4496 aes-128-cbc's in 3.00s
OpenSSL 1.1.1j  16 Feb 2021
built on: Fri Feb 19 22:05:09 2021 UTC
options:bn(64,32) rc4(char) des(long) aes(partial) idea(int) blowfish(ptr)
compiler: armv7a-hardfloat-linux-gnueabi-gcc -fPIC -pthread -Wa,--noexecstack -Wall -O2 -pipe -march=armv7-a -mfpu=neon-vfpv4 -mfloat-abi=hard -fno-strict-aliasing -Wa,--noexecstack -DOPENSSL_USE_NODELETE -DOPEN
SSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DZLIB -DNDEBUG -
DL_ENDIAN  -DOPENSSL_NO_BUF_FREELISTS
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-cbc      24276.10k    30304.47k    30511.53k    27025.07k    26951.68k    24554.15k

real    0m18,025s
user    0m15,880s
sys     0m0,060s

AF_ALG:

r2-gentoo ~ # time openssl speed -evp aes-128-cbc -elapsed
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128-cbc for 3s on 16 size blocks: 63195 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 64 size blocks: 62880 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 61864 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 59010 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 29029 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 16384 size blocks: 16145 aes-128-cbc's in 3.00s
OpenSSL 1.1.1j  16 Feb 2021
built on: Mon Feb 22 11:56:18 2021 UTC
options:bn(64,32) rc4(char) des(long) aes(partial) idea(int) blowfish(ptr)
compiler: armv7a-hardfloat-linux-gnueabi-gcc -fPIC -pthread -Wa,--noexecstack -Wall -O2 -pipe -march=armv7-a -mfpu=neon-vfpv4 -mfloat-abi=hard -fno-strict-aliasing -Wa,--noexecstack -DOPENSSL_USE_NODELETE -DOPEN
SSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DZLIB -DNDEBUG -
DL_ENDIAN  -DOPENSSL_NO_BUF_FREELISTS
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-cbc        337.04k     1341.44k     5279.06k    20142.08k    79268.52k    88173.23k

real    0m18,543s
user    0m0,824s
sys     0m12,075s

user 0m15,880s - non-accelerated vs user 0m0,824s - accelerated

~2.94 times faster on 8192 bytes blocks and ~3.6 times faster on 16384 bytes blocks (single run)

similar difference whet testing aes-192-cbc and aes-256-cbc

r2-gentoo ~ # cat /proc/interrupts | grep aes
 51:     339897          0          0          0  MT_SYSIRQ  82 Level     mtk-aes
 52:          0          0          0          0  MT_SYSIRQ  83 Level     mtk-aes

As for me - a good way to accelerate out of box: on mainline kernel w/o additional modules and mainline openssl w/o patches/additional engines.

Openvpn/openssh(sandbox=no required) are not yet tested,

Also testing on standard @frank-w’s kernel config is probably needed.

P.S. tested on 5.9.0-rc8 kernel

61

so you need no recompile of openssl and no change in dts (for using eip97 engine imho now using different compatible/driver), right?

you can create a pull request (for LTS main branch), then i add the config-options

63

Yes, at least for my kernel config. Only openssl configuration is neeed to be changed.

Sure, what is the LTS-main branch, 5.4 or 5.10? I’ll can do for both, but 5.4 is low-priority.

I’ll appreciate for any additional tests by community.

64

5.10 is enough…can cherry-pick it to 5.4.

65

Done.

Compiled and tested with importconfig. Works with openssl speed -evp aes-256-cbc -elapsed -engine afalg (no engine specified required after modifying openssl conf).

Still need testing with openvpn/openssh if possible,

Tried to test openvpn:

time openvpn --engine afalg --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc

Got cipher error on 1st run:

Mon Mar 1 19:39:07 2021 cipher_ctx_update: EVP_CipherUpdate() failed

dmesg:

[ 2278.586698] ------------[ cut here ]------------
[ 2278.586737] WARNING: CPU: 2 PID: 4612 at lib/refcount.c:28 refcount_warn_saturate+0x13c/0x174
[ 2278.586742] refcount_t: underflow; use-after-free.
[ 2278.586747] Modules linked in: algif_skcipher af_alg i2c_dev lima gpu_sched spi_mt65xx pwm_mediatek mtk_pmic_keys [last unloaded: cryptodev]
[ 2278.586798] CPU: 2 PID: 4612 Comm: openvpn Tainted: G           O      5.10.18-bpi-r2-main #1
[ 2278.586803] Hardware name: Mediatek Cortex-A7 (Device Tree)
[ 2278.586809] Backtrace:  
[ 2278.586827] [<c0dc9cf8>] (dump_backtrace) from [<c0dca0a4>] (show_stack+0x20/0x24)
[ 2278.586837]  r7:0000001c r6:600d0013 r5:00000000 r4:c14ef938
[ 2278.586850] [<c0dca084>] (show_stack) from [<c0dcdd48>] (dump_stack+0xcc/0xe0)
[ 2278.586862] [<c0dcdc7c>] (dump_stack) from [<c0126840>] (__warn+0xfc/0x114)
[ 2278.586871]  r7:0000001c r6:c057a580 r5:00000009 r4:c1181000
[ 2278.586881] [<c0126744>] (__warn) from [<c0dca7b4>] (warn_slowpath_fmt+0xa8/0xc4)
[ 2278.586890]  r7:0000001c r6:c1181000 r5:c118103c r4:d3640000
[ 2278.586902] [<c0dca710>] (warn_slowpath_fmt) from [<c057a580>] (refcount_warn_saturate+0x13c/0x174)
[ 2278.586913]  r9:c75d61c0 r8:00000000 r7:000000f6 r6:d3640000 r5:0051d3f0 r4:c5b7fa80
[ 2278.586927] [<c057a444>] (refcount_warn_saturate) from [<c037dfa4>] (sys_io_submit+0x6d4/0x8d4)
[ 2278.586939] [<c037d8d0>] (sys_io_submit) from [<c0100060>] (ret_fast_syscall+0x0/0x54)
[ 2278.586945] Exception stack(0xd3641fa8 to 0xd3641ff0)
[ 2278.586955] 1fa0:                   00000000 00000000 b6fc7000 00000001 bed958e4 00000020
[ 2278.586965] 1fc0: 00000000 00000000 bed958e4 000000f6 b6f9a000 bed958e8 00000000 bed958f4
[ 2278.586972] 1fe0: bed958c0 bed958b0 b6f88698 b6c08e30
[ 2278.586982]  r10:000000f6 r9:d3640000 r8:c0100264 r7:000000f6 r6:bed958e4 r5:00000000
[ 2278.586988]  r4:00000000
[ 2278.586994] ---[ end trace 54f7a3c195c24c86 ]---

and kernel panic on 2nd run:

serial output:

r2-gentoo login: af_algaf_alg[ 2357.439515] Internal error: Oops: 5 [#1] SMP ARM                                                                                                                                    
[ 2357.444256] Modules linked in: algif_skcipher afdev lima gpu_sched spi_mt65xx pwm_mediatek mtk_pmic_keys [last unloaded: af_alg]                                                                                 
[ 2357.456905] CPU: 2 PID: 4629 Comm: openvpn Tainted: G        W  O      5.10.18-bpi-r2-main #1                                                                                                                    
[ 2357.465595] Hardware name: Mediatek Cortex-A7 (Device Tree)                                                                                                                                                      
[ 2357.471271] PC is at release_sock+0x3c/0x90                                                                                                                                                                      
[ 2357.475524] LR is at release_sock+0x24/0x90                                                                                                                                                                      
[ 2357.479779] pc : [<c0a7f440>]    lr : [<c0a7f428>]    psr: 600d0013                                                                                                                                              
[ 2357.486148] sp : d54cdd80  ip : d54cdd80  fp : d54cdd94                                                                                                                                                          
[ 2357.491456] r10: c47ad6c0  r9 : d54cc000  r8 : 00000000                                                                                                                                                          
[ 2357.496765] r7 : 00000030  r6 : c5f66c00  r5 : c5f66c70  r4 : c5f66c00                                                                                                                                           
[ 2357.503390] r3 : 00000000  r2 : 00000007  r1 : 00000007  r0 : c5f66c70                                                                                                                                           
[ 2357.510018] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none                                                                                                                                    
[ 2357.517268] Control: 10c5387d  Table: 954bc06a  DAC: 00000051                                                                                                                                                    
[ 2357.523116] Process openvpn (pid: 4629, stack limit = 0x777cd319)                                                                                                                                                
[ 2357.529314] Stack: (0xd54cdd80 to 0xd54ce000)                                                                                                                                                                    
[ 2357.533748] dd80: 00000000 c5f65000 d54cddb4 d54cdd98 bf04208c c0a7f410 c81aea80 d54cdf40                                                                                                                        
[ 2357.542059] dda0: 00000010 00000030 d54cddd4 d54cddb8 bf042300 bf04200c d54cdf40 00000010                                                                                                                        
[ 2357.550373] ddc0: c81aea80 00000030 d54cde34 d54cddd8 c0a787c0 bf0422e8 00000000 00000051                                                                                                                        
[ 2357.558685] dde0: d54cde34 d54cddf0 c0a7a40c c0576960 d54cde40 d54cdf48 e06fc5c4 00000000                                                                                                                        
[ 2357.566999] de00: beb028e8 92899668 e06fc5c4 00000000 d54cdf40 d54cc000 c81aea80 00000000                                                                                                                        
[ 2357.575304] de20: 00000000 00000128 d54cdf2c d54cde38 c0a7a4ac c0a78604 00000000 92899668                                                                                                                        
[ 2357.583613] de40: 00000000 00597038 00000010 c15bb6c0 c157dd08 c75d62c0 00000051 beb02998                                                                                                                        
[ 2357.591926] de60: 00000001 00000001 c0378f10 d54cded8 c75d61c0 00000001 d54cc000 beb02998                                                                                                                        
[ 2357.600240] de80: d54cc000 000000f5 d54cdeb4 d54cde98 c037b15c c037ad78 00000001 c75d61c0                                                                                                                        
[ 2357.608554] dea0: 00000001 c037b3d0 d54cdf5c d54cdeb8 c037b3d0 c01a2a9c d54cded8 c036fcf8                                                                                                                        
[ 2357.616868] dec0: 00000000 c2ee7db0 00000000 beb02998 00000000 00000000 00000001 00000008                                                                                                                        
[ 2357.625183] dee0: 00000004 00000000 00000000 d54cdee0 00000000 c033dbd0 d54cdf14 d54cdf08                                                                                                                        
[ 2357.633495] df00: c033dbd0 92899668 d54cc000 beb028f4 00000000 c81aea80 c0100264 d54cc000                                                                                                                        
[ 2357.641806] df20: d54cdf94 d54cdf30 c0a7a928 c0a7a438 00000000 00000000 00000000 fffffff7                                                                                                                        
[ 2357.650116] df40: 00000000 00000000 00000005 00000000 00000010 d54cde44 00000001 c037b2d0                                                                                                                        
[ 2357.658430] df60: c47ad6c0 92899668 00000030 00000000 00000000 92899668 0058d0a0 0059bebc                                                                                                                        
[ 2357.666743] df80: b6f12840 00000128 d54cdfa4 d54cdf98 c0a7a980 c0a7a8d0 00000000 d54cdfa8                                                                                                                        
[ 2357.675053] dfa0: c0100060 c0a7a970 0058d0a0 0059bebc 00000006 beb028f4 00000000 00000000                                                                                                                        
[ 2357.683364] dfc0: 0058d0a0 0059bebc b6f12840 00000128 b6ee3000 beb028e8 00000001 beb028f4                                                                                                                        
[ 2357.691676] dfe0: b6ee3054 beb028a8 b6ed1598 b6b57458 600d0010 00000006 000000000                                                                                                                                
[ 2357.699983] Backtrace:                                                                                                                                                                                           
[ 2357.702512] [<c0a7f404>] (release_sock) from [<bf04208c>] (skcipher_check_key+0x8c/0x90 [algif_skcipher])                                                                                                        
[ 2357.712241]  r5:c5f65000 r4:00000000                                                                                                                                                                             
[ 2357.715889] [<bf042000>] (skcipher_check_key [algif_skcipher]) from [<bf042300>] (skcipher_sendmsg_nokey+0x24/0x54 [algif_skcipher])                                                                             
[ 2357.727991]  r7:00000030 r6:00000010 r5:d54cdf40 r4:c81aea80                                                                                                                                                     
[ 2357.733760] [<bf0422dc>] (skcipher_sendmsg_nokey [algif_skcipher]) from [<c0a787c0>] (____sys_sendmsg+0x1c8/0x260)                                                                                               
[ 2357.744282]  r7:00000030 r6:c81aea80 r5:00000010 r4:d54cdf40                                                                                                                                                     
[ 2357.750053] [<c0a785f8>] (____sys_sendmsg) from [<c0a7a4ac>] (___sys_sendmsg+0x80/0xac)                                                                                                                          
[ 2357.758194]  r10:00000128 r9:00000000 r8:00000000 r7:c81aea80 r6:d54cc000 r5:d54cdf40                                                                                                                            
[ 2357.766147]  r4:00000000                                                                                                                                                                                         
[ 2357.768746] [<c0a7a42c>] (___sys_sendmsg) from [<c0a7a928>] (__sys_sendmsg+0x64/0xa0)                                                                                                                            
[ 2357.776713]  r9:d54cc000 r8:c0100264 r7:c81aea80 r6:00000000 r5:beb028f4 r4:d54cc000                                                                                                                             
[ 2357.784585] [<c0a7a8c4>] (__sys_sendmsg) from [<c0a7a980>] (sys_sendmsg+0x1c/0x20)                                                                                                                               
[ 2357.792284]  r7:00000128 r6:b6f12840 r5:0059bebc r4:0058d0a0                                                                                                                                                     
[ 2357.798050] [<c0a7a964>] (sys_sendmsg) from [<c0100060>] (ret_fast_syscall+0x0/0x54)                                                                                                                             
[ 2357.805921] Exception stack(0xd54cdfa8 to 0xd54cdff0)                                                                                                                                                            
[ 2357.811064] dfa0:                   0058d0a0 0059bebc 00000006 beb028f4 00000000 00000000                                                                                                                        
[ 2357.819378] dfc0: 0058d0a0 0059bebc b6f12840 00000128 b6ee3000 beb028e8 00000001 beb028f4                                                                                                                        
[ 2357.827682] dfe0: b6ee3054 beb028a8 b6ed1598 b6b57458                                                                                                                                                            
[ 2357.832821] Code: 0a000001 e1a00004 ebffffaf e5943020 (e5933048)                                                                                                                                                 
[ 2357.839071] ---[ end trace 54f7a3c195c24c87 ]---                                                                                                                                                                 
[ 2357.843798] Kernel panic - not syncing: Fatal exception in interrupt                                                                                                                                             
[ 2357.843820] CPU0: stopping                                                                                                                                                                                       
[ 2357.843833] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G      D W  O      5.10.18-bpi-r2-main #1                                                                                                                     
[ 2357.843836] Hardware name: Mediatek Cortex-A7 (Device Tree)                                                                                                                                                      
[ 2357.843839] Backtrace:                                                                                                                                                                                           
[ 2357.843862] [<c0dc9cf8>] (dump_backtrace) from [<c0dca0a4>] (show_stack+0x20/0x24)                                                                                                                               
[ 2357.843870]  r7:00000000 r6:60030193 r5:00000000 r4:c14ef938                                                                                                                                                     
[ 2357.843880] [<c0dca084>] (show_stack) from [<c0dcdd48>] (dump_stack+0xcc/0xe0)                                                                                                                                   
[ 2357.843890] [<c0dcdc7c>] (dump_stack) from [<c0110290>] (do_handle_IPI+0x328/0x35c)                                                                                                                              
[ 2357.843897]  r7:00000000 r6:00000000 r5:c1572e40 r4:c15bb640                                                                                                                                                     
[ 2357.843905] [<c010ff68>] (do_handle_IPI) from [<c01102ec>] (ipi_handler+0x28/0x30)                                                                                                                               
[ 2357.843913]  r9:c1401ed0 r8:c2029000 r7:00000000 r6:1e1f2000 r5:c2030e00 r4:00000014                                                                                                                             
[ 2357.843923] [<c01102c4>] (ipi_handler) from [<c01946d4>] (handle_percpu_devid_fasteoi_ipi+0x90/0x160)                                                                                                            
[ 2357.843933] [<c0194644>] (handle_percpu_devid_fasteoi_ipi) from [<c018d76c>] (generic_handle_irq+0x44/0x54)                                                                                                      
[ 2357.843939]  r7:00000000 r6:00000001 r5:00000000 r4:c13a6c24                                                                                                                                                     
[ 2357.843948] [<c018d728>] (generic_handle_irq) from [<c018de7c>] (__handle_domain_irq+0x6c/0xc0)                                                                                                                  
[ 2357.843957] [<c018de10>] (__handle_domain_irq) from [<c010135c>] (gic_handle_irq+0x8c/0xa0)                                                                                                                      
[ 2357.843965]  r9:c1401ed0 r8:e100200c r7:c13a6c30 r6:e1002000 r5:c14ef9f8 r4:c14057cc                                                                                                                             
[ 2357.843973] [<c01012d0>] (gic_handle_irq) from [<c0100b0c>] (__irq_svc+0x6c/0x90)                                                                                                                                
[ 2357.843977] Exception stack(0xc1401ed0 to 0xc1401f18)                                                                                                                                                            
[ 2357.843983] 1ec0:                                     00000000 01779ccc df59a304 c011ee40                                                                                                                        
[ 2357.843991] 1ee0: c1573ba0 00000000 c1404f10 c1404f58 c1572634 c114496c e07fcd00 c1401f2c                                                                                                                        
[ 2357.843997] 1f00: c1401f30 c1401f20 c01096e4 c01096e8 60030013 ffffffff                                                                                                                                          
[ 2357.844005]  r9:c1400000 r8:c1572634 r7:c1401f04 r6:ffffffff r5:60030013 r4:c01096e8                                                                                                                             
[ 2357.844018] [<c01096a0>] (arch_cpu_idle) from [<c0de1148>] (default_idle_call+0x48/0x10c)                                                                                                                        
[ 2357.844027] [<c0de1100>] (default_idle_call) from [<c015f500>] (do_idle+0xe4/0x150)                                                                                                                              
[ 2357.844032]  r5:00000000 r4:c1400000                                                                                                                                                                             
[ 2357.844039] [<c015f41c>] (do_idle) from [<c015f83c>] (cpu_startup_entry+0x28/0x2c)                                                                                                                               
[ 2357.844047]  r9:c1357a54 r8:00000000 r7:c1357a54 r6:c1404ec0 r5:c1400000 r4:000000d8                                                                                                                             
[ 2357.844055] [<c015f814>] (cpu_startup_entry) from [<c0dda448>] (rest_init+0xbc/0xc4)                                                                                                                             
[ 2357.844067] [<c0dda38c>] (rest_init) from [<c1300ad8>] (arch_call_rest_init+0x18/0x1c)                                                                                                                           
[ 2357.844072]  r5:c1400000 r4:c15bb040                                                                                                                                                                             
[ 2357.844081] [<c1300ac0>] (arch_call_rest_init) from [<c1301078>] (start_kernel+0x524/0x55c)                                                                                                                      
[ 2357.844090] [<c1300b54>] (start_kernel) from [<00000000>] (0x0)                                                                                                                                                  
[ 2357.844096] CPU3: stopping                                                                                                                                                                                       
[ 2357.844104] CPU: 3 PID: 4536 Comm: emerge Tainted: G      D W  O      5.10.18-bpi-r2-main #1                                                                                                                     
[ 2357.844107] Hardware name: Mediatek Cortex-A7 (Device Tree)                                                                                                                                                      
[ 2357.844109] Backtrace:                                                                                                                                                                                           
[ 2357.844121] [<c0dc9cf8>] (dump_backtrace) from [<c0dca0a4>] (show_stack+0x20/0x24)                                                                                                                               
[ 2357.844128]  r7:00000000 r6:60030193 r5:00000000 r4:c14ef938                                                                                                                                                     
[ 2357.844136] [<c0dca084>] (show_stack) from [<c0dcdd48>] (dump_stack+0xcc/0xe0)                                                                                                                                   
[ 2357.844145] [<c0dcdc7c>] (dump_stack) from [<c0110290>] (do_handle_IPI+0x328/0x35c)                                                                                                                              
[ 2357.844151]  r7:00000000 r6:00000003 r5:c1572e40 r4:c15bb640                                                                                                                                                     
[ 2357.844159] [<c010ff68>] (do_handle_IPI) from [<c01102ec>] (ipi_handler+0x28/0x30)                                                                                                                               
[ 2357.844167]  r9:ca16dfb0 r8:c2029000 r7:00000000 r6:1e22e000 r5:c2030e00 r4:00000014                                                                                                                             
[ 2357.844175] [<c01102c4>] (ipi_handler) from [<c01946d4>] (handle_percpu_devid_fasteoi_ipi+0x90/0x160)                                                                                                            
[ 2357.844184] [<c0194644>] (handle_percpu_devid_fasteoi_ipi) from [<c018d76c>] (generic_handle_irq+0x44/0x54)                                                                                                      
[ 2357.844190]  r7:00000000 r6:00000001 r5:00000000 r4:c13a6c24                                                                                                                                                     
[ 2357.844200] [<c018d728>] (generic_handle_irq) from [<c018de7c>] (__handle_domain_irq+0x6c/0xc0)                                                                                                                  
[ 2357.844208] [<c018de10>] (__handle_domain_irq) from [<c010135c>] (gic_handle_irq+0x8c/0xa0)                                                                                                                      
[ 2357.844216]  r9:ca16dfb0 r8:e100200c r7:c13a6c30 r6:e1002000 r5:c14ef9f8 r4:c14057cc                                                                                                                             
[ 2357.844223] [<c01012d0>] (gic_handle_irq) from [<c0100eb4>] (__irq_usr+0x54/0x80)                                                                                                                                
[ 2357.844227] Exception stack(0xca16dfb0 to 0xca16dff8)                                                                                                                                                            
[ 2357.844232] dfa0:                                     b6569e60 b660acc4 00000040 0000003f                                                                                                                        
[ 2357.844240] dfc0: 00000030 b660aca0 7e486a30 00517f64 b660aca0 b6f22000 7e486a30 00000040                                                                                                                        
[ 2357.844246] dfe0: b6f30798 beacb800 00001e2c b6cd8180 80030010 ffffffff                                                                                                                                          
[ 2357.844254]  r9:b6f22000 r8:10c5387d r7:10c5387d r6:ffffffff r5:80030010 r4:b6cd8180                                                                                                                             
[ 2357.844259] CPU1: stopping                                                                                                                                                                                       
[ 2357.844266] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G      D W  O      5.10.18-bpi-r2-main #1                                                                                                                     
[ 2357.844269] Hardware name: Mediatek Cortex-A7 (Device Tree)                                                                                                                                                      
[ 2357.844271] Backtrace:                                                                                                                                                                                           
[ 2357.844283] [<c0dc9cf8>] (dump_backtrace) from [<c0dca0a4>] (show_stack+0x20/0x24)                                                                                                                               
[ 2357.844290]  r7:00000000 r6:60030193 r5:00000000 r4:c14ef938                                                                                                                                                     
[ 2357.844298] [<c0dca084>] (show_stack) from [<c0dcdd48>] (dump_stack+0xcc/0xe0)                                                                                                                                   
[ 2357.844306] [<c0dcdc7c>] (dump_stack) from [<c0110290>] (do_handle_IPI+0x328/0x35c)                                                                                                                              
[ 2357.844313]  r7:00000000 r6:00000001 r5:c1572e40 r4:c15bb640                                                                                                                                                     
[ 2357.844320] [<c010ff68>] (do_handle_IPI) from [<c01102ec>] (ipi_handler+0x28/0x30)                                                                                                                               
[ 2357.844328]  r9:c2171f30 r8:c2029000 r7:00000000 r6:1e206000 r5:c2030e00 r4:00000014    357.844376]  r9:c2171f30 r8:e100200c r7:c13a6c30 r6:e1002000 r5:c14ef9f8 r4:c14057cc                                                                                                                             
[ 2357.844383] [<c01012d0>] (gic_handle_irq) from [<c0100b0c>] (__irq_svc+0x6c/0x90)                                                                                                                                
[ 2357.844387] Exception stack(0xc2171f30 to 0xc2171f78)                                                                                                                                                            
[ 2357.844393] 1f20:                                     00000000 00d5c648 df5ae304 c011ee40                                                                                                                        
[ 2357.844400] 1f40: c1573ba0 00000001 c1404f10 c1404f58 c1572634 c114496c 00000000 c2171f8c                                                                                                                        
[ 2357.844407] 1f60: c2171f90 c2171f80 c01096e4 c01096e8 60030013 ffffffff                                                                                                                                          
[ 2357.844414]  r9:c2170000 r8:c1572634 r7:c2171f64 r6:ffffffff r5:60030013 r4:c01096e8                                                                                                                             
[ 2357.844424] [<c01096a0>] (arch_cpu_idle) from [<c0de1148>] (default_idle_call+0x48/0x10c)                                                                                                                        
[ 2357.844433] [<c0de1100>] (default_idle_call) from [<c015f500>] (do_idle+0xe4/0x150)                                                                                                                              
[ 2357.844437]  r5:00000001 r4:c2170000                                                                                                                                                                             
[ 2357.844444] [<c015f41c>] (do_idle) from [<c015f83c>] (cpu_startup_entry+0x28/0x2c)                                                                                                                               
[ 2357.844452]  r9:410fc073 r8:8000406a r7:c15bb650 r6:10c0387d r5:00000001 r4:00000091                                                                                                                             
[ 2357.844460] [<c015f814>] (cpu_startup_entry) from [<c0110bcc>] (secondary_start_kernel+0x170/0x194)                                                                                                              
[ 2357.844468] [<c0110a5c>] (secondary_start_kernel) from [<801017cc>] (0x801017cc)                                                                                                                                 
[ 2357.844472]  r5:00000051 r4:8216806a                                                                                                                                                                             
[ 2358.971480] Rebooting in 3 seconds..357.844337] [<c01102c4>] (ipi_handler) from [<c01946d4>] (handle_percpu_devid_fasteoi_ipi+0x90/0x160)                                                                                                            
[ 2357.844345] [<c0194644>] (handle_percpu_devid_fasteoi_ipi) from [<c018d76c>] (generic_handle_irq+0x44/0x54)                                                                                                      
[ 2357.844351]  r7:00000000 r6:00000001 r5:00000000 r4:c13a6c24                                                                                                                                                     
[ 2357.844360] [<c018d728>] (generic_handle_irq) from [<c018de7c>] (__handle_domain_irq+0x6c/0xc0)                                                                                                                  
[ 2357.844368] [<c018de10>] (__handle_domain_irq) from [<c010135c>] (gic_handle_irq+0x8c/0xa0)                                                                                                                      
[ 2357.844376]  r9:c2171f30 r8:e100200c r7:c13a6c30 r6:e1002000 r5:c14ef9f8 r4:c14057cc                                                                                                                             
[ 2357.844383] [<c01012d0>] (gic_handle_irq) from [<c0100b0c>] (__irq_svc+0x6c/0x90)                                                                                                                                
[ 2357.844387] Exception stack(0xc2171f30 to 0xc2171f78)                                                                                                                                                            
[ 2357.844393] 1f20:                                     00000000 00d5c648 df5ae304 c011ee40                                                                                                                        
[ 2357.844400] 1f40: c1573ba0 00000001 c1404f10 c1404f58 c1572634 c114496c 00000000 c2171f8c                                                                                                                        
[ 2357.844407] 1f60: c2171f90 c2171f80 c01096e4 c01096e8 60030013 ffffffff                                                                                                                                          
[ 2357.844414]  r9:c2170000 r8:c1572634 r7:c2171f64 r6:ffffffff r5:60030013 r4:c01096e8                                                                                                                             
[ 2357.844424] [<c01096a0>] (arch_cpu_idle) from [<c0de1148>] (default_idle_call+0x48/0x10c)                                                                                                                        
[ 2357.844433] [<c0de1100>] (default_idle_call) from [<c015f500>] (do_idle+0xe4/0x150)                                                                                                                              
[ 2357.844437]  r5:00000001 r4:c2170000                                                                                                                                                                             
[ 2357.844444] [<c015f41c>] (do_idle) from [<c015f83c>] (cpu_startup_entry+0x28/0x2c)                                                                                                                               
[ 2357.844452]  r9:410fc073 r8:8000406a r7:c15bb650 r6:10c0387d r5:00000001 r4:00000091                                                                                                                             
[ 2357.844460] [<c015f814>] (cpu_startup_entry) from [<c0110bcc>] (secondary_start_kernel+0x170/0x194)                                                                                                              
[ 2357.844468] [<c0110a5c>] (secondary_start_kernel) from [<801017cc>] (0x801017cc)                                                                                                                                 
[ 2357.844472]  r5:00000051 r4:8216806a                                                                                                                                                                             
[ 2358.971480] Rebooting in 3 seconds..

UPD:

openvpn test success depends on --tun-mtu in works great with --tun-mtu 8192, and fails with --tun-mtu 16384

66

Some openvpn perfomance tests:

setup:

server:

AF_ALG:

openvpn --engine afalg --dev tun --proto udp --port 11000 --secret ./static.key --tun-mtu 8192 --ifconfig 192.168.222.11 192.168.222.10 --cipher aes-256-cbc

SW:

openvpn --dev tun --proto udp --port 11000 --secret ./static.key --tun-mtu 8192 --ifconfig 192.168.222.11 192.168.222.10 --cipher aes-256-cbc

client:

openvpn --dev tun --proto udp --port 11000 --secret ./static.key --tun-mtu 8192 --ifconfig 192.168.222.10 192.168.222.11 --cipher aes-256-cbc --remote <server_ip>

results:

upload/download

network w/o openvpn - using wifi:

-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.0.1.150, port 59184
[  5] local 10.0.1.2 port 5201 connected to 10.0.1.150 port 59186
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  10.8 MBytes  90.7 Mbits/sec                  
[  5]   1.00-2.00   sec  13.3 MBytes   112 Mbits/sec                  
[  5]   2.00-3.00   sec  11.7 MBytes  98.4 Mbits/sec                  
[  5]   3.00-4.00   sec  10.8 MBytes  90.2 Mbits/sec                  
[  5]   4.00-5.00   sec  9.66 MBytes  81.1 Mbits/sec                  
[  5]   5.00-6.00   sec  10.6 MBytes  88.8 Mbits/sec                  
[  5]   6.00-7.00   sec  10.4 MBytes  87.6 Mbits/sec                  
[  5]   7.00-8.00   sec  9.96 MBytes  83.5 Mbits/sec                  
[  5]   8.00-9.00   sec  9.92 MBytes  83.2 Mbits/sec                  
[  5]   9.00-10.00  sec  10.2 MBytes  85.8 Mbits/sec                  
[  5]  10.00-10.02  sec   214 KBytes  78.6 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.02  sec   108 MBytes  90.1 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.0.1.150, port 59196
[  5] local 10.0.1.2 port 5201 connected to 10.0.1.150 port 59198
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  31.2 MBytes   262 Mbits/sec   22    498 KBytes       
[  5]   1.00-2.00   sec  18.8 MBytes   157 Mbits/sec    0    564 KBytes       
[  5]   2.00-3.00   sec  23.8 MBytes   199 Mbits/sec    0    609 KBytes       
[  5]   3.00-4.00   sec  22.5 MBytes   189 Mbits/sec    0    641 KBytes       
[  5]   4.00-5.00   sec  20.0 MBytes   168 Mbits/sec    0    659 KBytes       
[  5]   5.00-6.00   sec  21.2 MBytes   178 Mbits/sec    0    667 KBytes       
[  5]   6.00-7.00   sec  21.2 MBytes   178 Mbits/sec    4    499 KBytes       
[  5]   7.00-8.00   sec  18.8 MBytes   157 Mbits/sec    0    533 KBytes       
[  5]   8.00-9.00   sec  21.2 MBytes   178 Mbits/sec    0    553 KBytes       
[  5]   9.00-10.00  sec  21.2 MBytes   178 Mbits/sec    0    567 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   220 MBytes   184 Mbits/sec   26             sender

AF_ALG:

-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.222.10, port 52138
[  5] local 192.168.222.11 port 5201 connected to 192.168.222.10 port 52140
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  2.74 MBytes  23.0 Mbits/sec                  
[  5]   1.00-2.00   sec  2.99 MBytes  25.1 Mbits/sec                  
[  5]   2.00-3.00   sec  3.22 MBytes  27.0 Mbits/sec                  
[  5]   3.00-4.00   sec  3.09 MBytes  25.9 Mbits/sec                  
[  5]   4.00-5.00   sec  2.95 MBytes  24.7 Mbits/sec                  
[  5]   5.00-6.00   sec  3.27 MBytes  27.4 Mbits/sec                  
[  5]   6.00-7.00   sec  3.20 MBytes  26.8 Mbits/sec                  
[  5]   7.00-8.00   sec  3.22 MBytes  27.0 Mbits/sec                  
[  5]   8.00-9.00   sec  3.24 MBytes  27.1 Mbits/sec                  
[  5]   9.00-10.00  sec  3.13 MBytes  26.3 Mbits/sec                  
[  5]  10.00-10.00  sec  5.23 KBytes  14.1 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  31.0 MBytes  26.0 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.222.10, port 52152
[  5] local 192.168.222.11 port 5201 connected to 192.168.222.10 port 52154
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  13.8 MBytes   115 Mbits/sec    2   92.8 KBytes       
[  5]   1.00-2.00   sec  2.50 MBytes  21.0 Mbits/sec    4   91.5 KBytes       
[  5]   2.00-3.00   sec  2.50 MBytes  21.0 Mbits/sec    4   79.7 KBytes       
[  5]   3.00-4.00   sec  2.50 MBytes  21.0 Mbits/sec    2   71.9 KBytes       
[  5]   4.00-5.00   sec  2.50 MBytes  21.0 Mbits/sec    0   92.8 KBytes       
[  5]   5.00-6.00   sec  2.50 MBytes  21.0 Mbits/sec    2   82.3 KBytes       
[  5]   6.00-7.00   sec  2.50 MBytes  21.0 Mbits/sec    4   73.2 KBytes       
[  5]   7.00-8.00   sec  2.50 MBytes  21.0 Mbits/sec    0   94.1 KBytes       
[  5]   8.00-9.00   sec  2.50 MBytes  21.0 Mbits/sec    2   84.9 KBytes       
[  5]   9.00-10.00  sec  2.50 MBytes  21.0 Mbits/sec    2   74.5 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  36.2 MBytes  30.4 Mbits/sec   22             sender

SW:

-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.222.10, port 52204
[  5] local 192.168.222.11 port 5201 connected to 192.168.222.10 port 52206
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  4.52 MBytes  37.9 Mbits/sec                  
[  5]   1.00-2.00   sec  5.03 MBytes  42.2 Mbits/sec                  
[  5]   2.00-3.00   sec  6.01 MBytes  50.4 Mbits/sec                  
[  5]   3.00-4.00   sec  5.22 MBytes  43.8 Mbits/sec                  
[  5]   4.00-5.00   sec  5.83 MBytes  48.9 Mbits/sec                  
[  5]   5.00-6.00   sec  4.56 MBytes  38.2 Mbits/sec                  
[  5]   6.00-7.00   sec  4.36 MBytes  36.6 Mbits/sec                  
[  5]   7.00-8.00   sec  3.47 MBytes  29.1 Mbits/sec                  
[  5]   8.00-9.00   sec  4.66 MBytes  39.1 Mbits/sec                  
[  5]   9.00-10.00  sec  4.81 MBytes  40.3 Mbits/sec                  
[  5]  10.00-10.00  sec  24.8 KBytes  46.5 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  48.5 MBytes  40.7 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.222.10, port 52212
[  5] local 192.168.222.11 port 5201 connected to 192.168.222.10 port 52214
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  15.0 MBytes   126 Mbits/sec   11   95.4 KBytes       
[  5]   1.00-2.00   sec  3.75 MBytes  31.5 Mbits/sec    1   98.0 KBytes       
[  5]   2.00-3.00   sec  5.00 MBytes  41.9 Mbits/sec    6   67.9 KBytes       
[  5]   3.00-4.00   sec  3.75 MBytes  31.5 Mbits/sec    8   84.9 KBytes       
[  5]   4.00-5.00   sec  3.75 MBytes  31.5 Mbits/sec   15   88.9 KBytes       
[  5]   5.00-6.00   sec  3.75 MBytes  31.5 Mbits/sec    4   92.8 KBytes       
[  5]   6.00-7.00   sec  5.00 MBytes  41.9 Mbits/sec    1   95.4 KBytes       
[  5]   7.00-8.00   sec  3.75 MBytes  31.5 Mbits/sec   20   52.3 KBytes       
[  5]   8.00-9.00   sec  3.75 MBytes  31.5 Mbits/sec   19   47.0 KBytes       
[  5]   9.00-10.00  sec  5.00 MBytes  42.0 Mbits/sec    6   67.9 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  52.5 MBytes  44.0 Mbits/sec   91             sender

Looks strange, bun in this case HW accelerated openvpn seems to be slower than software-only. Probably it depends on small buffer size (i might be wrong - i think it depends on mtu size)

I couldn’t to force openssh to use afalg engine after recompiling with --with-sandbox=no so for now i’m unable to benchmark ssh.

P.S. openvpn is clearly uses aes_mtk - it’s visible from/proc/interrupts

UPD2:

after openvpn tuning i finnaly got some nicer results :slight_smile:

added --fragment 0 --mssfix 0 to both client and server configs

HW:

Accepted connection from 192.168.222.10, port 53218
[  5] local 192.168.222.11 port 5201 connected to 192.168.222.10 port 53220
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  9.33 MBytes  78.3 Mbits/sec                  
[  5]   1.00-2.00   sec  10.6 MBytes  88.7 Mbits/sec                  
[  5]   2.00-3.00   sec  9.66 MBytes  81.1 Mbits/sec                  
[  5]   3.00-4.00   sec  11.9 MBytes  99.5 Mbits/sec                  
[  5]   4.00-5.00   sec  11.0 MBytes  92.6 Mbits/sec                  
[  5]   5.00-6.00   sec  11.4 MBytes  95.7 Mbits/sec                  
[  5]   6.00-7.00   sec  10.8 MBytes  90.2 Mbits/sec                  
[  5]   7.00-8.00   sec  11.1 MBytes  93.0 Mbits/sec                  
[  5]   8.00-9.00   sec  10.9 MBytes  91.1 Mbits/sec                  
[  5]   9.00-10.00  sec  10.8 MBytes  90.7 Mbits/sec                  
[  5]  10.00-10.06  sec   707 KBytes  97.7 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.06  sec   108 MBytes  90.1 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.222.10, port 53222
[  5] local 192.168.222.11 port 5201 connected to 192.168.222.10 port 53224
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  20.0 MBytes   168 Mbits/sec    1    294 KBytes       
[  5]   1.00-2.00   sec  8.75 MBytes  73.4 Mbits/sec    0    413 KBytes       
[  5]   2.00-3.00   sec  10.0 MBytes  83.9 Mbits/sec    0    501 KBytes       
[  5]   3.00-4.00   sec  8.75 MBytes  73.4 Mbits/sec    0    572 KBytes       
[  5]   4.00-5.00   sec  8.75 MBytes  73.4 Mbits/sec    4    350 KBytes       
[  5]   5.00-6.00   sec  10.0 MBytes  83.9 Mbits/sec    0    453 KBytes       
[  5]   6.00-7.00   sec  8.75 MBytes  73.4 Mbits/sec    0    533 KBytes       
[  5]   7.00-8.00   sec  10.0 MBytes  83.9 Mbits/sec   10    318 KBytes       
[  5]   8.00-9.00   sec  8.75 MBytes  73.4 Mbits/sec    0    421 KBytes       
[  5]   9.00-10.00  sec  8.75 MBytes  73.4 Mbits/sec    0    509 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   102 MBytes  85.9 Mbits/sec   15             sender
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

SW:

Accepted connection from 192.168.222.10, port 53132
[  5] local 192.168.222.11 port 5201 connected to 192.168.222.10 port 53134
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  8.75 MBytes  73.4 Mbits/sec                  
[  5]   1.00-2.00   sec  8.86 MBytes  74.3 Mbits/sec                  
[  5]   2.00-3.00   sec  8.30 MBytes  69.6 Mbits/sec                  
[  5]   3.00-4.00   sec  9.44 MBytes  79.2 Mbits/sec                  
[  5]   4.00-5.00   sec  8.43 MBytes  70.7 Mbits/sec                  
[  5]   5.00-6.00   sec  8.36 MBytes  70.1 Mbits/sec                  
[  5]   6.00-7.00   sec  8.42 MBytes  70.7 Mbits/sec                  
[  5]   7.00-8.00   sec  8.02 MBytes  67.3 Mbits/sec                  
[  5]   8.00-9.00   sec  9.18 MBytes  77.0 Mbits/sec                  
[  5]   9.00-10.00  sec  8.38 MBytes  70.3 Mbits/sec                  
[  5]  10.00-10.03  sec   358 KBytes  85.7 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.03  sec  86.5 MBytes  72.3 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.222.10, port 53140
[  5] local 192.168.222.11 port 5201 connected to 192.168.222.10 port 53142
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  18.8 MBytes   157 Mbits/sec    4    278 KBytes       
[  5]   1.00-2.00   sec  10.0 MBytes  83.9 Mbits/sec    0    397 KBytes       
[  5]   2.00-3.00   sec  8.75 MBytes  73.4 Mbits/sec    4    270 KBytes       
[  5]   3.00-4.00   sec  10.0 MBytes  83.9 Mbits/sec    0    390 KBytes       
[  5]   4.00-5.00   sec  10.0 MBytes  83.9 Mbits/sec    2    350 KBytes       
[  5]   5.00-6.00   sec  8.75 MBytes  73.4 Mbits/sec    0    445 KBytes       
[  5]   6.00-7.00   sec  8.75 MBytes  73.4 Mbits/sec    9    175 KBytes       
[  5]   7.00-8.00   sec  10.0 MBytes  83.9 Mbits/sec    0    334 KBytes       
[  5]   8.00-9.00   sec  8.75 MBytes  73.4 Mbits/sec   33    231 KBytes       
[  5]   9.00-10.00  sec  8.75 MBytes  73.4 Mbits/sec    4    246 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec   102 MBytes  85.9 Mbits/sec   56             sender
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

UPD2:

When using --fragment 0 --mssfix 0, the speed is highly depends on --tun-mtu

–tun-mtu 1500:

upload
[  5]   0.00-10.00  sec  35.9 MBytes  30.1 Mbits/sec                  receiver
download
[  5]   0.00-10.00  sec  36.2 MBytes  30.4 Mbits/sec   43             sender

–tun-mtu 16000:

upload
[  5]   0.00-10.11  sec   121 MBytes   101 Mbits/sec                  receiver
download
[  5]   0.00-10.01  sec   144 MBytes   120 Mbits/sec   30             sender

–tun-mtu 16000 - no acceleration:

upload
[  5]   0.00-10.05  sec   112 MBytes  93.4 Mbits/sec                  receiver
download
[  5]   0.00-10.01  sec   126 MBytes   106 Mbits/sec   11             sender

full server/client command:

openvpn --dev tun --proto udp --port 11000 --secret ./static.key --tun-mtu 16000 --ifconfig 192.168.222.11 192.168.222.10 --cipher aes-256-cbc --fragment 0 --mssfix 0

openvpn --dev tun --proto udp --port 11000 --secret ./static.key --tun-mtu 16000 --ifconfig 192.168.222.10 192.168.222.11 --cipher aes-256-cbc --fragment 0 --mssfix 0 --remote 10.0.1.2

UPD3:

testing with --tun-mtu 16384 crashes server with same dmesg as openvpn test above, so --tun-mtu 16000 seems to be working configuration near it’s hw limit

setting ‘–tun-mtu 32768’ in sw mode didn’t gave any speed advantages compairing to --tun-mtu 16000 in sw mode

67

same test performed using LAN

SW mtu 16000:

vim3 ~ # iperf3 -c 192.168.222.11
Connecting to host 192.168.222.11, port 5201
[  5] local 192.168.222.10 port 47436 connected to 192.168.222.11 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  14.1 MBytes   118 Mbits/sec    0    934 KBytes       
[  5]   1.00-2.00   sec  11.2 MBytes  94.4 Mbits/sec    2    857 KBytes       
[  5]   2.00-3.00   sec  12.5 MBytes   105 Mbits/sec    0    950 KBytes       
[  5]   3.00-4.00   sec  13.8 MBytes   115 Mbits/sec    2    763 KBytes       
[  5]   4.00-5.00   sec  12.5 MBytes   105 Mbits/sec    1    623 KBytes       
[  5]   5.00-6.00   sec  12.5 MBytes   105 Mbits/sec    0    763 KBytes       
[  5]   6.00-7.00   sec  12.5 MBytes   105 Mbits/sec    0    888 KBytes       
[  5]   7.00-8.00   sec  12.5 MBytes   105 Mbits/sec    1    763 KBytes       
[  5]   8.00-9.00   sec  11.2 MBytes  94.4 Mbits/sec    0    888 KBytes       
[  5]   9.00-10.00  sec  12.5 MBytes   105 Mbits/sec    3    716 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   125 MBytes   105 Mbits/sec    9             sender
[  5]   0.00-10.02  sec   122 MBytes   102 Mbits/sec                  receiver

iperf Done.
vim3 ~ # iperf3 -c 192.168.222.11 -R
Connecting to host 192.168.222.11, port 5201
Reverse mode, remote host 192.168.222.11 is sending
[  5] local 192.168.222.10 port 47440 connected to 192.168.222.11 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  11.1 MBytes  93.1 Mbits/sec                  
[  5]   1.00-2.00   sec  11.5 MBytes  96.7 Mbits/sec                  
[  5]   2.00-3.00   sec  11.6 MBytes  97.3 Mbits/sec                  
[  5]   3.00-4.00   sec  11.6 MBytes  97.0 Mbits/sec                  
[  5]   4.00-5.00   sec  11.6 MBytes  97.6 Mbits/sec                  
[  5]   5.00-6.00   sec  11.7 MBytes  97.7 Mbits/sec                  
[  5]   6.00-7.00   sec  11.6 MBytes  97.6 Mbits/sec                  
[  5]   7.00-8.00   sec  11.7 MBytes  98.1 Mbits/sec                  
[  5]   8.00-9.00   sec  11.7 MBytes  98.1 Mbits/sec                  
[  5]   9.00-10.00  sec  11.6 MBytes  97.6 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec   126 MBytes   106 Mbits/sec    6             sender
[  5]   0.00-10.00  sec   116 MBytes  97.1 Mbits/sec                  receiver MBytes  98.1 Mbits/sec                  
[  5]   8.00-9.00   sec  11.7 MBytes  98.1 Mbits/sec                  
[  5]   9.00-10.00  sec  11.6 MBytes  97.6 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec   126 MBytes   106 Mbits/sec    6             sender
[  5]   0.00-10.00  sec   116 MBytes  97.1 Mbits/sec                  receiver

iperf Done.

HW, mtu 16000:

vim3 ~ # iperf3 -c 192.168.222.11
Connecting to host 192.168.222.11, port 5201
[  5] local 192.168.222.10 port 47444 connected to 192.168.222.11 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  17.7 MBytes   148 Mbits/sec    0   1.11 MBytes       
[  5]   1.00-2.00   sec  16.2 MBytes   136 Mbits/sec    0   1.96 MBytes       
[  5]   2.00-3.00   sec  17.5 MBytes   147 Mbits/sec    4   2.48 MBytes       
[  5]   3.00-4.00   sec  17.5 MBytes   147 Mbits/sec    6   1.51 MBytes       
[  5]   4.00-5.00   sec  17.5 MBytes   147 Mbits/sec    0   1.63 MBytes       
[  5]   5.00-6.00   sec  15.0 MBytes   126 Mbits/sec   50   1.25 MBytes       
[  5]   6.00-7.00   sec  16.2 MBytes   136 Mbits/sec    0   1.35 MBytes       
[  5]   7.00-8.00   sec  16.2 MBytes   136 Mbits/sec    0   1.38 MBytes       
[  5]   8.00-9.00   sec  15.0 MBytes   126 Mbits/sec    0   1.48 MBytes       
[  5]   9.00-10.00  sec  15.0 MBytes   126 Mbits/sec    0   1.55 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   164 MBytes   138 Mbits/sec   60             sender
[  5]   0.00-10.00  sec   162 MBytes   136 Mbits/sec                  receiver

iperf Done.
vim3 ~ # iperf3 -c 192.168.222.11 -R
Connecting to host 192.168.222.11, port 5201
Reverse mode, remote host 192.168.222.11 is sending
[  5] local 192.168.222.10 port 47448 connected to 192.168.222.11 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  13.1 MBytes   110 Mbits/sec                  
[  5]   1.00-2.00   sec  13.6 MBytes   114 Mbits/sec                  
[  5]   2.00-3.00   sec  13.6 MBytes   114 Mbits/sec                  
[  5]   3.00-4.00   sec  13.6 MBytes   114 Mbits/sec                  
[  5]   4.00-5.00   sec  13.2 MBytes   110 Mbits/sec                  
[  5]   5.00-6.00   sec  14.0 MBytes   118 Mbits/sec                  
[  5]   6.00-7.00   sec  13.4 MBytes   113 Mbits/sec                  
[  5]   7.00-8.00   sec  13.6 MBytes   114 Mbits/sec                  
[  5]   8.00-9.00   sec  13.7 MBytes   115 Mbits/sec                  
[  5]   9.00-10.00  sec  13.7 MBytes   115 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec   146 MBytes   123 Mbits/sec    8             sender
[  5]   0.00-10.00  sec   136 MBytes   114 Mbits/sec                  receiver

iperf Done.

In this case HW acceleration definitely gives ~30% boost for upload and ~15% for download.

Raw lan speed 870 to 920 mbits

68

So for mtu 1500 (or smaller like used in internet,my 1492,or manually reduced against connection issues) there is no benefit,right?

69

Not exactly: encrypted traffic still goes through LAN/WAN/WiFi with mtu 1492/1500, 16000 is only for virtual tun/tap interface. The only hint i see - the firewall needs to apply mss-fix in case when traffic should be routed to some real interfaces (any home router always do it when uses pppoe/pptp ,and mtu is 1492 for example). It still needs to be tested on some real cases ;).

P.S. Also in case 16000 in not a hw limit, but only some driver bug and it can be increased in some future - it may give some more improvements.

70

devcrypto testing:

I’ve tried to use cryptodev driver. After compiling kernel module and rebuilding openssl with devcrypto engine, i got:

r2-gentoo ~ # openssl engine -t -c
(devcrypto) /dev/crypto engine
 [AES-128-CBC, AES-192-CBC, AES-256-CBC, AES-128-CTR, AES-192-CTR, AES-256-CTR, AES-128-ECB, AES-192-ECB, AES-256-ECB, MD5, SHA1, SHA224, SHA256, SHA384, SHA512]
     [ available ]
(dynamic) Dynamic engine loading support
     [ unavailable ]
(afalg) AFALG engine support
 [AES-128-CBC, AES-192-CBC, AES-256-CBC]
     [ available ]

quick benchmark showed that if faster than AF_ALG in singlethread and multithread mode:

r2-gentoo ~ # openssl speed -elapsed -evp aes-256-cbc -engine devcrypto
engine "devcrypto" set.
...
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-cbc        524.01k     2119.77k     8263.25k    30785.19k    83902.46k    86731.43k


r2-gentoo ~ # openssl speed -elapsed -evp aes-256-cbc -engine devcrypto -multi 2
Forked child 0
Forked child 1
...
evp               1037.79k     4108.90k    16176.68k    60337.15k   140787.71k   140170.58k


r2-gentoo ~ # openssl speed -elapsed -evp aes-256-cbc --engine afalg
engine "afalg" set.
...
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-cbc        486.05k     1948.84k     7123.29k    18607.79k    64984.41k    79779.16k

r2-gentoo ~ # openssl speed -elapsed -evp aes-256-cbc --engine afalg -multi 2
Forked child 0
Forked child 1
engine "afalg" set.
evp                609.21k     2646.38k    10122.07k    34820.10k   119229.10k   142447.96k

but i failed to run openvpn, and self test showed:

openvpn --test-crypto --secret ./static.key --cipher AES-256-CBC --tun-mtu 100 --engine devcrypto  
Tue Mar  2 21:25:04 2021 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
Tue Mar  2 21:25:04 2021 OpenVPN 2.4.9 armv7a-hardfloat-linux-gnueabi [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Apr 29 2020
Tue Mar  2 21:25:04 2021 library versions: OpenSSL 1.1.1j  16 Feb 2021, LZO 2.10
Tue Mar  2 21:25:04 2021 OpenVPN 2.4.9 armv7a-hardfloat-linux-gnueabi [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Apr 29 2020
Tue Mar  2 21:25:04 2021 Initializing OpenSSL support for engine 'devcrypto'
Tue Mar  2 21:25:04 2021 Entering OpenVPN crypto self-test mode.
Tue Mar  2 21:25:04 2021 TESTING ENCRYPT/DECRYPT of packet length=1
Tue Mar  2 21:25:04 2021 TESTING ENCRYPT/DECRYPT of packet length=2
Tue Mar  2 21:25:04 2021 TESTING ENCRYPT/DECRYPT of packet length=3
Tue Mar  2 21:25:04 2021 TESTING ENCRYPT/DECRYPT of packet length=4
Tue Mar  2 21:25:04 2021 TESTING ENCRYPT/DECRYPT of packet length=5
Tue Mar  2 21:25:04 2021 TESTING ENCRYPT/DECRYPT of packet length=6
Tue Mar  2 21:25:04 2021 TESTING ENCRYPT/DECRYPT of packet length=7
Tue Mar  2 21:25:04 2021 TESTING ENCRYPT/DECRYPT of packet length=8
Tue Mar  2 21:25:04 2021 Authenticate/Decrypt packet error: cipher final failed
Tue Mar  2 21:25:04 2021 SELF TEST FAILED, src.len=8 buf.len=0
Tue Mar  2 21:25:04 2021 Exiting due to fatal error

Taking into account good results with AF_ALG, i think it’s a devcrypto issue(?)

BTW maximum working mtu with AF_ALG is 16375

71

Have you used latest cryptodev?

72

Yep, i used github version