You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@nuttx.apache.org by "zhhyu7 (via GitHub)" <gi...@apache.org> on 2026/01/19 09:37:45 UTC
[PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
zhhyu7 opened a new pull request, #18011:
URL: https://github.com/apache/nuttx/pull/18011
## Summary
The main content of this submission is to limit both the TX/RX buffers of TCP/UDP to throttled IOBs, avoiding impacts on the sending and receiving of control-type messages.
## Impact
- **TCP**: `tcp_wrbuffer.c`, `tcp_recvfrom.c`, `tcp_send_buffered.c`
- **UDP**: `udp_wrbuffer.c`
- **ICMP/ICMPv6**: `icmp_input.c`, `icmpv6_input.c`
- **Packet**: `pkt_input.c`, `pkt_sendmsg_buffered.c`
## Testing
sim:matter with very small amount of IOB configuration
```
CONFIG_MM_IOB=y
CONFIG_IOB_NBUFFERS=40
CONFIG_IOB_BUFSIZE=196
CONFIG_IOB_ALIGNMENT=4
CONFIG_IOB_SECTION=""
CONFIG_IOB_NCHAINS=40
CONFIG_IOB_THROTTLE=10
```
NuttX test log:
```
NuttShell (NSH) NuttX-12.12.0
MOTD: username=admin password=Administrator
nsh> iperf -c 10.0.1.1
IP: 10.0.1.2
mode=tcp-client sip=10.0.1.2:5001,dip=10.0.1.1:5001, interval=3, time=30
Interval Transfer Bandwidth
0.00- 3.01 sec 167133184 Bytes 444.21 Mbits/sec
3.01- 6.02 sec 160907264 Bytes 427.66 Mbits/sec
6.02- 9.03 sec 167264256 Bytes 444.56 Mbits/sec
9.03- 12.04 sec 159039488 Bytes 422.70 Mbits/sec
12.04- 15.05 sec 165265408 Bytes 439.24 Mbits/sec
15.05- 18.06 sec 162758656 Bytes 432.58 Mbits/sec
18.06- 21.07 sec 166150144 Bytes 441.60 Mbits/sec
21.07- 24.08 sec 168116224 Bytes 446.82 Mbits/sec
24.08- 27.09 sec 163971072 Bytes 435.80 Mbits/sec
27.09- 30.10 sec 168886272 Bytes 448.87 Mbits/sec
0.00- 30.10 sec 1649491968 Bytes 438.40 Mbits/sec
iperf exit
nsh>
nsh> iperf -s
IP: 10.0.1.2
mode=tcp-server sip=10.0.1.2:5001,dip=0.0.0.0:5001, interval=3, time=0
accept: 10.0.1.1:37436
Interval Transfer Bandwidth
0.00- 3.01 sec 212808200 Bytes 565.60 Mbits/sec
3.01- 6.02 sec 206625040 Bytes 549.17 Mbits/sec
6.02- 9.03 sec 204516800 Bytes 543.57 Mbits/sec
closed by the peer: 10.0.1.1:37436
iperf exit
nsh>
nsh> iperf -c 10.0.1.1 -u
IP: 10.0.1.2
mode=udp-client sip=10.0.1.2:5001,dip=10.0.1.1:5001, interval=3, time=30
Interval Transfer Bandwidth
0.00- 3.01 sec 248220416 Bytes 659.72 Mbits/sec
3.01- 6.02 sec 252431808 Bytes 670.92 Mbits/sec
6.02- 9.03 sec 251402880 Bytes 668.18 Mbits/sec
9.03- 12.04 sec 248893120 Bytes 661.51 Mbits/sec
12.04- 15.05 sec 248382336 Bytes 660.15 Mbits/sec
15.05- 18.06 sec 252202176 Bytes 670.30 Mbits/sec
18.06- 21.07 sec 247859776 Bytes 658.76 Mbits/sec
21.07- 24.08 sec 252090304 Bytes 670.01 Mbits/sec
24.08- 27.09 sec 252523072 Bytes 671.16 Mbits/sec
27.09- 30.10 sec 251651648 Bytes 668.84 Mbits/sec
0.00- 30.10 sec 2505657536 Bytes 665.96 Mbits/sec
iperf exit
nsh>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "zhhyu7 (via GitHub)" <gi...@apache.org>.
zhhyu7 commented on code in PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#discussion_r2704226394
##########
net/udp/udp_callback.c:
##########
@@ -80,7 +80,8 @@ static uint16_t udp_datahandler(FAR struct net_driver_s *dev,
conn_lock(&conn->sconn);
#if CONFIG_NET_RECV_BUFSIZE > 0
- if (conn->readahead && conn->readahead->io_pktlen > conn->rcvbufs)
+ if (conn->readahead && conn->readahead->io_pktlen > conn->rcvbufs &&
+ iob_navail(true) == 0)
Review Comment:
> why add iob_navail check
when throttling IOB exists , save it to the receive queue, otherwise, directly release this packet, to avoid affecting other services, especially the packet reception process, due to UDP service not reading readahead data for a long time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "fdcavalcanti (via GitHub)" <gi...@apache.org>.
fdcavalcanti commented on PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#issuecomment-3784321145
@zhhyu7 I tried on the `esp32-devkitc` and `esp32c6-devkit` because it has easier debug. Also, I enabled DEBUG_ASSERTIONS. On IPERF start, it immediately crashed:
- ESP32:
```
nsh> iperf -c 192.168.0.125 -i 3 -t 10 &
iperf [12:100]
nsh> IP: 192.168.0.85
mode=tcp-client sip=192.168.0.85:5001,dip=192.168.0.125:5001, interval=3, time=10
Interval Transfer Bandwidth
[CPU1] dump_assert_info: Current Version: NuttX 10.4.0 21c19b7824 Jan 22 2026 09:56:18 xtensa
[CPU1] dump_assert_info: Assertion failed ((wrb)->wb_sent) <= ((wrb)->wb_iob->io_pktlen): at file: tcp/tcp_send_buffered.c:1173 task(CPU1): iperf_traffic process: iperf 0x4011a890
...
```
- ESP32-C6:
```
nsh> iperf -c 192.168.0.125 -i 3 -t 10 &
iperf [9:100]
nsh> IP: 192.168.0.3
mode=tcp-client sip=192.168.0.3:5001,dip=192.168.0.125:5001, interval=3, time=10
Interval Transfer Bandwidth
dump_assert_info: Current Version: NuttX 10.4.0 21c19b7824 Jan 22 2026 10:04:02 risc-v
dump_assert_info: Assertion failed ((wrb)->wb_sent) <= ((wrb)->wb_iob->io_pktlen): at file: tcp/tcp_send_buffered.c:1173 task: lpwork process: Kernel 0x4200702e
...
```
Seems they crash at the same place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "zhhyu7 (via GitHub)" <gi...@apache.org>.
zhhyu7 commented on code in PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#discussion_r2704226394
##########
net/udp/udp_callback.c:
##########
@@ -80,7 +80,8 @@ static uint16_t udp_datahandler(FAR struct net_driver_s *dev,
conn_lock(&conn->sconn);
#if CONFIG_NET_RECV_BUFSIZE > 0
- if (conn->readahead && conn->readahead->io_pktlen > conn->rcvbufs)
+ if (conn->readahead && conn->readahead->io_pktlen > conn->rcvbufs &&
+ iob_navail(true) == 0)
Review Comment:
> why add iob_navail check
when throttling IOB exists , save it to the receive queue, otherwise, directly release this packet, to avoid affecting other services, especially the packet reception process, due to UDP service not reading pre-read data for a long time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "fdcavalcanti (via GitHub)" <gi...@apache.org>.
fdcavalcanti commented on PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#issuecomment-3783983545
Yes it seems we have issues. Ping is fine but IPERF speed drops and stays there.
I read the IOB status before and after the test. Seems the IOBs are exhausted.
```
nsh> cat /proc/iobinfo
ntotal nfree nwait nthrottle
160 160 0 136
nsh> renew wlan0
nsh> ifconfig
wlan0 Link encap:Ethernet HWaddr 60:55:f9:f7:87:6c at RUNNING mtu 1500
inet addr:192.168.0.8 DRaddr:192.168.0.1 Mask:255.255.255.0
nsh> ping 192.168.0.1
PING 192.168.0.1 56 bytes of data
56 bytes from 192.168.0.1: icmp_seq=0 time=10.0 ms
56 bytes from 192.168.0.1: icmp_seq=1 time=0.0 ms
56 bytes from 192.168.0.1: icmp_seq=2 time=0.0 ms
56 bytes from 192.168.0.1: icmp_seq=3 time=0.0 ms
56 bytes from 192.168.0.1: icmp_seq=4 time=0.0 ms
56 bytes from 192.168.0.1: icmp_seq=5 time=0.0 ms
56 bytes from 192.168.0.1: icmp_seq=6 time=0.0 ms
56 bytes from 192.168.0.1: icmp_seq=7 time=0.0 ms
56 bytes from 192.168.0.1: icmp_seq=8 time=0.0 ms
56 bytes from 192.168.0.1: icmp_seq=9 time=0.0 ms
10 packets transmitted, 10 received, 0% packet loss, time 10100 ms
rtt min/avg/max/mdev = 0.000/1.000/10.000/3.000 ms
nsh>
nsh> iperf -c 192.168.0.125 -i 3 -t 20 &
iperf [10:100]
nsh> IP: 192.168.0.8
mode=tcp-client sip=192.168.0.8:5001,dip=192.168.0.125:5001, interval=3, time=20
Interval Transfer Bandwidth
0.00- 3.01 sec 81920 Bytes 0.22 Mbits/sec
3.01- 6.02 sec 0 Bytes 0.00 Mbits/sec
6.02- 9.03 sec 0 Bytes 0.00 Mbits/sec
9.03- 12.04 sec 0 Bytes 0.00 Mbits/sec
12.04- 15.05 sec 0 Bytes 0.00 Mbits/sec
15.05- 18.06 sec 0 Bytes 0.00 Mbits/sec
18.06- 21.07 sec 0 Bytes 0.00 Mbits/sec
0.00- 21.07 sec 81920 Bytes 0.03 Mbits/sec
nsh> cat /proc/iobinfo
ntotal nfree nwait nthrottle
160 24 0 0
nsh>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "fdcavalcanti (via GitHub)" <gi...@apache.org>.
fdcavalcanti commented on PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#issuecomment-3784176858
The wireless interfacing is done on `arch/risc-v/src/common/espressif/esp_wlan_netdev.c`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 commented on code in PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#discussion_r2704000625
##########
net/udp/udp_callback.c:
##########
@@ -80,7 +80,8 @@ static uint16_t udp_datahandler(FAR struct net_driver_s *dev,
conn_lock(&conn->sconn);
#if CONFIG_NET_RECV_BUFSIZE > 0
- if (conn->readahead && conn->readahead->io_pktlen > conn->rcvbufs)
+ if (conn->readahead && conn->readahead->io_pktlen > conn->rcvbufs &&
+ iob_navail(true) == 0)
Review Comment:
why add iob_navail check
##########
net/icmp/icmp_input.c:
##########
@@ -153,7 +153,7 @@ static uint16_t icmp_datahandler(FAR struct net_driver_s *dev,
/* Copy the ICMP message into the I/O buffer chain (without waiting) */
ret = iob_clone_partial(dev->d_iob, dev->d_iob->io_pktlen,
- 0, iob, 0, true, false);
+ 0, iob, 0, false, false);
Review Comment:
why change to false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "PetervdPerk-NXP (via GitHub)" <gi...@apache.org>.
PetervdPerk-NXP commented on PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#issuecomment-3784108810
> @PetervdPerk-NXP It seems to be related to the network card not entering the RUNNING state, the protocol stack will check the RUNNING flag when selecting the sending network card.
This for pointing this out indeed it's that RUNNING flag doesn't get active to make ethernet work. Adding `netdev_carrier` calls in the imxrt ifup/ifdown functions makes DHCP work and I see data. I've tracked this change in #18110
However, with this PR my network is even locking up faster and ifconfig locks up as well. Unfortunely I can't debug this anymore because of the regresssion defined in #18109
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "zhhyu7 (via GitHub)" <gi...@apache.org>.
zhhyu7 commented on code in PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#discussion_r2704226394
##########
net/udp/udp_callback.c:
##########
@@ -80,7 +80,8 @@ static uint16_t udp_datahandler(FAR struct net_driver_s *dev,
conn_lock(&conn->sconn);
#if CONFIG_NET_RECV_BUFSIZE > 0
- if (conn->readahead && conn->readahead->io_pktlen > conn->rcvbufs)
+ if (conn->readahead && conn->readahead->io_pktlen > conn->rcvbufs &&
+ iob_navail(true) == 0)
Review Comment:
> why add iob_navail check
when throttle IOB exists , save it to the receive queue, otherwise, directly release this packet, to avoid affecting other services, especially the packet reception process, due to UDP service not reading readahead data for a long time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "zhhyu7 (via GitHub)" <gi...@apache.org>.
zhhyu7 commented on PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#issuecomment-3782818891
> @zhhyu7 I just tried this commit and the one before but compared to NuttX 12.12 the network regressed competely where it's not working anymore on the NXP i.MXRT.
>
> ```
> eth0 Link encap:Ethernet HWaddr 0a:0e:82:92:b3:7f at UP mtu 1504
> inet addr:192.168.2.133 DRaddr:192.168.2.1 Mask:255.255.255.0
>
> can0 Link encap:UNSPEC at DOWN mtu 72
> inet addr:0.0.0.0 DRaddr:0.0.0.0 Mask:0.0.0.0
>
> can1 Link encap:UNSPEC at DOWN mtu 72
> inet addr:0.0.0.0 DRaddr:0.0.0.0 Mask:0.0.0.0
>
> can2 Link encap:UNSPEC at DOWN mtu 72
> inet addr:0.0.0.0 DRaddr:0.0.0.0 Mask:0.0.0.0
>
> IPv4 TCP UDP ICMP CAN
> Received 0002 0000 0002 0000 0000
> Dropped 0000 0000 0000 0000 0000
> IPv4 VHL: 0000 Frg: 0000
> Checksum 0000 0000 0000 ---- ----
> TCP ACK: 0000 SYN: 0000
> RST: 0000 0000
> Type 0000 ---- ---- 0000 ----
> Sent 0000 0000 0000 0000 0000
> Rexmit ---- 0000 ---- ---- ----
> nsh> arp_send: ERROR: Unreachable: ff02a8c0
> psock_udp_sendto: ERROR: Not reachable
> arp_send: ERROR: Unreachable: ff02a8c0
> psock_udp_sendto: ERROR: Not reachable
> arp_send: ERROR: Unreachable: ff02a8c0
> psock_udp_sendto: ERROR: Not reachable
> arp_send: ERROR: Unreachable: ff02a8c0
> ```
eth0 Link encap:Ethernet HWaddr 0a:0e:82:92:b3:7f at **UP** mtu 1504
@PetervdPerk-NXP It seems to be related to the network card not entering the RUNNING state, the protocol stack will check the RUNNING flag when selecting the sending network card.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "zhhyu7 (via GitHub)" <gi...@apache.org>.
zhhyu7 commented on PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#issuecomment-3784381088
> @zhhyu7 I tried on the `esp32-devkitc` and `esp32c6-devkit` because it has easier debug. Also, I enabled DEBUG_ASSERTIONS. On IPERF start, it immediately crashed:
>
> * ESP32:
>
> ```
> nsh> iperf -c 192.168.0.125 -i 3 -t 10 &
> iperf [12:100]
> nsh> IP: 192.168.0.85
>
> mode=tcp-client sip=192.168.0.85:5001,dip=192.168.0.125:5001, interval=3, time=10
>
> Interval Transfer Bandwidth
>
> [CPU1] dump_assert_info: Current Version: NuttX 10.4.0 21c19b7824 Jan 22 2026 09:56:18 xtensa
> [CPU1] dump_assert_info: Assertion failed ((wrb)->wb_sent) <= ((wrb)->wb_iob->io_pktlen): at file: tcp/tcp_send_buffered.c:1173 task(CPU1): iperf_traffic process: iperf 0x4011a890
> ...
> ```
>
> * ESP32-C6:
>
> ```
> nsh> iperf -c 192.168.0.125 -i 3 -t 10 &
> iperf [9:100]
> nsh> IP: 192.168.0.3
>
> mode=tcp-client sip=192.168.0.3:5001,dip=192.168.0.125:5001, interval=3, time=10
>
> Interval Transfer Bandwidth
>
> dump_assert_info: Current Version: NuttX 10.4.0 21c19b7824 Jan 22 2026 10:04:02 risc-v
> dump_assert_info: Assertion failed ((wrb)->wb_sent) <= ((wrb)->wb_iob->io_pktlen): at file: tcp/tcp_send_buffered.c:1173 task: lpwork process: Kernel 0x4200702e
> ...
> ```
>
> Seems they crash at the same place.
@fdcavalcanti Get, Will iperf return to normal if this patch is reverted? Then I'll revert this patch first, and after optimizing the implementation, I'll find several more devices locally for detailed verification before proceeding further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "zhhyu7 (via GitHub)" <gi...@apache.org>.
zhhyu7 commented on PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#issuecomment-3784452426
> Yes, if I revert the commit IPERF is able to complete.
https://github.com/apache/nuttx/pull/18113 this is the revert PR. I will subsequently conduct verification based on the above devices first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "fdcavalcanti (via GitHub)" <gi...@apache.org>.
fdcavalcanti commented on PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#issuecomment-3784408332
Yes, if I revert the commit IPERF is able to complete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "PetervdPerk-NXP (via GitHub)" <gi...@apache.org>.
PetervdPerk-NXP commented on PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#issuecomment-3805166969
@fdcavalcanti which branch you're testing, unfortunately I'm facing other regressions on NuttX main that completely makes enet unusable for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "fdcavalcanti (via GitHub)" <gi...@apache.org>.
fdcavalcanti commented on PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#issuecomment-3804941630
Hi @zhhyu7 this is excellent input, thanks!
I can confirm IPERF is working blazing fast with your recommendations, but there are some issues:
I tried `CONFIG_NET_SEND_BUFSIZE` of 12000 (your recommendation) and also 20000 which is the value I got from the equation you showed.
Here's my base IOB config:
```
CONFIG_IOB_BUFSIZE=400
CONFIG_IOB_NBUFFERS=100
CONFIG_IOB_NCHAINS=32
CONFIG_IOB_THROTTLE=40
```
- `CONFIG_NET_SEND_BUFSIZE=20000`
It did not transmit properly and seems we have lost some buffers after IPERF finished.
```
nsh> iperf -c 192.168.0.125 -i 3 -t 20 &
iperf [12:100]
nsh> IP: 192.168.0.85
mode=tcp-client sip=192.168.0.85:5001,dip=192.168.0.125:5001, interval=3, time=20
Interval Transfer Bandwidth
0.00- 3.01 sec 131072 Bytes 0.35 Mbits/sec
3.01- 6.02 sec 0 Bytes 0.00 Mbits/sec
6.02- 9.03 sec 0 Bytes 0.00 Mbits/sec
9.03- 12.04 sec 0 Bytes 0.00 Mbits/sec
12.04- 15.05 sec 0 Bytes 0.00 Mbits/sec
15.05- 18.06 sec 0 Bytes 0.00 Mbits/sec
18.06- 21.07 sec 0 Bytes 0.00 Mbits/sec
0.00- 21.07 sec 131072 Bytes 0.05 Mbits/sec
nsh> cat /proc/iobinfo
ntotal nfree nwait nthrottle
100 41 0 1
```
- `CONFIG_NET_SEND_BUFSIZE=12000`
It starts very slow then picks up speed. If I call IPERF again (without rebooting the board) it runs fine. You can see below that after the first run, some IOBs are "lost" but after the second run, there are no changes. I imagine those lost buffers are related to the slow start of the first IPERF run.
I then rebooted the board and tried again. It had the same results regarding throughput but did not lose any IOB. Here's the log of the first run:
```
nsh> iperf -c 192.168.0.125 -i 3 -t 20 &
iperf [12:100]
nsh> IP: 192.168.0.85
mode=tcp-client sip=192.168.0.85:5001,dip=192.168.0.125:5001, interval=3, time=20
Interval Transfer Bandwidth
0.00- 3.01 sec 114688 Bytes 0.30 Mbits/sec
3.01- 6.02 sec 0 Bytes 0.00 Mbits/sec
6.02- 9.03 sec 4063232 Bytes 10.80 Mbits/sec
9.03- 12.04 sec 4276224 Bytes 11.37 Mbits/sec
12.04- 15.05 sec 4259840 Bytes 11.32 Mbits/sec
15.05- 18.06 sec 4210688 Bytes 11.19 Mbits/sec
18.06- 21.07 sec 4259840 Bytes 11.32 Mbits/sec
0.00- 21.07 sec 21184512 Bytes 8.04 Mbits/sec
iperf exit
nsh> cat /proc/iobinfo
ntotal nfree nwait nthrottle
100 98 0 58
nsh>
nsh>
nsh> iperf -c 192.168.0.125 -i 3 -t 20 &
iperf [15:100]
nsh> IP: 192.168.0.85
mode=tcp-client sip=192.168.0.85:5001,dip=192.168.0.125:5001, interval=3, time=20
Interval Transfer Bandwidth
0.00- 3.01 sec 4128768 Bytes 10.97 Mbits/sec
3.01- 6.02 sec 4210688 Bytes 11.19 Mbits/sec
6.02- 9.03 sec 4292608 Bytes 11.41 Mbits/sec
9.03- 12.04 sec 4407296 Bytes 11.71 Mbits/sec
12.04- 15.05 sec 4194304 Bytes 11.15 Mbits/sec
15.05- 18.06 sec 4243456 Bytes 11.28 Mbits/sec
18.06- 21.07 sec 4341760 Bytes 11.54 Mbits/sec
0.00- 21.07 sec 29818880 Bytes 11.32 Mbits/sec
iperf exit
cat /proc/iobinfo
ntotal nfree nwait nthrottle
100 98 0 58
nsh>
```
After this, I kept it running for 5 minutes and it had very stable throughput (better then we ever had). Seems we are very near closure here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "zhhyu7 (via GitHub)" <gi...@apache.org>.
zhhyu7 commented on PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#issuecomment-3784228902
> @zhhyu7 The wireless interfacing is done on `arch/risc-v/src/common/espressif/esp_wlan_netdev.c`.
>
> * esp_wlan_netdev.c: defines the struct wireless_ops_s and netdev_ops_s.
>
> * Defines the esp_wlan_priv_s which is the interface for the Wi-Fi peripheral (STA and SoftAP each has one esp_wlan_priv_s.)
> * Defines esp_wlan_initialize which is called from board level to initialize the wireless interface.
> * esp_wifi_utils.c: generic functions that are moved to make the code cleaner and reusable. Includes Wi-Fi scan routines and callback events.
> * esp_wifi_event_handler.c: deals with events generated from the Wi-Fi driver.
@fdcavalcanti If netdev_upperhalf is used, it should be fine. Can we see what the backtrace of each thread are when it gets stuck?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "fdcavalcanti (via GitHub)" <gi...@apache.org>.
fdcavalcanti commented on PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#issuecomment-3779824600
Well in this case it is better to revert and investigate further. I can assist with more tests using IPERF over Wi-Fi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "zhhyu7 (via GitHub)" <gi...@apache.org>.
zhhyu7 commented on PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#issuecomment-3803965898
> 0x4201e974: psock_tcp_send at tcp_send_buffered.c:1534 (discriminator 1)
@fdcavalcanti The total amount of IOB configured for the board is too small. When iperf writes 16k of data at once, if the write queue only uses unthrottled IOBs, there is a relatively high probability that the data in the write queue will exhaust the unthrottled IOBs but still be requesting a write buffer. At this point, the thread will block here, unable to trigger the subsequent tcp_send_txnotify, causing the data not to be sent out, and thus unable to receive an ack to release these IOBs, so it will never recover. You can try configuring CONFIG_NET_SEND_BUFSIZE=12000 (CONFIG_IOB_BUFSIZE * (CONFIG_IOB_NBUFFERS - CONFIG_IOB_THROTTLE) - 4k), it should no longer get stuck.
Please help to include this patch and the configuration of CONFIG_NET_SEND_BUFSIZE for verification when you have time. Thank you very much.
I am synchronously optimizing the scenario where the TCP buffer writes too much data in a single operation (exceeding the total number of valid IOBs).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "zhhyu7 (via GitHub)" <gi...@apache.org>.
zhhyu7 commented on PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#issuecomment-3782850274
> Well in this case it is better to revert and investigate further. I can assist with more tests using IPERF over Wi-Fi.
@fdcavalcanti Thanks, If the test reveals that the robustness of the network protocol stack is worse than before this PR was merged, we can first revert this commit, and I will also conduct more detailed analysis and testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "PetervdPerk-NXP (via GitHub)" <gi...@apache.org>.
PetervdPerk-NXP commented on PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#issuecomment-3779027853
@zhhyu7 I just tried this commit and the one before but compared to NuttX 12.12 the network regressed competely where it's not working anymore on the NXP i.MXRT.
```
eth0 Link encap:Ethernet HWaddr 0a:0e:82:92:b3:7f at UP mtu 1504
inet addr:192.168.2.133 DRaddr:192.168.2.1 Mask:255.255.255.0
can0 Link encap:UNSPEC at DOWN mtu 72
inet addr:0.0.0.0 DRaddr:0.0.0.0 Mask:0.0.0.0
can1 Link encap:UNSPEC at DOWN mtu 72
inet addr:0.0.0.0 DRaddr:0.0.0.0 Mask:0.0.0.0
can2 Link encap:UNSPEC at DOWN mtu 72
inet addr:0.0.0.0 DRaddr:0.0.0.0 Mask:0.0.0.0
IPv4 TCP UDP ICMP CAN
Received 0002 0000 0002 0000 0000
Dropped 0000 0000 0000 0000 0000
IPv4 VHL: 0000 Frg: 0000
Checksum 0000 0000 0000 ---- ----
TCP ACK: 0000 SYN: 0000
RST: 0000 0000
Type 0000 ---- ---- 0000 ----
Sent 0000 0000 0000 0000 0000
Rexmit ---- 0000 ---- ---- ----
nsh> arp_send: ERROR: Unreachable: ff02a8c0
psock_udp_sendto: ERROR: Not reachable
arp_send: ERROR: Unreachable: ff02a8c0
psock_udp_sendto: ERROR: Not reachable
arp_send: ERROR: Unreachable: ff02a8c0
psock_udp_sendto: ERROR: Not reachable
arp_send: ERROR: Unreachable: ff02a8c0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "zhhyu7 (via GitHub)" <gi...@apache.org>.
zhhyu7 commented on code in PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#discussion_r2704210403
##########
net/icmp/icmp_input.c:
##########
@@ -153,7 +153,7 @@ static uint16_t icmp_datahandler(FAR struct net_driver_s *dev,
/* Copy the ICMP message into the I/O buffer chain (without waiting) */
ret = iob_clone_partial(dev->d_iob, dev->d_iob->io_pktlen,
- 0, iob, 0, true, false);
+ 0, iob, 0, false, false);
Review Comment:
> why change to false
try not to restrict the packet of control types, the main reason is that the number of such messages is generally relatively small, and second, the business logic is often relatively simple, so the messages will be consumed immediately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "xiaoxiang781216 (via GitHub)" <gi...@apache.org>.
xiaoxiang781216 merged PR #18011:
URL: https://github.com/apache/nuttx/pull/18011
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "zhhyu7 (via GitHub)" <gi...@apache.org>.
zhhyu7 commented on PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#issuecomment-3784129926
> esp32c3-devkit:wifi
@fdcavalcanti May I ask which network card driver is used, arch/risc-v/src/esp32c3-legacy/esp32c3_wlan.c?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "fdcavalcanti (via GitHub)" <gi...@apache.org>.
fdcavalcanti commented on PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#issuecomment-3784345819
Here's the backtrace for C6:
```
dump_tasks: PID GROUP PRI POLICY TYPE NPX STATE EVENT SIGMASK STACKBASE STACKSIZE COMMAND
dump_tasks: ---- --- --- -------- ------- --- ------- ---------- ---------------- 0x40819380 2048 irq
dump_task: 0 0 0 FIFO Kthread - Ready 0000000000000000 0x40826e40 2016 Idle_Task
dump_task: 1 0 100 RR Kthread - Running 0000000000000000 0x40827ab8 1960 lpwork 0x40816320 0x40816368
dump_task: 2 2 100 RR Task - Waiting Semaphore 0000000000000000 0x40828668 1976 nsh_main
dump_task: 3 0 223 RR Kthread - Waiting Semaphore 0000000000000000 0x40828fc0 1984 hr_timer
dump_task: 4 0 253 RR Kthread - Waiting MQ empty 0000000000000000 0x4082aa58 6600 wifi
dump_task: 5 0 100 RR Kthread - Waiting Semaphore 0000000000000000 0x408331e8 1960 netdev-wlan0 0x40833060 0
dump_task: 9 9 100 RR Task - Waiting Semaphore 0000000000000000 0x408345b8 1928 iperf -c 192.168.0.125 -i 3 -t 10
dump_task: 10 9 100 RR pthread - Waiting Semaphore 0000000000000000 0x4083a0b0 4064 iperf_traffic 0x42040bba 0x40834c20
dump_task: 11 9 101 RR pthread - Waiting Signal 0000000000000000 0x4083b0b8 4056 iperf_report 0x42041216 0x40834c20
$ ./tools/btdecode.sh esp32c6 bt
Backtrace for task 9:
0x42010620: sys_call0 at syscall.h:161
(inlined by) up_switch_context at riscv_switchcontext.c:83
0x40834b90: ?? ??:0
0x4200caa6: nxsem_wait at sem_wait.c:202
0x42008a4c: nxsem_wait_uninterruptible at sem_wait.c:314 (discriminator 1)
0x4202fd58: pthread_join at pthread_join.c:149
0x4204157c: iperf_start at iperf.c:965
0x42037ccc: iperf_main at iperf_main.c:325
0x4200c916: nxtask_startup at task_startup.c:72 (discriminator 1)
0x420098a6: nxtask_start at task_start.c:72
Backtrace for task 5:
0x42010620: sys_call0 at syscall.h:161
(inlined by) up_switch_context at riscv_switchcontext.c:83
0x40833930: ?? ??:0
0x4200caa6: nxsem_wait at sem_wait.c:202
0x420a896e: netdev_upper_loop at netdev_upperhalf.c:816 (discriminator 1)
0x4200988c: nxtask_start at task_start.c:112
Backtrace for task 4:
0x42010620: sys_call0 at syscall.h:161
(inlined by) up_switch_context at riscv_switchcontext.c:83
0x4082c360: ?? ??:0
0x42049b68: file_mq_timedreceive_internal at mq_receive.c:195
0x42049c20: file_mq_receive at mq_receive.c:488
0x4204ee66: queue_recv_wrapper at esp_wifi_adapter.c:1153
0x4080c28e: ppTask at ??:?
Backtrace for task 3:
0x42010620: sys_call0 at syscall.h:161
(inlined by) up_switch_context at riscv_switchcontext.c:83
0x40829710: ?? ??:0
0x4200caa6: nxsem_wait at sem_wait.c:202
0x42008a4c: nxsem_wait_uninterruptible at sem_wait.c:314 (discriminator 1)
0x408029cc: esp_hr_timer_thread at esp_hr_timer.c:133 (discriminator 2)
0x4200988c: nxtask_start at task_start.c:112
Backtrace for task 2:
0x42010620: sys_call0 at syscall.h:161
(inlined by) up_switch_context at riscv_switchcontext.c:83
0x40828ba0: ?? ??:0
0x4200caa6: nxsem_wait at sem_wait.c:202
0x4200b9cc: uart_readv at serial.c:1261
0x42003fdc: file_readv_compat at fs_read.c:116
(inlined by) file_readv at fs_read.c:221
0x42004086: nx_readv at fs_read.c:304
0x420040d0: readv at fs_read.c:369
0x42004108: read at fs_read.c:404
0x42010bfc: readline_getc at readline_fd.c:79
0x42015b74: readline_common at readline_common.c:519
0x42010cc2: readline_fd at readline_fd.c:238
0x4201094e: nsh_session at nsh_session.c:233
0x4201080c: nsh_consolemain at nsh_consolemain.c:81
0x420107ac: nsh_main at nsh_main.c:82
0x4200c916: nxtask_startup at task_startup.c:72 (discriminator 1)
0x420098a6: nxtask_start at task_start.c:72
Backtrace for task 1:
0x420306ce: sched_backtrace at sched_backtrace.c:104
0x4200e654: sched_dumpstack at sched_dumpstack.c:71
0x4200653a: dump_backtrace at assert.c:459
0x42006dde: nxsched_foreach at sched_foreach.c:69 (discriminator 2)
0x42006962: dump_fatal_info at assert.c:778
(inlined by) _assert at assert.c:915
0x4200c404: __assert at lib_assert.c:39
0x4201e1cc: psock_send_eventhandler at tcp_send_buffered.c:456
0x42023268: devif_conn_event at devif_callback.c:438
0x42021ec4: tcp_callback at tcp_callback.c:310
0x420a98e6: tcp_poll at tcp_devpoll.c:132
0x420a90e8: devif_poll_tcp_connections at devif_poll.c:679
(inlined by) devif_poll_connections at devif_poll.c:951
0x420a917a: devif_iob_poll at devif_poll.c:1059
(inlined by) devif_poll at devif_poll.c:1142
0x420a83a0: netdev_upper_tx at netdev_upperhalf.c:362
(inlined by) netdev_upper_txavail_work at netdev_upperhalf.c:391
0x420a83fc: netdev_upper_txavail at netdev_upperhalf.c:887
0x4201f9a0: netdev_txnotify_dev at netdev_txnotify.c:124
0x42020ea4: tcp_timer_expiry at tcp_timer.c:159
0x42007cb2: up_irq_save at irq.h:772
(inlined by) spin_lock_irqsave_nopreempt at spinlock.h:522
(inlined by) work_thread at kwork_thread.c:256
0x4200988c: nxtask_start at task_start.c:112
Backtrace for task 0:
0x4200fc30: up_idle at esp_idle.c:226
0x40827590: ?? ??:0
Backtrace for task 10:
0x42010620: sys_call0 at syscall.h:161
(inlined by) up_switch_context at riscv_switchcontext.c:83
0x4083add0: ?? ??:0
0x4200caa6: nxsem_wait at sem_wait.c:202
0x42008a4c: nxsem_wait_uninterruptible at sem_wait.c:314 (discriminator 1)
0x42036de0: iob_allocwait at iob_alloc.c:207
(inlined by) iob_timedalloc at iob_alloc.c:273
0x42023682: net_iobtimedalloc at net_lock.c:362
(inlined by) net_iobtimedalloc at net.h:524
0x420227d6: tcp_wrbuffer_timedalloc at tcp_wrbuffer.c:103 (discriminator 1)
0x4201e974: psock_tcp_send at tcp_send_buffered.c:1534 (discriminator 1)
0x4201c73e: inet_send at inet_sockif.c:1734 (discriminator 1)
(inlined by) inet_send at inet_sockif.c:1676 (discriminator 1)
0x4201c8ce: inet_sendmsg at inet_sockif.c:1883
0x4201bd0a: psock_sendmsg at sendmsg.c:98
0x4201bac0: psock_sendto at sendto.c:135
0x4201bb12: sendto at sendto.c:247
0x4201ba74: send at send.c:164
0x42040ef4: iperf_tcp_client at iperf.c:806
0x42040aa6: iperf_run_client at iperf.c:477
0x42040bee: iperf_task_traffic at iperf.c:873
0x420324da: pthread_startup at pthread_create.c:61 (discriminator 1)
0x4204a07e: pthread_start at pthread_create.c:153
Backtrace for task 11:
0x42010620: sys_call0 at syscall.h:161
(inlined by) up_switch_context at riscv_switchcontext.c:83
0x4083bfb0: ?? ??:0
0x4204a690: clock_nanosleep at sig_nanosleep.c:196
0x42034f86: sleep at lib_sleep.c:118
0x420412d8: iperf_report_task at iperf.c:317
0x420324da: pthread_startup at pthread_create.c:61 (discriminator 1)
0x4204a07e: pthread_start at pthread_create.c:153
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] net: limit TCP and UDP send/recv buffer usage with throttled IOB [nuttx]
Posted by "fdcavalcanti (via GitHub)" <gi...@apache.org>.
fdcavalcanti commented on PR #18011:
URL: https://github.com/apache/nuttx/pull/18011#issuecomment-3805182052
> @fdcavalcanti which branch you're testing, unfortunately I'm facing other regressions on NuttX main that completely makes enet unusable for me.
All right, thanks. I'm testing this branch (rebased) with the latest comments from @zhhyu7.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org