You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nuttx.apache.org by Reto G��hwiler <gr...@gmail.com> on 2020/02/19 10:32:53 UTC
STM32H7 Ethernet problems
Hello Everyone,
I am working for Hexagon Mining in Baar (some of our people were on the last nuttx summit). Our project involves nuttx and runs on a stm32h743zi. Our nuttx version was recently (mid January) updated.
We recently recognised that the ethernet of our platform dies after some time. After some investigation it looks like that the DMA of the ethernet ends up in a deadlock since no further descriptors are available and also not freed. It all starts with a packet which is not processed right away but in the next interrupt handling cycle after the next ETH_DMACSR_RI appeared. It then either collapses or will have a massive slow down. In case of a collapse everything will stop (incl. ping) since the chain is stopped around the DMA.
So far it only happened during a longer data transfer. The device works fine over days just sitting there and responding to the broadcasts on the network. To verify the problems we also used the TCPblaster example with a minimum code base from our side. There, it only happens if multiple threads are used. One thread alone is handled well as it seems. The TCPblaster worked fine on the stm32f7.
I am wondering if anyone had this or a similar problem before. If you need more information please let me know.
Thanks in advance for your thoughts and helps. Best Regards,
Reto Gähwiler
Re: STM32H7 Ethernet problems
Posted by Adam Feuer <ad...@starcat.io>.
Xiang,
Thanks for the concrete example of how to break IOB deadlock. If that is
what's causing my problem, I will try it out.
cheers
adam
On Wed, Feb 19, 2020 at 6:11 PM Xiang Xiao <xi...@gmail.com>
wrote:
> Here is a demo fix for one of IOB deadlock recently :
>
> commit 2d0baa779d997f39b8121f5965f8125184e80d71
> Author: chao.an <an...@xiaomi.com>
> Date: Thu Jan 16 14:20:09 2020 -0300
>
> net/udp: break the network lock to avoid deadlock
>
> Author: chao.an <an...@xiaomi.com>
>
> net/udp: break the network lock to avoid deadlock
>
> network deadlock when udp sendto() storm is coming
>
> net/close: force wait tx drain to complete
>
> atomic send() and close() will causes data to be discarded
> directly
>
> Signed-off-by: chao.an <an...@xiaomi.com>
>
> diff --git a/net/udp/udp_psock_sendto_buffered.c
> b/net/udp/udp_psock_sendto_buffered.c
> index 0dcf892759..fbc44de0f1 100644
> --- a/net/udp/udp_psock_sendto_buffered.c
> +++ b/net/udp/udp_psock_sendto_buffered.c
> @@ -75,6 +75,7 @@
> #include "neighbor/neighbor.h"
> #include "udp/udp.h"
> #include "devif/devif.h"
> +#include "utils/utils.h"
>
>
> /****************************************************************************
> * Pre-processor Definitions
> @@ -713,8 +714,21 @@ ssize_t psock_udp_sendto(FAR struct socket
> *psock, FAR const void *buf,
> }
> else
> {
> + unsigned int count;
> + int blresult;
> +
> + /* iob_copyin might wait for buffers to be freed, but if
> + * network is locked this might never happen, since network
> + * driver is also locked, therefore we need to break the lock
> + */
> +
> + blresult = net_breaklock(&count);
> ret = iob_copyin(wrb->wb_iob, (FAR uint8_t *)buf, len, 0, false,
> IOBUSER_NET_SOCK_UDP);
> + if (blresult >= 0)
> + {
> + net_restorelock(count);
> + }
> }
>
> if (ret < 0)
>
> The problem is that iob_copybin may allocate more IOB buffer
> internally and will wait if IOB isn't available.
> the old code call it without breaking netlock, then the other path
> can't get the netlock again after the pending IOB finish the sending
> and return to the pool.
> Hoping this case can give some tips.
>
> Thanks
> Xiang
>
> On Thu, Feb 20, 2020 at 6:50 AM Gregory Nutt <sp...@gmail.com> wrote:
> >
> >
> > > This sounds a lot like the problem I'm having with the SAMA5D36 Gigabit
> > > ethernet... I'm running into some kind of deadlock on long transfers
> that
> > > send packets very quickly. NuttX seems to run out of IOBs and then
> can't
> > > send or respond to network packets.
> > >
> > > I tried increasing the low priority worker threads to 2 (and also 3)
> but
> > > neither of them solved the problem.
> > >
> > > I'll look at the net_lock() to see if there's a way to release it.
> > >
> > > If you find a solution, I would love to know it! If I find one, I'll
> post
> > > it here.
> >
> > The first step in debugging a deadlock is to find what is stuck waiting
> > for what resource.
> >
> > Then find the logic that provides the resource that is being waited on.
> >
> > Then figure out why that logic is not running. Most likely, it would be
> > waiting the low priority work queue.
> >
> > I have had to solve lots of problems like this. It is not really so
> > difficult once you unstand the above things.
> >
> >
> >
>
--
Adam Feuer <ad...@starcat.io>
Re: STM32H7 Ethernet problems
Posted by Reto G��hwiler <gr...@gmail.com>.
@Xiao: Thanks for your example, I'll see how I can apply it to our problem.
On 2020/02/20 02:10:49, Xiang Xiao <xi...@gmail.com> wrote:
> Here is a demo fix for one of IOB deadlock recently :
>
> commit 2d0baa779d997f39b8121f5965f8125184e80d71
> Author: chao.an <an...@xiaomi.com>
> Date: Thu Jan 16 14:20:09 2020 -0300
>
> net/udp: break the network lock to avoid deadlock
>
> Author: chao.an <an...@xiaomi.com>
>
> net/udp: break the network lock to avoid deadlock
>
> network deadlock when udp sendto() storm is coming
>
> net/close: force wait tx drain to complete
>
> atomic send() and close() will causes data to be discarded directly
>
> Signed-off-by: chao.an <an...@xiaomi.com>
>
> diff --git a/net/udp/udp_psock_sendto_buffered.c
> b/net/udp/udp_psock_sendto_buffered.c
> index 0dcf892759..fbc44de0f1 100644
> --- a/net/udp/udp_psock_sendto_buffered.c
> +++ b/net/udp/udp_psock_sendto_buffered.c
> @@ -75,6 +75,7 @@
> #include "neighbor/neighbor.h"
> #include "udp/udp.h"
> #include "devif/devif.h"
> +#include "utils/utils.h"
>
> /****************************************************************************
> * Pre-processor Definitions
> @@ -713,8 +714,21 @@ ssize_t psock_udp_sendto(FAR struct socket
> *psock, FAR const void *buf,
> }
> else
> {
> + unsigned int count;
> + int blresult;
> +
> + /* iob_copyin might wait for buffers to be freed, but if
> + * network is locked this might never happen, since network
> + * driver is also locked, therefore we need to break the lock
> + */
> +
> + blresult = net_breaklock(&count);
> ret = iob_copyin(wrb->wb_iob, (FAR uint8_t *)buf, len, 0, false,
> IOBUSER_NET_SOCK_UDP);
> + if (blresult >= 0)
> + {
> + net_restorelock(count);
> + }
> }
>
> if (ret < 0)
>
> The problem is that iob_copybin may allocate more IOB buffer
> internally and will wait if IOB isn't available.
> the old code call it without breaking netlock, then the other path
> can't get the netlock again after the pending IOB finish the sending
> and return to the pool.
> Hoping this case can give some tips.
>
> Thanks
> Xiang
>
> On Thu, Feb 20, 2020 at 6:50 AM Gregory Nutt <sp...@gmail.com> wrote:
> >
> >
> > > This sounds a lot like the problem I'm having with the SAMA5D36 Gigabit
> > > ethernet... I'm running into some kind of deadlock on long transfers that
> > > send packets very quickly. NuttX seems to run out of IOBs and then can't
> > > send or respond to network packets.
> > >
> > > I tried increasing the low priority worker threads to 2 (and also 3) but
> > > neither of them solved the problem.
> > >
> > > I'll look at the net_lock() to see if there's a way to release it.
> > >
> > > If you find a solution, I would love to know it! If I find one, I'll post
> > > it here.
> >
> > The first step in debugging a deadlock is to find what is stuck waiting
> > for what resource.
> >
> > Then find the logic that provides the resource that is being waited on.
> >
> > Then figure out why that logic is not running. Most likely, it would be
> > waiting the low priority work queue.
> >
> > I have had to solve lots of problems like this. It is not really so
> > difficult once you unstand the above things.
> >
> >
> >
>
Re: STM32H7 Ethernet problems
Posted by Xiang Xiao <xi...@gmail.com>.
Here is a demo fix for one of IOB deadlock recently :
commit 2d0baa779d997f39b8121f5965f8125184e80d71
Author: chao.an <an...@xiaomi.com>
Date: Thu Jan 16 14:20:09 2020 -0300
net/udp: break the network lock to avoid deadlock
Author: chao.an <an...@xiaomi.com>
net/udp: break the network lock to avoid deadlock
network deadlock when udp sendto() storm is coming
net/close: force wait tx drain to complete
atomic send() and close() will causes data to be discarded directly
Signed-off-by: chao.an <an...@xiaomi.com>
diff --git a/net/udp/udp_psock_sendto_buffered.c
b/net/udp/udp_psock_sendto_buffered.c
index 0dcf892759..fbc44de0f1 100644
--- a/net/udp/udp_psock_sendto_buffered.c
+++ b/net/udp/udp_psock_sendto_buffered.c
@@ -75,6 +75,7 @@
#include "neighbor/neighbor.h"
#include "udp/udp.h"
#include "devif/devif.h"
+#include "utils/utils.h"
/****************************************************************************
* Pre-processor Definitions
@@ -713,8 +714,21 @@ ssize_t psock_udp_sendto(FAR struct socket
*psock, FAR const void *buf,
}
else
{
+ unsigned int count;
+ int blresult;
+
+ /* iob_copyin might wait for buffers to be freed, but if
+ * network is locked this might never happen, since network
+ * driver is also locked, therefore we need to break the lock
+ */
+
+ blresult = net_breaklock(&count);
ret = iob_copyin(wrb->wb_iob, (FAR uint8_t *)buf, len, 0, false,
IOBUSER_NET_SOCK_UDP);
+ if (blresult >= 0)
+ {
+ net_restorelock(count);
+ }
}
if (ret < 0)
The problem is that iob_copybin may allocate more IOB buffer
internally and will wait if IOB isn't available.
the old code call it without breaking netlock, then the other path
can't get the netlock again after the pending IOB finish the sending
and return to the pool.
Hoping this case can give some tips.
Thanks
Xiang
On Thu, Feb 20, 2020 at 6:50 AM Gregory Nutt <sp...@gmail.com> wrote:
>
>
> > This sounds a lot like the problem I'm having with the SAMA5D36 Gigabit
> > ethernet... I'm running into some kind of deadlock on long transfers that
> > send packets very quickly. NuttX seems to run out of IOBs and then can't
> > send or respond to network packets.
> >
> > I tried increasing the low priority worker threads to 2 (and also 3) but
> > neither of them solved the problem.
> >
> > I'll look at the net_lock() to see if there's a way to release it.
> >
> > If you find a solution, I would love to know it! If I find one, I'll post
> > it here.
>
> The first step in debugging a deadlock is to find what is stuck waiting
> for what resource.
>
> Then find the logic that provides the resource that is being waited on.
>
> Then figure out why that logic is not running. Most likely, it would be
> waiting the low priority work queue.
>
> I have had to solve lots of problems like this. It is not really so
> difficult once you unstand the above things.
>
>
>
Re: STM32H7 Ethernet problems
Posted by Adam Feuer <ad...@starcat.io>.
Thanks Greg. I will try to track down the deadlock this way.
-adam
On Wed, Feb 19, 2020 at 2:50 PM Gregory Nutt <sp...@gmail.com> wrote:
>
> > This sounds a lot like the problem I'm having with the SAMA5D36 Gigabit
> > ethernet... I'm running into some kind of deadlock on long transfers that
> > send packets very quickly. NuttX seems to run out of IOBs and then can't
> > send or respond to network packets.
> >
> > I tried increasing the low priority worker threads to 2 (and also 3) but
> > neither of them solved the problem.
> >
> > I'll look at the net_lock() to see if there's a way to release it.
> >
> > If you find a solution, I would love to know it! If I find one, I'll post
> > it here.
>
> The first step in debugging a deadlock is to find what is stuck waiting
> for what resource.
>
> Then find the logic that provides the resource that is being waited on.
>
> Then figure out why that logic is not running. Most likely, it would be
> waiting the low priority work queue.
>
> I have had to solve lots of problems like this. It is not really so
> difficult once you unstand the above things.
>
>
>
>
--
Adam Feuer <ad...@starcat.io>
Re: STM32H7 Ethernet problems
Posted by Reto G��hwiler <gr...@gmail.com>.
Just a short summary: Your suggested fix in stm32_freeframe was resolving the issue as it seems for the moment. So far it seems to handle our traffic and several ping-floods well.
The suggested iobs solution do not address our problem since we do not use the buffered functions.
On 2020/02/22 07:19:16, Jukka Laitinen <ju...@iki.fi> wrote:
> Hi!
>
> On 21.2.2020 16.08, Reto Gähwiler wrote:
> > Hi Jukka,
> >
> > First of all, thanks for your time and hints. Just applied your suggested change to the stm32_freebuffer.
>
> Don't mention, I am sort of responsible for these bugs originally ;)
>
> > About the second suggested fix in stm32_freesegment, how do you mean that? The stm32_get_next_rxdesc just increments the descriptor, this should be independent from advancing the rx tail pointer, shouldn't it? But the tail pointer should be advanced be the just modified rx-descriptor!
>
> In this function the stm32_get_next_rxdesc will return the pointer to the next descriptor to be freed. In case there are no more RX descriptors to be freed, this pointer will point to the START of the FIRST free descriptor, and the tail pointer should point to the END of the LAST free descriptor, which is the same ptr. The DMA fills the descriptors up to this pointer (so also the last desccriptor is available).
>
> So currently the tail pointer lags one desc behind, and actually the last free rx desc can't be filled in by dma (since the tail pointer must point to the END of the last free RX desc, and now it points to the START of it).
>
> The code, IMHO, should be just like this:
>
> /* Get the next RX descriptor in the chain */
>
> rxdesc = stm32_get_next_rxdesc(priv, rxdesc);
>
> /* Update the tail pointer */
>
> stm32_putreg((uintptr_t)rxdesc, STM32_ETH_DMACRXDTPR);
>
> If you enable the network error debug outputs, it is worthwhile to also clear the RBU flag in the same function later after it has been checked:
> if (..RBU) {
> /* Clear the RBU flag */
>
> stm32_putreg(ETH_DMACSR_RBU, STM32_ETH_DMACSR);
>
> Since that bit will not clear by itself, and you'll be spammed by the err...
>
> Another, but independent, source of networking problems is then having enough IOBs and also enough ethernet buffers. About these you'll easily find out by enabling the corresponding CONFIG_DEBUGs.
>
> -Jukka
>
> >
> > Will let you guys know on Monday how things are with the little change in the stm32_freebuffer. Have a nice weekend,
> > Reto
> >
> > On 2020/02/21 08:07:41, Jukka Laitinen <ju...@iki.fi> wrote:
> >> Hi,
> >>
> >> Reviewing the ethernet driver, I can see couple of bugs:
> >>
> >> 1) In stm32_freeframe, it should free all the buffers, and not just the first one. So remove the "if ((txdesc->des3 & ETH_TDES3_RD_FD) != 0)"
> >>
> >> That may cause it run out of buffers.
> >>
> >> 2) In stm32_freesegment, the order of getting the next descriptor, and updating the tail pointer is wrong. It should first call the stm32_get_next_rxdesc and only after that the stm32_putreg((uintptr_t)rxdesc, STM32_ETH_DMACRXDTPR);
> >>
> >> This is not fatal, but leads to the driver not using one of the desciptors and buffers at all (there is one less buffer in use and wasted memory for that).
> >>
> >> I am very sorry, but I am unable to provide any patches currently.
> >>
> >> Regards,
> >> Jukka
> >>
> >>
> >> On 20.2.2020 9.03, Reto Gähwiler wrote:
> >>> @Gregory: Thanks for your responses and input, will see what I can do.
> >>>
> >>> On 2020/02/19 22:50:02, Gregory Nutt <sp...@gmail.com> wrote:
> >>>>
> >>>>> This sounds a lot like the problem I'm having with the SAMA5D36 Gigabit
> >>>>> ethernet... I'm running into some kind of deadlock on long transfers that
> >>>>> send packets very quickly. NuttX seems to run out of IOBs and then can't
> >>>>> send or respond to network packets.
> >>>>>
> >>>>> I tried increasing the low priority worker threads to 2 (and also 3) but
> >>>>> neither of them solved the problem.
> >>>>>
> >>>>> I'll look at the net_lock() to see if there's a way to release it.
> >>>>>
> >>>>> If you find a solution, I would love to know it! If I find one, I'll post
> >>>>> it here.
> >>>>
> >>>> The first step in debugging a deadlock is to find what is stuck waiting
> >>>> for what resource.
> >>>>
> >>>> Then find the logic that provides the resource that is being waited on.
> >>>>
> >>>> Then figure out why that logic is not running. Most likely, it would be
> >>>> waiting the low priority work queue.
> >>>>
> >>>> I have had to solve lots of problems like this. It is not really so
> >>>> difficult once you unstand the above things.
> >>>>
> >>>>
> >>>>
> >>>>
> >>
>
Re: STM32H7 Ethernet problems
Posted by Jukka Laitinen <ju...@iki.fi>.
Hi!
On 21.2.2020 16.08, Reto Gähwiler wrote:
> Hi Jukka,
>
> First of all, thanks for your time and hints. Just applied your suggested change to the stm32_freebuffer.
Don't mention, I am sort of responsible for these bugs originally ;)
> About the second suggested fix in stm32_freesegment, how do you mean that? The stm32_get_next_rxdesc just increments the descriptor, this should be independent from advancing the rx tail pointer, shouldn't it? But the tail pointer should be advanced be the just modified rx-descriptor!
In this function the stm32_get_next_rxdesc will return the pointer to the next descriptor to be freed. In case there are no more RX descriptors to be freed, this pointer will point to the START of the FIRST free descriptor, and the tail pointer should point to the END of the LAST free descriptor, which is the same ptr. The DMA fills the descriptors up to this pointer (so also the last desccriptor is available).
So currently the tail pointer lags one desc behind, and actually the last free rx desc can't be filled in by dma (since the tail pointer must point to the END of the last free RX desc, and now it points to the START of it).
The code, IMHO, should be just like this:
/* Get the next RX descriptor in the chain */
rxdesc = stm32_get_next_rxdesc(priv, rxdesc);
/* Update the tail pointer */
stm32_putreg((uintptr_t)rxdesc, STM32_ETH_DMACRXDTPR);
If you enable the network error debug outputs, it is worthwhile to also clear the RBU flag in the same function later after it has been checked:
if (..RBU) {
/* Clear the RBU flag */
stm32_putreg(ETH_DMACSR_RBU, STM32_ETH_DMACSR);
Since that bit will not clear by itself, and you'll be spammed by the err...
Another, but independent, source of networking problems is then having enough IOBs and also enough ethernet buffers. About these you'll easily find out by enabling the corresponding CONFIG_DEBUGs.
-Jukka
>
> Will let you guys know on Monday how things are with the little change in the stm32_freebuffer. Have a nice weekend,
> Reto
>
> On 2020/02/21 08:07:41, Jukka Laitinen <ju...@iki.fi> wrote:
>> Hi,
>>
>> Reviewing the ethernet driver, I can see couple of bugs:
>>
>> 1) In stm32_freeframe, it should free all the buffers, and not just the first one. So remove the "if ((txdesc->des3 & ETH_TDES3_RD_FD) != 0)"
>>
>> That may cause it run out of buffers.
>>
>> 2) In stm32_freesegment, the order of getting the next descriptor, and updating the tail pointer is wrong. It should first call the stm32_get_next_rxdesc and only after that the stm32_putreg((uintptr_t)rxdesc, STM32_ETH_DMACRXDTPR);
>>
>> This is not fatal, but leads to the driver not using one of the desciptors and buffers at all (there is one less buffer in use and wasted memory for that).
>>
>> I am very sorry, but I am unable to provide any patches currently.
>>
>> Regards,
>> Jukka
>>
>>
>> On 20.2.2020 9.03, Reto Gähwiler wrote:
>>> @Gregory: Thanks for your responses and input, will see what I can do.
>>>
>>> On 2020/02/19 22:50:02, Gregory Nutt <sp...@gmail.com> wrote:
>>>>
>>>>> This sounds a lot like the problem I'm having with the SAMA5D36 Gigabit
>>>>> ethernet... I'm running into some kind of deadlock on long transfers that
>>>>> send packets very quickly. NuttX seems to run out of IOBs and then can't
>>>>> send or respond to network packets.
>>>>>
>>>>> I tried increasing the low priority worker threads to 2 (and also 3) but
>>>>> neither of them solved the problem.
>>>>>
>>>>> I'll look at the net_lock() to see if there's a way to release it.
>>>>>
>>>>> If you find a solution, I would love to know it! If I find one, I'll post
>>>>> it here.
>>>>
>>>> The first step in debugging a deadlock is to find what is stuck waiting
>>>> for what resource.
>>>>
>>>> Then find the logic that provides the resource that is being waited on.
>>>>
>>>> Then figure out why that logic is not running. Most likely, it would be
>>>> waiting the low priority work queue.
>>>>
>>>> I have had to solve lots of problems like this. It is not really so
>>>> difficult once you unstand the above things.
>>>>
>>>>
>>>>
>>>>
>>
Re: STM32H7 Ethernet problems
Posted by Reto G��hwiler <gr...@gmail.com>.
Hi Jukka,
First of all, thanks for your time and hints. Just applied your suggested change to the stm32_freebuffer.
About the second suggested fix in stm32_freesegment, how do you mean that? The stm32_get_next_rxdesc just increments the descriptor, this should be independent from advancing the rx tail pointer, shouldn't it? But the tail pointer should be advanced be the just modified rx-descriptor!
Will let you guys know on Monday how things are with the little change in the stm32_freebuffer. Have a nice weekend,
Reto
On 2020/02/21 08:07:41, Jukka Laitinen <ju...@iki.fi> wrote:
> Hi,
>
> Reviewing the ethernet driver, I can see couple of bugs:
>
> 1) In stm32_freeframe, it should free all the buffers, and not just the first one. So remove the "if ((txdesc->des3 & ETH_TDES3_RD_FD) != 0)"
>
> That may cause it run out of buffers.
>
> 2) In stm32_freesegment, the order of getting the next descriptor, and updating the tail pointer is wrong. It should first call the stm32_get_next_rxdesc and only after that the stm32_putreg((uintptr_t)rxdesc, STM32_ETH_DMACRXDTPR);
>
> This is not fatal, but leads to the driver not using one of the desciptors and buffers at all (there is one less buffer in use and wasted memory for that).
>
> I am very sorry, but I am unable to provide any patches currently.
>
> Regards,
> Jukka
>
>
> On 20.2.2020 9.03, Reto Gähwiler wrote:
> > @Gregory: Thanks for your responses and input, will see what I can do.
> >
> > On 2020/02/19 22:50:02, Gregory Nutt <sp...@gmail.com> wrote:
> >>
> >>> This sounds a lot like the problem I'm having with the SAMA5D36 Gigabit
> >>> ethernet... I'm running into some kind of deadlock on long transfers that
> >>> send packets very quickly. NuttX seems to run out of IOBs and then can't
> >>> send or respond to network packets.
> >>>
> >>> I tried increasing the low priority worker threads to 2 (and also 3) but
> >>> neither of them solved the problem.
> >>>
> >>> I'll look at the net_lock() to see if there's a way to release it.
> >>>
> >>> If you find a solution, I would love to know it! If I find one, I'll post
> >>> it here.
> >>
> >> The first step in debugging a deadlock is to find what is stuck waiting
> >> for what resource.
> >>
> >> Then find the logic that provides the resource that is being waited on.
> >>
> >> Then figure out why that logic is not running. Most likely, it would be
> >> waiting the low priority work queue.
> >>
> >> I have had to solve lots of problems like this. It is not really so
> >> difficult once you unstand the above things.
> >>
> >>
> >>
> >>
>
Re: STM32H7 Ethernet problems
Posted by Jukka Laitinen <ju...@iki.fi>.
Hi,
Reviewing the ethernet driver, I can see couple of bugs:
1) In stm32_freeframe, it should free all the buffers, and not just the first one. So remove the "if ((txdesc->des3 & ETH_TDES3_RD_FD) != 0)"
That may cause it run out of buffers.
2) In stm32_freesegment, the order of getting the next descriptor, and updating the tail pointer is wrong. It should first call the stm32_get_next_rxdesc and only after that the stm32_putreg((uintptr_t)rxdesc, STM32_ETH_DMACRXDTPR);
This is not fatal, but leads to the driver not using one of the desciptors and buffers at all (there is one less buffer in use and wasted memory for that).
I am very sorry, but I am unable to provide any patches currently.
Regards,
Jukka
On 20.2.2020 9.03, Reto Gähwiler wrote:
> @Gregory: Thanks for your responses and input, will see what I can do.
>
> On 2020/02/19 22:50:02, Gregory Nutt <sp...@gmail.com> wrote:
>>
>>> This sounds a lot like the problem I'm having with the SAMA5D36 Gigabit
>>> ethernet... I'm running into some kind of deadlock on long transfers that
>>> send packets very quickly. NuttX seems to run out of IOBs and then can't
>>> send or respond to network packets.
>>>
>>> I tried increasing the low priority worker threads to 2 (and also 3) but
>>> neither of them solved the problem.
>>>
>>> I'll look at the net_lock() to see if there's a way to release it.
>>>
>>> If you find a solution, I would love to know it! If I find one, I'll post
>>> it here.
>>
>> The first step in debugging a deadlock is to find what is stuck waiting
>> for what resource.
>>
>> Then find the logic that provides the resource that is being waited on.
>>
>> Then figure out why that logic is not running. Most likely, it would be
>> waiting the low priority work queue.
>>
>> I have had to solve lots of problems like this. It is not really so
>> difficult once you unstand the above things.
>>
>>
>>
>>
STM32H7 Ethernet problems
Posted by Jukka Laitinen <ju...@iki.fi>.
Hi,
Reviewing the ethernet driver, I can see couple of bugs:
1) In stm32_freeframe, it should free all the buffers, and not just the
first one. So remove the "if ((txdesc->des3 & ETH_TDES3_RD_FD) != 0)"
That may cause it run out of buffers
2) In stm32_freesegment, the order of getting the next descriptor, and
updating the tail pointer is wrong. It should first call the
stm32_get_next_rxdesc and only after that the stm32_putreg((uintptr_t)rxdesc,
STM32_ETH_DMACRXDTPR);
This is not fatal, but leads to the driver not using one of the desciptors
and buffers at all (there is one less buffer in use and wasted memory for
that).
I am very sorry, but I am unable to provide any patches currently.
Regards,
Jukka
On 20.2.2020 9.03, Reto Gähwiler wrote:
> @Gregory: Thanks for your responses and input, will see what I can do.
>
> On 2020/02/19 22:50:02, Gregory Nutt <sp...@gmail.com> wrote:
>>
>>> This sounds a lot like the problem I'm having with the SAMA5D36 Gigabit
>>> ethernet... I'm running into some kind of deadlock on long transfers
that
>>> send packets very quickly. NuttX seems to run out of IOBs and then can't
>>> send or respond to network packets.
>>>
>>> I tried increasing the low priority worker threads to 2 (and also 3) but
>>> neither of them solved the problem.
>>>
>>> I'll look at the net_lock() to see if there's a way to release it.
>>>
>>> If you find a solution, I would love to know it! If I find one, I'll
post
>>> it here.
>>
>> The first step in debugging a deadlock is to find what is stuck waiting
>> for what resource.
>>
>> Then find the logic that provides the resource that is being waited on.
>>
>> Then figure out why that logic is not running. Most likely, it would be
>> waiting the low priority work queue.
>>
>> I have had to solve lots of problems like this. It is not really so
>> difficult once you unstand the above things.
>>
>>
>>
>>
Re: STM32H7 Ethernet problems
Posted by Gregory Nutt <sp...@gmail.com>.
> This sounds a lot like the problem I'm having with the SAMA5D36 Gigabit
> ethernet... I'm running into some kind of deadlock on long transfers that
> send packets very quickly. NuttX seems to run out of IOBs and then can't
> send or respond to network packets.
>
> I tried increasing the low priority worker threads to 2 (and also 3) but
> neither of them solved the problem.
>
> I'll look at the net_lock() to see if there's a way to release it.
>
> If you find a solution, I would love to know it! If I find one, I'll post
> it here.
The first step in debugging a deadlock is to find what is stuck waiting
for what resource.
Then find the logic that provides the resource that is being waited on.
Then figure out why that logic is not running. Most likely, it would be
waiting the low priority work queue.
I have had to solve lots of problems like this. It is not really so
difficult once you unstand the above things.
Re: STM32H7 Ethernet problems
Posted by Reto G��hwiler <gr...@gmail.com>.
Hi Adam,
I saw your post and was thinking about linking to it. From what I understand so far, it happens only if there are multiple connections running. Some bursts of arp messages or flood pings will eventually also break things. Just a matter of timing. The whole collapse starts with no free buffer (network warning and error messages need to be actiaved in the config) after which the DMA will be locked down.
rgds, Reto
On 2020/02/19 22:18:51, Adam Feuer <ad...@starcat.io> wrote:
> Reto,
>
> This sounds a lot like the problem I'm having with the SAMA5D36 Gigabit
> ethernet... I'm running into some kind of deadlock on long transfers that
> send packets very quickly. NuttX seems to run out of IOBs and then can't
> send or respond to network packets.
>
> I tried increasing the low priority worker threads to 2 (and also 3) but
> neither of them solved the problem.
>
> I'll look at the net_lock() to see if there's a way to release it.
>
> If you find a solution, I would love to know it! If I find one, I'll post
> it here.
>
> cheers
> adam
>
> On Wed, Feb 19, 2020 at 6:49 AM Gregory Nutt <sp...@gmail.com> wrote:
>
> > Hi, Reto,
> > > I am working for Hexagon Mining in Baar (some of our people were on the
> > last nuttx summit). Our project involves nuttx and runs on a stm32h743zi.
> > Our nuttx version was recently (mid January) updated.
> > >
> > > We recently recognised that the ethernet of our platform dies after some
> > time. After some investigation it looks like that the DMA of the ethernet
> > ends up in a deadlock since no further descriptors are available and also
> > not freed. It all starts with a packet which is not processed right away
> > but in the next interrupt handling cycle after the next ETH_DMACSR_RI
> > appeared. It then either collapses or will have a massive slow down. In
> > case of a collapse everything will stop (incl. ping) since the chain is
> > stopped around the DMA.
> > >
> > > So far it only happened during a longer data transfer. The device works
> > fine over days just sitting there and responding to the broadcasts on the
> > network. To verify the problems we also used the TCPblaster example with a
> > minimum code base from our side. There, it only happens if multiple threads
> > are used. One thread alone is handled well as it seems. The TCPblaster
> > worked fine on the stm32f7.
> > >
> > > I am wondering if anyone had this or a similar problem before. If you
> > need more information please let me know.
> >
> > I don't know anything about your specific deadlock, but errors like
> > these have occurred and been fixed in the past. The deadlock normally
> > occurs like this:
> >
> > 1. Some network activity is started and runs on the low priority work
> > queue. Most networking occurs FIFO on the low priority worker thread.
> > 2. That network task task the network lock (net_lock()) giving it
> > exclusive access to the network
> > 3. Then it waits for some event or resource with the network locked.
> > 4. The task that will provide the event or resource also requires the
> > network lock --> Deadlock
> >
> > With IOBs, there are other related kinds of deadlocks that are possible.
> >
> > 1. Some network activity is started and runs on the low priority work
> > queue.
> > 2. The network task needs IOBs but we are out of IOBs so that network
> > task unlocks the network (allowing network activity) but blocks in
> > the low priority work queue waiting for a free IOB.
> > 3. The task the will release the IOB is also queued for execution on
> > the low priority work queue. But since the queue is block because
> > the network task is waiting on the working queue, the IOB cannot be
> > release --> Deadlock
> >
> > There are a couple of solutions to this latter IOB case: First you can
> > analyze the deadlock and find the culprit. Then modify the design so
> > that the deadlock cannot occur.
> >
> > If this is the situation, that a really simple fix is to increase the
> > number of low priority worker threads. By default, there is only one so
> > the FIFO nature of the single work queue tends to deadlock. But if you
> > increase the number of threads to two these is much less likelihood of
> > deadlocking in this way.
> >
> > Greg
> >
> >
> >
>
> --
> Adam Feuer <ad...@starcat.io>
>
Re: STM32H7 Ethernet problems
Posted by Adam Feuer <ad...@starcat.io>.
Reto,
This sounds a lot like the problem I'm having with the SAMA5D36 Gigabit
ethernet... I'm running into some kind of deadlock on long transfers that
send packets very quickly. NuttX seems to run out of IOBs and then can't
send or respond to network packets.
I tried increasing the low priority worker threads to 2 (and also 3) but
neither of them solved the problem.
I'll look at the net_lock() to see if there's a way to release it.
If you find a solution, I would love to know it! If I find one, I'll post
it here.
cheers
adam
On Wed, Feb 19, 2020 at 6:49 AM Gregory Nutt <sp...@gmail.com> wrote:
> Hi, Reto,
> > I am working for Hexagon Mining in Baar (some of our people were on the
> last nuttx summit). Our project involves nuttx and runs on a stm32h743zi.
> Our nuttx version was recently (mid January) updated.
> >
> > We recently recognised that the ethernet of our platform dies after some
> time. After some investigation it looks like that the DMA of the ethernet
> ends up in a deadlock since no further descriptors are available and also
> not freed. It all starts with a packet which is not processed right away
> but in the next interrupt handling cycle after the next ETH_DMACSR_RI
> appeared. It then either collapses or will have a massive slow down. In
> case of a collapse everything will stop (incl. ping) since the chain is
> stopped around the DMA.
> >
> > So far it only happened during a longer data transfer. The device works
> fine over days just sitting there and responding to the broadcasts on the
> network. To verify the problems we also used the TCPblaster example with a
> minimum code base from our side. There, it only happens if multiple threads
> are used. One thread alone is handled well as it seems. The TCPblaster
> worked fine on the stm32f7.
> >
> > I am wondering if anyone had this or a similar problem before. If you
> need more information please let me know.
>
> I don't know anything about your specific deadlock, but errors like
> these have occurred and been fixed in the past. The deadlock normally
> occurs like this:
>
> 1. Some network activity is started and runs on the low priority work
> queue. Most networking occurs FIFO on the low priority worker thread.
> 2. That network task task the network lock (net_lock()) giving it
> exclusive access to the network
> 3. Then it waits for some event or resource with the network locked.
> 4. The task that will provide the event or resource also requires the
> network lock --> Deadlock
>
> With IOBs, there are other related kinds of deadlocks that are possible.
>
> 1. Some network activity is started and runs on the low priority work
> queue.
> 2. The network task needs IOBs but we are out of IOBs so that network
> task unlocks the network (allowing network activity) but blocks in
> the low priority work queue waiting for a free IOB.
> 3. The task the will release the IOB is also queued for execution on
> the low priority work queue. But since the queue is block because
> the network task is waiting on the working queue, the IOB cannot be
> release --> Deadlock
>
> There are a couple of solutions to this latter IOB case: First you can
> analyze the deadlock and find the culprit. Then modify the design so
> that the deadlock cannot occur.
>
> If this is the situation, that a really simple fix is to increase the
> number of low priority worker threads. By default, there is only one so
> the FIFO nature of the single work queue tends to deadlock. But if you
> increase the number of threads to two these is much less likelihood of
> deadlocking in this way.
>
> Greg
>
>
>
--
Adam Feuer <ad...@starcat.io>
Re: STM32H7 Ethernet problems
Posted by Gregory Nutt <sp...@gmail.com>.
Hi, Reto,
> I am working for Hexagon Mining in Baar (some of our people were on the last nuttx summit). Our project involves nuttx and runs on a stm32h743zi. Our nuttx version was recently (mid January) updated.
>
> We recently recognised that the ethernet of our platform dies after some time. After some investigation it looks like that the DMA of the ethernet ends up in a deadlock since no further descriptors are available and also not freed. It all starts with a packet which is not processed right away but in the next interrupt handling cycle after the next ETH_DMACSR_RI appeared. It then either collapses or will have a massive slow down. In case of a collapse everything will stop (incl. ping) since the chain is stopped around the DMA.
>
> So far it only happened during a longer data transfer. The device works fine over days just sitting there and responding to the broadcasts on the network. To verify the problems we also used the TCPblaster example with a minimum code base from our side. There, it only happens if multiple threads are used. One thread alone is handled well as it seems. The TCPblaster worked fine on the stm32f7.
>
> I am wondering if anyone had this or a similar problem before. If you need more information please let me know.
I don't know anything about your specific deadlock, but errors like
these have occurred and been fixed in the past. The deadlock normally
occurs like this:
1. Some network activity is started and runs on the low priority work
queue. Most networking occurs FIFO on the low priority worker thread.
2. That network task task the network lock (net_lock()) giving it
exclusive access to the network
3. Then it waits for some event or resource with the network locked.
4. The task that will provide the event or resource also requires the
network lock --> Deadlock
With IOBs, there are other related kinds of deadlocks that are possible.
1. Some network activity is started and runs on the low priority work
queue.
2. The network task needs IOBs but we are out of IOBs so that network
task unlocks the network (allowing network activity) but blocks in
the low priority work queue waiting for a free IOB.
3. The task the will release the IOB is also queued for execution on
the low priority work queue. But since the queue is block because
the network task is waiting on the working queue, the IOB cannot be
release --> Deadlock
There are a couple of solutions to this latter IOB case: First you can
analyze the deadlock and find the culprit. Then modify the design so
that the deadlock cannot occur.
If this is the situation, that a really simple fix is to increase the
number of low priority worker threads. By default, there is only one so
the FIFO nature of the single work queue tends to deadlock. But if you
increase the number of threads to two these is much less likelihood of
deadlocking in this way.
Greg