Re: Serial exceptions --- MINA DEVELOPER FYI

[Julien Vermillard <jv...@archean.fr>]

Hi,
Look like we can add some lock in the port discovery/opening code.
I'll check if RXTX is supposed to be thread safe on this portion.
for the port in use problem, I never seen it. On what kind of operating
system you are trying ?
Julien
 
Le Tue, 28 Jul 2009 11:08:25 -0400,
boB Gage <bg...@merge.com> a écrit :

> Ah ha!!   :-)  I appreciate the feedback!
> 
> Thanks!!
> 
> Emmanuel Lecharny wrote:
> >
> > Hi,
> >
> > I see you are posting mails here, may be expecting some help from
> > the only guy who knows hwat's inside this part of the code : he is 
> > enjoying 3 weeks off in China atm...
> >
> > Just to give you some feedback :)
> >
> > boB Gage wrote:
> >> Found a work-around for my immediate issue, but it speaks of a
> >> much deeper problem in the Mina and/or rxtx layers.
> >>
> >> I have added a static lock around my chunk of code that calls the 
> >> IoConnector.connect() method.    I have locked code lazily, so the 
> >> critical code includes the bulk of my open() method; this consists
> >> of building the IoConnector, configuring it, calling connect(),
> >> and waiting on the ConnectFuture returned (if not null).     It
> >> may be that a much tighter lock is all that is needed -- that is
> >> beyond the scope of my work-around.
> >>
> >> With no two ports attempting an open operation concurrently, the 
> >> symptoms (Exceptions and loss of the serial port) have stopped.
> >> I have two tests going on (different pieces of hardware), and both
> >> have passed 12 hours now without showing any problems.
> >>
> >> Apparently that maybe-collision the logs were hinting at was a
> >> real collision (though I misnamed IoSession.open() as the culprit)
> >> -- whether it's Mina or rxtx is beyond my knowledge.   But
> >> apparently two threads attempting to open different serial ports
> >> at the same time *can* cause fatal problems.   [ It's very timing
> >> based; I've seen runs go 6 hours before failing; seen others fail
> >> in under 10 minutes. ]
> >>
> >> A repeating (ie unrecoverable) SerialPortUnavailableException is
> >> the one consistent symptom.   I cannot explain why early tests got
> >> a single PortInUseException first, nor why later tests got a 
> >> recoverable SerialPortUnavailableException before complete failure
> >> -- but the RejectedExecutionExceptions were definitely unrelated
> >> red herrings.
> >>
> >> It seems likely this is a serial-specific issue, as I have run
> >> many similar tests using socket connections without seeing this
> >> behaviour.
> >>
> >> Thanks again for all the help!!
> >> boB
> >>
> >> PS.  Opps, keep forgetting to say:   Mina 2.0.0-M6,   RxTx 2.1.7 r2
> >>
> >>
> >>
> >> boB Gage wrote:
> >>> Have made some more progress on this issue. The 
> >>> RejectedExecutionException's were my fault and have been 
> >>> eliminated.   [ Executor handling periodic requests was getting
> >>> shut down and re-started, rather than destroyed and recreated. ]
> >>>
> >>> I still, however, have the case where after some un-constant
> >>> number of successful serial port open/timeout/close cycles I
> >>> start seeing SerialPortUnavailableExceptions.
> >>>
> >>> With my newest code I'm seeing a new pattern (twice in two runs):
> >>>   * A number of successful open/timeout/close cycles on the port
> >>>   * A single SerialPortUnavailableException
> >>>   * Several more successful open/timeout/close cycles on the port
> >>>   * A series of SerialPortUnavailableExceptions, one for every 
> >>> connection attempt until the program is stopped and restarted.
> >>>
> >>> Two pieces of new behavior here.    1) I've never before seen the 
> >>> port go unavailable and come back again (the first single
> >>> exception). 2) I'm no longer seeing a PortInUseException at the
> >>> edge of the port going unavailable.
> >>>
> >>> Some more factoids that may or may not help:
> >>>   * With one serial port, this problem is very rare (I've seen
> >>> runs over 24 hours).  With each additional serial port, the
> >>> occurrence rate goes up.
> >>>   * Logs make it look like the problem *MIGHT* be a collision 
> >>> between two IoSession.open() calls happening concurrently
> >>> (different ports, of course).   Could there be something
> >>> non-thread safe in Mina's serial layer and/or the rxtx library
> >>> it's using??
> >>>   * The message text of the SerialPortUnavailableExceptions reads 
> >>> "Serial port not found"
> >>>
> >>> Thanks in advance,
> >>> boB Gage
> >>>
> >>>
> >>>
> >>> boB Gage wrote:
> >>>> I'm seeing three different exceptions on the serial ports and am 
> >>>> unsure what they exactly mean.
> >>>>
> >>>> Can anyone provide details as to *why* each of these may be
> >>>> throw:
> >>>>   * PortInUseException
> >>>>   * SerialPortUnavailableException
> >>>>   * RejectedExecutionException
> >>>>
> >>>> I have seen each of the preceding exceptions while trying to
> >>>> open a serial port.    The first two make the open fail, the
> >>>> third does not (though the open operation has not completely
> >>>> succeeded either -- no data flows on the port.)
> >>>>
> >>>> There are two specific cases that I have seen, both while 
> >>>> attempting to connect to a device that's temporarily unavailable 
> >>>> (ie powered off):
> >>>>
> >>>>   1) My program opens the port, times out, closes it, waits, and 
> >>>> tries again (proper behavior).    This cycle continues fine for 
> >>>> awhile (time varies) then a single PortInUseException is
> >>>> thrown. On the next, and every following, attempt a 
> >>>> SerialPortUnavailableException is thrown.    The 
> >>>> SerialPortUnavailableException continues until my program is
> >>>> shut down and restarted -- then it immediately can open the port
> >>>> again.
> >>>>
> >>>>   2) Same situation, same timeout cycles, but for some reason a 
> >>>> RejectedExecutionException is thrown.   The "open" still returns 
> >>>> true, but no data flows on the port.   Eventually the attempt
> >>>> times out.   One the next, and every following, attempt the 
> >>>> RejectedExecutionException is thrown and the behavior does not
> >>>> change.
> >>>>
> >>>> In both cases, turning on the far end device after the first 
> >>>> exception has no affect.    The port, once failed stays that 
> >>>> way.    Stopping and restarting our program has immediately
> >>>> cleared up the issue each and every time.    Unfortunately,
> >>>> stopping & restarting our problem is not a viable solution.
> >>>>
> >>>> I have seen timing changes have some effect.   With a 5-second 
> >>>> delay between connection attempts only 27 attempts succeeded
> >>>> (timed out) over a period of about 8 minutes before throwing the 
> >>>> PortInUseException.    Changing that delay to 20 seconds, over
> >>>> 220 attempts were made over a period of 6+ hours before the
> >>>> exception was thrown.
> >>>>
> >>>> Any clues???????
> >>>>
> >>>> Thanks in advance!!!
> >>>>
> >>>
> >>
> >>
> >
> >
> 

Re: Serial exceptions --- MINA DEVELOPER FYI

[boB Gage <bg...@merge.com>]

The app I'm developing is for both Linux and windows platforms.    This 
behavior was tracked and fixed on a Linux box.

boB


Julien Vermillard wrote:
> Hi,
> Look like we can add some lock in the port discovery/opening code.
> I'll check if RXTX is supposed to be thread safe on this portion.
> for the port in use problem, I never seen it. On what kind of operating
> system you are trying ?
> Julien
>  
> Le Tue, 28 Jul 2009 11:08:25 -0400,
> boB Gage <bg...@merge.com> a écrit :
>
>   
>> Ah ha!!   :-)  I appreciate the feedback!
>>
>> Thanks!!
>>
>> Emmanuel Lecharny wrote:
>>     
>>> Hi,
>>>
>>> I see you are posting mails here, may be expecting some help from
>>> the only guy who knows hwat's inside this part of the code : he is 
>>> enjoying 3 weeks off in China atm...
>>>
>>> Just to give you some feedback :)
>>>
>>> boB Gage wrote:
>>>       
>>>> Found a work-around for my immediate issue, but it speaks of a
>>>> much deeper problem in the Mina and/or rxtx layers.
>>>>
>>>> I have added a static lock around my chunk of code that calls the 
>>>> IoConnector.connect() method.    I have locked code lazily, so the 
>>>> critical code includes the bulk of my open() method; this consists
>>>> of building the IoConnector, configuring it, calling connect(),
>>>> and waiting on the ConnectFuture returned (if not null).     It
>>>> may be that a much tighter lock is all that is needed -- that is
>>>> beyond the scope of my work-around.
>>>>
>>>> With no two ports attempting an open operation concurrently, the 
>>>> symptoms (Exceptions and loss of the serial port) have stopped.
>>>> I have two tests going on (different pieces of hardware), and both
>>>> have passed 12 hours now without showing any problems.
>>>>
>>>> Apparently that maybe-collision the logs were hinting at was a
>>>> real collision (though I misnamed IoSession.open() as the culprit)
>>>> -- whether it's Mina or rxtx is beyond my knowledge.   But
>>>> apparently two threads attempting to open different serial ports
>>>> at the same time *can* cause fatal problems.   [ It's very timing
>>>> based; I've seen runs go 6 hours before failing; seen others fail
>>>> in under 10 minutes. ]
>>>>
>>>> A repeating (ie unrecoverable) SerialPortUnavailableException is
>>>> the one consistent symptom.   I cannot explain why early tests got
>>>> a single PortInUseException first, nor why later tests got a 
>>>> recoverable SerialPortUnavailableException before complete failure
>>>> -- but the RejectedExecutionExceptions were definitely unrelated
>>>> red herrings.
>>>>
>>>> It seems likely this is a serial-specific issue, as I have run
>>>> many similar tests using socket connections without seeing this
>>>> behaviour.
>>>>
>>>> Thanks again for all the help!!
>>>> boB
>>>>
>>>> PS.  Opps, keep forgetting to say:   Mina 2.0.0-M6,   RxTx 2.1.7 r2
>>>>
>>>>
>>>>
>>>> boB Gage wrote:
>>>>         
>>>>> Have made some more progress on this issue. The 
>>>>> RejectedExecutionException's were my fault and have been 
>>>>> eliminated.   [ Executor handling periodic requests was getting
>>>>> shut down and re-started, rather than destroyed and recreated. ]
>>>>>
>>>>> I still, however, have the case where after some un-constant
>>>>> number of successful serial port open/timeout/close cycles I
>>>>> start seeing SerialPortUnavailableExceptions.
>>>>>
>>>>> With my newest code I'm seeing a new pattern (twice in two runs):
>>>>>   * A number of successful open/timeout/close cycles on the port
>>>>>   * A single SerialPortUnavailableException
>>>>>   * Several more successful open/timeout/close cycles on the port
>>>>>   * A series of SerialPortUnavailableExceptions, one for every 
>>>>> connection attempt until the program is stopped and restarted.
>>>>>
>>>>> Two pieces of new behavior here.    1) I've never before seen the 
>>>>> port go unavailable and come back again (the first single
>>>>> exception). 2) I'm no longer seeing a PortInUseException at the
>>>>> edge of the port going unavailable.
>>>>>
>>>>> Some more factoids that may or may not help:
>>>>>   * With one serial port, this problem is very rare (I've seen
>>>>> runs over 24 hours).  With each additional serial port, the
>>>>> occurrence rate goes up.
>>>>>   * Logs make it look like the problem *MIGHT* be a collision 
>>>>> between two IoSession.open() calls happening concurrently
>>>>> (different ports, of course).   Could there be something
>>>>> non-thread safe in Mina's serial layer and/or the rxtx library
>>>>> it's using??
>>>>>   * The message text of the SerialPortUnavailableExceptions reads 
>>>>> "Serial port not found"
>>>>>
>>>>> Thanks in advance,
>>>>> boB Gage
>>>>>
>>>>>
>>>>>
>>>>> boB Gage wrote:
>>>>>           
>>>>>> I'm seeing three different exceptions on the serial ports and am 
>>>>>> unsure what they exactly mean.
>>>>>>
>>>>>> Can anyone provide details as to *why* each of these may be
>>>>>> throw:
>>>>>>   * PortInUseException
>>>>>>   * SerialPortUnavailableException
>>>>>>   * RejectedExecutionException
>>>>>>
>>>>>> I have seen each of the preceding exceptions while trying to
>>>>>> open a serial port.    The first two make the open fail, the
>>>>>> third does not (though the open operation has not completely
>>>>>> succeeded either -- no data flows on the port.)
>>>>>>
>>>>>> There are two specific cases that I have seen, both while 
>>>>>> attempting to connect to a device that's temporarily unavailable 
>>>>>> (ie powered off):
>>>>>>
>>>>>>   1) My program opens the port, times out, closes it, waits, and 
>>>>>> tries again (proper behavior).    This cycle continues fine for 
>>>>>> awhile (time varies) then a single PortInUseException is
>>>>>> thrown. On the next, and every following, attempt a 
>>>>>> SerialPortUnavailableException is thrown.    The 
>>>>>> SerialPortUnavailableException continues until my program is
>>>>>> shut down and restarted -- then it immediately can open the port
>>>>>> again.
>>>>>>
>>>>>>   2) Same situation, same timeout cycles, but for some reason a 
>>>>>> RejectedExecutionException is thrown.   The "open" still returns 
>>>>>> true, but no data flows on the port.   Eventually the attempt
>>>>>> times out.   One the next, and every following, attempt the 
>>>>>> RejectedExecutionException is thrown and the behavior does not
>>>>>> change.
>>>>>>
>>>>>> In both cases, turning on the far end device after the first 
>>>>>> exception has no affect.    The port, once failed stays that 
>>>>>> way.    Stopping and restarting our program has immediately
>>>>>> cleared up the issue each and every time.    Unfortunately,
>>>>>> stopping & restarting our problem is not a viable solution.
>>>>>>
>>>>>> I have seen timing changes have some effect.   With a 5-second 
>>>>>> delay between connection attempts only 27 attempts succeeded
>>>>>> (timed out) over a period of about 8 minutes before throwing the 
>>>>>> PortInUseException.    Changing that delay to 20 seconds, over
>>>>>> 220 attempts were made over a period of 6+ hours before the
>>>>>> exception was thrown.
>>>>>>
>>>>>> Any clues???????
>>>>>>
>>>>>> Thanks in advance!!!
>>>>>>
>>>>>>             
>>>>         
>>>