You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@plc4x.apache.org by "Stefano Bossi (Jira)" <ji...@apache.org> on 2021/02/01 21:46:00 UTC
[jira] [Commented] (PLC4X-278) Double Reading Error
[ https://issues.apache.org/jira/browse/PLC4X-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276679#comment-17276679 ]
Stefano Bossi commented on PLC4X-278:
-------------------------------------
I did some more test to better understand the problem; this time I have increased the logs in order to understand if the double reading came from my code or from the library.
The class which read the data from the PLC is now in TRACE and a typical and success reading is something like:
{code:java}
2021-02-01 20:40:55.00590 TRACE DbFetcher - Adding pressLoadArray02 %DB2:1696.0:INT[16]
2021-02-01 20:40:55.00591 TRACE DbFetcher - Adding CellValueMax0 %DB2:116.0:INT
2021-02-01 20:40:55.00591 TRACE DbFetcher - Adding CellValueMax2 %DB2:120.0:INT
2021-02-01 20:40:55.00591 TRACE DbFetcher - Adding CellValueMax1 %DB2:118.0:INT
2021-02-01 20:40:55.00591 TRACE DbFetcher - Adding CellValueMax4 %DB2:124.0:INT
2021-02-01 20:40:55.00591 TRACE DbFetcher - Adding N_Pezzo %DB2:7354.0:DINT
2021-02-01 20:40:55.00591 TRACE DbFetcher - Adding CellValueMax3 %DB2:122.0:INT
2021-02-01 20:40:55.00592 TRACE DbFetcher - Adding CellValue_5 %DB2:3334.0:INT[87]
2021-02-01 20:40:55.00650 TRACE DbFetcher - pressLoadArray02 = [384,385,386,387,388,389,390,391,392,393,394,395,396,397,398,399]
2021-02-01 20:40:55.00650 TRACE DbFetcher - CellValueMax0 = 0
2021-02-01 20:40:55.00650 TRACE DbFetcher - CellValueMax2 = 0
2021-02-01 20:40:55.00651 TRACE DbFetcher - CellValueMax1 = 0
2021-02-01 20:40:55.00651 TRACE DbFetcher - CellValueMax4 = 0
2021-02-01 20:40:55.00651 TRACE DbFetcher - N_Pezzo = -12333
2021-02-01 20:40:55.00651 TRACE DbFetcher - CellValueMax3 = 0
2021-02-01 20:40:55.00651 TRACE DbFetcher - CellValue_5 = [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86]
2021-02-01 20:40:55.00652 TRACE DbFetcher - Adding CellValue_5 %DB2:3508.0:INT[110]
2021-02-01 20:40:55.00680 TRACE DbFetcher - CellValue_5 = [87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196]
2021-02-01 20:40:55.00681 TRACE DbFetcher - Adding CellValue_5 %DB2:3728.0:INT[110]
2021-02-01 20:40:55.00735 TRACE DbFetcher - CellValue_5 = [197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306]
{code}
The first part of the log is when the code Add the requests for the reading (e.g. TRACE DbFetcher - Adding CellValueMax0 %DB2:116.0:INT) the second part is the trace of the read values (e.g. TRACE DbFetcher - CellValueMax0 = 0 )
All the reading values are added and then the request is fired to the PLC and the data came back.
An internal check was performed to control that the response will not exceed the max PDU length.
This is a failure read (i.e. a double read)
{code:java}
2021-02-01 20:40:55.00735 TRACE DbFetcher - Adding CellValue_5 %DB2:3948.0:INT[93]
2021-02-01 20:40:55.00736 TRACE DbFetcher - Adding T_Wait_Del %DB2:7362.0:DINT
2021-02-01 20:40:55.00736 TRACE DbFetcher - Adding EncSpeedSafety %DB2:78.0:REAL
2021-02-01 20:40:55.00736 TRACE DbFetcher - Adding ArchiveReport %DB2:36.0:BOOL
2021-02-01 20:40:55.00736 TRACE DbFetcher - Adding TemBroSx %DB2:54.0:REAL
2021-02-01 20:40:55.00736 TRACE DbFetcher - Adding IDX %DB2:38.0:INT
2021-02-01 20:40:55.00736 TRACE DbFetcher - Adding T_LogColpo %DB2:7346.0:INT
2021-02-01 20:40:55.00736 TRACE DbFetcher - Adding CellValueOffset %DB2:96.0:INT[5]
2021-02-01 20:40:55.00736 TRACE DbFetcher - Adding EncPosAct %DB2:70.0:REAL
2021-02-01 20:40:55.00736 TRACE DbFetcher - Adding TrasferimentoCurva %DB2:34.0:INT
2021-02-01 20:40:55.00779 WARN DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:?]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:?]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276) ~[?:?]
at sun.nio.ch.IOUtil.read(IOUtil.java:233) ~[?:?]
at sun.nio.ch.IOUtil.read(IOUtil.java:223) ~[?:?]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:358) ~[?:?]
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253) ~[netty-buffer-4.1.54.Final.jar:4.1.54.Final]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1134) ~[netty-buffer-4.1.54.Final.jar:4.1.54.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.54.Final.jar:4.1.54.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.54.Final.jar:4.1.54.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.54.Final.jar:4.1.54.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
2021-02-01 20:41:00.00736 WARN CachedDriverManager - Watchdog detected a long borrowed connection, will be forcefully closed!
2021-02-01 20:41:00.00738 WARN CachedPlcConnection - Request finished with exception. Reporting Connection as Broken
java.util.concurrent.CancellationException: null
at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2396) ~[?:?]
at org.apache.plc4x.java.utils.connectionpool2.CachedPlcConnection.lambda$wrapWithTimeout$0(CachedPlcConnection.java:84) ~[plc4j-connection-cache-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
2021-02-01 20:41:00.00745 WARN CachedDriverManager - Broken Connection was returned, although it is not borrowed, currently.
2021-02-01 20:41:00.00748 ERROR DbFetcher - Time out Exception in reading DataBlock
java.util.concurrent.TimeoutException: null
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at it.fox.chronos.plc.observers.DbFetcher.onObservableChanged(DbFetcher.java:93) [chronos-0.0.3.jar:0.0.3]
at it.fox.chronos.plc.Poller.notifyObservers(Poller.java:173) [chronos-0.0.3.jar:0.0.3]
at it.fox.chronos.plc.Poller.run(Poller.java:110) [chronos-0.0.3.jar:0.0.3]
at java.lang.Thread.run(Thread.java:834) [?:?]
2021-02-01 20:41:00.00748 ERROR DbFetcher - Time out Exception in reading DataBlock
java.util.concurrent.TimeoutException: null
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at it.fox.chronos.plc.observers.DbFetcher.onObservableChanged(DbFetcher.java:93) [chronos-0.0.3.jar:0.0.3]
at it.fox.chronos.plc.Poller.notifyObservers(Poller.java:173) [chronos-0.0.3.jar:0.0.3]
at it.fox.chronos.plc.Poller.run(Poller.java:110) [chronos-0.0.3.jar:0.0.3]
at java.lang.Thread.run(Thread.java:834) [?:?]
2021-02-01 20:41:00.00751 INFO PlcDriverManager - Instantiating new PLC Driver Manager with class loader jdk.internal.loader.ClassLoaders$AppClassLoader@c77c2e
2021-02-01 20:41:00.00752 INFO PlcDriverManager - Registering available drivers...
2021-02-01 20:41:00.00758 INFO PlcDriverManager - Registering driver for Protocol s7 (Siemens S7 (Basic))
2021-02-01 20:41:00.00786 INFO TcpChannelFactory - Configuring Bootstrap with Configuration{local-rack=1, local-slot=1, remote-rack=0, remot-slot=0, pduSize=1024, maxAmqCaller=8, maxAmqCallee=8, controllerType='null'}
2021-02-01 20:41:00.00824 INFO S7ProtocolLogic - S7 Driver running in ACTIVE mode.
{code}
My code would like to read:
{code:java}
2021-02-01 20:40:55.00735 TRACE DbFetcher - Adding CellValue_5 %DB2:3948.0:INT[93]
2021-02-01 20:40:55.00736 TRACE DbFetcher - Adding T_Wait_Del %DB2:7362.0:DINT
2021-02-01 20:40:55.00736 TRACE DbFetcher - Adding EncSpeedSafety %DB2:78.0:REAL
2021-02-01 20:40:55.00736 TRACE DbFetcher - Adding ArchiveReport %DB2:36.0:BOOL
2021-02-01 20:40:55.00736 TRACE DbFetcher - Adding TemBroSx %DB2:54.0:REAL
2021-02-01 20:40:55.00736 TRACE DbFetcher - Adding IDX %DB2:38.0:INT
2021-02-01 20:40:55.00736 TRACE DbFetcher - Adding T_LogColpo %DB2:7346.0:INT
2021-02-01 20:40:55.00736 TRACE DbFetcher - Adding CellValueOffset %DB2:96.0:INT[5]
2021-02-01 20:40:55.00736 TRACE DbFetcher - Adding EncPosAct %DB2:70.0:REAL
2021-02-01 20:40:55.00736 TRACE DbFetcher - Adding TrasferimentoCurva %DB2:34.0:INT
{code}
A total of *10 variables* of different types, BUT on the wire, wireshark recorded TWO readings, the first one with *4 variables* and the second one with the remaining *6 variables*.
Here is the jpeg of wireshark trace:
!Screenshot 2021-02-01 at 22.35.20.png|width=1295,height=289!
The first packet contains:
{code:text}
S7 Communication
Header: (Job)
Parameter: (Read Var)
Function: Read Var (0x04)
Item count: 4
Item [1]: (DB 2.DBX 7346.0 INT 1)
Item [2]: (DB 2.DBX 96.0 INT 5)
Item [3]: (DB 2.DBX 70.0 REAL 1)
Item [4]: (DB 2.DBX 34.0 INT 1)
{code}
and the second one the others 6 vars:
{code:java}
S7 Communication
Header: (Job)
Parameter: (Read Var)
Function: Read Var (0x04)
Item count: 6
Item [1]: (DB 2.DBX 3948.0 INT 93)
Item [2]: (DB 2.DBX 7362.0 DINT 1)
Item [3]: (DB 2.DBX 78.0 REAL 1)
Item [4]: (DB 2.DBX 36.0 BIT 1)
Item [5]: (DB 2.DBX 54.0 REAL 1)
Item [6]: (DB 2.DBX 38.0 INT 1)
{code}
I have attached the wireshark reads of this reading and of the RST of the PLC which refuse to answer.
Somewhere in the code the library decide to fragment the reading in two.
The code I use to read the data is:
{code:java}
try (PlcConnection plcConnection = plcCachedDriverManager.getConnection(plcConnectionString)) {
Builder builder = plcConnection.readRequestBuilder();
optimizedDataBlock.getPlcQueries().stream().forEach(plcQuery->builder.addItem(plcQuery.getVarName(), plcQuery.getAddress()));
optimizedDataBlock.getPlcQueries().stream().forEach(plcQuery->logger.trace("Reading request with: {} {}", plcQuery.getVarName(), plcQuery.getAddress()));
PlcReadRequest readRequest = builder.build();
PlcReadResponse response = readRequest.execute().get(CONNECTION_TIME_OUT, TimeUnit.MILLISECONDS);
readRequest.getFieldNames().stream().forEach(s->logger.trace("{} = {}", s, response.getPlcValue(s)));
addResponseToDataBlock(dataBlockResponse, response);
} catch (PlcConnectionException e){
logger.error("Connection exception in reading DataBlock", e);
isGood = false;
} catch (CancellationException e){
logger.error("Reading Data Block thread canceled", e);
isGood = false;
} catch (IllegalStateException e){
logger.error("Error in Netty state machine during reading Data Block", e);
isGood = false;
} catch (TimeoutException timeoutException) {
logger.error("Time out Exception in reading DataBlock", timeoutException);
isGood = false;
} catch (ExecutionException e){
logger.error("Execution exception in reading DataBlock", e);
isGood = false;
} catch (Exception e){
logger.error("Generic, and very BAD, exception in reading DataBlock", e);
isGood = false;
}
{code}
As you can see the reads are blocking ones.
Any ideas on how to solve the issue ?
Regards,
S.
> Double Reading Error
> --------------------
>
> Key: PLC4X-278
> URL: https://issues.apache.org/jira/browse/PLC4X-278
> Project: Apache PLC4X
> Issue Type: Bug
> Components: Driver-S7
> Affects Versions: 0.8.0
> Environment: Client on Linux box:
> Linux RevPi33574 4.9.76-rt60-v7+ #1 SMP PREEMPT RT Tue, 12 Mar 2019 15:19:36 +0100 armv7l GNU/Linux
> PLC Siemens S7 1200 (CPU 1214C DC/DC/DC Firmware Version V1.0)
> Reporter: Stefano Bossi
> Priority: Major
> Attachments: Screenshot 2021-02-01 at 22.35.20.png, trace-01-30-10-20-28-1611998428-drop01.pcap, trace-01-30-10-20-28-1611998428-drop01.png
>
>
> Dear developers,
> I think I have found the reason of some disconnection from the PLC (Siemens 1200 S7) I see on the logs of a software I am writing. Let me explain the scenario.
> The software reads a lot of data from the PLC a polling variable and a complete DataBlock I am using a Raspberry like a client an a Siemens S7 1200 PLC.
> The reading are usually fine but randomly I have an error in the logs coming from low level Driver.
> I am using 0.8.0-SNAPSHOT coming from [pool2|https://github.com/apache/plc4x/tree/feature/integrate-pool2] experimental feature.
> The error I read on the logs is:
> {code:txt}
> 2021-01-30 10:27:58 WARN CachedDriverManager - Watchdog detected a long borrowed connection, will be forcefully closed!
> 2021-01-30 10:27:58 WARN CachedDriverManager - Broken Connection was returned, although it is not borrowed, currently.
> 2021-01-30 10:27:58 ERROR DbFetcher - Time out Exception in reading DataBlock
> java.util.concurrent.TimeoutException: null
> at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
> at it.fox.chronos.plc.observers.DbFetcher.onObservableChanged(DbFetcher.java:92) [chronos-0.0.3.jar:0.0.3]
> at it.fox.chronos.plc.Poller.notifyObservers(Poller.java:177) [chronos-0.0.3.jar:0.0.3]
> at it.fox.chronos.plc.Poller.run(Poller.java:105) [chronos-0.0.3.jar:0.0.3]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> 2021-01-30 10:27:58 INFO PlcDriverManager - Instantiating new PLC Driver Manager with class loader jdk.internal.loader.ClassLoaders$AppClassLoader@c77c2e
> 2021-01-30 10:27:58 WARN CachedPlcConnection - Request finished with exception. Reporting Connection as Broken
> java.util.concurrent.CancellationException: null
> at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2396) ~[?:?]
> at org.apache.plc4x.java.utils.connectionpool2.CachedPlcConnection.lambda$wrapWithTimeout$0(CachedPlcConnection.java:84) ~[plc4j-connection-cache-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
> at java.lang.Thread.run(Thread.java:834) ~[?:?]
> 2021-01-30 10:27:58 INFO PlcDriverManager - Registering available drivers...
> 2021-01-30 10:27:58 WARN CachedDriverManager - Broken Connection was returned, although it is not borrowed, currently.
> 2021-01-30 10:27:58 INFO PlcDriverManager - Registering driver for Protocol s7 (Siemens S7 (Basic))
> 2021-01-30 10:27:58 INFO TcpChannelFactory - Configuring Bootstrap with Configuration{local-rack=1, local-slot=1, remote-rack=0, remot-slot=0, pduSize=1024, maxAmqCaller=8, maxAmqCallee=8, controllerType='null'}
> 2021-01-30 10:27:58 INFO S7ProtocolLogic - S7 Driver running in ACTIVE mode.
> {code}
> At the first sight it seems that a connection became stale and the watchdog drop it but I think this is just the effect of an error on the wire.
> Here is the screenshot of the capture on the wire (I attach the capture to the ticket too).
> !trace-01-30-10-20-28-1611998428-drop01.png!
> At 2021-01-30 10:27:53.899233 there's a request for 7 variables in the DB2:
> {code:java}
> Frame 9: 157 bytes on wire (1256 bits), 157 bytes captured (1256 bits)
> Ethernet II, Src: KUNBUS_01:5e:53 (c8:3e:a7:01:5e:53), Dst: SiemensN_02:19:4c (00:1c:06:02:19:4c)
> Internet Protocol Version 4, Src: 192.168.1.190, Dst: 192.168.1.192
> Transmission Control Protocol, Src Port: 36240, Dst Port: 102, Seq: 94, Ack: 736, Len: 103
> TPKT, Version: 3, Length: 103
> ISO 8073/X.224 COTP Connection-Oriented Transport Protocol
> S7 Communication
> Header: (Job)
> Parameter: (Read Var)
> Function: Read Var (0x04)
> Item count: 7
> Item [1]: (DB 2.DBX 3238.0 INT 47)
> Item [2]: (DB 2.DBX 7348.0 BIT 1)
> Item [3]: (DB 2.DBX 24.0 DINT 1)
> Item [4]: (DB 2.DBX 7366.0 DINT 1)
> Item [5]: (DB 2.DBX 66.0 REAL 1)
> Item [6]: (DB 2.DBX 86.0 INT 5)
> Item [7]: (DB 2.DBX 74.0 REAL 1)
> {code}
> Without waiting the response, immediately after this request, there's an another request for an another variable, a single REAL again from the same DB2:
> {code:java}
> Frame 10: 85 bytes on wire (680 bits), 85 bytes captured (680 bits)
> Ethernet II, Src: KUNBUS_01:5e:53 (c8:3e:a7:01:5e:53), Dst: SiemensN_02:19:4c (00:1c:06:02:19:4c)
> Internet Protocol Version 4, Src: 192.168.1.190, Dst: 192.168.1.192
> Transmission Control Protocol, Src Port: 36240, Dst Port: 102, Seq: 197, Ack: 736, Len: 31
> TPKT, Version: 3, Length: 31
> ISO 8073/X.224 COTP Connection-Oriented Transport Protocol
> S7 Communication
> Header: (Job)
> Parameter: (Read Var)
> Function: Read Var (0x04)
> Item count: 1
> Item [1]: (DB 2.DBX 4938.0 REAL 24)
> {code}
> I think this is the root cause of the error and the real reason because the watchdog stand up and drop the connection.
> The PLC solicited by this double request send a couple of RST to the library and the connection dye.
> I have many of this disconnections, more or less one every 15 minutes. All the good request has a query and an answer, all the double request receive a RST from the PLC.
> My code perform only synchronous request in this way:
> {code:java}
> PlcReadResponse response = readRequest.execute().get(5000, TimeUnit.MILLISECONDS);
> {code}
> I think the double request came from the pool2 or the driver (sorry I am not able to understand that).
> Do you think is this a bug? Is it something fixable ?
> Regards,
> Stefano Bossi
--
This message was sent by Atlassian Jira
(v8.3.4#803005)