You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@synapse.apache.org by kimhorn <ki...@icsglobal.net> on 2009/05/12 07:47:37 UTC
Data Loss Issue VFS
In testing the CDATA issues (another stream) we have come across a more
serious problem.
We have found that synapse is loosing parts of files that have imbedded CR
and spaces. These are fixed format files. The script at the bottom is used
to reproduce the problem. Logging the property extracted shows the loss of
data. As the files are not small (96KB) and are full of spaces it is hard to
demonstrate or shows the full logs. Here is the start and end of the data
file only; it has 100's of records in between. The data has a "*" at the
beginning and at the end.
<Data Start>
*000000141973601M95202YBA000000014197 341316840
010402030000010
000190000000000000000000000000000000000000000000000000000000000
006770000000000000000000000000000000000000000000000020000000000
* <Data END>
Looking at the log the last part of the file is gone when Property DATA is
logged.
-----------------------------------------------------------------------------
2009-05-12 15:23:50,825 [-] [vfs-Worker-5] INFO LogMediator DATA =
*000000141973601M95202YBA000000014197 341316840
010402030000010
000190000000000000000000000000000000000000000000000000000000000
2009-05-12 15:23:54,387 [-] [vfs-Worker-5] DEBUG LogMediator End : Log
mediator
Looking at the XML created we get:
----------------------------------
<?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><data>*000000141973601M95202YBA000000014197
000190000000000000000000000000000000000000000000000000000000000</data></soapenv:Body></soapenv:Envelope>
Clearly the last part of the file has gone. I thought the error could be in
the Javascript but it appears the data is already lost at the property when
extracted from the VFS payload. Any suggestions on whats happening here ?
Could this be an issue with the XPATH property setting code?
Version of Synapse is one of recent Stable Snapshots.
Thanks
Kim
<definitions xmlns="http://ws.apache.org/ns/synapse">
<proxy name="FileCheckProxyPRS" transports="vfs">
<parameter
name="transport.vfs.FileURI">file:///C:/work/sftpagent/testdata/prs/ndc</parameter>
<parameter name="transport.vfs.ContentType">text/plain</parameter>
<parameter name="transport.vfs.FileNamePattern">.*req</parameter>
<parameter name="transport.PollInterval">15</parameter>
<parameter
name="transport.vfs.MoveAfterProcess">file:///C:/work/sftpagent/testdata/prs/ndc/archive</parameter>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<target inSequence="inSequencePRS"/>
</proxy>
<sequence name="inSequencePRS">
<log level="custom">
<property name="MSG" value=">>>>>>> IN"/>
</log>
<log level="custom">
<property xmlns:axis2ns="http://ws.apache.org/commons/ns/payload"
name="DATA" expression="//axis2ns:text"/>
</log>
<property xmlns:axis2ns="http://ws.apache.org/commons/ns/payload"
name="claimData" expression="//axis2ns:text"/>
<log level="full"/>
<script language="js"><![CDATA[ var claimData =
mc.getProperty("claimData").toString();
mc.setPayloadXML(<data>{claimData}</data>);
]]></script>
<send>
<endpoint>
<address uri="vfs:file:///C:/work/sftpagent/testdata/prs"/>
</endpoint>
</send>
</sequence>
</definitions>
--
View this message in context: http://www.nabble.com/Data-Loss-Issue-VFS-tp23496565p23496565.html
Sent from the Synapse - Dev mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
RE: Data Loss Issue VFS
Posted by Kim Horn <ki...@icsglobal.net>.
We got around this now by zipping the files and changing the send/receive web services; thus CData not necessary, for now, and works... Also I think may have been an issue with the particular snapshot I used; as I see it has other problems.
I do think that CData should be addressed ASAP, in 1.3, so Synapse can work with standard web services. The issue is also one of maintainability. If the synapse we use has to be patched, to add in different libraries for CDATA, after your releases, then things soon gets out of hand, patching patches and multiple releases of components. Very easy for an incompatibility to happen and as we will have a different Synapse to everyone else makes it difficult to report issues.
So we will still wait for CData to be implemented before using Synapse in production as customers cannot be expected to change their services, and one main aim of ESB's is legacy integration which means CData becomes a critical feature.
Thanks
Kim
_____________________________________
Kim Horn
Enterprise Architect
ICSGlobal Limited / THELMA
kim.horn@icsglobal.net
Switch: +61 2 9247 2111
Direct: +61 2 9258 7235
Mobile: +61 437 898 060
Fax: +61 2 9247 6122
www.icsglobal.net
Level 26, 201 Kent Street
Sydney NSW 2000 AUSTRALIA
NOTICE: This e-mail and any attachments are private and confidential and may contain legally privileged information. If you are not an authorised recipient, the copying or distribution of this e-mail and any attachments is prohibited and you must not read, print or act in reliance on this e-mail or attachments. This notice should not be removed.
-----Original Message-----
From: Andreas Veithen [mailto:andreas.veithen@gmail.com]
Sent: Thursday, 14 May 2009 6:19 PM
To: dev@synapse.apache.org
Subject: Re: Data Loss Issue VFS
Kim,
I did a quick test, but I'm unable to reproduce this. To investigate
this, we will definitely need an example file that reproduces this
problem.
Andreas
On Tue, May 12, 2009 at 09:21, kimhorn <ki...@icsglobal.net> wrote:
>
> The synapse.xml is in the original post. We have tried synapse with very
> large files and other types of data but have never succeeded with this data
> type. The data contains many spaces, carriage returns, line feeds etc. I
> must admit the results I am getting are weird, but I can't see what I am
> doing wrong. I gave up on this a month ago and now have some time to look
> into it again.
>
> Currently our existing systems (and our clients) must use CData to wrap this
> data for sends and receives but unfortunately I cannot even get this far to
> test this with Synapse. Hence why I added the "*" to see where the
> whitespace are disappearing in the data.
>
> I will also attach a data file, when I can, but can't do that at the moment
> as all the data has private info in it and it is not easy to sanitise these
> types of files.
>
> Thanks
> Kim
>
>
>
> pzfreo wrote:
>>
>> Kim
>>
>> Can you please post a simple Synapse.xml and a file that demonstrates
>> this. I'm quite surprised because I have had successful use of Synapse
>> with similar record oriented data to yours.
>>
>> Paul
>>
>> On Tue, May 12, 2009 at 6:53 AM, kimhorn <ki...@icsglobal.net> wrote:
>>>
>>> I should also mention that not only the data at the end of the file is
>>> missing but data in the middle is also gone in a number of places.
>>>
>>>
>>> kimhorn wrote:
>>>>
>>>> In testing the CDATA issues (another stream) we have come across a more
>>>> serious problem.
>>>> We have found that synapse is loosing parts of files that have imbedded
>>>> CR
>>>> and spaces. These are fixed format files. The script at the bottom is
>>>> used
>>>> to reproduce the problem. Logging the property extracted shows the loss
>>>> of
>>>> data. As the files are not small (96KB) and are full of spaces it is
>>>> hard
>>>> to demonstrate or shows the full logs. Here is the start and end of the
>>>> data file only; it has 100's of records in between. The data has a "*"
>>>> at
>>>> the beginning and at the end.
>>>>
>>>> <Data Start>
>>>> *000000141973601M95202YBA000000014197 341316840
>>>>
>>>> 010402030000010
>>>> 000190000000000000000000000000000000000000000000000000000000000
>>>>
>>>> 006770000000000000000000000000000000000000000000000020000000000
>>>>
>>>> * <Data END>
>>>>
>>>>
>>>> Looking at the log the last part of the file is gone when Property DATA
>>>> is
>>>> logged.
>>>> -----------------------------------------------------------------------------
>>>>
>>>> 2009-05-12 15:23:50,825 [-] [vfs-Worker-5] INFO LogMediator DATA =
>>>> *000000141973601M95202YBA000000014197 341316840
>>>>
>>>> 010402030000010
>>>> 000190000000000000000000000000000000000000000000000000000000000
>>>>
>>>>
>>>> 2009-05-12 15:23:54,387 [-] [vfs-Worker-5] DEBUG LogMediator End : Log
>>>> mediator
>>>>
>>>> Looking at the XML created we get:
>>>> ----------------------------------
>>>>
>>>> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope
>>>> xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><data>*000000141973601M95202YBA000000014197
>>>>
>>>> 000190000000000000000000000000000000000000000000000000000000000</data></soapenv:Body></soapenv:Envelope>
>>>>
>>>>
>>>> Clearly the last part of the file has gone. I thought the error could be
>>>> in the Javascript but it appears the data is already lost at the
>>>> property
>>>> when extracted from the VFS payload. Any suggestions on whats happening
>>>> here ?
>>>>
>>>> Could this be an issue with the XPATH property setting code?
>>>>
>>>> Version of Synapse is one of recent Stable Snapshots.
>>>>
>>>> Thanks
>>>> Kim
>>>>
>>>> <definitions xmlns="http://ws.apache.org/ns/synapse">
>>>> <proxy name="FileCheckProxyPRS" transports="vfs">
>>>> <parameter
>>>> name="transport.vfs.FileURI">file:///C:/work/sftpagent/testdata/prs/ndc</parameter>
>>>> <parameter name="transport.vfs.ContentType">text/plain</parameter>
>>>> <parameter name="transport.vfs.FileNamePattern">.*req</parameter>
>>>> <parameter name="transport.PollInterval">15</parameter>
>>>> <parameter
>>>> name="transport.vfs.MoveAfterProcess">file:///C:/work/sftpagent/testdata/prs/ndc/archive</parameter>
>>>> <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
>>>> <target inSequence="inSequencePRS"/>
>>>> </proxy>
>>>> <sequence name="inSequencePRS">
>>>> <log level="custom">
>>>> <property name="MSG" value=">>>>>>> IN"/>
>>>> </log>
>>>> <log level="custom">
>>>> <property xmlns:axis2ns="http://ws.apache.org/commons/ns/payload"
>>>> name="DATA" expression="//axis2ns:text"/>
>>>> </log>
>>>> <property xmlns:axis2ns="http://ws.apache.org/commons/ns/payload"
>>>> name="claimData" expression="//axis2ns:text"/>
>>>> <log level="full"/>
>>>> <script language="js"><![CDATA[ var claimData =
>>>> mc.getProperty("claimData").toString();
>>>> mc.setPayloadXML(<data>{claimData}</data>);
>>>> ]]></script>
>>>> <send>
>>>> <endpoint>
>>>> <address uri="vfs:file:///C:/work/sftpagent/testdata/prs"/>
>>>> </endpoint>
>>>> </send>
>>>> </sequence>
>>>> </definitions>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Data-Loss-Issue-VFS-tp23496565p23496604.html
>>> Sent from the Synapse - Dev mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
>>> For additional commands, e-mail: dev-help@synapse.apache.org
>>>
>>>
>>
>>
>>
>> --
>> Paul Fremantle
>> Co-Founder and CTO, WSO2
>> Apache Synapse PMC Chair
>> OASIS WS-RX TC Co-chair
>>
>> blog: http://pzf.fremantle.org
>> paul@wso2.com
>>
>> "Oxygenating the Web Service Platform", www.wso2.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
>> For additional commands, e-mail: dev-help@synapse.apache.org
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Data-Loss-Issue-VFS-tp23496565p23497440.html
> Sent from the Synapse - Dev mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
> For additional commands, e-mail: dev-help@synapse.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
Re: Data Loss Issue VFS
Posted by Andreas Veithen <an...@gmail.com>.
Kim,
I did a quick test, but I'm unable to reproduce this. To investigate
this, we will definitely need an example file that reproduces this
problem.
Andreas
On Tue, May 12, 2009 at 09:21, kimhorn <ki...@icsglobal.net> wrote:
>
> The synapse.xml is in the original post. We have tried synapse with very
> large files and other types of data but have never succeeded with this data
> type. The data contains many spaces, carriage returns, line feeds etc. I
> must admit the results I am getting are weird, but I can't see what I am
> doing wrong. I gave up on this a month ago and now have some time to look
> into it again.
>
> Currently our existing systems (and our clients) must use CData to wrap this
> data for sends and receives but unfortunately I cannot even get this far to
> test this with Synapse. Hence why I added the "*" to see where the
> whitespace are disappearing in the data.
>
> I will also attach a data file, when I can, but can't do that at the moment
> as all the data has private info in it and it is not easy to sanitise these
> types of files.
>
> Thanks
> Kim
>
>
>
> pzfreo wrote:
>>
>> Kim
>>
>> Can you please post a simple Synapse.xml and a file that demonstrates
>> this. I'm quite surprised because I have had successful use of Synapse
>> with similar record oriented data to yours.
>>
>> Paul
>>
>> On Tue, May 12, 2009 at 6:53 AM, kimhorn <ki...@icsglobal.net> wrote:
>>>
>>> I should also mention that not only the data at the end of the file is
>>> missing but data in the middle is also gone in a number of places.
>>>
>>>
>>> kimhorn wrote:
>>>>
>>>> In testing the CDATA issues (another stream) we have come across a more
>>>> serious problem.
>>>> We have found that synapse is loosing parts of files that have imbedded
>>>> CR
>>>> and spaces. These are fixed format files. The script at the bottom is
>>>> used
>>>> to reproduce the problem. Logging the property extracted shows the loss
>>>> of
>>>> data. As the files are not small (96KB) and are full of spaces it is
>>>> hard
>>>> to demonstrate or shows the full logs. Here is the start and end of the
>>>> data file only; it has 100's of records in between. The data has a "*"
>>>> at
>>>> the beginning and at the end.
>>>>
>>>> <Data Start>
>>>> *000000141973601M95202YBA000000014197 341316840
>>>>
>>>> 010402030000010
>>>> 000190000000000000000000000000000000000000000000000000000000000
>>>>
>>>> 006770000000000000000000000000000000000000000000000020000000000
>>>>
>>>> * <Data END>
>>>>
>>>>
>>>> Looking at the log the last part of the file is gone when Property DATA
>>>> is
>>>> logged.
>>>> -----------------------------------------------------------------------------
>>>>
>>>> 2009-05-12 15:23:50,825 [-] [vfs-Worker-5] INFO LogMediator DATA =
>>>> *000000141973601M95202YBA000000014197 341316840
>>>>
>>>> 010402030000010
>>>> 000190000000000000000000000000000000000000000000000000000000000
>>>>
>>>>
>>>> 2009-05-12 15:23:54,387 [-] [vfs-Worker-5] DEBUG LogMediator End : Log
>>>> mediator
>>>>
>>>> Looking at the XML created we get:
>>>> ----------------------------------
>>>>
>>>> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope
>>>> xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><data>*000000141973601M95202YBA000000014197
>>>>
>>>> 000190000000000000000000000000000000000000000000000000000000000</data></soapenv:Body></soapenv:Envelope>
>>>>
>>>>
>>>> Clearly the last part of the file has gone. I thought the error could be
>>>> in the Javascript but it appears the data is already lost at the
>>>> property
>>>> when extracted from the VFS payload. Any suggestions on whats happening
>>>> here ?
>>>>
>>>> Could this be an issue with the XPATH property setting code?
>>>>
>>>> Version of Synapse is one of recent Stable Snapshots.
>>>>
>>>> Thanks
>>>> Kim
>>>>
>>>> <definitions xmlns="http://ws.apache.org/ns/synapse">
>>>> <proxy name="FileCheckProxyPRS" transports="vfs">
>>>> <parameter
>>>> name="transport.vfs.FileURI">file:///C:/work/sftpagent/testdata/prs/ndc</parameter>
>>>> <parameter name="transport.vfs.ContentType">text/plain</parameter>
>>>> <parameter name="transport.vfs.FileNamePattern">.*req</parameter>
>>>> <parameter name="transport.PollInterval">15</parameter>
>>>> <parameter
>>>> name="transport.vfs.MoveAfterProcess">file:///C:/work/sftpagent/testdata/prs/ndc/archive</parameter>
>>>> <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
>>>> <target inSequence="inSequencePRS"/>
>>>> </proxy>
>>>> <sequence name="inSequencePRS">
>>>> <log level="custom">
>>>> <property name="MSG" value=">>>>>>> IN"/>
>>>> </log>
>>>> <log level="custom">
>>>> <property xmlns:axis2ns="http://ws.apache.org/commons/ns/payload"
>>>> name="DATA" expression="//axis2ns:text"/>
>>>> </log>
>>>> <property xmlns:axis2ns="http://ws.apache.org/commons/ns/payload"
>>>> name="claimData" expression="//axis2ns:text"/>
>>>> <log level="full"/>
>>>> <script language="js"><![CDATA[ var claimData =
>>>> mc.getProperty("claimData").toString();
>>>> mc.setPayloadXML(<data>{claimData}</data>);
>>>> ]]></script>
>>>> <send>
>>>> <endpoint>
>>>> <address uri="vfs:file:///C:/work/sftpagent/testdata/prs"/>
>>>> </endpoint>
>>>> </send>
>>>> </sequence>
>>>> </definitions>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Data-Loss-Issue-VFS-tp23496565p23496604.html
>>> Sent from the Synapse - Dev mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
>>> For additional commands, e-mail: dev-help@synapse.apache.org
>>>
>>>
>>
>>
>>
>> --
>> Paul Fremantle
>> Co-Founder and CTO, WSO2
>> Apache Synapse PMC Chair
>> OASIS WS-RX TC Co-chair
>>
>> blog: http://pzf.fremantle.org
>> paul@wso2.com
>>
>> "Oxygenating the Web Service Platform", www.wso2.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
>> For additional commands, e-mail: dev-help@synapse.apache.org
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Data-Loss-Issue-VFS-tp23496565p23497440.html
> Sent from the Synapse - Dev mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
> For additional commands, e-mail: dev-help@synapse.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
Re: Data Loss Issue VFS
Posted by kimhorn <ki...@icsglobal.net>.
The synapse.xml is in the original post. We have tried synapse with very
large files and other types of data but have never succeeded with this data
type. The data contains many spaces, carriage returns, line feeds etc. I
must admit the results I am getting are weird, but I can't see what I am
doing wrong. I gave up on this a month ago and now have some time to look
into it again.
Currently our existing systems (and our clients) must use CData to wrap this
data for sends and receives but unfortunately I cannot even get this far to
test this with Synapse. Hence why I added the "*" to see where the
whitespace are disappearing in the data.
I will also attach a data file, when I can, but can't do that at the moment
as all the data has private info in it and it is not easy to sanitise these
types of files.
Thanks
Kim
pzfreo wrote:
>
> Kim
>
> Can you please post a simple Synapse.xml and a file that demonstrates
> this. I'm quite surprised because I have had successful use of Synapse
> with similar record oriented data to yours.
>
> Paul
>
> On Tue, May 12, 2009 at 6:53 AM, kimhorn <ki...@icsglobal.net> wrote:
>>
>> I should also mention that not only the data at the end of the file is
>> missing but data in the middle is also gone in a number of places.
>>
>>
>> kimhorn wrote:
>>>
>>> In testing the CDATA issues (another stream) we have come across a more
>>> serious problem.
>>> We have found that synapse is loosing parts of files that have imbedded
>>> CR
>>> and spaces. These are fixed format files. The script at the bottom is
>>> used
>>> to reproduce the problem. Logging the property extracted shows the loss
>>> of
>>> data. As the files are not small (96KB) and are full of spaces it is
>>> hard
>>> to demonstrate or shows the full logs. Here is the start and end of the
>>> data file only; it has 100's of records in between. The data has a "*"
>>> at
>>> the beginning and at the end.
>>>
>>> <Data Start>
>>> *000000141973601M95202YBA000000014197 341316840
>>>
>>> 010402030000010
>>> 000190000000000000000000000000000000000000000000000000000000000
>>>
>>> 006770000000000000000000000000000000000000000000000020000000000
>>>
>>> * <Data END>
>>>
>>>
>>> Looking at the log the last part of the file is gone when Property DATA
>>> is
>>> logged.
>>> -----------------------------------------------------------------------------
>>>
>>> 2009-05-12 15:23:50,825 [-] [vfs-Worker-5] INFO LogMediator DATA =
>>> *000000141973601M95202YBA000000014197 341316840
>>>
>>> 010402030000010
>>> 000190000000000000000000000000000000000000000000000000000000000
>>>
>>>
>>> 2009-05-12 15:23:54,387 [-] [vfs-Worker-5] DEBUG LogMediator End : Log
>>> mediator
>>>
>>> Looking at the XML created we get:
>>> ----------------------------------
>>>
>>> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope
>>> xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><data>*000000141973601M95202YBA000000014197
>>>
>>> 000190000000000000000000000000000000000000000000000000000000000</data></soapenv:Body></soapenv:Envelope>
>>>
>>>
>>> Clearly the last part of the file has gone. I thought the error could be
>>> in the Javascript but it appears the data is already lost at the
>>> property
>>> when extracted from the VFS payload. Any suggestions on whats happening
>>> here ?
>>>
>>> Could this be an issue with the XPATH property setting code?
>>>
>>> Version of Synapse is one of recent Stable Snapshots.
>>>
>>> Thanks
>>> Kim
>>>
>>> <definitions xmlns="http://ws.apache.org/ns/synapse">
>>> <proxy name="FileCheckProxyPRS" transports="vfs">
>>> <parameter
>>> name="transport.vfs.FileURI">file:///C:/work/sftpagent/testdata/prs/ndc</parameter>
>>> <parameter name="transport.vfs.ContentType">text/plain</parameter>
>>> <parameter name="transport.vfs.FileNamePattern">.*req</parameter>
>>> <parameter name="transport.PollInterval">15</parameter>
>>> <parameter
>>> name="transport.vfs.MoveAfterProcess">file:///C:/work/sftpagent/testdata/prs/ndc/archive</parameter>
>>> <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
>>> <target inSequence="inSequencePRS"/>
>>> </proxy>
>>> <sequence name="inSequencePRS">
>>> <log level="custom">
>>> <property name="MSG" value=">>>>>>> IN"/>
>>> </log>
>>> <log level="custom">
>>> <property xmlns:axis2ns="http://ws.apache.org/commons/ns/payload"
>>> name="DATA" expression="//axis2ns:text"/>
>>> </log>
>>> <property xmlns:axis2ns="http://ws.apache.org/commons/ns/payload"
>>> name="claimData" expression="//axis2ns:text"/>
>>> <log level="full"/>
>>> <script language="js"><![CDATA[ var claimData =
>>> mc.getProperty("claimData").toString();
>>> mc.setPayloadXML(<data>{claimData}</data>);
>>> ]]></script>
>>> <send>
>>> <endpoint>
>>> <address uri="vfs:file:///C:/work/sftpagent/testdata/prs"/>
>>> </endpoint>
>>> </send>
>>> </sequence>
>>> </definitions>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Data-Loss-Issue-VFS-tp23496565p23496604.html
>> Sent from the Synapse - Dev mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
>> For additional commands, e-mail: dev-help@synapse.apache.org
>>
>>
>
>
>
> --
> Paul Fremantle
> Co-Founder and CTO, WSO2
> Apache Synapse PMC Chair
> OASIS WS-RX TC Co-chair
>
> blog: http://pzf.fremantle.org
> paul@wso2.com
>
> "Oxygenating the Web Service Platform", www.wso2.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
> For additional commands, e-mail: dev-help@synapse.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/Data-Loss-Issue-VFS-tp23496565p23497440.html
Sent from the Synapse - Dev mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
Re: Data Loss Issue VFS
Posted by Paul Fremantle <pz...@gmail.com>.
Kim
Can you please post a simple Synapse.xml and a file that demonstrates
this. I'm quite surprised because I have had successful use of Synapse
with similar record oriented data to yours.
Paul
On Tue, May 12, 2009 at 6:53 AM, kimhorn <ki...@icsglobal.net> wrote:
>
> I should also mention that not only the data at the end of the file is
> missing but data in the middle is also gone in a number of places.
>
>
> kimhorn wrote:
>>
>> In testing the CDATA issues (another stream) we have come across a more
>> serious problem.
>> We have found that synapse is loosing parts of files that have imbedded CR
>> and spaces. These are fixed format files. The script at the bottom is used
>> to reproduce the problem. Logging the property extracted shows the loss of
>> data. As the files are not small (96KB) and are full of spaces it is hard
>> to demonstrate or shows the full logs. Here is the start and end of the
>> data file only; it has 100's of records in between. The data has a "*" at
>> the beginning and at the end.
>>
>> <Data Start>
>> *000000141973601M95202YBA000000014197 341316840
>>
>> 010402030000010
>> 000190000000000000000000000000000000000000000000000000000000000
>>
>> 006770000000000000000000000000000000000000000000000020000000000
>>
>> * <Data END>
>>
>>
>> Looking at the log the last part of the file is gone when Property DATA is
>> logged.
>> -----------------------------------------------------------------------------
>>
>> 2009-05-12 15:23:50,825 [-] [vfs-Worker-5] INFO LogMediator DATA =
>> *000000141973601M95202YBA000000014197 341316840
>>
>> 010402030000010
>> 000190000000000000000000000000000000000000000000000000000000000
>>
>>
>> 2009-05-12 15:23:54,387 [-] [vfs-Worker-5] DEBUG LogMediator End : Log
>> mediator
>>
>> Looking at the XML created we get:
>> ----------------------------------
>>
>> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope
>> xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><data>*000000141973601M95202YBA000000014197
>>
>> 000190000000000000000000000000000000000000000000000000000000000</data></soapenv:Body></soapenv:Envelope>
>>
>>
>> Clearly the last part of the file has gone. I thought the error could be
>> in the Javascript but it appears the data is already lost at the property
>> when extracted from the VFS payload. Any suggestions on whats happening
>> here ?
>>
>> Could this be an issue with the XPATH property setting code?
>>
>> Version of Synapse is one of recent Stable Snapshots.
>>
>> Thanks
>> Kim
>>
>> <definitions xmlns="http://ws.apache.org/ns/synapse">
>> <proxy name="FileCheckProxyPRS" transports="vfs">
>> <parameter
>> name="transport.vfs.FileURI">file:///C:/work/sftpagent/testdata/prs/ndc</parameter>
>> <parameter name="transport.vfs.ContentType">text/plain</parameter>
>> <parameter name="transport.vfs.FileNamePattern">.*req</parameter>
>> <parameter name="transport.PollInterval">15</parameter>
>> <parameter
>> name="transport.vfs.MoveAfterProcess">file:///C:/work/sftpagent/testdata/prs/ndc/archive</parameter>
>> <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
>> <target inSequence="inSequencePRS"/>
>> </proxy>
>> <sequence name="inSequencePRS">
>> <log level="custom">
>> <property name="MSG" value=">>>>>>> IN"/>
>> </log>
>> <log level="custom">
>> <property xmlns:axis2ns="http://ws.apache.org/commons/ns/payload"
>> name="DATA" expression="//axis2ns:text"/>
>> </log>
>> <property xmlns:axis2ns="http://ws.apache.org/commons/ns/payload"
>> name="claimData" expression="//axis2ns:text"/>
>> <log level="full"/>
>> <script language="js"><![CDATA[ var claimData =
>> mc.getProperty("claimData").toString();
>> mc.setPayloadXML(<data>{claimData}</data>);
>> ]]></script>
>> <send>
>> <endpoint>
>> <address uri="vfs:file:///C:/work/sftpagent/testdata/prs"/>
>> </endpoint>
>> </send>
>> </sequence>
>> </definitions>
>>
>>
>>
>>
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Data-Loss-Issue-VFS-tp23496565p23496604.html
> Sent from the Synapse - Dev mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
> For additional commands, e-mail: dev-help@synapse.apache.org
>
>
--
Paul Fremantle
Co-Founder and CTO, WSO2
Apache Synapse PMC Chair
OASIS WS-RX TC Co-chair
blog: http://pzf.fremantle.org
paul@wso2.com
"Oxygenating the Web Service Platform", www.wso2.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
Re: Data Loss Issue VFS
Posted by kimhorn <ki...@icsglobal.net>.
I should also mention that not only the data at the end of the file is
missing but data in the middle is also gone in a number of places.
kimhorn wrote:
>
> In testing the CDATA issues (another stream) we have come across a more
> serious problem.
> We have found that synapse is loosing parts of files that have imbedded CR
> and spaces. These are fixed format files. The script at the bottom is used
> to reproduce the problem. Logging the property extracted shows the loss of
> data. As the files are not small (96KB) and are full of spaces it is hard
> to demonstrate or shows the full logs. Here is the start and end of the
> data file only; it has 100's of records in between. The data has a "*" at
> the beginning and at the end.
>
> <Data Start>
> *000000141973601M95202YBA000000014197 341316840
>
> 010402030000010
> 000190000000000000000000000000000000000000000000000000000000000
>
> 006770000000000000000000000000000000000000000000000020000000000
>
> * <Data END>
>
>
> Looking at the log the last part of the file is gone when Property DATA is
> logged.
> -----------------------------------------------------------------------------
>
> 2009-05-12 15:23:50,825 [-] [vfs-Worker-5] INFO LogMediator DATA =
> *000000141973601M95202YBA000000014197 341316840
>
> 010402030000010
> 000190000000000000000000000000000000000000000000000000000000000
>
>
> 2009-05-12 15:23:54,387 [-] [vfs-Worker-5] DEBUG LogMediator End : Log
> mediator
>
> Looking at the XML created we get:
> ----------------------------------
>
> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope
> xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><data>*000000141973601M95202YBA000000014197
>
> 000190000000000000000000000000000000000000000000000000000000000</data></soapenv:Body></soapenv:Envelope>
>
>
> Clearly the last part of the file has gone. I thought the error could be
> in the Javascript but it appears the data is already lost at the property
> when extracted from the VFS payload. Any suggestions on whats happening
> here ?
>
> Could this be an issue with the XPATH property setting code?
>
> Version of Synapse is one of recent Stable Snapshots.
>
> Thanks
> Kim
>
> <definitions xmlns="http://ws.apache.org/ns/synapse">
> <proxy name="FileCheckProxyPRS" transports="vfs">
> <parameter
> name="transport.vfs.FileURI">file:///C:/work/sftpagent/testdata/prs/ndc</parameter>
> <parameter name="transport.vfs.ContentType">text/plain</parameter>
> <parameter name="transport.vfs.FileNamePattern">.*req</parameter>
> <parameter name="transport.PollInterval">15</parameter>
> <parameter
> name="transport.vfs.MoveAfterProcess">file:///C:/work/sftpagent/testdata/prs/ndc/archive</parameter>
> <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
> <target inSequence="inSequencePRS"/>
> </proxy>
> <sequence name="inSequencePRS">
> <log level="custom">
> <property name="MSG" value=">>>>>>> IN"/>
> </log>
> <log level="custom">
> <property xmlns:axis2ns="http://ws.apache.org/commons/ns/payload"
> name="DATA" expression="//axis2ns:text"/>
> </log>
> <property xmlns:axis2ns="http://ws.apache.org/commons/ns/payload"
> name="claimData" expression="//axis2ns:text"/>
> <log level="full"/>
> <script language="js"><![CDATA[ var claimData =
> mc.getProperty("claimData").toString();
> mc.setPayloadXML(<data>{claimData}</data>);
> ]]></script>
> <send>
> <endpoint>
> <address uri="vfs:file:///C:/work/sftpagent/testdata/prs"/>
> </endpoint>
> </send>
> </sequence>
> </definitions>
>
>
>
>
>
>
>
--
View this message in context: http://www.nabble.com/Data-Loss-Issue-VFS-tp23496565p23496604.html
Sent from the Synapse - Dev mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org