You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@synapse.apache.org by "Cislaghi Gianni (JIRA)" <ji...@apache.org> on 2008/08/04 09:03:44 UTC
[jira] Created: (SYNAPSE-420) FIX4.4 repeting grouping
FIX4.4 repeting grouping
------------------------
Key: SYNAPSE-420
URL: https://issues.apache.org/jira/browse/SYNAPSE-420
Project: Synapse
Issue Type: Bug
Components: Transports
Reporter: Cislaghi Gianni
Synapse 1.2 FIX transport layer does not support FIX repeating groups.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
Re: [jira] Created: (SYNAPSE-420) FIX4.4 repeting grouping
Posted by Andreas Veithen <an...@skynet.be>.
If the only problem is a method with package access, then I would
suggest to implement a workaround by invoking that method using
reflection. Indeed, reflection allows you to circumvent access
restrictions (simply call setAccessible on the Method object). Once a
new version including a fix for QFJ-330 is released, you can clean up
the code in the FIX transport.
Andreas
On 15 août 08, at 22:10, Asanka Abeysinghe wrote:
> Hi,
> I did fix the issue QFJ-330 and provided the patch. The change is to
> make one package access method to public access inside core/src/main/
> java/quickfix/FieldMap.java. This method returns a list of groups
> (List<Group> getGroups(int field)). And the group contains the
> **delimiter field**. Purpose of accessing the DD is to get the
> delimiter key for the repeating group, with this approach we don't
> need to access the DD.
> With this change current implementation of FIX to AXIOM will remain
> with CDATA and binary support it will add new routines to create and
> parse the repeating groups using recursions.
> Do we have to wait till QFJ accept QFJ-330 patch? if not we have to
> deploy the updated quickfixj-core-<version>.jar in a public maven
> repo and provide a download url for the synapse FIX users to
> download the updated library.
> Asanka A.
>
>
>
>
>
Re: [jira] Created: (SYNAPSE-420) FIX4.4 repeting grouping
Posted by Asanka Abeysinghe <as...@wso2.com>.
Hiranya Jayathilaka wrote:
>
>
> In that case I don't think there will be any performance degradations.
> Am I correct?
>
Yes, no impact on the performance because we are not accessing the DD.
Group object it self contains the delimiter that is using to set as the
first tag of the group. (Patch is attached with
https://issues.apache.org/jira/browse/SYNAPSE-420)
Asanka A.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
Re: [jira] Created: (SYNAPSE-420) FIX4.4 repeting grouping
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
Hi,
I'm +1 for fixing this issue in QFJ side. I feel that losing CDATA sections
in the infoset along with the binary support is a too much of a price to
pay.
On Sat, Aug 16, 2008 at 1:40 AM, Asanka Abeysinghe <as...@wso2.com> wrote:
> Hi,
> I did fix the issue QFJ-330 and provided the patch. The change is to make
> one package access method to public access inside
> core/src/main/java/quickfix/FieldMap.java. This method returns a list of
> groups (List<Group> getGroups(int field)). And the group contains the ***delimiter
> field***. Purpose of accessing the DD is to get the delimiter key for the
> repeating group, with this approach we don't need to access the DD.
>
In that case I don't think there will be any performance degradations. Am I
correct?
>
> With this change current implementation of FIX to AXIOM will remain with
> CDATA and binary support it will add new routines to create and parse the
> repeating groups using recursions.
>
+1
>
> Do we have to wait till QFJ accept QFJ-330 patch? if not we have to deploy
> the updated quickfixj-core-<version>.jar in a public maven repo and provide
> a download url for the synapse FIX users to download the updated library.
>
I don't think QFJ folks would accept the patch until their next release. I
submitted a patch few months back and it wasn't accepted until their latest
release. So I think it's best to look for other options like hosting the
modified library in a public Maven repo.
Thanks
Best Regards,
Hiranya
> Asanka A.
>
>
>
>
>
>
Re: [jira] Created: (SYNAPSE-420) FIX4.4 repeting grouping
Posted by Asanka Abeysinghe <as...@wso2.com>.
Hi,
I did fix the issue QFJ-330 and provided the patch. The change is to
make one package access method to public access inside
core/src/main/java/quickfix/FieldMap.java. This method returns a list of
groups (List<Group> getGroups(int field)). And the group contains the
***delimiter field***. Purpose of accessing the DD is to get the
delimiter key for the repeating group, with this approach we don't need
to access the DD.
With this change current implementation of FIX to AXIOM will remain with
CDATA and binary support it will add new routines to create and parse
the repeating groups using recursions.
Do we have to wait till QFJ accept QFJ-330 patch? if not we have to
deploy the updated quickfixj-core-<version>.jar in a public maven repo
and provide a download url for the synapse FIX users to download the
updated library.
Asanka A.
Re: [jira] Created: (SYNAPSE-420) FIX4.4 repeting grouping
Posted by Asanka Abeysinghe <as...@wso2.com>.
> Any possibility to use the DD once for each type of message and cache
> results? We could ship a binary patch against QFJ until such time they
> accept and include any fix you make into its code base
There are more than 150 message types in FIX 4.4 and FIX 5.0 got some
additional messages, and each message type will contains its own
repeating groups, we can keep it but it will take significant time to
load to the working-store/cache, and this working-store will take
considerable amount of memory space to store.
> How large can a FIX message get? Can it contain binary? attachments
> etc? This will let us determine if reading the message into a String
> is appropriate. For example a JMS Text message can be read into a
> String, as always its also loaded into memory.
Average size of a Order will be around 250bytes but a Order-list message
can have N number of orders (Order-list use to send batch orders and
frequently use). FIX contains DATA fields that can send binary (before
FAST came people used it). Slandered FIX doesn't allow to send
attachments ,even an email is a message of FIX (35=C).
I will look at the fixes on QFJ side.
Asanka A.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
Re: [jira] Created: (SYNAPSE-420) FIX4.4 repeting grouping
Posted by "Asankha C. Perera" <as...@wso2.com>.
Asanka
> So we have two options
> [1] Fix the bug in QFJ side and use data dictionary to identify the
> repeating groups. Accessing the DD for each FIX message will effect
> our message processing time because I noticed that API calls for the
> data dictionary is very slow.
Any possibility to use the DD once for each type of message and cache
results? We could ship a binary patch against QFJ until such time they
accept and include any fix you make into its code base
> [2] Change the way Synapse do the FIX transformation to XML infoset .
> - Current implementation access the qfj.Message and iterate through
> the data structures provided by qfj and build the AXIOM tree. My
> proposal is to use a simple way and it will cut down the 100+ code to
> 14 lines of code.
>
> This is the code snippet
> String fixToXML = message.toXML().replaceAll("\n","");
> InputStream is = new
> ByteArrayInputStream(fixToXML.getBytes());
> XMLStreamReader parser =
> XMLInputFactory.newInstance().createXMLStreamReader(is);
> StAXOMBuilder builder = new StAXOMBuilder(parser);
> OMElement documentElement = builder.getDocumentElement();
> SOAPFactory soapFactory = new SOAP11Factory();
>
> documentElement.addAttribute(soapFactory.createOMAttribute(FIXConstants.FIX_MESSAGE_INCOMING_SESSION,
> null, sessionID));
> System.out.println (documentElement.toString());
> SOAPEnvelope envelope = soapFactory.getDefaultEnvelope();
> envelope.getBody().addChild(documentElement);
> msgCtx.setEnvelope(envelope);
>
> This will basically create the AXIOM tree using the qfj.Message.toXML().
> Few changes will apply for the current implementation
> 1. Will not have CDATA section in the xml infoset
> 2. Attribute "id" in the element "Field" will rename as "tag"
> 3. Will not check binary content (FIX is not made to send binary data,
> and from my past experience with the FIX users none of them sent
> binary with FIX , FAST is there for binary transports for financial
> messaging)
>
> I did apply this change to my local workspace and it is working
> perfectly with the FIX scenarios we have and it supports ***repeating
> groups*** as well.
> WDYT ?
How large can a FIX message get? Can it contain binary? attachments etc?
This will let us determine if reading the message into a String is
appropriate. For example a JMS Text message can be read into a String,
as always its also loaded into memory.
asankha
--
Asankha C. Perera
WSO2 - http://wso2.org
http://esbmagic.blogspot.com
Re: [jira] Created: (SYNAPSE-420) FIX4.4 repeting grouping
Posted by Andreas Veithen <an...@skynet.be>.
Concerning the CDATA question, you might have a look at SYNAPSE-280.
Andreas
On 15 août 08, at 11:54, Hiranya Jayathilaka wrote:
> Hi,
>
> When the FIX transport was first implemented we considered the idea
> of using the Message#toXML method in the QFJ API to convert FIX
> messages into XML. But since this method only returns a String,
> formatted as XML and since according to the spec a user 'might' send
> binary data over FIX we decided to give up on that idea and went on
> to use the AXIOM API to construct the XML infoset for FIX messages.
>
> If the users are more interested in using repeating groups than
> sending binary data over FIX then I think it makes sense to make the
> suggested change. Eventhough this is a critical design decision it
> can be implemented with minimal code changes as Asanka has
> mentioned. (Only a couple of methods in the FIXUtils class need to
> be changed)
>
> Anyway I don't understand how the CDATA sections in the XML infoset
> get lost. I think the string retuned by the toXML method contains
> CDATA sections. Can you explain a bit how that happens? Also how
> costly is it to fix this issue in QFJ side and use a data dictionary
> based approach?
>
> Thanks
>
> Best Regards,
> Hiranya
>
>
>
> However I don't understand how the CDATA sections get lost. I think
> the XML string returned by toXML contains CDATA sections.
>
> On Fri, Aug 15, 2008 at 12:45 PM, Asanka Abeysinghe
> <as...@wso2.com> wrote:
> Hi ,
> I didn't receive any response yet from QFJ guys. This is a critical
> bug because without repeating group support we cannot say Synapse
> supports FIX messaging, almost every FIX user use repeating group
> even a small broker firm.
> So we have two options
> [1] Fix the bug in QFJ side and use data dictionary to identify the
> repeating groups. Accessing the DD for each FIX message will effect
> our message processing time because I noticed that API calls for the
> data dictionary is very slow.
> [2] Change the way Synapse do the FIX transformation to XML
> infoset . - Current implementation access the qfj.Message and
> iterate through the data structures provided by qfj and build the
> AXIOM tree. My proposal is to use a simple way and it will cut down
> the 100+ code to 14 lines of code.
>
> This is the code snippet
> String fixToXML = message.toXML().replaceAll("\n","");
> InputStream is = new
> ByteArrayInputStream(fixToXML.getBytes());
> XMLStreamReader parser =
> XMLInputFactory.newInstance().createXMLStreamReader(is);
> StAXOMBuilder builder = new StAXOMBuilder(parser);
> OMElement documentElement = builder.getDocumentElement();
> SOAPFactory soapFactory = new SOAP11Factory();
>
> documentElement
> .addAttribute
> (soapFactory
> .createOMAttribute(FIXConstants.FIX_MESSAGE_INCOMING_SESSION, null,
> sessionID));
> System.out.println (documentElement.toString());
> SOAPEnvelope envelope = soapFactory.getDefaultEnvelope();
> envelope.getBody().addChild(documentElement);
> msgCtx.setEnvelope(envelope);
>
> This will basically create the AXIOM tree using the
> qfj.Message.toXML().
> Few changes will apply for the current implementation
> 1. Will not have CDATA section in the xml infoset
> 2. Attribute "id" in the element "Field" will rename as "tag"
> 3. Will not check binary content (FIX is not made to send binary
> data, and from my past experience with the FIX users none of them
> sent binary with FIX , FAST is there for binary transports for
> financial messaging)
>
> I did apply this change to my local workspace and it is working
> perfectly with the FIX scenarios we have and it supports **repeating
> groups** as well.
> WDYT ?
> If every one is happy with this change I can provide the relevant
> patch.
> Asanka A.
>
>
>> There is a blocker in the QFJ side on accessing repeating groups, I
>> have created the relevant issues on QFJ issue list (Jiras -
>> QFj-330, QFj-329), we have to wait for the QFj devs comments to
>> proceed.
>> Asanka A.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
>> For additional commands, e-mail: dev-help@synapse.apache.org
>>
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
Re: [jira] Created: (SYNAPSE-420) FIX4.4 repeting grouping
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
Hi,
When the FIX transport was first implemented we considered the idea of using
the Message#toXML method in the QFJ API to convert FIX messages into XML.
But since this method only returns a String, formatted as XML and since
according to the spec a user 'might' send binary data over FIX we decided to
give up on that idea and went on to use the AXIOM API to construct the XML
infoset for FIX messages.
If the users are more interested in using repeating groups than sending
binary data over FIX then I think it makes sense to make the suggested
change. Eventhough this is a critical design decision it can be implemented
with minimal code changes as Asanka has mentioned. (Only a couple of methods
in the FIXUtils class need to be changed)
Anyway I don't understand how the CDATA sections in the XML infoset get
lost. I think the string retuned by the toXML method contains CDATA
sections. Can you explain a bit how that happens? Also how costly is it to
fix this issue in QFJ side and use a data dictionary based approach?
Thanks
Best Regards,
Hiranya
However I don't understand how the CDATA sections get lost. I think the XML
string returned by toXML contains CDATA sections.
On Fri, Aug 15, 2008 at 12:45 PM, Asanka Abeysinghe <as...@wso2.com>wrote:
> Hi ,
> I didn't receive any response yet from QFJ guys. This is a critical bug
> because without repeating group support we cannot say Synapse supports FIX
> messaging, almost every FIX user use repeating group even a small broker
> firm.
> So we have two options
> [1] Fix the bug in QFJ side and use data dictionary to identify the
> repeating groups. Accessing the DD for each FIX message will effect our
> message processing time because I noticed that API calls for the data
> dictionary is very slow.
> [2] Change the way Synapse do the FIX transformation to XML infoset . -
> Current implementation access the qfj.Message and iterate through the data
> structures provided by qfj and build the AXIOM tree. My proposal is to use a
> simple way and it will cut down the 100+ code to 14 lines of code.
>
> This is the code snippet
> String fixToXML = message.toXML().replaceAll("\n","");
> InputStream is = new ByteArrayInputStream(fixToXML.getBytes());
> XMLStreamReader parser =
> XMLInputFactory.newInstance().createXMLStreamReader(is);
> StAXOMBuilder builder = new StAXOMBuilder(parser);
> OMElement documentElement = builder.getDocumentElement();
> SOAPFactory soapFactory = new SOAP11Factory();
>
> documentElement.addAttribute(soapFactory.createOMAttribute(FIXConstants.FIX_MESSAGE_INCOMING_SESSION,
> null, sessionID));
> System.out.println (documentElement.toString());
> SOAPEnvelope envelope = soapFactory.getDefaultEnvelope();
> envelope.getBody().addChild(documentElement);
> msgCtx.setEnvelope(envelope);
>
> This will basically create the AXIOM tree using the qfj.Message.toXML().
> Few changes will apply for the current implementation
> 1. Will not have CDATA section in the xml infoset
> 2. Attribute "id" in the element "Field" will rename as "tag"
> 3. Will not check binary content (FIX is not made to send binary data, and
> from my past experience with the FIX users none of them sent binary with FIX
> , FAST is there for binary transports for financial messaging)
>
> I did apply this change to my local workspace and it is working perfectly
> with the FIX scenarios we have and it supports ***repeating groups*** as
> well.
> WDYT ?
> If every one is happy with this change I can provide the relevant patch.
> Asanka A.
>
>
> There is a blocker in the QFJ side on accessing repeating groups, I have
> created the relevant issues on QFJ issue list (Jiras - QFj-330, QFj-329), we
> have to wait for the QFj devs comments to proceed.
> Asanka A.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
> For additional commands, e-mail: dev-help@synapse.apache.org
>
>
>
>
Re: [jira] Created: (SYNAPSE-420) FIX4.4 repeting grouping
Posted by Asanka Abeysinghe <as...@wso2.com>.
Hi ,
I didn't receive any response yet from QFJ guys. This is a critical bug
because without repeating group support we cannot say Synapse supports
FIX messaging, almost every FIX user use repeating group even a small
broker firm.
So we have two options
[1] Fix the bug in QFJ side and use data dictionary to identify the
repeating groups. Accessing the DD for each FIX message will effect our
message processing time because I noticed that API calls for the data
dictionary is very slow.
[2] Change the way Synapse do the FIX transformation to XML infoset . -
Current implementation access the qfj.Message and iterate through the
data structures provided by qfj and build the AXIOM tree. My proposal is
to use a simple way and it will cut down the 100+ code to 14 lines of code.
This is the code snippet
String fixToXML = message.toXML().replaceAll("\n","");
InputStream is = new ByteArrayInputStream(fixToXML.getBytes());
XMLStreamReader parser =
XMLInputFactory.newInstance().createXMLStreamReader(is);
StAXOMBuilder builder = new StAXOMBuilder(parser);
OMElement documentElement = builder.getDocumentElement();
SOAPFactory soapFactory = new SOAP11Factory();
documentElement.addAttribute(soapFactory.createOMAttribute(FIXConstants.FIX_MESSAGE_INCOMING_SESSION,
null, sessionID));
System.out.println (documentElement.toString());
SOAPEnvelope envelope = soapFactory.getDefaultEnvelope();
envelope.getBody().addChild(documentElement);
msgCtx.setEnvelope(envelope);
This will basically create the AXIOM tree using the qfj.Message.toXML().
Few changes will apply for the current implementation
1. Will not have CDATA section in the xml infoset
2. Attribute "id" in the element "Field" will rename as "tag"
3. Will not check binary content (FIX is not made to send binary data,
and from my past experience with the FIX users none of them sent binary
with FIX , FAST is there for binary transports for financial messaging)
I did apply this change to my local workspace and it is working
perfectly with the FIX scenarios we have and it supports ***repeating
groups*** as well.
WDYT ?
If every one is happy with this change I can provide the relevant patch.
Asanka A.
> There is a blocker in the QFJ side on accessing repeating groups, I
> have created the relevant issues on QFJ issue list (Jiras - QFj-330,
> QFj-329), we have to wait for the QFj devs comments to proceed.
> Asanka A.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
> For additional commands, e-mail: dev-help@synapse.apache.org
>
>
Re: [jira] Created: (SYNAPSE-420) FIX4.4 repeting grouping
Posted by Asanka Abeysinghe <as...@wso2.com>.
>
There is a blocker in the QFJ side on accessing repeating groups, I have
created the relevant issues on QFJ issue list (Jiras - QFj-330,
QFj-329), we have to wait for the QFj devs comments to proceed.
Asanka A.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
[jira] Resolved: (SYNAPSE-420) FIX4.4 repeting grouping
Posted by "Hiranya Jayathilaka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SYNAPSE-420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hiranya Jayathilaka resolved SYNAPSE-420.
-----------------------------------------
Resolution: Fixed
Fix Version/s: NIGHTLY
Assignee: Hiranya Jayathilaka
Committed the patch from Asanka A. Fixed on SVN trunk.
> FIX4.4 repeting grouping
> ------------------------
>
> Key: SYNAPSE-420
> URL: https://issues.apache.org/jira/browse/SYNAPSE-420
> Project: Synapse
> Issue Type: Bug
> Components: Transports
> Reporter: Cislaghi Gianni
> Assignee: Hiranya Jayathilaka
> Fix For: NIGHTLY
>
> Attachments: SYNAPSE-420.patch
>
>
> Synapse 1.2 FIX transport layer does not support FIX repeating groups.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
[jira] Updated: (SYNAPSE-420) FIX4.4 repeting grouping
Posted by "Asanka Abeysinghe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SYNAPSE-420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Asanka Abeysinghe updated SYNAPSE-420:
--------------------------------------
Attachment: SYNAPSE-420.patch
Attached the patch that resolve the above issue.
Using reflections to access the private members of the QFJ library (as Andreas recommended) , put the relevant comments to do a code cleanup once QFJ-330 get committed to the code base.
With this patch Synapse enables FIX repeating groups and it works with previous FIX scenarios.
Changes Made:
modules/transports/src/main/java/org/apache/synapse/transport/fix/FIXConstants.java - New constants added for groups
modules/transports/src/main/java/org/apache/synapse/transport/fix/FIXUtils.java - new routines added to create and extract repeating groups
> FIX4.4 repeting grouping
> ------------------------
>
> Key: SYNAPSE-420
> URL: https://issues.apache.org/jira/browse/SYNAPSE-420
> Project: Synapse
> Issue Type: Bug
> Components: Transports
> Reporter: Cislaghi Gianni
> Attachments: SYNAPSE-420.patch
>
>
> Synapse 1.2 FIX transport layer does not support FIX repeating groups.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org