You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by "Romain Manni-Bucau (JIRA)" <ji...@apache.org> on 2011/05/19 23:18:47 UTC
[jira] [Created] (CAMEL-3998) improve stax/jax components
improve stax/jax components
---------------------------
Key: CAMEL-3998
URL: https://issues.apache.org/jira/browse/CAMEL-3998
Project: Camel
Issue Type: Improvement
Components: camel-core
Reporter: Romain Manni-Bucau
Priority: Minor
it could be pretty cool to have an iterator to be able to iterate over a filtered tree of a xml document avoiding to load all the document in memory.
it can be nice to do it loading sub objects using jaxb.
i did a version here: http://code.google.com/p/rmannibucau/source/browse/camel/camel-stax/src/main/java/org/apache/camel/stax/expression/StAXJAXBIteratorExpression.java (i added some converters to help too).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-3998) improve stax/jax components
Posted by "Willem Jiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051766#comment-13051766 ]
Willem Jiang commented on CAMEL-3998:
-------------------------------------
It sounds great, and we love contribution and share our experiences. It is how the Open Source works :)
> improve stax/jax components
> ---------------------------
>
> Key: CAMEL-3998
> URL: https://issues.apache.org/jira/browse/CAMEL-3998
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Reporter: Romain Manni-Bucau
> Assignee: Christian Müller
> Fix For: 2.9.0
>
>
> it could be pretty cool to have an iterator to be able to iterate over a filtered tree of a xml document avoiding to load all the document in memory.
> it can be nice to do it loading sub objects using jaxb.
> i did a version here: http://code.google.com/p/rmannibucau/source/browse/camel/camel-stax/src/main/java/org/apache/camel/stax/expression/StAXJAXBIteratorExpression.java (i added some converters to help too).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-3998) improve stax/jax components
Posted by "Claus Ibsen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139589#comment-13139589 ]
Claus Ibsen commented on CAMEL-3998:
------------------------------------
I like the StAXBuilder idea.
> improve stax/jax components
> ---------------------------
>
> Key: CAMEL-3998
> URL: https://issues.apache.org/jira/browse/CAMEL-3998
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Reporter: Romain Manni-Bucau
> Assignee: Christian Müller
> Fix For: Future
>
>
> it could be pretty cool to have an iterator to be able to iterate over a filtered tree of a xml document avoiding to load all the document in memory.
> it can be nice to do it loading sub objects using jaxb.
> i did a version here: http://code.google.com/p/rmannibucau/source/browse/camel/camel-stax/src/main/java/org/apache/camel/stax/expression/StAXJAXBIteratorExpression.java (i added some converters to help too).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-3998) improve stax/jax components
Posted by "Christian Müller (Commented JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139740#comment-13139740 ]
Christian Müller commented on CAMEL-3998:
-----------------------------------------
I would like to see this in Camel 2.9.0. At present, it's one of the most important features I miss in Camel (working with very large XML files).
Unfortunately I did had enough time to work on it.
Claus, feel free to assign this issue to yourself if you would like to work on it.
> improve stax/jax components
> ---------------------------
>
> Key: CAMEL-3998
> URL: https://issues.apache.org/jira/browse/CAMEL-3998
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Reporter: Romain Manni-Bucau
> Assignee: Christian Müller
> Fix For: Future
>
>
> it could be pretty cool to have an iterator to be able to iterate over a filtered tree of a xml document avoiding to load all the document in memory.
> it can be nice to do it loading sub objects using jaxb.
> i did a version here: http://code.google.com/p/rmannibucau/source/browse/camel/camel-stax/src/main/java/org/apache/camel/stax/expression/StAXJAXBIteratorExpression.java (i added some converters to help too).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CAMEL-3998) improve stax/jax components
Posted by "Christian Müller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Müller reassigned CAMEL-3998:
---------------------------------------
Assignee: Christian Müller
> improve stax/jax components
> ---------------------------
>
> Key: CAMEL-3998
> URL: https://issues.apache.org/jira/browse/CAMEL-3998
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Reporter: Romain Manni-Bucau
> Assignee: Christian Müller
> Priority: Minor
>
> it could be pretty cool to have an iterator to be able to iterate over a filtered tree of a xml document avoiding to load all the document in memory.
> it can be nice to do it loading sub objects using jaxb.
> i did a version here: http://code.google.com/p/rmannibucau/source/browse/camel/camel-stax/src/main/java/org/apache/camel/stax/expression/StAXJAXBIteratorExpression.java (i added some converters to help too).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-3998) improve stax/jax components
Posted by "Claus Ibsen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139588#comment-13139588 ]
Claus Ibsen commented on CAMEL-3998:
------------------------------------
There is a token based solution now
http://camel.465427.n5.nabble.com/HEADS-UP-Splitting-big-XML-files-using-XPath-tp4949834p4949834.html
> improve stax/jax components
> ---------------------------
>
> Key: CAMEL-3998
> URL: https://issues.apache.org/jira/browse/CAMEL-3998
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Reporter: Romain Manni-Bucau
> Assignee: Christian Müller
> Fix For: Future
>
>
> it could be pretty cool to have an iterator to be able to iterate over a filtered tree of a xml document avoiding to load all the document in memory.
> it can be nice to do it loading sub objects using jaxb.
> i did a version here: http://code.google.com/p/rmannibucau/source/browse/camel/camel-stax/src/main/java/org/apache/camel/stax/expression/StAXJAXBIteratorExpression.java (i added some converters to help too).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CAMEL-3998) improve stax/jax
components
Posted by "Christian Müller (Issue Comment Edited JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139740#comment-13139740 ]
Christian Müller edited comment on CAMEL-3998 at 10/30/11 8:24 PM:
-------------------------------------------------------------------
I would like to see this in Camel 2.9.0. At present, it's one of the most important features I miss in Camel (working with very large XML files).
Unfortunately I didn't had enough time to work on it.
Claus, feel free to assign this issue to yourself if you would like to work on it.
was (Author: muellerc):
I would like to see this in Camel 2.9.0. At present, it's one of the most important features I miss in Camel (working with very large XML files).
Unfortunately I did had enough time to work on it.
Claus, feel free to assign this issue to yourself if you would like to work on it.
> improve stax/jax components
> ---------------------------
>
> Key: CAMEL-3998
> URL: https://issues.apache.org/jira/browse/CAMEL-3998
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Reporter: Romain Manni-Bucau
> Assignee: Christian Müller
> Fix For: Future
>
>
> it could be pretty cool to have an iterator to be able to iterate over a filtered tree of a xml document avoiding to load all the document in memory.
> it can be nice to do it loading sub objects using jaxb.
> i did a version here: http://code.google.com/p/rmannibucau/source/browse/camel/camel-stax/src/main/java/org/apache/camel/stax/expression/StAXJAXBIteratorExpression.java (i added some converters to help too).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-3998) improve stax/jax components
Posted by "Christian Müller (Commented JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139738#comment-13139738 ]
Christian Müller commented on CAMEL-3998:
-----------------------------------------
In my opinion, we should also support the following:
{code:java}
from("direct:start")
.choice()
.when().stax("//foo/bar[@name='Max']")
.to("..")
.when().stax("//foo/bar[@name='Moritz']")
.to("..")
{code}
or
{code:java}
from("direct:start")
.choice()
.when(stax("//foo/bar[@name='Max']"))
.to("..")
.when(stax("//foo/bar[@name='Moritz']"))
.to("..")
{code}
and
{code:java}
from("direct:start")
.filter().stax("//foo/bar[@name='Max']")
.to("..")
{code}
or
{code:java}
from("direct:start")
.filter(stax("//foo/bar[@name='Max']"))
.to("..")
{code}
> improve stax/jax components
> ---------------------------
>
> Key: CAMEL-3998
> URL: https://issues.apache.org/jira/browse/CAMEL-3998
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Reporter: Romain Manni-Bucau
> Assignee: Christian Müller
> Fix For: Future
>
>
> it could be pretty cool to have an iterator to be able to iterate over a filtered tree of a xml document avoiding to load all the document in memory.
> it can be nice to do it loading sub objects using jaxb.
> i did a version here: http://code.google.com/p/rmannibucau/source/browse/camel/camel-stax/src/main/java/org/apache/camel/stax/expression/StAXJAXBIteratorExpression.java (i added some converters to help too).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CAMEL-3998) improve stax/jax components
Posted by "Christian Müller (Updated JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Müller updated CAMEL-3998:
------------------------------------
Fix Version/s: (was: 2.9.0)
Future
> improve stax/jax components
> ---------------------------
>
> Key: CAMEL-3998
> URL: https://issues.apache.org/jira/browse/CAMEL-3998
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Reporter: Romain Manni-Bucau
> Assignee: Christian Müller
> Fix For: Future
>
>
> it could be pretty cool to have an iterator to be able to iterate over a filtered tree of a xml document avoiding to load all the document in memory.
> it can be nice to do it loading sub objects using jaxb.
> i did a version here: http://code.google.com/p/rmannibucau/source/browse/camel/camel-stax/src/main/java/org/apache/camel/stax/expression/StAXJAXBIteratorExpression.java (i added some converters to help too).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CAMEL-3998) improve stax/jax components
Posted by "Christian Müller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Müller updated CAMEL-3998:
------------------------------------
Priority: Major (was: Minor)
Fix Version/s: 2.9.0
I increased the priority of this issue because at present we do not have any "out of the box" solution to split very large XML files.
> improve stax/jax components
> ---------------------------
>
> Key: CAMEL-3998
> URL: https://issues.apache.org/jira/browse/CAMEL-3998
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Reporter: Romain Manni-Bucau
> Assignee: Christian Müller
> Fix For: 2.9.0
>
>
> it could be pretty cool to have an iterator to be able to iterate over a filtered tree of a xml document avoiding to load all the document in memory.
> it can be nice to do it loading sub objects using jaxb.
> i did a version here: http://code.google.com/p/rmannibucau/source/browse/camel/camel-stax/src/main/java/org/apache/camel/stax/expression/StAXJAXBIteratorExpression.java (i added some converters to help too).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-3998) improve stax/jax components
Posted by "Christian Müller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051697#comment-13051697 ]
Christian Müller commented on CAMEL-3998:
-----------------------------------------
In one of our projects, we need the ability to split a large XML file (3 - 10 GB) into its individual records. Because XPath reads the entire document into memory, it's not an option for us. For the time being, we will develop our own custom Splitter which use internally StAX to stream the file and return an Iterator. For each call of hasNext()/next() on the Iterator interface, we will read and return the next record.
I think we will share this solution so that in Camel 2.9.0, the user should be able to do something like (in the same way as the xpath support):
{code:java}
from("direct:start")
.split(stax("//foo/bar"))
.convertBodyTo(String.class) // may be not necessary, depending on the stax response
.to("mock:result")
{code}
The following should also be possible:
{code:java}
StaxBuilder staxBuilder = new StaxBuilder("//foo/bar");
from("direct:start")
.split(staxBuilder)
.convertBodyTo(String.class) // may be not necessary, depending on the stax response
.to("mock:result")
{code}
To support namespaces, the following should be implemented:
{code:java}
Namespaces ns = new Namespaces("c", "http://acme.com/cheese");
from("direct:start")
.split(stax("//foo/bar", ns))
.convertBodyTo(String.class) // may be not necessary, depending on the stax response
.to("mock:result")
{code}
and also:
{code:java}
Namespaces ns = new Namespaces("c", "http://acme.com/cheese");
StaxBuilder staxBuilder = new StaxBuilder("//foo/bar");
staxBuilder.setNamespaces(ns.getNamespaces());
from("direct:start")
.split(staxBuilder)
.convertBodyTo(String.class) // may be not necessary, depending on the stax response
.to("mock:result")
{code}
What do you think?
> improve stax/jax components
> ---------------------------
>
> Key: CAMEL-3998
> URL: https://issues.apache.org/jira/browse/CAMEL-3998
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Reporter: Romain Manni-Bucau
> Priority: Minor
>
> it could be pretty cool to have an iterator to be able to iterate over a filtered tree of a xml document avoiding to load all the document in memory.
> it can be nice to do it loading sub objects using jaxb.
> i did a version here: http://code.google.com/p/rmannibucau/source/browse/camel/camel-stax/src/main/java/org/apache/camel/stax/expression/StAXJAXBIteratorExpression.java (i added some converters to help too).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira