You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Claus Ibsen <cl...@gmail.com> on 2011/12/16 05:33:24 UTC
Re: .split(beanExpression(new MyCustomIteratorFactory(), "iterator"))
Hi
You can look at this example instead from the same wiki page - Using a
Pojo to do the splitting
Then in the POJO, you can return an Iterator instead of a List, in
case you want to do the splitting on-demand in a streaming fashion.
This avoids preparing all the messages up-front, and thus do not eat up memory.
For example its the same principle used by the splitting big XML
files, we added in Camel 2.9, using the tokenizers.
http://davsclaus.blogspot.com/2011/11/splitting-big-xml-files-with-apache.html
Or the stax component
http://davsclaus.blogspot.com/2011/11/splitting-big-xml-files-with-apache_24.html
On Thu, Dec 15, 2011 at 9:23 PM, marks1900 <ma...@yahoo.com.au> wrote:
> On the following page: http://camel.apache.org/splitter.html , there is an
> excerpt that I have questions about.
>
> ++ Excerpt - Start ++
>
> import static org.apache.camel.builder.ExpressionBuilder.beanExpression;
> from("direct:streaming")
> .split(beanExpression(new MyCustomIteratorFactory(), "iterator"))
> .streaming().to("activemq:my.parts")
>
> ++ Excerpt - End ++
>
> Can someone provided me with same code for the MyCustomIteratorFactory
> class?
>
> Basically I have one huge file, that I want to split into many jms messages.
>
> The following is an outline of my current thoughts...
>
> public void configure() throws Exception
> {
> from(
> "file://file-item-list/general?initialDelay=1000&delay=5000&delete=true"
> )
> .transacted( "PROPAGATION_REQUIRED" )
> .split(ExpressionBuilder.beanExpression(new
> MyCustomFileChunkIterator(), "iterator")) // split big file of items into
> smaller chunks
> .streaming()
> .process( generalPreprocessorBean ) // Reads in a smaller list
> and outputs a request to process the smaller subset of items
> .to("activemq:queue:file-item-sublist-request");
> }
>
>
>
> * Unfortunately I cannot subscribe to the mailing list as, so if someone can
> bounce this request to the mailing list that would be great.
>
> <us...@camel.apache.org>:
> Remote host said: 552 spam score (5.8) exceeded threshold
> (FREEMAIL_FORGED_REPLYTO,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,REPTO_QUOTE_YAHOO,SPF_NEUTRAL
> ) [BODY]
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/split-beanExpression-new-MyCustomIteratorFactory-iterator-tp5078583p5078583.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: .split(beanExpression(new MyCustomIteratorFactory(), "iterator"))
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Dec 16, 2011 at 8:16 AM, marks1900 <ma...@yahoo.com.au> wrote:
> Unfortunately, I am targeting a release of ServiceMix and I am not dealing
> with XML.
>
> ServiceMix 4.3 = Camel 2.6.0
> ServiceMix 4.4 = Camel 2.8.3
>
> The following is some sample code with some parts stripped and modified for
> this post (Warning: I have not tested the below code).
>
> Does it look like I am heading down the correct path?
>
Yes. I suggest to develop the iterator in pure Java, and create an
unit test to test that it works as expected.
The iterator does not need to depend on neither Camel, nor SMX.
>
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> // The Camel Route
>
> // The input files I am dealing with are standard .txt and .csv list dumps
> and not xml.
> public void configure() throws Exception
> {
> from(
> "file://file-item-list/general?initialDelay=1000&delay=5000&delete=true" )
> .to( "activemq:queue:my-queue-01" )
> .split( ExpressionBuilder.beanExpression(
> new MyFileIteratorFactory(), "createIterator" ) ) // split big file of
> items into smaller chunks
> .streaming()
> .to( "activemq:queue:my-queue-02" );
> }
>
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> public class MyFileIteratorFactory
> {
>
> int batchSize;
>
> public MyFileIteratorFactory()
> {
> this.batchSize = 10000;
> }
>
> public MyFileIteratorFactory( int batchSize )
> {
> this.batchSize = batchSize;
> }
>
> public Iterator<String> createIterator( final InputStream inputStream )
> {
> return new MyFileIterator( inputStream, batchSize );
> }
>
> /**
> * {@inheritDoc}
> */
> @Override
> public String toString()
> {
> return "MyFileIteratorFactory []";
> }
>
> }
>
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> public class MyFileIterator implements Iterator<String>
> {
>
>
>
> int batchSize;
>
> BufferedReader bufferedReader;
>
> String nextItemList;
>
>
> public MyFileIterator( final InputStream inputStream, int batchSize )
> {
>
> bufferedReader = new BufferedReader( new InputStreamReader(
> inputStream ) );
> this.batchSize = batchSize;
> tick();
>
> }
>
> @Override
> public boolean hasNext()
> {
> return nextItemList != null;
> }
>
> @Override
> public String next()
> {
>
> if ( null == nextItemList )
> {
> throw new NoSuchElementException();
> }
>
> String resultValue = nextItemList;
>
> tick();
>
> return resultValue;
> }
>
> @Override
> public void remove()
> {
> throw new UnsupportedOperationException();
> }
>
> protected void tick()
> {
> try
> {
> StringBuilder output = null;
> String currentLine;
> int currentLineItemCountInOutput = 0;
>
> readLoop: while ( ( currentLine = bufferedReader.readLine() )
> != null )
> {
>
> if ( null == output )
> {
> output = new StringBuilder();
> }
> else
> {
> output.append( "," );
> }
>
> output.append( MyFileTool.processLine( currentLine ) );
>
> currentLineItemCountInOutput =
> MyFileTool.getLineItemCountFromLineLength( output.length() );
> if ( currentLineItemCountInOutput >= batchSize )
> {
> break readLoop;
> }
> }
>
>
> if ( null == output )
> {
> nextItemList = null;
> bufferedReader.close();
> }
> else
> {
> nextItemList = output.toString();
> }
>
> }
> catch ( IOException e )
> {
> throw new IllegalArgumentException( e );
> }
> }
>
> }
>
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> public final class MyFileTool
> {
> // ...
> }
>
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/split-Iterator-streaming-tp5078583p5079537.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: .split(beanExpression(new MyCustomIteratorFactory(),
"iterator"))
Posted by marks1900 <ma...@yahoo.com.au>.
Unfortunately, I am targeting a release of ServiceMix and I am not dealing
with XML.
ServiceMix 4.3 = Camel 2.6.0
ServiceMix 4.4 = Camel 2.8.3
The following is some sample code with some parts stripped and modified for
this post (Warning: I have not tested the below code).
Does it look like I am heading down the correct path?
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
// The Camel Route
// The input files I am dealing with are standard .txt and .csv list dumps
and not xml.
public void configure() throws Exception
{
from(
"file://file-item-list/general?initialDelay=1000&delay=5000&delete=true" )
.to( "activemq:queue:my-queue-01" )
.split( ExpressionBuilder.beanExpression(
new MyFileIteratorFactory(), "createIterator" ) ) // split big file of
items into smaller chunks
.streaming()
.to( "activemq:queue:my-queue-02" );
}
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
public class MyFileIteratorFactory
{
int batchSize;
public MyFileIteratorFactory()
{
this.batchSize = 10000;
}
public MyFileIteratorFactory( int batchSize )
{
this.batchSize = batchSize;
}
public Iterator<String> createIterator( final InputStream inputStream )
{
return new MyFileIterator( inputStream, batchSize );
}
/**
* {@inheritDoc}
*/
@Override
public String toString()
{
return "MyFileIteratorFactory []";
}
}
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
public class MyFileIterator implements Iterator<String>
{
int batchSize;
BufferedReader bufferedReader;
String nextItemList;
public MyFileIterator( final InputStream inputStream, int batchSize )
{
bufferedReader = new BufferedReader( new InputStreamReader(
inputStream ) );
this.batchSize = batchSize;
tick();
}
@Override
public boolean hasNext()
{
return nextItemList != null;
}
@Override
public String next()
{
if ( null == nextItemList )
{
throw new NoSuchElementException();
}
String resultValue = nextItemList;
tick();
return resultValue;
}
@Override
public void remove()
{
throw new UnsupportedOperationException();
}
protected void tick()
{
try
{
StringBuilder output = null;
String currentLine;
int currentLineItemCountInOutput = 0;
readLoop: while ( ( currentLine = bufferedReader.readLine() )
!= null )
{
if ( null == output )
{
output = new StringBuilder();
}
else
{
output.append( "," );
}
output.append( MyFileTool.processLine( currentLine ) );
currentLineItemCountInOutput =
MyFileTool.getLineItemCountFromLineLength( output.length() );
if ( currentLineItemCountInOutput >= batchSize )
{
break readLoop;
}
}
if ( null == output )
{
nextItemList = null;
bufferedReader.close();
}
else
{
nextItemList = output.toString();
}
}
catch ( IOException e )
{
throw new IllegalArgumentException( e );
}
}
}
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
public final class MyFileTool
{
// ...
}
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--
View this message in context: http://camel.465427.n5.nabble.com/split-Iterator-streaming-tp5078583p5079537.html
Sent from the Camel - Users mailing list archive at Nabble.com.