You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Bob Jolliffe <bo...@gmail.com> on 2012/07/08 15:40:38 UTC
looping http fetches of paginated data
I've got a problem which I can't quite figure out how to solve. I
want to configure a route which fetches xml data (representing health
facilities) from a repository using a REST http get, transform the
data and load into another system. Typically triggered from a quartz
endpoint running every 24 hours.
I have found that this is quite straightforward using quartz, http and
xslt components. But ... the data coming from the repository is
paginated so I can't fetch it in a single request.
Is there anyway to create a route in camel DSL (xml or java) which
makes a series of requests to the http service, incrementing an http
parameter eg. Page=xx, until some test condition is true in the xml
result eg. count('/Facilities/Facility')=0
Thanks for any pointers.
Regards
Bob
Re: looping http fetches of paginated data
Posted by Claus Ibsen <cl...@gmail.com>.
On Mon, Jul 9, 2012 at 1:11 PM, Pontus Ullgren <ul...@gmail.com> wrote:
> Just realised that you did not have the number of pages that you
> needed to fetch before hand but wanted to loop based on a condition.
> Then loop will not help you.
>
Yeah in the future we will improve loop to act like a while where you
can evaluate an expression.
> I would go with creating your own processor that does all the collection.
>
Yeah that would be fine. Or use the dynamic router EIP, where you can just keep
routing back to the http.
http://camel.apache.org/dynamic-router.html
> // Pontus
>
> On Mon, Jul 9, 2012 at 1:07 PM, Pontus Ullgren <ul...@gmail.com> wrote:
>> Hello,
>>
>> You can use loop (http://camel.apache.org/loop.html) and storing
>> information from each request in the exchange.
>>
>> However I personally like to keep my camel routes as simple as
>> possible and avoid building logic such as this into the routes.
>> So I would recommend you to create your own processor that does all
>> the data collection and then enriches the message with the complete
>> message.
>>
>> // Pontus
>>
>>
>>
>> On Sun, Jul 8, 2012 at 3:40 PM, Bob Jolliffe <bo...@gmail.com> wrote:
>>> I've got a problem which I can't quite figure out how to solve. I
>>> want to configure a route which fetches xml data (representing health
>>> facilities) from a repository using a REST http get, transform the
>>> data and load into another system. Typically triggered from a quartz
>>> endpoint running every 24 hours.
>>>
>>> I have found that this is quite straightforward using quartz, http and
>>> xslt components. But ... the data coming from the repository is
>>> paginated so I can't fetch it in a single request.
>>>
>>> Is there anyway to create a route in camel DSL (xml or java) which
>>> makes a series of requests to the http service, incrementing an http
>>> parameter eg. Page=xx, until some test condition is true in the xml
>>> result eg. count('/Facilities/Facility')=0
>>>
>>> Thanks for any pointers.
>>>
>>> Regards
>>> Bob
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: looping http fetches of paginated data
Posted by Pontus Ullgren <ul...@gmail.com>.
Just realised that you did not have the number of pages that you
needed to fetch before hand but wanted to loop based on a condition.
Then loop will not help you.
I would go with creating your own processor that does all the collection.
// Pontus
On Mon, Jul 9, 2012 at 1:07 PM, Pontus Ullgren <ul...@gmail.com> wrote:
> Hello,
>
> You can use loop (http://camel.apache.org/loop.html) and storing
> information from each request in the exchange.
>
> However I personally like to keep my camel routes as simple as
> possible and avoid building logic such as this into the routes.
> So I would recommend you to create your own processor that does all
> the data collection and then enriches the message with the complete
> message.
>
> // Pontus
>
>
>
> On Sun, Jul 8, 2012 at 3:40 PM, Bob Jolliffe <bo...@gmail.com> wrote:
>> I've got a problem which I can't quite figure out how to solve. I
>> want to configure a route which fetches xml data (representing health
>> facilities) from a repository using a REST http get, transform the
>> data and load into another system. Typically triggered from a quartz
>> endpoint running every 24 hours.
>>
>> I have found that this is quite straightforward using quartz, http and
>> xslt components. But ... the data coming from the repository is
>> paginated so I can't fetch it in a single request.
>>
>> Is there anyway to create a route in camel DSL (xml or java) which
>> makes a series of requests to the http service, incrementing an http
>> parameter eg. Page=xx, until some test condition is true in the xml
>> result eg. count('/Facilities/Facility')=0
>>
>> Thanks for any pointers.
>>
>> Regards
>> Bob
Re: looping http fetches of paginated data
Posted by Pontus Ullgren <ul...@gmail.com>.
Hello,
You can use loop (http://camel.apache.org/loop.html) and storing
information from each request in the exchange.
However I personally like to keep my camel routes as simple as
possible and avoid building logic such as this into the routes.
So I would recommend you to create your own processor that does all
the data collection and then enriches the message with the complete
message.
// Pontus
On Sun, Jul 8, 2012 at 3:40 PM, Bob Jolliffe <bo...@gmail.com> wrote:
> I've got a problem which I can't quite figure out how to solve. I
> want to configure a route which fetches xml data (representing health
> facilities) from a repository using a REST http get, transform the
> data and load into another system. Typically triggered from a quartz
> endpoint running every 24 hours.
>
> I have found that this is quite straightforward using quartz, http and
> xslt components. But ... the data coming from the repository is
> paginated so I can't fetch it in a single request.
>
> Is there anyway to create a route in camel DSL (xml or java) which
> makes a series of requests to the http service, incrementing an http
> parameter eg. Page=xx, until some test condition is true in the xml
> result eg. count('/Facilities/Facility')=0
>
> Thanks for any pointers.
>
> Regards
> Bob
Re: looping http fetches of paginated data
Posted by "Sam (Stephen Samuel)" <sa...@gmail.com>.
What I would do is have a queue that triggers a HTTP fetch. The message on
this queue contains the pagination info, eg the start and limit. After each
request, it can then put another message back on the queue with updated
pagination info if there is more left to process. The initial quartz job
can put the "start" message onto the queue to begin the process. This queue
doesn't need to be anything fancy, it could just be a seda
http://camel.apache.org/seda.html
On Sun, Jul 8, 2012 at 2:40 PM, Bob Jolliffe <bo...@gmail.com> wrote:
> I've got a problem which I can't quite figure out how to solve. I
> want to configure a route which fetches xml data (representing health
> facilities) from a repository using a REST http get, transform the
> data and load into another system. Typically triggered from a quartz
> endpoint running every 24 hours.
>
> I have found that this is quite straightforward using quartz, http and
> xslt components. But ... the data coming from the repository is
> paginated so I can't fetch it in a single request.
>
> Is there anyway to create a route in camel DSL (xml or java) which
> makes a series of requests to the http service, incrementing an http
> parameter eg. Page=xx, until some test condition is true in the xml
> result eg. count('/Facilities/Facility')=0
>
> Thanks for any pointers.
>
> Regards
> Bob
>
--
-Sam