Apache Camel Routing: Best practice how to poll an external REST API

[simon_dietschi <si...@siemens.com>]

Hi Camel folks,

I am currently looking for a simple approach achieving the following in
Apache Camel / Spring Boot:

1. Send a search job request to an external REST/JSON API
2. Get a search processing job ID from the API
3. Poll the API until it returns a JSON response containing a specific
status = done
4. Once status done has been returned, proceed with the retrieval of the
search result (separate part, considered to be straight forward)

The first solution one may think of is to implement the Polling Consumer
pattern, which in my opinion requires too much overhead code (need to
implement Component, Endpoint, Consumer,...) and more important this pattern
does the contrary - it returns a message once a message is available (e.g.
https://github.com/Palo-IT/Examples/tree/master/camel-custom-feedly-component).

Another approach would be a simple Processor where I loop a defined number
of times calling the API or using the DSL -> doWhile...

As I was unable to find a good code snippet, it would be cool if you guys
can share your thoughts or point me to a good sample.

Cheers,
Simon



--
View this message in context: http://camel.465427.n5.nabble.com/Apache-Camel-Routing-Best-practice-how-to-poll-an-external-REST-API-tp5809926.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Apache Camel Routing: Best practice how to poll an external REST API

[simon_dietschi <si...@siemens.com>]

Hi shuston,

Thanks for your suggestion using multiple queues as another approach for
polling the results.

In the meantime I ended up implementing some sort of combined future
pattern, where I keep polling by simply calling the endpoint; something
similar to this example here:

https://stackoverflow.com/questions/40251528/how-to-use-executorservice-to-poll-until-a-result-arrives

Cheers,
Simon



--
View this message in context: http://camel.465427.n5.nabble.com/Apache-Camel-Routing-Best-practice-how-to-poll-an-external-REST-API-tp5809926p5810172.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Apache Camel Routing: Best practice how to poll an external REST API

[Steve Huston <sh...@riverace.com>]

I don’t have example code readily available, but one possible approach would be:

- Send job request REST/JSON and get job ID back
- Add ID to a queue of not-yet-done jobs
- A separate route pulls job IDs from the not-done queue and polls
    - If not done, put it back on the queue
    - If done, put ID on a different queue for done jobs
- A third route pulls job IDs from the done queue and retrieves the search result

You’d want some delay or use a timer in combination with the not-done job queue.

-Steve

> On Aug 8, 2017, at 4:12 AM, simon_dietschi <si...@siemens.com> wrote:
> 
> Hi Camel folks,
> 
> I am currently looking for a simple approach achieving the following in
> Apache Camel / Spring Boot:
> 
> 1. Send a search job request to an external REST/JSON API
> 2. Get a search processing job ID from the API
> 3. Poll the API until it returns a JSON response containing a specific
> status = done
> 4. Once status done has been returned, proceed with the retrieval of the
> search result (separate part, considered to be straight forward)
> 
> The first solution one may think of is to implement the Polling Consumer
> pattern, which in my opinion requires too much overhead code (need to
> implement Component, Endpoint, Consumer,...) and more important this pattern
> does the contrary - it returns a message once a message is available (e.g.
> https://github.com/Palo-IT/Examples/tree/master/camel-custom-feedly-component).
> 
> Another approach would be a simple Processor where I loop a defined number
> of times calling the API or using the DSL -> doWhile...
> 
> As I was unable to find a good code snippet, it would be cool if you guys
> can share your thoughts or point me to a good sample.
> 
> Cheers,
> Simon
> 
> 
> 
> --
> View this message in context: http://camel.465427.n5.nabble.com/Apache-Camel-Routing-Best-practice-how-to-poll-an-external-REST-API-tp5809926.html
> Sent from the Camel - Users mailing list archive at Nabble.com.