Elasticsearch component - bulk requests with ids

[slush <de...@gmail.com>]

Hello! I'm new to the camel forums. First off, thank you to all the
contributors. Camel is a wonderful framework.

While doing some work recently I came across an issue with the elasticsearch
component. Specifically, I need to be able to send a bulk request where the
documents in the bulk request have application-provided (not
automatically-generated, as is what happens by default with the
elasticsearch-java client) document ids.

I see that the INDEX operation supports an index header as of version 2.15.0
(which is yet to be released). However, this still wouldn't satisfy my need,
as I want to send bulk updates, not individual index requests for
speed/efficiency.

So, I've temporarily forked the elasticsearch component and added support
for my use case and others. To do this, 

My question is whether this is a suitable change to the component. If so,
I'll attempt to create a ticket in the issue tracker and a pull request for
the changes I've made after I'm done finalizing them (I haven't read through
the contribution guide yet, i'm sure I need tests, doc, coding standards,
etc.).

Additionally, is it common for components to provide other helper classes,
like AggregationStrategies? I've written an aggregation strategy that will
collect individual IndexRequests into a BulkRequest so that the change I've
made in the Elasticsearch component can support my needs. This is easily
reproducible by others and follows after the ArrayListAggregationStrategy
example in the camel guide, but I think the use case would be common so it
may be good to include with the patch as well.

Thank you!
Derek Abdine



--
View this message in context: http://camel.465427.n5.nabble.com/Elasticsearch-component-bulk-requests-with-ids-tp5760522.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Elasticsearch component - bulk requests with ids

[Claus Ibsen <cl...@gmail.com>]

Hi

Sure we love contributions
http://camel.apache.org/contributing.html

So continue your work, and when you think you got something ready,
then you can provide a patch / PR. And we favor unit tests +
documentation to be included as well.

On Wed, Dec 10, 2014 at 10:49 PM, slush <de...@gmail.com> wrote:
> Hello! I'm new to the camel forums. First off, thank you to all the
> contributors. Camel is a wonderful framework.
>
> While doing some work recently I came across an issue with the elasticsearch
> component. Specifically, I need to be able to send a bulk request where the
> documents in the bulk request have application-provided (not
> automatically-generated, as is what happens by default with the
> elasticsearch-java client) document ids.
>
> I see that the INDEX operation supports an index header as of version 2.15.0
> (which is yet to be released). However, this still wouldn't satisfy my need,
> as I want to send bulk updates, not individual index requests for
> speed/efficiency.
>
> So, I've temporarily forked the elasticsearch component and added support
> for my use case and others. To do this,
>
> My question is whether this is a suitable change to the component. If so,
> I'll attempt to create a ticket in the issue tracker and a pull request for
> the changes I've made after I'm done finalizing them (I haven't read through
> the contribution guide yet, i'm sure I need tests, doc, coding standards,
> etc.).
>
> Additionally, is it common for components to provide other helper classes,
> like AggregationStrategies? I've written an aggregation strategy that will
> collect individual IndexRequests into a BulkRequest so that the change I've
> made in the Elasticsearch component can support my needs. This is easily
> reproducible by others and follows after the ArrayListAggregationStrategy
> example in the camel guide, but I think the use case would be common so it
> may be good to include with the patch as well.
>
> Thank you!
> Derek Abdine
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Elasticsearch-component-bulk-requests-with-ids-tp5760522.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



-- 
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/

Re: Elasticsearch component - bulk requests with ids

[dalouuu <ph...@gmail.com>]

After some digging in the code I managed to make it work. 
I am using camel-elasticsearch 2.16.2
It would be useful to others to add something about this in the
documentation page -

Here's how I did it : 

For each exchange containing a document to index I define the following
headers :
org.apache.camel.component.elasticsearch.ElasticsearchConstants.PARAM_INDEX_ID
(
org.apache.camel.component.elasticsearch.ElasticsearchConstants.PARAM_INDEX_NAME 
org.apache.camel.component.elasticsearch.ElasticsearchConstants.PARAM_INDEX_TYPE
org.apache.camel.component.elasticsearch.ElasticsearchConstants.PARAM_CONSISTENCY_LEVEL
org.apache.camel.component.elasticsearch.ElasticsearchConstants.PARAM_REPLICATION_TYPE

I also put an IndexRequest object in my body by calling static method
org.apache.camel.component.elasticsearch.converter.ElasticsearchActionRequestConverter.toIndexRequest
([map of values to index], exchange)

Then I simply forward the exchange to
org.apache.camel.component.elasticsearch.aggregation.BulkRequestAggregationStrategy





--
View this message in context: http://camel.465427.n5.nabble.com/Elasticsearch-component-bulk-requests-with-ids-tp5760522p5788269.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Elasticsearch component - bulk requests with ids

[dalouuu <ph...@gmail.com>]

Hello,

Did you publish any patch ? I also need this feature. 

Regards,





--
View this message in context: http://camel.465427.n5.nabble.com/Elasticsearch-component-bulk-requests-with-ids-tp5760522p5788202.html
Sent from the Camel - Users mailing list archive at Nabble.com.