You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hop.apache.org by HG <ha...@gmail.com> on 2022/01/18 16:20:35 UTC
elasticsearch output
Hi,
I see many outputs like Cassandra and MongoDb.
However the most important one is missing : elasticsearch.
Apache Flink does have a sink for it.
Is it possible to get it working in HOP too?
Regards Hans-Peter
Re: elasticsearch output
Posted by Dan <da...@dankeeley.co.uk>.
Ok just asking, sounds like it'll be easier to scale with beam then anyway!
Sent from my phone
On Tue, 18 Jan 2022, 5:52 pm HG, <ha...@gmail.com> wrote:
> We are dealing with 6000 messages per second. Of a considerable size ( up
> to 30M)
> We need to use the bulk API.
> Also fail over must work smoothly.
>
> But perhaps that it is all possible.
>
> Regards Hans
>
> On Tue, Jan 18, 2022, 18:26 Dan <da...@dankeeley.co.uk> wrote:
>
>> Out of curiosity, could you not just use the API directly? It's a
>> possibility anyway!
>>
>> On Tue, 18 Jan 2022 at 17:21, Hans Van Akelyen <
>> hans.van.akelyen@gmail.com> wrote:
>>
>>> Hi Hans-Peter,
>>>
>>> The Beam API has Elasticsearch support (version 7.9.2) so we could add
>>> it.
>>> I have created a jira case for you to implement these Read and Write
>>> Transforms [1].
>>>
>>> Feel free to create tickets for any features you would like to have or
>>> need to make your project successful.
>>>
>>> Kind regards,
>>> Hans
>>>
>>> [1] https://issues.apache.org/jira/browse/HOP-3690
>>>
>>
Re: elasticsearch output
Posted by HG <ha...@gmail.com>.
We are dealing with 6000 messages per second. Of a considerable size ( up
to 30M)
We need to use the bulk API.
Also fail over must work smoothly.
But perhaps that it is all possible.
Regards Hans
On Tue, Jan 18, 2022, 18:26 Dan <da...@dankeeley.co.uk> wrote:
> Out of curiosity, could you not just use the API directly? It's a
> possibility anyway!
>
> On Tue, 18 Jan 2022 at 17:21, Hans Van Akelyen <ha...@gmail.com>
> wrote:
>
>> Hi Hans-Peter,
>>
>> The Beam API has Elasticsearch support (version 7.9.2) so we could add it.
>> I have created a jira case for you to implement these Read and Write
>> Transforms [1].
>>
>> Feel free to create tickets for any features you would like to have or
>> need to make your project successful.
>>
>> Kind regards,
>> Hans
>>
>> [1] https://issues.apache.org/jira/browse/HOP-3690
>>
>
Re: elasticsearch output
Posted by Dan <da...@dankeeley.co.uk>.
Out of curiosity, could you not just use the API directly? It's a
possibility anyway!
On Tue, 18 Jan 2022 at 17:21, Hans Van Akelyen <ha...@gmail.com>
wrote:
> Hi Hans-Peter,
>
> The Beam API has Elasticsearch support (version 7.9.2) so we could add it.
> I have created a jira case for you to implement these Read and Write
> Transforms [1].
>
> Feel free to create tickets for any features you would like to have or
> need to make your project successful.
>
> Kind regards,
> Hans
>
> [1] https://issues.apache.org/jira/browse/HOP-3690
>
Re: elasticsearch output
Posted by HG <ha...@gmail.com>.
I have created one.
Hopefully the correct way.
On Tue, Jan 18, 2022, 23:15 Sergio Ramazzina (SERASOFT) <
sergio.ramazzina@serasoft.it> wrote:
> Hi Hans,
>
> I think it would be good to provide also the non Beam version of the
> ElasticSearch adapter that will be useful for any other cases. Can we
> ceate a JIRA issue also for that?
>
> Kind Regards
> Sergio
>
> Il 18/01/2022 18:21, Hans Van Akelyen ha scritto:
> > Hi Hans-Peter,
> >
> > The Beam API has Elasticsearch support (version 7.9.2) so we could add
> > it.
> > I have created a jira case for you to implement these Read and Write
> > Transforms [1].
> >
> > Feel free to create tickets for any features you would like to have or
> > need to make your project successful.
> >
> > Kind regards,
> > Hans
> >
> > [1] https://issues.apache.org/jira/browse/HOP-3690
>
Re: elasticsearch output
Posted by "Sergio Ramazzina (SERASOFT)" <se...@serasoft.it>.
Hi Hans,
I think it would be good to provide also the non Beam version of the
ElasticSearch adapter that will be useful for any other cases. Can we
ceate a JIRA issue also for that?
Kind Regards
Sergio
Il 18/01/2022 18:21, Hans Van Akelyen ha scritto:
> Hi Hans-Peter,
>
> The Beam API has Elasticsearch support (version 7.9.2) so we could add
> it.
> I have created a jira case for you to implement these Read and Write
> Transforms [1].
>
> Feel free to create tickets for any features you would like to have or
> need to make your project successful.
>
> Kind regards,
> Hans
>
> [1] https://issues.apache.org/jira/browse/HOP-3690
Re: elasticsearch output
Posted by Hans Van Akelyen <ha...@gmail.com>.
Hi Hans-Peter,
The Beam API has Elasticsearch support (version 7.9.2) so we could add it.
I have created a jira case for you to implement these Read and Write
Transforms [1].
Feel free to create tickets for any features you would like to have or need
to make your project successful.
Kind regards,
Hans
[1] https://issues.apache.org/jira/browse/HOP-3690
Re: elasticsearch output
Posted by Francois Papon <fr...@openobject.fr>.
Hi,
We need t be careful about the ES client license version if we want to
use it for the integration (version <= 7.10.2).
regards,
François
On 18/01/2022 17:20, HG wrote:
> Hi,
>
> I see many outputs like Cassandra and MongoDb.
> However the most important one is missing : elasticsearch.
> Apache Flink does have a sink for it.
> Is it possible to get it working in HOP too?
>
> Regards Hans-Peter