You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Fidea Lidea <li...@gmail.com> on 2024/03/12 10:28:46 UTC
Read data from elasticsearch using Java flink
Hi ,
I am trying to read data from elasticsearch & store in a stream.
Could you please share a few examples to *read*/get all data from
Elasticsearch using java.
Thanks,
Re:Re: Re: Read data from elasticsearch using Java flink
Posted by Xuyang <xy...@163.com>.
Hi, Nida.
About using connector with Java by Flink DataStream api, you can mainly reference these doc[1][2].
However, Elaticsearch connector only supports sink currently. What you need it to build a custom ES
connector with that patch[3] yourself. The following steps may help you.
1. Download the source code of the ES connector and checkout the version that is corresponding to
Flink that you use. (For example: 1.17.0)
2. Cherry pick that patch to your repo. If serious conflicts occur during the CP process, it is recommended
to use higher versions of Flink and ES connector.
3. use 'mvn' to package your custom ES connector.
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/
[2] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#data-sources
[3] https://github.com/apache/flink-connector-elasticsearch/pull/62
--
Best！
Xuyang
在 2024-03-22 13:21:04，"Fidea Lidea" <li...@gmail.com> 写道：
Hi Xuyang
I am new to Flink & I don't know how to implement this dependency into the code.
Can you please share some examples so that I can refer those.
Thanks
Nida
On Fri, Mar 22, 2024 at 7:02 AM Xuyang <xy...@163.com> wrote:
Hi, Nida.
Can you explain more details about "unable to use it." ? Did you get an exception after using it?
--
Best！
Xuyang
在 2024-03-21 21:14:53，"Fidea Lidea" <li...@gmail.com> 写道：
Thank you Xuyang.
I added the above flink-connector-elasticsearch dependency in my project.
But I am unable to use it.
Can you please share a few sample codes which are using this dependency?
Thanks
Nida
On Tue, Mar 12, 2024 at 5:37 PM Xuyang <xy...@163.com> wrote:
Hi, Fidea.
Currently, elasticsearch is not supported to be used as a source. You can see the jira[1] for more details.
You can also cherry pick this pr[2] to your own branch and build a custom elasticsearch connector to use it directly.
[1] https://issues.apache.org/jira/browse/FLINK-25568
[2] https://github.com/apache/flink-connector-elasticsearch/pull/62
--
Best！
Xuyang
At 2024-03-12 18:28:46, "Fidea Lidea" <li...@gmail.com> wrote:
Hi ,
I am trying to read data from elasticsearch & store in a stream.
Could you please share a few examples to read/get all data from Elasticsearch using java.
Thanks,
Re: Re: Read data from elasticsearch using Java flink
Posted by Fidea Lidea <li...@gmail.com>.
Hi Xuyang
I am new to Flink & I don't know how to implement this dependency into the
code.
Can you please share some examples so that I can refer those.
Thanks
Nida
On Fri, Mar 22, 2024 at 7:02 AM Xuyang <xy...@163.com> wrote:
> Hi, Nida.
> Can you explain more details about "unable to use it." ? Did you get an
> exception after using it?
>
>
> --
> Best！
> Xuyang
>
>
> 在 2024-03-21 21:14:53，"Fidea Lidea" <li...@gmail.com> 写道：
>
> Thank you Xuyang.
> I added the above flink-connector-elasticsearch dependency in my project.
> But I am unable to use it.
> Can you please share a few sample codes which are using this dependency?
>
> Thanks
> Nida
>
> On Tue, Mar 12, 2024 at 5:37 PM Xuyang <xy...@163.com> wrote:
>
>> Hi, Fidea.
>> Currently, elasticsearch is not supported to be used as a source. You can
>> see the jira[1] for more details.
>> You can also cherry pick this pr[2] to your own branch and build a custom
>> elasticsearch connector to use it directly.
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-25568
>> [2] https://github.com/apache/flink-connector-elasticsearch/pull/62
>>
>> --
>> Best！
>> Xuyang
>>
>>
>> At 2024-03-12 18:28:46, "Fidea Lidea" <li...@gmail.com> wrote:
>>
>> Hi ,
>>
>> I am trying to read data from elasticsearch & store in a stream.
>> Could you please share a few examples to *read*/get all data from
>> Elasticsearch using java.
>>
>>
>> Thanks,
>>
>>
>>
>>
Re:Re: Read data from elasticsearch using Java flink
Posted by Xuyang <xy...@163.com>.
Hi, Nida.
Can you explain more details about "unable to use it." ? Did you get an exception after using it?
--
Best！
Xuyang
在 2024-03-21 21:14:53，"Fidea Lidea" <li...@gmail.com> 写道：
Thank you Xuyang.
I added the above flink-connector-elasticsearch dependency in my project.
But I am unable to use it.
Can you please share a few sample codes which are using this dependency?
Thanks
Nida
On Tue, Mar 12, 2024 at 5:37 PM Xuyang <xy...@163.com> wrote:
Hi, Fidea.
Currently, elasticsearch is not supported to be used as a source. You can see the jira[1] for more details.
You can also cherry pick this pr[2] to your own branch and build a custom elasticsearch connector to use it directly.
[1] https://issues.apache.org/jira/browse/FLINK-25568
[2] https://github.com/apache/flink-connector-elasticsearch/pull/62
--
Best！
Xuyang
At 2024-03-12 18:28:46, "Fidea Lidea" <li...@gmail.com> wrote:
Hi ,
I am trying to read data from elasticsearch & store in a stream.
Could you please share a few examples to read/get all data from Elasticsearch using java.
Thanks,
Re: Read data from elasticsearch using Java flink
Posted by Fidea Lidea <li...@gmail.com>.
Thank you Xuyang.
I added the above flink-connector-elasticsearch dependency in my project.
But I am unable to use it.
Can you please share a few sample codes which are using this dependency?
Thanks
Nida
On Tue, Mar 12, 2024 at 5:37 PM Xuyang <xy...@163.com> wrote:
> Hi, Fidea.
> Currently, elasticsearch is not supported to be used as a source. You can
> see the jira[1] for more details.
> You can also cherry pick this pr[2] to your own branch and build a custom
> elasticsearch connector to use it directly.
>
> [1] https://issues.apache.org/jira/browse/FLINK-25568
> [2] https://github.com/apache/flink-connector-elasticsearch/pull/62
>
> --
> Best！
> Xuyang
>
>
> At 2024-03-12 18:28:46, "Fidea Lidea" <li...@gmail.com> wrote:
>
> Hi ,
>
> I am trying to read data from elasticsearch & store in a stream.
> Could you please share a few examples to *read*/get all data from
> Elasticsearch using java.
>
>
> Thanks,
>
>
>
>
Re:Read data from elasticsearch using Java flink
Posted by Xuyang <xy...@163.com>.
Hi, Fidea.
Currently, elasticsearch is not supported to be used as a source. You can see the jira[1] for more details.
You can also cherry pick this pr[2] to your own branch and build a custom elasticsearch connector to use it directly.
[1] https://issues.apache.org/jira/browse/FLINK-25568
[2] https://github.com/apache/flink-connector-elasticsearch/pull/62
--
Best！
Xuyang
At 2024-03-12 18:28:46, "Fidea Lidea" <li...@gmail.com> wrote:
Hi ,
I am trying to read data from elasticsearch & store in a stream.
Could you please share a few examples to read/get all data from Elasticsearch using java.
Thanks,