You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by santosh_rajaguru <sa...@gmail.com> on 2015/10/13 15:08:08 UTC

Apache flink with Elastic Search

Hi all,

Is there any possibility to query Elastic search using flink?


Thanks and Regards,
Santosh



--
View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Apache-flink-with-Elastic-Search-tp8435.html
Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.

Re: Apache flink with Elastic Search

Posted by santosh_rajaguru <sa...@gmail.com>.

Thanks Martin for the input.



--
View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Apache-flink-with-Elastic-Search-tp8435p8474.html
Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.

Re: Apache flink with Elastic Search

Posted by Martin Liesenberg <ma...@gmail.com>.

Elasticsearch does not support streaming in the sense, that you subscribe
to a search query and as new documents come in, you will get any matching
your query.
There is a rather long discussion in one of the issues in the ES repository
on GitHub concerning this (active for 4 years now) [1]

It does however provide you with an API to scroll through a large set of
results, so you can retrieve them in chunks which fit your application. [2]
That is supported by the hadoop connector via [3] which is used as far as I
can tell in almost all of the different connectors (hive, cascading, mr,
spark)

Best regards,
Martin

[1] https://github.com/elastic/elasticsearch/issues/1242
[2]
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html
[3]
https://github.com/elastic/elasticsearch-hadoop/blob/03c056142a5ab7422b81bb1f519fd67a9581405f/mr/src/main/java/org/elasticsearch/hadoop/rest/ScrollQuery.java

Robert Metzger <rm...@apache.org> schrieb am Mi., 14. Okt. 2015 um
17:49 Uhr:

> I think the Hadoop connector of ElasticSearch only works for batch jobs.
> In my understanding, elasticsearch also allows "streaming" standing search
> queries.
>
> On Wed, Oct 14, 2015 at 10:41 AM, santosh_rajaguru <sa...@gmail.com>
> wrote:
>
> > Thanks flavio.
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Apache-flink-with-Elastic-Search-tp8435p8445.html
> > Sent from the Apache Flink Mailing List archive. mailing list archive at
> > Nabble.com.
> >
>

Re: Apache flink with Elastic Search

Posted by Robert Metzger <rm...@apache.org>.

I think the Hadoop connector of ElasticSearch only works for batch jobs.
In my understanding, elasticsearch also allows "streaming" standing search
queries.

On Wed, Oct 14, 2015 at 10:41 AM, santosh_rajaguru <sa...@gmail.com>
wrote:

> Thanks flavio.
>
>
>
> --
> View this message in context:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Apache-flink-with-Elastic-Search-tp8435p8445.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>

Re: Apache flink with Elastic Search

Posted by santosh_rajaguru <sa...@gmail.com>.

Thanks flavio.



--
View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Apache-flink-with-Elastic-Search-tp8435p8445.html
Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.

Re: Apache flink with Elastic Search

Posted by Flavio Pompermaier <po...@okkam.it>.

You can always use the Hadoop connector of Elasticsearch ;)
On 14 Oct 2015 08:32, "Aljoscha Krettek" <al...@apache.org> wrote:

> Hi,
> the Elasticsearch connector can only be used for writing right now. If
> there is need we could also think about adding a connector for reading,
> though.
>
> Cheers,
> Aljoscha
> > On 13 Oct 2015, at 16:01, Sachin Goel <sa...@gmail.com> wrote:
> >
> > Hi Santosh
> > There is an Elastic search connector under streaming connectors. Can you
> > check if that satisfies your needs?
> > Aljoscha will be able to provide more details in case you need them.
> >
> > Cheers!
> > Sachin
> >
> > -- Sachin Goel
> > Computer Science, IIT Delhi
> > m. +91-9871457685
> >
> > On Tue, Oct 13, 2015 at 6:38 PM, santosh_rajaguru <sa...@gmail.com>
> wrote:
> >
> >> Hi all,
> >>
> >> Is there any possibility to query Elastic search using flink?
> >>
> >>
> >> Thanks and Regards,
> >> Santosh
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Apache-flink-with-Elastic-Search-tp8435.html
> >> Sent from the Apache Flink Mailing List archive. mailing list archive at
> >> Nabble.com.
> >>
>
>

Re: Apache flink with Elastic Search

Posted by Aljoscha Krettek <al...@apache.org>.

Hi,
the Elasticsearch connector can only be used for writing right now. If there is need we could also think about adding a connector for reading, though.

Cheers,
Aljoscha
> On 13 Oct 2015, at 16:01, Sachin Goel <sa...@gmail.com> wrote:
> 
> Hi Santosh
> There is an Elastic search connector under streaming connectors. Can you
> check if that satisfies your needs?
> Aljoscha will be able to provide more details in case you need them.
> 
> Cheers!
> Sachin
> 
> -- Sachin Goel
> Computer Science, IIT Delhi
> m. +91-9871457685
> 
> On Tue, Oct 13, 2015 at 6:38 PM, santosh_rajaguru <sa...@gmail.com> wrote:
> 
>> Hi all,
>> 
>> Is there any possibility to query Elastic search using flink?
>> 
>> 
>> Thanks and Regards,
>> Santosh
>> 
>> 
>> 
>> --
>> View this message in context:
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Apache-flink-with-Elastic-Search-tp8435.html
>> Sent from the Apache Flink Mailing List archive. mailing list archive at
>> Nabble.com.
>>

Re: Apache flink with Elastic Search

Posted by Sachin Goel <sa...@gmail.com>.

Hi Santosh
There is an Elastic search connector under streaming connectors. Can you
check if that satisfies your needs?
Aljoscha will be able to provide more details in case you need them.

Cheers!
Sachin

-- Sachin Goel
Computer Science, IIT Delhi
m. +91-9871457685

On Tue, Oct 13, 2015 at 6:38 PM, santosh_rajaguru <sa...@gmail.com> wrote:

> Hi all,
>
> Is there any possibility to query Elastic search using flink?
>
>
> Thanks and Regards,
> Santosh
>
>
>
> --
> View this message in context:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Apache-flink-with-Elastic-Search-tp8435.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>