You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by jackylau <li...@gmail.com> on 2020/06/19 07:45:53 UTC

[DISCUSS] Drop connectors for 5.x and restart the flink es source connector

Hi all:
     when i coding the es source connector here 
https://github.com/liuyongvs/flink/commit/c397a759d05956629a27bf850458dd4e70330189
for the elasticsearch source connector. The doc is here
https://cwiki.apache.org/confluence/display/FLINK/FLIP-127%3A+Support+Elasticsearch+Source+Connector
,and i find  problem of SearchHits at 

*currentScrollWindowHits = searchResponse.getHits().getHits()
*
And the SearchHits is a Interface in es5 but it is a class in es 6,7 and if
i use it in flink es connector base(the es dependency is 5). it will throw
this. 
*Caused by: java.lang.IncompatibleClassChangeError: Found class
org.elasticsearch.search.SearchHits, but interface was expected.***

To fix it : we can do this ways

1) move the logic to ApiCallBridge such as define ElasticsearchResponse or
Tuple2<String, String[]>, but it wll make the code weirdly

class ElasticsearchResponse
{ 
    String scroll; 
    String[] result // convert ervery es connector
searchResponse.getHits().getHits() to this result 
}


if user want to add some thing, it will need modify this
 

2) just support es 6,7 and upgrade flink-es-connector-base es dependency
version to 6 and drop flink-es-connector-5. And i found this discussion of
dropping es connector 2 and 5 here
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-connectors-for-Elasticsearch-2-x-and-5-x-td32662.html
.

 The es5 connector just support DataStream api currently .And Is it possible
to drop es5 connector and upgrade es-connector-base to es6?

I am looking forward all your response
Best !



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: [DISCUSS] Drop connectors for 5.x and restart the flink es source connector

Posted by Robert Metzger <rm...@apache.org>.
The discussion on dropping the ES5 connector was not conclusive, when we
discussed it in February 2020. We wanted to revisit it for the 1.12 release.

From maven central, we have the following download numbers
ES2: 500 downloads
ES5: 10500 downloads (the es5_2.10:1.3.1 had 8000 downloads last month. I
there's a CI system or something downloading all these)
ES6: 4200 downloads
ES7: 1800 downloads

For 1.10.0 we had the following numbers:
ES5: 500
ES6: 525
ES7: 840

Based on these numbers, I would advise against removing the ES5 connector
for the 1.12 release.



On Fri, Jun 19, 2020 at 9:53 AM Jark Wu <im...@gmail.com> wrote:

> I'm fine with dropping support for es5.
>
> forward to dev@.
>
> Best,
> Jark
>
>
>
> On Fri, 19 Jun 2020 at 15:46, jackylau <li...@gmail.com> wrote:
>
> > Hi all:
> >      when i coding the es source connector here
> >
> >
> https://github.com/liuyongvs/flink/commit/c397a759d05956629a27bf850458dd4e70330189
> > for the elasticsearch source connector. The doc is here
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-127%3A+Support+Elasticsearch+Source+Connector
> > ,and
> > <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-127%3A+Support+Elasticsearch+Source+Connector,and
> >
> > i find  problem of SearchHits at
> >
> > *currentScrollWindowHits = searchResponse.getHits().getHits()
> > *
> > And the SearchHits is a Interface in es5 but it is a class in es 6,7 and
> if
> > i use it in flink es connector base(the es dependency is 5). it will
> throw
> > this.
> > *Caused by: java.lang.IncompatibleClassChangeError: Found class
> > org.elasticsearch.search.SearchHits, but interface was expected.***
> >
> > To fix it : we can do this ways
> >
> > 1) move the logic to ApiCallBridge such as define ElasticsearchResponse
> or
> > Tuple2<String, String[]>, but it wll make the code weirdly
> >
> > class ElasticsearchResponse
> > {
> >     String scroll;
> >     String[] result // convert ervery es connector
> > searchResponse.getHits().getHits() to this result
> > }
> >
> >
> > if user want to add some thing, it will need modify this
> >
> >
> > 2) just support es 6,7 and upgrade flink-es-connector-base es dependency
> > version to 6 and drop flink-es-connector-5. And i found this discussion
> of
> > dropping es connector 2 and 5 here
> >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-connectors-for-Elasticsearch-2-x-and-5-x-td32662.html
> > .
> >
> >  The es5 connector just support DataStream api currently .And Is it
> > possible
> > to drop es5 connector and upgrade es-connector-base to es6?
> >
> > I am looking forward all your response
> > Best !
> >
> >
> >
> > --
> > Sent from:
> > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
> >
>

Re: [DISCUSS] Drop connectors for 5.x and restart the flink es source connector

Posted by Robert Metzger <rm...@apache.org>.
The discussion on dropping the ES5 connector was not conclusive, when we
discussed it in February 2020. We wanted to revisit it for the 1.12 release.

From maven central, we have the following download numbers
ES2: 500 downloads
ES5: 10500 downloads (the es5_2.10:1.3.1 had 8000 downloads last month. I
there's a CI system or something downloading all these)
ES6: 4200 downloads
ES7: 1800 downloads

For 1.10.0 we had the following numbers:
ES5: 500
ES6: 525
ES7: 840

Based on these numbers, I would advise against removing the ES5 connector
for the 1.12 release.



On Fri, Jun 19, 2020 at 9:53 AM Jark Wu <im...@gmail.com> wrote:

> I'm fine with dropping support for es5.
>
> forward to dev@.
>
> Best,
> Jark
>
>
>
> On Fri, 19 Jun 2020 at 15:46, jackylau <li...@gmail.com> wrote:
>
> > Hi all:
> >      when i coding the es source connector here
> >
> >
> https://github.com/liuyongvs/flink/commit/c397a759d05956629a27bf850458dd4e70330189
> > for the elasticsearch source connector. The doc is here
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-127%3A+Support+Elasticsearch+Source+Connector
> > ,and
> > <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-127%3A+Support+Elasticsearch+Source+Connector,and
> >
> > i find  problem of SearchHits at
> >
> > *currentScrollWindowHits = searchResponse.getHits().getHits()
> > *
> > And the SearchHits is a Interface in es5 but it is a class in es 6,7 and
> if
> > i use it in flink es connector base(the es dependency is 5). it will
> throw
> > this.
> > *Caused by: java.lang.IncompatibleClassChangeError: Found class
> > org.elasticsearch.search.SearchHits, but interface was expected.***
> >
> > To fix it : we can do this ways
> >
> > 1) move the logic to ApiCallBridge such as define ElasticsearchResponse
> or
> > Tuple2<String, String[]>, but it wll make the code weirdly
> >
> > class ElasticsearchResponse
> > {
> >     String scroll;
> >     String[] result // convert ervery es connector
> > searchResponse.getHits().getHits() to this result
> > }
> >
> >
> > if user want to add some thing, it will need modify this
> >
> >
> > 2) just support es 6,7 and upgrade flink-es-connector-base es dependency
> > version to 6 and drop flink-es-connector-5. And i found this discussion
> of
> > dropping es connector 2 and 5 here
> >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-connectors-for-Elasticsearch-2-x-and-5-x-td32662.html
> > .
> >
> >  The es5 connector just support DataStream api currently .And Is it
> > possible
> > to drop es5 connector and upgrade es-connector-base to es6?
> >
> > I am looking forward all your response
> > Best !
> >
> >
> >
> > --
> > Sent from:
> > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
> >
>

Re: [DISCUSS] Drop connectors for 5.x and restart the flink es source connector

Posted by Jark Wu <im...@gmail.com>.
I'm fine with dropping support for es5.

forward to dev@.

Best,
Jark



On Fri, 19 Jun 2020 at 15:46, jackylau <li...@gmail.com> wrote:

> Hi all:
>      when i coding the es source connector here
>
> https://github.com/liuyongvs/flink/commit/c397a759d05956629a27bf850458dd4e70330189
> for the elasticsearch source connector. The doc is here
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-127%3A+Support+Elasticsearch+Source+Connector
> ,and
> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-127%3A+Support+Elasticsearch+Source+Connector,and>
> i find  problem of SearchHits at
>
> *currentScrollWindowHits = searchResponse.getHits().getHits()
> *
> And the SearchHits is a Interface in es5 but it is a class in es 6,7 and if
> i use it in flink es connector base(the es dependency is 5). it will throw
> this.
> *Caused by: java.lang.IncompatibleClassChangeError: Found class
> org.elasticsearch.search.SearchHits, but interface was expected.***
>
> To fix it : we can do this ways
>
> 1) move the logic to ApiCallBridge such as define ElasticsearchResponse or
> Tuple2<String, String[]>, but it wll make the code weirdly
>
> class ElasticsearchResponse
> {
>     String scroll;
>     String[] result // convert ervery es connector
> searchResponse.getHits().getHits() to this result
> }
>
>
> if user want to add some thing, it will need modify this
>
>
> 2) just support es 6,7 and upgrade flink-es-connector-base es dependency
> version to 6 and drop flink-es-connector-5. And i found this discussion of
> dropping es connector 2 and 5 here
>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-connectors-for-Elasticsearch-2-x-and-5-x-td32662.html
> .
>
>  The es5 connector just support DataStream api currently .And Is it
> possible
> to drop es5 connector and upgrade es-connector-base to es6?
>
> I am looking forward all your response
> Best !
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>

Re: [DISCUSS] Drop connectors for 5.x and restart the flink es source connector

Posted by Jark Wu <im...@gmail.com>.
I'm fine with dropping support for es5.

forward to dev@.

Best,
Jark



On Fri, 19 Jun 2020 at 15:46, jackylau <li...@gmail.com> wrote:

> Hi all:
>      when i coding the es source connector here
>
> https://github.com/liuyongvs/flink/commit/c397a759d05956629a27bf850458dd4e70330189
> for the elasticsearch source connector. The doc is here
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-127%3A+Support+Elasticsearch+Source+Connector
> ,and
> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-127%3A+Support+Elasticsearch+Source+Connector,and>
> i find  problem of SearchHits at
>
> *currentScrollWindowHits = searchResponse.getHits().getHits()
> *
> And the SearchHits is a Interface in es5 but it is a class in es 6,7 and if
> i use it in flink es connector base(the es dependency is 5). it will throw
> this.
> *Caused by: java.lang.IncompatibleClassChangeError: Found class
> org.elasticsearch.search.SearchHits, but interface was expected.***
>
> To fix it : we can do this ways
>
> 1) move the logic to ApiCallBridge such as define ElasticsearchResponse or
> Tuple2<String, String[]>, but it wll make the code weirdly
>
> class ElasticsearchResponse
> {
>     String scroll;
>     String[] result // convert ervery es connector
> searchResponse.getHits().getHits() to this result
> }
>
>
> if user want to add some thing, it will need modify this
>
>
> 2) just support es 6,7 and upgrade flink-es-connector-base es dependency
> version to 6 and drop flink-es-connector-5. And i found this discussion of
> dropping es connector 2 and 5 here
>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-connectors-for-Elasticsearch-2-x-and-5-x-td32662.html
> .
>
>  The es5 connector just support DataStream api currently .And Is it
> possible
> to drop es5 connector and upgrade es-connector-base to es6?
>
> I am looking forward all your response
> Best !
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>