[jira] [Commented] (FLUME-2921) Support Elasticsearch 2.0+

["Mike Percy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-2921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369983#comment-15369983 ] 

Mike Percy commented on FLUME-2921:
-----------------------------------

Moving some of this discussion off of reviewboard to make it easier to follow.

My concern about this update is that ES exposes Guava as a dependency and requires Guava 18. This patch updates Guava to a version that is incompatible with Hadoop (Hadoop is on Guava 11). Most Flume sinks inherit Hadoop dependencies. Therefore HBase, Hive, etc inherit that as well. Guava 15 removed some classes.

Flume's use of Guava itself doesn't concern me. It's the lack of compatibility and undefined behavior that would result from putting different versions of Guava in the same classpath. Flume ships all of its deps together, in the same classpath.

In short, we can't upgrade ElasticSearch to a version that requires Guava > 11 unless a solution to this issue is found.

The most obvious fix (this is a lot of work) is to refactor Flume's plugin architecture to support something like OSGI or Spring DM so that individual sources and sinks get their own classloaders. This solves the dependency problem but introduces new problems, such as debuggability.

There may be other solutions to this problem. I'm not sure.

Another approach is to delay this upgrade until ES 3.0, when Guava support is apparently dropped. Or maybe an older version of the ES client, like 1.x, depends on an old version of Guava.

> Support Elasticsearch 2.0+
> --------------------------
>
>                 Key: FLUME-2921
>                 URL: https://issues.apache.org/jira/browse/FLUME-2921
>             Project: Flume
>          Issue Type: Improvement
>          Components: Sinks+Sources
>            Reporter: Lior Zeno
>            Assignee: Lior Zeno
>             Fix For: v1.7.0
>
>         Attachments: FLUME-2921-0.patch, FLUME-2921-1.patch
>
>
> Elasticsearch sink supports an ancient version of ES. We should make the sink work with newer versions of Elasticsearch.
> I attached a patch for that. Please note that this involves upgrading netty and guava.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)