You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by se...@apache.org on 2016/10/05 22:17:01 UTC
[11/17] flink git commit: [FLINK-4739] [elasticsearch connector]
Adding packaging details for the Elasticsearch connector
[FLINK-4739] [elasticsearch connector] Adding packaging details for the Elasticsearch connector
When an uber-jar containing an Elasticsearch sink is executed, an
IllegalArgumentException may occur, which is caused by conflicting files of
Elasticsearch and it's dependencies in META-INF/services.
This commit adds further clarification to the documentation on how to build a
porper uber-jar that can be properly executed by adapting the pom file.
This closes #2591
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/beb31fc1
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/beb31fc1
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/beb31fc1
Branch: refs/heads/master
Commit: beb31fc1c5a6f5db060e5a4e1d64eacb70632900
Parents: 8000b64
Author: Steffen Hausmann <st...@hausmann-family.de>
Authored: Mon Oct 3 22:32:01 2016 +0200
Committer: Stephan Ewen <se...@apache.org>
Committed: Wed Oct 5 19:36:14 2016 +0200
----------------------------------------------------------------------
docs/dev/connectors/elasticsearch2.md | 32 ++++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/beb31fc1/docs/dev/connectors/elasticsearch2.md
----------------------------------------------------------------------
diff --git a/docs/dev/connectors/elasticsearch2.md b/docs/dev/connectors/elasticsearch2.md
index 8eed690..5f4267e 100644
--- a/docs/dev/connectors/elasticsearch2.md
+++ b/docs/dev/connectors/elasticsearch2.md
@@ -139,3 +139,35 @@ This now provides a list of Elasticsearch Nodes
to which the sink should connect via a `TransportClient`.
More information about Elasticsearch can be found [here](https://elastic.co).
+
+
+#### Packaging the Elasticsearch Connector into an Uber-jar
+
+For the execution of your Flink program,
+it is recommended to build a so-called uber-jar (executable jar) containing all your dependencies
+(see [here]({{site.baseurl}}/dev/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution) for further information).
+
+However,
+when an uber-jar containing an Elasticsearch sink is executed,
+an `IllegalArgumentException` may occur,
+which is caused by conflicting files of Elasticsearch and it's dependencies
+in `META-INF/services`:
+
+```
+IllegalArgumentException[An SPI class of type org.apache.lucene.codecs.PostingsFormat with name 'Lucene50' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: [es090, completion090, XBloomFilter]]
+```
+
+If the uber-jar is build by means of maven,
+this issue can be avoided by adding the following bits to the pom file:
+
+```
+<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
+ <resource>META-INF/services/org.apache.lucene.codecs.Codec</resource>
+</transformer>
+<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
+ <resource>META-INF/services/org.apache.lucene.codecs.DocValuesFormat</resource>
+</transformer>
+<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
+ <resource>META-INF/services/org.apache.lucene.codecs.PostingsFormat</resource>
+</transformer>
+```