You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/09/30 15:24:11 UTC
[GitHub] [beam] KarthikPerikala opened a new issue, #23448: [Bug]: Apache Beam ElasticIO Connector not working for ElasticSearch Cluster version 8.X
KarthikPerikala opened a new issue, #23448:
URL: https://github.com/apache/beam/issues/23448
### What happened?
We have google cloud dataflow application which reads data from big query and ingest the documents to elastic search index
Apache Beam IO Connector Writing methods is not working with ElasticSearch Cluster (8.X Version) .
The dataflow application is throwing the following errors(compatibility error)
```
Error message from worker: java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.lang.IllegalArgumentException: The Elasticsearch version to connect to is 8.x. This version of the ElasticsearchIO is only compatible with Elasticsearch v7.x, v6.x, v5.x and v2.x
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:194)
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:165)
org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:63)
org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:50)
org.apache.beam.runners.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:87)
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.create(IntrinsicMapTaskExecutorFactory.java:125)
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:358)
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:311)
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140)
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120)
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107)
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.beam.sdk.util.UserCodeException: java.lang.IllegalArgumentException: The Elasticsearch version to connect to is 8.x. This version of the ElasticsearchIO is only compatible with Elasticsearch v7.x, v6.x, v5.x and v2.x
org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn$DoFnInvoker.invokeSetup(Unknown Source)
org.apache.beam.runners.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.deserializeCopy(DoFnInstanceManagers.java:80)
org.apache.beam.runners.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.peek(DoFnInstanceManagers.java:62)
org.apache.beam.runners.dataflow.worker.UserParDoFnFactory.create(UserParDoFnFactory.java:95)
org.apache.beam.runners.dataflow.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:75)
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.createParDoOperation(IntrinsicMapTaskExecutorFactory.java:264)
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.access$000(IntrinsicMapTaskExecutorFactory.java:86)
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:183)
... 14 more
Caused by: java.lang.IllegalArgumentException: The Elasticsearch version to connect to is 8.x. This version of the ElasticsearchIO is only compatible with Elasticsearch v7.x, v6.x, v5.x and v2.x
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument(Preconditions.java:189)
org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.getBackendVersion(ElasticsearchIO.java:1485)
org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.setup(ElasticsearchIO.java:1287)
```
Please find the details of the apache beam/dataflow job that was executed
- Cloud Environment : Google Cloud Platform
- Apache Beam SDKs : Apache Beam SDK for Java 2.41.0,Apache Beam SDK for Java 2.25.0
-Apache Beam ElasticIO Connector : beam-sdks-java-io-elasticsearch versions 2.25.0
### Issue Priority
Priority: 2
### Issue Component
Component: io-java-elasticsearch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev closed issue #23448: [Bug]: Apache Beam ElasticIO Connector not working for ElasticSearch Cluster version 8.X
Posted by GitBox <gi...@apache.org>.
aromanenko-dev closed issue #23448: [Bug]: Apache Beam ElasticIO Connector not working for ElasticSearch Cluster version 8.X
URL: https://github.com/apache/beam/issues/23448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on issue #23448: [Bug]: Apache Beam ElasticIO Connector not working for ElasticSearch Cluster version 8.X
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on issue #23448:
URL: https://github.com/apache/beam/issues/23448#issuecomment-1313419961
Hi @karthikprogrammer ! Thanks for update. So, I close this issue then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] egalpin commented on issue #23448: [Bug]: Apache Beam ElasticIO Connector not working for ElasticSearch Cluster version 8.X
Posted by GitBox <gi...@apache.org>.
egalpin commented on issue #23448:
URL: https://github.com/apache/beam/issues/23448#issuecomment-1263901448
Hi @karthikprogrammer. To use the 2.41.0 ES connector with the 2.25.0 SDK, you would need to build the jar for the ES module from the 2.41.0 branch and specify using that local version instead of the version from 2.25.0. I can’t guarantee that this will work, however, but you could try it.
I previously recorded some steps I took to do a similar thing with the pubsub module, which could easily be adapted to the ES module (just would be no need to using a custom sink experiment):
https://lists.apache.org/thread/c6929ms0bjxtcw9ho4tdb5y3t8wnwnfy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] karthikprogrammer commented on issue #23448: [Bug]: Apache Beam ElasticIO Connector not working for ElasticSearch Cluster version 8.X
Posted by GitBox <gi...@apache.org>.
karthikprogrammer commented on issue #23448:
URL: https://github.com/apache/beam/issues/23448#issuecomment-1310979123
> Is it still a bug?
Hi @aromanenko-dev . After upgrading Apache Beam SDK and Apache Beam ElasticIO Connector dependencies to the latest versions no issues are observed . We have tested reading and writing data to Elastic Cluster With version 8.X.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] karthikprogrammer commented on issue #23448: [Bug]: Apache Beam ElasticIO Connector not working for ElasticSearch Cluster version 8.X
Posted by GitBox <gi...@apache.org>.
karthikprogrammer commented on issue #23448:
URL: https://github.com/apache/beam/issues/23448#issuecomment-1263866973
Hi Team ,
We have used the latest versions for Apache Beam SDK's and Apache Beam Elastic IO Connectors .
Dataflow is able to index the documents in the Elastic Cluster version 8.X
Cloud Environment : Google Cloud Platform
Pipeline RunnerType : DataflowRunner
Apache Beam SDKs Used : Apache Beam SDK for Java 2.41.0 instead of Apache Beam SDK for Java 2.25.0
Apache Beam ElasticIO Connector : beam-sdks-java-io-elasticsearch versions 2.41.0 instead of beam-sdks-java-io-elasticsearch versions 2.25.0
Need guidance to make our apache beam pipelines work for elastic search clusters with the previous versions of apache beam sdks & connectors
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on issue #23448: [Bug]: Apache Beam ElasticIO Connector not working for ElasticSearch Cluster version 8.X
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on issue #23448:
URL: https://github.com/apache/beam/issues/23448#issuecomment-1302367735
Is it still a bug?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org