You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Akshay Bhardwaj <ak...@gmail.com> on 2019/05/10 06:52:27 UTC
Spark Elasticsearch Connector | Index and Update
Hi All,
Are there any users who have integrated spark structured streaming with
elastic search 6.x?
In elastic search doc, ElasticSearch Hadoop Configuration
<https://www.elastic.co/guide/en/elasticsearch/hadoop/6.6/configuration.html>
There is a property es.write.option which is defined as follows
es.write.operation (default index)
The write operation elasticsearch-hadoop should perform - can be any of:
1. index (default)
- new data is added while existing data (based on its id) is replaced
(reindexed).
1. create
- adds new data - if the data already exists (based on its id), an
exception is thrown.
1. update
- updates existing data (based on its id). If no data is found, an
exception is thrown.
1. upsert
- known as merge or insert if the data does not exist, updates if the
data exists (based on its id).
*Say if I have 2 documents in the partition, and based on a field I want to
index the document, and based on another field I want to update the
document with an inline script.*
*Is there a possibility to do this in the same writeStream for Elastic
Search in spark structured streaming?*
Akshay Bhardwaj
+91-97111-33849