You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Akshay Bhardwaj <ak...@gmail.com> on 2019/05/10 06:52:27 UTC

Spark Elasticsearch Connector | Index and Update

Hi All,

Are there any users who have integrated spark structured streaming  with
elastic search 6.x?

In elastic search doc, ElasticSearch Hadoop Configuration
<https://www.elastic.co/guide/en/elasticsearch/hadoop/6.6/configuration.html>
There is a property es.write.option which is defined as follows

es.write.operation (default index)
The write operation elasticsearch-hadoop should perform - can be any of:


   1. index (default)
   - new data is added while existing data (based on its id) is replaced
         (reindexed).
      1. create
   - adds new data - if the data already exists (based on its id), an
         exception is thrown.
      1. update
   - updates existing data (based on its id). If no data is found, an
         exception is thrown.
      1. upsert
   - known as merge or insert if the data does not exist, updates if the
         data exists (based on its id).


*Say if I have 2 documents in the partition, and based on a field I want to
index the document, and based on another field I want to update the
document with an inline script.*

*Is there a possibility to do this in the same writeStream for Elastic
Search in spark structured streaming?*

Akshay Bhardwaj
+91-97111-33849