[jira] [Commented] (GEARPUMP-167) Create Elasticsearch sink and example

["Karol Brejna (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/GEARPUMP-167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004425#comment-16004425 ] 

Karol Brejna commented on GEARPUMP-167:
---------------------------------------

[~yanghua] I started developing PoC on the following branch: https://github.com/karol-brejna-i/incubator-gearpump/tree/GEARPUMP-167-elasticsearch-sink

My idea was to have a simple sink that would just accept a JSON document to be stored in elasticsearch (not an object that would need to be serialized to JSON).

I anticipated two cases for the incoming message:
* Tuple2[String, String] - id, body - message gets stored under given id, so it's actually an upsert 
* String - body - message is stored under new id
* A sequence of any of above

I even did some short "demo" that had:
* small app subscribing to Twitter and putting messages to Kafka
* a DAG that consumed messages from Kafka, did wordcount, stored data to elasticsearch
* Kibana visualization


The code was written in Java. I planned to convert it to Scala but got distracted by the last release...

[~yanghua] I'd be glad to hear your idea on this. Maybe we could cooperate on this feature.

> Create Elasticsearch sink and example
> -------------------------------------
>
>                 Key: GEARPUMP-167
>                 URL: https://issues.apache.org/jira/browse/GEARPUMP-167
>             Project: Apache Gearpump
>          Issue Type: Task
>          Components: connectors
>            Reporter: Karol Brejna
>            Assignee: Karol Brejna
>              Labels: feature
>   Original Estimate: 18h
>  Remaining Estimate: 18h
>
> Create a simple Elasticsearch sink that enables indexing messages as Elasticsearch docs.
> Write an example demonstrating usage of the sink
> Write a doc on setting up local development env (installing and confifuring elastic, etc.) for the users of the sink



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)