You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Mubarak Seyed (JIRA)" <ji...@apache.org> on 2012/07/15 07:36:34 UTC
[jira] [Created] (FLUME-1371) ElasticSearch Sink
Mubarak Seyed created FLUME-1371:
------------------------------------
Summary: ElasticSearch Sink
Key: FLUME-1371
URL: https://issues.apache.org/jira/browse/FLUME-1371
Project: Flume
Issue Type: New Feature
Components: Sinks+Sources
Reporter: Mubarak Seyed
Motivation:
Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
Design:
ElasticSearch supports RESTful style API to publish events for indexing.
There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
[1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1371) ElasticSearch Sink
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland updated FLUME-1371:
--------------------------------
Assignee: Cameron Gandevia
> ElasticSearch Sink
> ------------------
>
> Key: FLUME-1371
> URL: https://issues.apache.org/jira/browse/FLUME-1371
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.3.0
> Reporter: Mubarak Seyed
> Assignee: Cameron Gandevia
> Labels: noob
> Fix For: v1.3.0
>
> Attachments: FLUME-1371.patch
>
>
> Motivation:
> Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
> Design:
> ElasticSearch supports RESTful style API to publish events for indexing.
> There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
> [1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1371) ElasticSearch Sink
Posted by "Cameron Gandevia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cameron Gandevia updated FLUME-1371:
------------------------------------
Attachment: (was: FLUME-1371.patch)
> ElasticSearch Sink
> ------------------
>
> Key: FLUME-1371
> URL: https://issues.apache.org/jira/browse/FLUME-1371
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.3.0
> Reporter: Mubarak Seyed
> Labels: noob
> Attachments: FLUME-1371.patch
>
>
> Motivation:
> Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
> Design:
> ElasticSearch supports RESTful style API to publish events for indexing.
> There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
> [1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1371) ElasticSearch Sink
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483184#comment-13483184 ]
Hari Shreedharan commented on FLUME-1371:
-----------------------------------------
[~brocknoland] - You might need to make some manual changes to poms to make sure it commits to all branches without issue.
> ElasticSearch Sink
> ------------------
>
> Key: FLUME-1371
> URL: https://issues.apache.org/jira/browse/FLUME-1371
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.3.0
> Reporter: Mubarak Seyed
> Labels: noob
> Fix For: v1.3.0
>
> Attachments: FLUME-1371.patch
>
>
> Motivation:
> Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
> Design:
> ElasticSearch supports RESTful style API to publish events for indexing.
> There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
> [1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1371) ElasticSearch Sink
Posted by "Cameron Gandevia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cameron Gandevia updated FLUME-1371:
------------------------------------
Attachment: (was: FLUME-1371.patch)
> ElasticSearch Sink
> ------------------
>
> Key: FLUME-1371
> URL: https://issues.apache.org/jira/browse/FLUME-1371
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.3.0
> Reporter: Mubarak Seyed
> Labels: noob
> Attachments: FLUME-1371.patch
>
>
> Motivation:
> Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
> Design:
> ElasticSearch supports RESTful style API to publish events for indexing.
> There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
> [1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1371) ElasticSearch Sink
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484174#comment-13484174 ]
Brock Noland commented on FLUME-1371:
-------------------------------------
1.3 sounds good for this.
> ElasticSearch Sink
> ------------------
>
> Key: FLUME-1371
> URL: https://issues.apache.org/jira/browse/FLUME-1371
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.3.0
> Reporter: Mubarak Seyed
> Labels: noob
> Fix For: v1.3.0
>
> Attachments: FLUME-1371.patch, FLUME-1371.patch
>
>
> Motivation:
> Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
> Design:
> ElasticSearch supports RESTful style API to publish events for indexing.
> There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
> [1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1371) ElasticSearch Sink
Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475933#comment-13475933 ]
Otis Gospodnetic commented on FLUME-1371:
-----------------------------------------
+1!
Should Fix Version be set to 1.3.0 so this gets included in the 1.3.0 release?
[~gnoremac] Thanks! Are you actually using this somewhere and can you comments on the stability, performance, etc.?
> ElasticSearch Sink
> ------------------
>
> Key: FLUME-1371
> URL: https://issues.apache.org/jira/browse/FLUME-1371
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Reporter: Mubarak Seyed
> Labels: noob
> Attachments: FLUME-1371.patch
>
>
> Motivation:
> Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
> Design:
> ElasticSearch supports RESTful style API to publish events for indexing.
> There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
> [1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1371) ElasticSearch Sink
Posted by "Cameron Gandevia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cameron Gandevia updated FLUME-1371:
------------------------------------
Attachment: FLUME-1371.patch
First cut of an elasticsearch sink I plan on using. It is based off of the existing flumeelastic sink
> ElasticSearch Sink
> ------------------
>
> Key: FLUME-1371
> URL: https://issues.apache.org/jira/browse/FLUME-1371
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Reporter: Mubarak Seyed
> Labels: noob
> Attachments: FLUME-1371.patch
>
>
> Motivation:
> Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
> Design:
> ElasticSearch supports RESTful style API to publish events for indexing.
> There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
> [1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1371) ElasticSearch Sink
Posted by "Cameron Gandevia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cameron Gandevia updated FLUME-1371:
------------------------------------
Attachment: FLUME-1371.patch
> ElasticSearch Sink
> ------------------
>
> Key: FLUME-1371
> URL: https://issues.apache.org/jira/browse/FLUME-1371
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.3.0
> Reporter: Mubarak Seyed
> Labels: noob
> Attachments: FLUME-1371.patch
>
>
> Motivation:
> Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
> Design:
> ElasticSearch supports RESTful style API to publish events for indexing.
> There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
> [1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1371) ElasticSearch Sink
Posted by "Cameron Gandevia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cameron Gandevia updated FLUME-1371:
------------------------------------
Attachment: FLUME-1371.patch
> ElasticSearch Sink
> ------------------
>
> Key: FLUME-1371
> URL: https://issues.apache.org/jira/browse/FLUME-1371
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.3.0
> Reporter: Mubarak Seyed
> Labels: noob
> Attachments: FLUME-1371.patch, FLUME-1371.patch
>
>
> Motivation:
> Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
> Design:
> ElasticSearch supports RESTful style API to publish events for indexing.
> There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
> [1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1371) ElasticSearch Sink
Posted by "Jorn Argelo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483106#comment-13483106 ]
Jorn Argelo commented on FLUME-1371:
------------------------------------
As a humble user I'd like to say that it would be great to see this going into 1.3.0. That means I no longer need a rabbitmq or something like that to put stuff into Elastic Search.
> ElasticSearch Sink
> ------------------
>
> Key: FLUME-1371
> URL: https://issues.apache.org/jira/browse/FLUME-1371
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.3.0
> Reporter: Mubarak Seyed
> Labels: noob
> Attachments: FLUME-1371.patch
>
>
> Motivation:
> Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
> Design:
> ElasticSearch supports RESTful style API to publish events for indexing.
> There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
> [1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1371) ElasticSearch Sink
Posted by "Cameron Gandevia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cameron Gandevia updated FLUME-1371:
------------------------------------
Attachment: FLUME-1371.patch
> ElasticSearch Sink
> ------------------
>
> Key: FLUME-1371
> URL: https://issues.apache.org/jira/browse/FLUME-1371
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.3.0
> Reporter: Mubarak Seyed
> Labels: noob
> Fix For: v1.3.0
>
> Attachments: FLUME-1371.patch
>
>
> Motivation:
> Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
> Design:
> ElasticSearch supports RESTful style API to publish events for indexing.
> There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
> [1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1371) ElasticSearch Sink
Posted by "Cameron Gandevia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cameron Gandevia updated FLUME-1371:
------------------------------------
Attachment: FLUME-1371.patch
> ElasticSearch Sink
> ------------------
>
> Key: FLUME-1371
> URL: https://issues.apache.org/jira/browse/FLUME-1371
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.3.0
> Reporter: Mubarak Seyed
> Labels: noob
> Attachments: FLUME-1371.patch
>
>
> Motivation:
> Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
> Design:
> ElasticSearch supports RESTful style API to publish events for indexing.
> There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
> [1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1371) ElasticSearch Sink
Posted by "Cameron Gandevia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cameron Gandevia updated FLUME-1371:
------------------------------------
Attachment: FLUME-1371.patch
Fixed EventSerializer to push through Malformed JSON as text
> ElasticSearch Sink
> ------------------
>
> Key: FLUME-1371
> URL: https://issues.apache.org/jira/browse/FLUME-1371
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.3.0
> Reporter: Mubarak Seyed
> Labels: noob
> Fix For: v1.3.0
>
> Attachments: FLUME-1371.patch, FLUME-1371.patch
>
>
> Motivation:
> Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
> Design:
> ElasticSearch supports RESTful style API to publish events for indexing.
> There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
> [1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1371) ElasticSearch Sink
Posted by "Cameron Gandevia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cameron Gandevia updated FLUME-1371:
------------------------------------
Attachment: (was: FLUME-1371.patch)
> ElasticSearch Sink
> ------------------
>
> Key: FLUME-1371
> URL: https://issues.apache.org/jira/browse/FLUME-1371
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.3.0
> Reporter: Mubarak Seyed
> Labels: noob
> Attachments: FLUME-1371.patch
>
>
> Motivation:
> Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
> Design:
> ElasticSearch supports RESTful style API to publish events for indexing.
> There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
> [1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1371) ElasticSearch Sink
Posted by "Cameron Gandevia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cameron Gandevia updated FLUME-1371:
------------------------------------
Attachment: FLUME-1371.patch
> ElasticSearch Sink
> ------------------
>
> Key: FLUME-1371
> URL: https://issues.apache.org/jira/browse/FLUME-1371
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.3.0
> Reporter: Mubarak Seyed
> Labels: noob
> Attachments: FLUME-1371.patch
>
>
> Motivation:
> Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
> Design:
> ElasticSearch supports RESTful style API to publish events for indexing.
> There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
> [1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1371) ElasticSearch Sink
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Shreedharan updated FLUME-1371:
------------------------------------
Fix Version/s: v1.3.0
Tentatively setting Fix Version as 1.3.
[~brocknoland] - Please change if you feel this may not make it.
> ElasticSearch Sink
> ------------------
>
> Key: FLUME-1371
> URL: https://issues.apache.org/jira/browse/FLUME-1371
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.3.0
> Reporter: Mubarak Seyed
> Labels: noob
> Fix For: v1.3.0
>
> Attachments: FLUME-1371.patch
>
>
> Motivation:
> Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
> Design:
> ElasticSearch supports RESTful style API to publish events for indexing.
> There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
> [1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1371) ElasticSearch Sink
Posted by "Cameron Gandevia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cameron Gandevia updated FLUME-1371:
------------------------------------
Attachment: (was: FLUME-1371.patch)
> ElasticSearch Sink
> ------------------
>
> Key: FLUME-1371
> URL: https://issues.apache.org/jira/browse/FLUME-1371
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.3.0
> Reporter: Mubarak Seyed
> Labels: noob
> Fix For: v1.3.0
>
> Attachments: FLUME-1371.patch
>
>
> Motivation:
> Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
> Design:
> ElasticSearch supports RESTful style API to publish events for indexing.
> There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
> [1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1371) ElasticSearch Sink
Posted by "Cameron Gandevia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cameron Gandevia updated FLUME-1371:
------------------------------------
Attachment: (was: FLUME-1371.patch)
> ElasticSearch Sink
> ------------------
>
> Key: FLUME-1371
> URL: https://issues.apache.org/jira/browse/FLUME-1371
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.3.0
> Reporter: Mubarak Seyed
> Labels: noob
> Attachments: FLUME-1371.patch
>
>
> Motivation:
> Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
> Design:
> ElasticSearch supports RESTful style API to publish events for indexing.
> There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
> [1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1371) ElasticSearch Sink
Posted by "Cameron Gandevia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cameron Gandevia updated FLUME-1371:
------------------------------------
Attachment: (was: FLUME-1371.patch)
> ElasticSearch Sink
> ------------------
>
> Key: FLUME-1371
> URL: https://issues.apache.org/jira/browse/FLUME-1371
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.3.0
> Reporter: Mubarak Seyed
> Labels: noob
> Fix For: v1.3.0
>
> Attachments: FLUME-1371.patch
>
>
> Motivation:
> Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
> Design:
> ElasticSearch supports RESTful style API to publish events for indexing.
> There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
> [1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1371) ElasticSearch Sink
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485546#comment-13485546 ]
Hudson commented on FLUME-1371:
-------------------------------
Integrated in flume-trunk #319 (See [https://builds.apache.org/job/flume-trunk/319/])
FLUME-1371: ElasticSearch Sink (Revision 6d083df627e7513307d328dd9c92748bd055de0b)
Result = SUCCESS
brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=6d083df627e7513307d328dd9c92748bd055de0b
Files :
* flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/ElasticSearchSinkTest.java
* flume-ng-sinks/pom.xml
* flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/ElasticSearchDynamicSerializerTest.java
* flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java
* flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSinkConstants.java
* flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/ElasticSearchLogStashEventSerializerTest.java
* flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
* flume-ng-sinks/flume-ng-elasticsearch-sink/pom.xml
* pom.xml
* flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchLogStashEventSerializer.java
* flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchDynamicSerializer.java
* flume-ng-dist/pom.xml
* flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchSinkTest.java
* flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchEventSerializer.java
* flume-ng-doc/sphinx/FlumeUserGuide.rst
> ElasticSearch Sink
> ------------------
>
> Key: FLUME-1371
> URL: https://issues.apache.org/jira/browse/FLUME-1371
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Affects Versions: v1.3.0
> Reporter: Mubarak Seyed
> Assignee: Cameron Gandevia
> Labels: noob
> Fix For: v1.3.0
>
> Attachments: FLUME-1371.patch
>
>
> Motivation:
> Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.
> Design:
> ElasticSearch supports RESTful style API to publish events for indexing.
> There is an existing ??flumeelastic?? sink [1] in github but it does not support batching of events for indexing.
> [1] https://github.com/Aconex/elasticflume
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira