You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Satoshi Iijima <pa...@yahoo.co.jp> on 2014/05/01 12:15:53 UTC
Re: Review Request 20601: Flume-2273 ElasticSearchSink: Add handling for
header substitution in indexName and indexType
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20601/
-----------------------------------------------------------
(Updated May 1, 2014, 7:15 p.m.)
Review request for Flume.
Changes
-------
Add note that header substituion can be used for as an attack.
Repository: flume-git
Description
-------
The ElasticSearchSink would be improved by allowing for header substitution in the indexName and indexType property.
A use case is where the sink is an intermediate part of a chain and the index name is required to identify the message origin, at present it can only be a hardcoded value.
The HDFS sink already supports header substitution so a similar format would maintain consistency.
Example is below:
a1.sinks.k1.type = elasticsearch
a1.sinks.k1.indexName = %{index-name}
a1.sinks.k1.indexType = %{index-type}
a1.sinks.k1.hostNames = 127.0.0.1:9200
a1.sinks.k1.channel = c1
Diffs (updated)
-----
flume-ng-doc/sphinx/FlumeUserGuide.rst 7b918ed
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchIndexRequestBuilderFactory.java de84b95
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java 39b6db5
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/SimpleIndexNameBuilder.java 19079af
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/TimeBasedIndexNameBuilder.java a8603a4
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchRestClient.java ff95e30
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java e9ed0b4
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java 807a9c7
Diff: https://reviews.apache.org/r/20601/diff/
Testing
-------
Thanks,
Satoshi Iijima
Re: Review Request 20601: Flume-2273 ElasticSearchSink: Add handling for
header substitution in indexName and indexType
Posted by Edward Sargisson <es...@pobox.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20601/#review41920
-----------------------------------------------------------
flume-ng-doc/sphinx/FlumeUserGuide.rst
<https://reviews.apache.org/r/20601/#comment75574>
How about:
"header substitution is a handy to use the value of an event header to dynamically decide the indexName and indexType to use when storing the event. Caution should be used in using this feature as the event submitter now has control of the indexName and indexType. Furthermore, if the elasticsearch REST client is used then the event submitter has control of the URL path used."
- Edward Sargisson
On May 1, 2014, 3:22 a.m., Satoshi Iijima wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20601/
> -----------------------------------------------------------
>
> (Updated May 1, 2014, 3:22 a.m.)
>
>
> Review request for Flume.
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> The ElasticSearchSink would be improved by allowing for header substitution in the indexName and indexType property.
> A use case is where the sink is an intermediate part of a chain and the index name is required to identify the message origin, at present it can only be a hardcoded value.
> The HDFS sink already supports header substitution so a similar format would maintain consistency.
>
> Example is below:
>
> a1.sinks.k1.type = elasticsearch
> a1.sinks.k1.indexName = %{index-name}
> a1.sinks.k1.indexType = %{index-type}
> a1.sinks.k1.hostNames = 127.0.0.1:9200
> a1.sinks.k1.channel = c1
>
>
> Diffs
> -----
>
> flume-ng-doc/sphinx/FlumeUserGuide.rst 7b918ed
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchIndexRequestBuilderFactory.java de84b95
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java 39b6db5
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/SimpleIndexNameBuilder.java 19079af
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/TimeBasedIndexNameBuilder.java a8603a4
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchRestClient.java ff95e30
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java e9ed0b4
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java 807a9c7
>
> Diff: https://reviews.apache.org/r/20601/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Satoshi Iijima
>
>
Re: Review Request 20601: Flume-2273 ElasticSearchSink: Add handling for
header substitution in indexName and indexType
Posted by Satoshi Iijima <pa...@yahoo.co.jp>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20601/
-----------------------------------------------------------
(Updated May 23, 2014, 10:36 p.m.)
Review request for Flume.
Changes
-------
Final patch after incorporating Edward's comments.
Repository: flume-git
Description
-------
The ElasticSearchSink would be improved by allowing for header substitution in the indexName and indexType property.
A use case is where the sink is an intermediate part of a chain and the index name is required to identify the message origin, at present it can only be a hardcoded value.
The HDFS sink already supports header substitution so a similar format would maintain consistency.
Example is below:
a1.sinks.k1.type = elasticsearch
a1.sinks.k1.indexName = %{index-name}
a1.sinks.k1.indexType = %{index-type}
a1.sinks.k1.hostNames = 127.0.0.1:9200
a1.sinks.k1.channel = c1
Diffs (updated)
-----
flume-ng-doc/sphinx/FlumeUserGuide.rst 7b918ed
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchIndexRequestBuilderFactory.java de84b95
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java 39b6db5
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/SimpleIndexNameBuilder.java 19079af
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/TimeBasedIndexNameBuilder.java a8603a4
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchRestClient.java ff95e30
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java e9ed0b4
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java 807a9c7
Diff: https://reviews.apache.org/r/20601/diff/
Testing
-------
Thanks,
Satoshi Iijima
Re: Review Request 20601: Flume-2273 ElasticSearchSink: Add handling for
header substitution in indexName and indexType
Posted by Satoshi Iijima <pa...@yahoo.co.jp>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20601/
-----------------------------------------------------------
(Updated May 1, 2014, 7:22 p.m.)
Review request for Flume.
Changes
-------
minor fix in doc
Repository: flume-git
Description
-------
The ElasticSearchSink would be improved by allowing for header substitution in the indexName and indexType property.
A use case is where the sink is an intermediate part of a chain and the index name is required to identify the message origin, at present it can only be a hardcoded value.
The HDFS sink already supports header substitution so a similar format would maintain consistency.
Example is below:
a1.sinks.k1.type = elasticsearch
a1.sinks.k1.indexName = %{index-name}
a1.sinks.k1.indexType = %{index-type}
a1.sinks.k1.hostNames = 127.0.0.1:9200
a1.sinks.k1.channel = c1
Diffs (updated)
-----
flume-ng-doc/sphinx/FlumeUserGuide.rst 7b918ed
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchIndexRequestBuilderFactory.java de84b95
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java 39b6db5
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/SimpleIndexNameBuilder.java 19079af
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/TimeBasedIndexNameBuilder.java a8603a4
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchRestClient.java ff95e30
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java e9ed0b4
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java 807a9c7
Diff: https://reviews.apache.org/r/20601/diff/
Testing
-------
Thanks,
Satoshi Iijima