You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Satoshi Iijima <pa...@yahoo.co.jp> on 2014/04/23 05:05:39 UTC
Review Request 20601: Flume-2273 ElasticSearchSink: Add handling for header
substitution in indexName and indexType
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20601/
-----------------------------------------------------------
Review request for Flume.
Repository: flume-git
Description
-------
The ElasticSearchSink would be improved by allowing for header substitution in the indexName and indexType property.
A use case is where the sink is an intermediate part of a chain and the index name is required to identify the message origin, at present it can only be a hardcoded value.
The HDFS sink already supports header substitution so a similar format would maintain consistency.
Example is below:
a1.sinks.k1.type = elasticsearch
a1.sinks.k1.indexName = %{index-name}
a1.sinks.k1.indexType = %{index-type}
a1.sinks.k1.hostNames = 127.0.0.1:9200
a1.sinks.k1.channel = c1
Diffs
-----
flume-ng-doc/sphinx/FlumeUserGuide.rst 7b918ed
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchIndexRequestBuilderFactory.java de84b95
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java 39b6db5
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/SimpleIndexNameBuilder.java 19079af
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/TimeBasedIndexNameBuilder.java a8603a4
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchRestClient.java ff95e30
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java e9ed0b4
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java 807a9c7
Diff: https://reviews.apache.org/r/20601/diff/
Testing
-------
Thanks,
Satoshi Iijima
Re: Review Request 20601: Flume-2273 ElasticSearchSink: Add handling for
header substitution in indexName and indexType
Posted by Satoshi Iijima <pa...@yahoo.co.jp>.
> On April 30, 2014, 8:32 a.m., Edward Sargisson wrote:
> > flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchIndexRequestBuilderFactory.java, line 80
> > <https://reviews.apache.org/r/20601/diff/1/?file=565264#file565264line80>
> >
> > Is there anyway this substitution can be used as an attack? i.e. we are now allowing event submitters to provide part of the URL used to submit the event to elasticsearch (if using the REST client). Could an attacker craft something nefarious?
There is that possibility. But I think that the administrator of ElasticSearch and Flume has responsibility to judge whether header substitution is used in flume.conf.
I think that header substitution of hdfs-sink has a same issue, too.
- Satoshi
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20601/#review41786
-----------------------------------------------------------
On April 30, 2014, 7:26 p.m., Satoshi Iijima wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20601/
> -----------------------------------------------------------
>
> (Updated April 30, 2014, 7:26 p.m.)
>
>
> Review request for Flume.
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> The ElasticSearchSink would be improved by allowing for header substitution in the indexName and indexType property.
> A use case is where the sink is an intermediate part of a chain and the index name is required to identify the message origin, at present it can only be a hardcoded value.
> The HDFS sink already supports header substitution so a similar format would maintain consistency.
>
> Example is below:
>
> a1.sinks.k1.type = elasticsearch
> a1.sinks.k1.indexName = %{index-name}
> a1.sinks.k1.indexType = %{index-type}
> a1.sinks.k1.hostNames = 127.0.0.1:9200
> a1.sinks.k1.channel = c1
>
>
> Diffs
> -----
>
> flume-ng-doc/sphinx/FlumeUserGuide.rst 7b918ed
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchIndexRequestBuilderFactory.java de84b95
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java 39b6db5
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/SimpleIndexNameBuilder.java 19079af
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/TimeBasedIndexNameBuilder.java a8603a4
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchRestClient.java ff95e30
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java e9ed0b4
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java 807a9c7
>
> Diff: https://reviews.apache.org/r/20601/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Satoshi Iijima
>
>
Re: Review Request 20601: Flume-2273 ElasticSearchSink: Add handling for
header substitution in indexName and indexType
Posted by Satoshi Iijima <pa...@yahoo.co.jp>.
> On April 30, 2014, 8:32 a.m., Edward Sargisson wrote:
> > flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchIndexRequestBuilderFactory.java, line 80
> > <https://reviews.apache.org/r/20601/diff/1/?file=565264#file565264line80>
> >
> > Is there anyway this substitution can be used as an attack? i.e. we are now allowing event submitters to provide part of the URL used to submit the event to elasticsearch (if using the REST client). Could an attacker craft something nefarious?
>
> Satoshi Iijima wrote:
> There is that possibility. But I think that the administrator of ElasticSearch and Flume has responsibility to judge whether header substitution is used in flume.conf.
> I think that header substitution of hdfs-sink has a same issue, too.
I will add an explanation about the possibility of this issue in doc later.
I think this issue should be solved in other ticket if necessary.
- Satoshi
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20601/#review41786
-----------------------------------------------------------
On April 30, 2014, 7:26 p.m., Satoshi Iijima wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20601/
> -----------------------------------------------------------
>
> (Updated April 30, 2014, 7:26 p.m.)
>
>
> Review request for Flume.
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> The ElasticSearchSink would be improved by allowing for header substitution in the indexName and indexType property.
> A use case is where the sink is an intermediate part of a chain and the index name is required to identify the message origin, at present it can only be a hardcoded value.
> The HDFS sink already supports header substitution so a similar format would maintain consistency.
>
> Example is below:
>
> a1.sinks.k1.type = elasticsearch
> a1.sinks.k1.indexName = %{index-name}
> a1.sinks.k1.indexType = %{index-type}
> a1.sinks.k1.hostNames = 127.0.0.1:9200
> a1.sinks.k1.channel = c1
>
>
> Diffs
> -----
>
> flume-ng-doc/sphinx/FlumeUserGuide.rst 7b918ed
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchIndexRequestBuilderFactory.java de84b95
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java 39b6db5
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/SimpleIndexNameBuilder.java 19079af
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/TimeBasedIndexNameBuilder.java a8603a4
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchRestClient.java ff95e30
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java e9ed0b4
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java 807a9c7
>
> Diff: https://reviews.apache.org/r/20601/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Satoshi Iijima
>
>
Re: Review Request 20601: Flume-2273 ElasticSearchSink: Add handling for
header substitution in indexName and indexType
Posted by Edward Sargisson <es...@pobox.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20601/#review41786
-----------------------------------------------------------
flume-ng-doc/sphinx/FlumeUserGuide.rst
<https://reviews.apache.org/r/20601/#comment75384>
substituion should be spelt substitution.
I would probably also add that the date time will be appended to substituted values too.
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchIndexRequestBuilderFactory.java
<https://reviews.apache.org/r/20601/#comment75385>
Is there anyway this substitution can be used as an attack? i.e. we are now allowing event submitters to provide part of the URL used to submit the event to elasticsearch (if using the REST client). Could an attacker craft something nefarious?
- Edward Sargisson
On April 22, 2014, 8:05 p.m., Satoshi Iijima wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20601/
> -----------------------------------------------------------
>
> (Updated April 22, 2014, 8:05 p.m.)
>
>
> Review request for Flume.
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> The ElasticSearchSink would be improved by allowing for header substitution in the indexName and indexType property.
> A use case is where the sink is an intermediate part of a chain and the index name is required to identify the message origin, at present it can only be a hardcoded value.
> The HDFS sink already supports header substitution so a similar format would maintain consistency.
>
> Example is below:
>
> a1.sinks.k1.type = elasticsearch
> a1.sinks.k1.indexName = %{index-name}
> a1.sinks.k1.indexType = %{index-type}
> a1.sinks.k1.hostNames = 127.0.0.1:9200
> a1.sinks.k1.channel = c1
>
>
> Diffs
> -----
>
> flume-ng-doc/sphinx/FlumeUserGuide.rst 7b918ed
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchIndexRequestBuilderFactory.java de84b95
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java 39b6db5
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/SimpleIndexNameBuilder.java 19079af
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/TimeBasedIndexNameBuilder.java a8603a4
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchRestClient.java ff95e30
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java e9ed0b4
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java 807a9c7
>
> Diff: https://reviews.apache.org/r/20601/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Satoshi Iijima
>
>
Re: Review Request 20601: Flume-2273 ElasticSearchSink: Add handling for
header substitution in indexName and indexType
Posted by Edward Sargisson <es...@pobox.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20601/#review41920
-----------------------------------------------------------
flume-ng-doc/sphinx/FlumeUserGuide.rst
<https://reviews.apache.org/r/20601/#comment75574>
How about:
"header substitution is a handy to use the value of an event header to dynamically decide the indexName and indexType to use when storing the event. Caution should be used in using this feature as the event submitter now has control of the indexName and indexType. Furthermore, if the elasticsearch REST client is used then the event submitter has control of the URL path used."
- Edward Sargisson
On May 1, 2014, 3:22 a.m., Satoshi Iijima wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20601/
> -----------------------------------------------------------
>
> (Updated May 1, 2014, 3:22 a.m.)
>
>
> Review request for Flume.
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> The ElasticSearchSink would be improved by allowing for header substitution in the indexName and indexType property.
> A use case is where the sink is an intermediate part of a chain and the index name is required to identify the message origin, at present it can only be a hardcoded value.
> The HDFS sink already supports header substitution so a similar format would maintain consistency.
>
> Example is below:
>
> a1.sinks.k1.type = elasticsearch
> a1.sinks.k1.indexName = %{index-name}
> a1.sinks.k1.indexType = %{index-type}
> a1.sinks.k1.hostNames = 127.0.0.1:9200
> a1.sinks.k1.channel = c1
>
>
> Diffs
> -----
>
> flume-ng-doc/sphinx/FlumeUserGuide.rst 7b918ed
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchIndexRequestBuilderFactory.java de84b95
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java 39b6db5
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/SimpleIndexNameBuilder.java 19079af
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/TimeBasedIndexNameBuilder.java a8603a4
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchRestClient.java ff95e30
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java e9ed0b4
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java 807a9c7
>
> Diff: https://reviews.apache.org/r/20601/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Satoshi Iijima
>
>
Re: Review Request 20601: Flume-2273 ElasticSearchSink: Add handling for
header substitution in indexName and indexType
Posted by Satoshi Iijima <pa...@yahoo.co.jp>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20601/
-----------------------------------------------------------
(Updated May 23, 2014, 10:36 p.m.)
Review request for Flume.
Changes
-------
Final patch after incorporating Edward's comments.
Repository: flume-git
Description
-------
The ElasticSearchSink would be improved by allowing for header substitution in the indexName and indexType property.
A use case is where the sink is an intermediate part of a chain and the index name is required to identify the message origin, at present it can only be a hardcoded value.
The HDFS sink already supports header substitution so a similar format would maintain consistency.
Example is below:
a1.sinks.k1.type = elasticsearch
a1.sinks.k1.indexName = %{index-name}
a1.sinks.k1.indexType = %{index-type}
a1.sinks.k1.hostNames = 127.0.0.1:9200
a1.sinks.k1.channel = c1
Diffs (updated)
-----
flume-ng-doc/sphinx/FlumeUserGuide.rst 7b918ed
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchIndexRequestBuilderFactory.java de84b95
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java 39b6db5
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/SimpleIndexNameBuilder.java 19079af
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/TimeBasedIndexNameBuilder.java a8603a4
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchRestClient.java ff95e30
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java e9ed0b4
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java 807a9c7
Diff: https://reviews.apache.org/r/20601/diff/
Testing
-------
Thanks,
Satoshi Iijima
Re: Review Request 20601: Flume-2273 ElasticSearchSink: Add handling for
header substitution in indexName and indexType
Posted by Satoshi Iijima <pa...@yahoo.co.jp>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20601/
-----------------------------------------------------------
(Updated May 1, 2014, 7:22 p.m.)
Review request for Flume.
Changes
-------
minor fix in doc
Repository: flume-git
Description
-------
The ElasticSearchSink would be improved by allowing for header substitution in the indexName and indexType property.
A use case is where the sink is an intermediate part of a chain and the index name is required to identify the message origin, at present it can only be a hardcoded value.
The HDFS sink already supports header substitution so a similar format would maintain consistency.
Example is below:
a1.sinks.k1.type = elasticsearch
a1.sinks.k1.indexName = %{index-name}
a1.sinks.k1.indexType = %{index-type}
a1.sinks.k1.hostNames = 127.0.0.1:9200
a1.sinks.k1.channel = c1
Diffs (updated)
-----
flume-ng-doc/sphinx/FlumeUserGuide.rst 7b918ed
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchIndexRequestBuilderFactory.java de84b95
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java 39b6db5
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/SimpleIndexNameBuilder.java 19079af
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/TimeBasedIndexNameBuilder.java a8603a4
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchRestClient.java ff95e30
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java e9ed0b4
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java 807a9c7
Diff: https://reviews.apache.org/r/20601/diff/
Testing
-------
Thanks,
Satoshi Iijima
Re: Review Request 20601: Flume-2273 ElasticSearchSink: Add handling for
header substitution in indexName and indexType
Posted by Satoshi Iijima <pa...@yahoo.co.jp>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20601/
-----------------------------------------------------------
(Updated May 1, 2014, 7:15 p.m.)
Review request for Flume.
Changes
-------
Add note that header substituion can be used for as an attack.
Repository: flume-git
Description
-------
The ElasticSearchSink would be improved by allowing for header substitution in the indexName and indexType property.
A use case is where the sink is an intermediate part of a chain and the index name is required to identify the message origin, at present it can only be a hardcoded value.
The HDFS sink already supports header substitution so a similar format would maintain consistency.
Example is below:
a1.sinks.k1.type = elasticsearch
a1.sinks.k1.indexName = %{index-name}
a1.sinks.k1.indexType = %{index-type}
a1.sinks.k1.hostNames = 127.0.0.1:9200
a1.sinks.k1.channel = c1
Diffs (updated)
-----
flume-ng-doc/sphinx/FlumeUserGuide.rst 7b918ed
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchIndexRequestBuilderFactory.java de84b95
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java 39b6db5
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/SimpleIndexNameBuilder.java 19079af
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/TimeBasedIndexNameBuilder.java a8603a4
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchRestClient.java ff95e30
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java e9ed0b4
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java 807a9c7
Diff: https://reviews.apache.org/r/20601/diff/
Testing
-------
Thanks,
Satoshi Iijima
Re: Review Request 20601: Flume-2273 ElasticSearchSink: Add handling for
header substitution in indexName and indexType
Posted by Satoshi Iijima <pa...@yahoo.co.jp>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20601/
-----------------------------------------------------------
(Updated April 30, 2014, 7:26 p.m.)
Review request for Flume.
Changes
-------
I fixed the spell of "substituion" in doc.
Repository: flume-git
Description
-------
The ElasticSearchSink would be improved by allowing for header substitution in the indexName and indexType property.
A use case is where the sink is an intermediate part of a chain and the index name is required to identify the message origin, at present it can only be a hardcoded value.
The HDFS sink already supports header substitution so a similar format would maintain consistency.
Example is below:
a1.sinks.k1.type = elasticsearch
a1.sinks.k1.indexName = %{index-name}
a1.sinks.k1.indexType = %{index-type}
a1.sinks.k1.hostNames = 127.0.0.1:9200
a1.sinks.k1.channel = c1
Diffs (updated)
-----
flume-ng-doc/sphinx/FlumeUserGuide.rst 7b918ed
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchIndexRequestBuilderFactory.java de84b95
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java 39b6db5
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/SimpleIndexNameBuilder.java 19079af
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/TimeBasedIndexNameBuilder.java a8603a4
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchRestClient.java ff95e30
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java e9ed0b4
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java 807a9c7
Diff: https://reviews.apache.org/r/20601/diff/
Testing
-------
Thanks,
Satoshi Iijima