You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Roberto Coluccio <ro...@eng.it> on 2016/12/01 10:59:39 UTC
Re: Define and leverage custom constants in agent configuration
Hi Silvio,
thanks for your suggestion. It gives me a workaround, but it's not
exactly what I was hoping/asking about.
I'm going to assume what I'd want to do being just not doable.
Thank you,
Roberto
On 29/11/2016 09:26, Silvio Di gregorio wrote:
> with the "static interceptor" you can add key/value pairs in your
> flume event header:
>
> agentOne.sources.s1.interceptors = interOne
> agentOne.sources.s1.interceptors.interOne.type = static
> agentOne.sources.s1.interceptors.interOne.key = costante
> agentOne.sources.s1.interceptors.interOne.value = quello_che_vuoi
>
> I hope it is useful
> have nice day
> Silvio
>
> 2016-11-28 18:26 GMT+01:00 Roberto Coluccio <roberto.coluccio@eng.it
> <ma...@eng.it>>:
>
> Hello folks,
>
> I was wondering if it's possible to define custom constants in a
> Flume agent configuration file, so to avoid errors and replicated
> stuff when not needed. An example to make my point clear:
>
> assume I want to launch an agent of type /myAgent/ configured with
> file channel and hdfs sink.
>
> I want the file channel to persist data and checkpoint into
> "~/flume/myAgent1/data" and "~/flume/myAgent1/checkpoint"
>
> Also, I want the hdfs sink to write files with prefix "myAgent1"
>
> In this scenario, I should harcorde the "myAgent1" string 3 times,
> one for each configuration parameter. This is error prone and
> kinda frustrating.
>
> Is there a way to define a constant like "myConstant" in the
> configuration file, and leverage it as parameter in the other
> configuration lines?
>
> e.g.
>
> myConstant = myAgent1
> myAgent.channels.myChannel.dataDirs = ~/flume/${myConstant}/data
> myAgent.sinks.mySink.hdfs.filePrefix = ${myConstant}
>
> It seems like the use of flume events header keys, but I don't
> want to add interceptors or other components just to have
> parameters in my config file.
>
> Thanks for your help.
>
> Best regards,
>
> Roberto
>
>
>
--
**
**
**
**
**
*Roberto Coluccio*
Solution Designer
Direzione Ricerca e Innovazione
roberto.coluccio@eng.it <ma...@eng.it>
Interno 604135 | Esterno +39 0510435135
*Engineering Ingegneria Informatica spa*
Via Marconi, 10, 40122 Bologna
www.eng.it <http://www.eng.it/>
Re: Define and leverage custom constants in agent configuration
Posted by Bessenyei Balázs Donát <be...@apache.org>.
(+CC: dev@)
I have requested reviews and comments on the ticket.
If nobody disagrees, I'll commit that change soon.
Donát
2016-12-02 9:33 GMT+01:00 Roberto Coluccio <ro...@eng.it>:
> That would be a cool option, Donat!
>
> On 01/12/2016 19:41, Bessenyei Balázs Donát wrote:
>
> Hi Roberto,
>
> Do you mean something like the proposed change at
> https://issues.apache.org/jira/browse/FLUME-2993 ? Would that fit your
> requirements?
>
> Thank you,
>
> Donat
>
> On Dec 1, 2016 7:33 PM, "Shang Wu (CS)" <Sh...@symantec.com> wrote:
>
>> Hi Robert,
>>
>> Looks like what you want is more like a global configuration, and based
>> on my limited knowledge in Apache Flume, it is missing now.
>>
>> BR,
>>
>> Shang
>>
>> *From: *Roberto Coluccio <ro...@eng.it>
>> *Reply-To: *"user@flume.apache.org" <us...@flume.apache.org>
>> *Date: *Thursday, December 1, 2016 at 2:59 AM
>> *To: *"user@flume.apache.org" <us...@flume.apache.org>
>> *Subject: *Re: Define and leverage custom constants in agent
>> configuration
>>
>>
>>
>> Hi Silvio,
>>
>> thanks for your suggestion. It gives me a workaround, but it's not
>> exactly what I was hoping/asking about.
>>
>> I'm going to assume what I'd want to do being just not doable.
>>
>> Thank you,
>>
>> Roberto
>>
>>
>>
>> On 29/11/2016 09:26, Silvio Di gregorio wrote:
>>
>> with the "static interceptor" you can add key/value pairs in your flume
>> event header:
>>
>>
>>
>> agentOne.sources.s1.interceptors = interOne
>>
>> agentOne.sources.s1.interceptors.interOne.type = static
>>
>> agentOne.sources.s1.interceptors.interOne.key = costante
>>
>> agentOne.sources.s1.interceptors.interOne.value = quello_che_vuoi
>>
>>
>>
>> I hope it is useful
>>
>> have nice day
>>
>> Silvio
>>
>>
>>
>> 2016-11-28 18:26 GMT+01:00 Roberto Coluccio <ro...@eng.it>:
>>
>> Hello folks,
>>
>> I was wondering if it's possible to define custom constants in a Flume
>> agent configuration file, so to avoid errors and replicated stuff when not
>> needed. An example to make my point clear:
>>
>> assume I want to launch an agent of type *myAgent* configured with file
>> channel and hdfs sink.
>>
>> I want the file channel to persist data and checkpoint into
>> "~/flume/myAgent1/data" and "~/flume/myAgent1/checkpoint"
>>
>> Also, I want the hdfs sink to write files with prefix "myAgent1"
>>
>> In this scenario, I should harcorde the "myAgent1" string 3 times, one
>> for each configuration parameter. This is error prone and kinda frustrating.
>>
>> Is there a way to define a constant like "myConstant" in the
>> configuration file, and leverage it as parameter in the other configuration
>> lines?
>>
>> e.g.
>>
>> myConstant = myAgent1
>> myAgent.channels.myChannel.dataDirs = ~/flume/${myConstant}/data
>> myAgent.sinks.mySink.hdfs.filePrefix = ${myConstant}
>>
>> It seems like the use of flume events header keys, but I don't want to
>> add interceptors or other components just to have parameters in my config
>> file.
>>
>> Thanks for your help.
>>
>> Best regards,
>>
>> Roberto
>>
>>
>>
>>
>>
>> --
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *Roberto Coluccio*
>>
>> Solution Designer
>> Direzione Ricerca e Innovazione
>> roberto.coluccio@eng.it
>>
>> Interno 604135 | Esterno +39 0510435135 <+39%20051%20043%205135>
>>
>>
>>
>> *Engineering Ingegneria Informatica spa*
>> Via Marconi, 10, 40122 Bologna
>>
>> www.eng.it
>>
>>
>>
>
> --
>
>
>
>
>
>
>
>
>
> *Roberto Coluccio*
>
> Solution Designer
> Direzione Ricerca e Innovazione
> roberto.coluccio@eng.it
>
> Interno 604135 | Esterno +39 0510435135 <+39%20051%20043%205135>
>
>
> *Engineering Ingegneria Informatica spa*
> Via Marconi, 10, 40122 Bologna
>
> www.eng.it
>
>
>
Re: Define and leverage custom constants in agent configuration
Posted by Bessenyei Balázs Donát <be...@apache.org>.
(+CC: dev@)
I have requested reviews and comments on the ticket.
If nobody disagrees, I'll commit that change soon.
Donát
2016-12-02 9:33 GMT+01:00 Roberto Coluccio <ro...@eng.it>:
> That would be a cool option, Donat!
>
> On 01/12/2016 19:41, Bessenyei Balázs Donát wrote:
>
> Hi Roberto,
>
> Do you mean something like the proposed change at
> https://issues.apache.org/jira/browse/FLUME-2993 ? Would that fit your
> requirements?
>
> Thank you,
>
> Donat
>
> On Dec 1, 2016 7:33 PM, "Shang Wu (CS)" <Sh...@symantec.com> wrote:
>
>> Hi Robert,
>>
>> Looks like what you want is more like a global configuration, and based
>> on my limited knowledge in Apache Flume, it is missing now.
>>
>> BR,
>>
>> Shang
>>
>> *From: *Roberto Coluccio <ro...@eng.it>
>> *Reply-To: *"user@flume.apache.org" <us...@flume.apache.org>
>> *Date: *Thursday, December 1, 2016 at 2:59 AM
>> *To: *"user@flume.apache.org" <us...@flume.apache.org>
>> *Subject: *Re: Define and leverage custom constants in agent
>> configuration
>>
>>
>>
>> Hi Silvio,
>>
>> thanks for your suggestion. It gives me a workaround, but it's not
>> exactly what I was hoping/asking about.
>>
>> I'm going to assume what I'd want to do being just not doable.
>>
>> Thank you,
>>
>> Roberto
>>
>>
>>
>> On 29/11/2016 09:26, Silvio Di gregorio wrote:
>>
>> with the "static interceptor" you can add key/value pairs in your flume
>> event header:
>>
>>
>>
>> agentOne.sources.s1.interceptors = interOne
>>
>> agentOne.sources.s1.interceptors.interOne.type = static
>>
>> agentOne.sources.s1.interceptors.interOne.key = costante
>>
>> agentOne.sources.s1.interceptors.interOne.value = quello_che_vuoi
>>
>>
>>
>> I hope it is useful
>>
>> have nice day
>>
>> Silvio
>>
>>
>>
>> 2016-11-28 18:26 GMT+01:00 Roberto Coluccio <ro...@eng.it>:
>>
>> Hello folks,
>>
>> I was wondering if it's possible to define custom constants in a Flume
>> agent configuration file, so to avoid errors and replicated stuff when not
>> needed. An example to make my point clear:
>>
>> assume I want to launch an agent of type *myAgent* configured with file
>> channel and hdfs sink.
>>
>> I want the file channel to persist data and checkpoint into
>> "~/flume/myAgent1/data" and "~/flume/myAgent1/checkpoint"
>>
>> Also, I want the hdfs sink to write files with prefix "myAgent1"
>>
>> In this scenario, I should harcorde the "myAgent1" string 3 times, one
>> for each configuration parameter. This is error prone and kinda frustrating.
>>
>> Is there a way to define a constant like "myConstant" in the
>> configuration file, and leverage it as parameter in the other configuration
>> lines?
>>
>> e.g.
>>
>> myConstant = myAgent1
>> myAgent.channels.myChannel.dataDirs = ~/flume/${myConstant}/data
>> myAgent.sinks.mySink.hdfs.filePrefix = ${myConstant}
>>
>> It seems like the use of flume events header keys, but I don't want to
>> add interceptors or other components just to have parameters in my config
>> file.
>>
>> Thanks for your help.
>>
>> Best regards,
>>
>> Roberto
>>
>>
>>
>>
>>
>> --
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *Roberto Coluccio*
>>
>> Solution Designer
>> Direzione Ricerca e Innovazione
>> roberto.coluccio@eng.it
>>
>> Interno 604135 | Esterno +39 0510435135 <+39%20051%20043%205135>
>>
>>
>>
>> *Engineering Ingegneria Informatica spa*
>> Via Marconi, 10, 40122 Bologna
>>
>> www.eng.it
>>
>>
>>
>
> --
>
>
>
>
>
>
>
>
>
> *Roberto Coluccio*
>
> Solution Designer
> Direzione Ricerca e Innovazione
> roberto.coluccio@eng.it
>
> Interno 604135 | Esterno +39 0510435135 <+39%20051%20043%205135>
>
>
> *Engineering Ingegneria Informatica spa*
> Via Marconi, 10, 40122 Bologna
>
> www.eng.it
>
>
>
Re: Define and leverage custom constants in agent configuration
Posted by Roberto Coluccio <ro...@eng.it>.
That would be a cool option, Donat!
On 01/12/2016 19:41, Bessenyei Bal�zs Don�t wrote:
>
> Hi Roberto,
>
> Do you mean something like the proposed change at
> https://issues.apache.org/jira/browse/FLUME-2993 ? Would that fit your
> requirements?
>
> Thank you,
>
> Donat
>
>
> On Dec 1, 2016 7:33 PM, "Shang Wu (CS)" <Shang_Wu@symantec.com
> <ma...@symantec.com>> wrote:
>
> Hi Robert,
>
> Looks like what you want is more like a global configuration, and
> based on my limited knowledge in Apache Flume, it is missing now.
>
> BR,
>
> Shang
>
> *From: *Roberto Coluccio <roberto.coluccio@eng.it
> <ma...@eng.it>>
> *Reply-To: *"user@flume.apache.org <ma...@flume.apache.org>"
> <user@flume.apache.org <ma...@flume.apache.org>>
> *Date: *Thursday, December 1, 2016 at 2:59 AM
> *To: *"user@flume.apache.org <ma...@flume.apache.org>"
> <user@flume.apache.org <ma...@flume.apache.org>>
> *Subject: *Re: Define and leverage custom constants in agent
> configuration
>
> Hi Silvio,
>
> thanks for your suggestion. It gives me a workaround, but it's not
> exactly what I was hoping/asking about.
>
> I'm going to assume what I'd want to do being just not doable.
>
> Thank you,
>
> Roberto
>
> On 29/11/2016 09:26, Silvio Di gregorio wrote:
>
> with the "static interceptor" you can add key/value pairs in
> your flume event header:
>
> agentOne.sources.s1.interceptors = interOne
>
> agentOne.sources.s1.interceptors.interOne.type = static
>
> agentOne.sources.s1.interceptors.interOne.key = costante
>
> agentOne.sources.s1.interceptors.interOne.value = quello_che_vuoi
>
> I hope it is useful
>
> have nice day
>
> Silvio
>
> 2016-11-28 18:26 GMT+01:00 Roberto Coluccio
> <roberto.coluccio@eng.it <ma...@eng.it>>:
>
> Hello folks,
>
> I was wondering if it's possible to define custom
> constants in a Flume agent configuration file, so to avoid
> errors and replicated stuff when not needed. An example to
> make my point clear:
>
> assume I want to launch an agent of type /myAgent/
> configured with file channel and hdfs sink.
>
> I want the file channel to persist data and checkpoint
> into "~/flume/myAgent1/data" and "~/flume/myAgent1/checkpoint"
>
> Also, I want the hdfs sink to write files with prefix
> "myAgent1"
>
> In this scenario, I should harcorde the "myAgent1" string
> 3 times, one for each configuration parameter. This is
> error prone and kinda frustrating.
>
> Is there a way to define a constant like "myConstant" in
> the configuration file, and leverage it as parameter in
> the other configuration lines?
>
> e.g.
>
> myConstant = myAgent1
> myAgent.channels.myChannel.dataDirs =
> ~/flume/${myConstant}/data
> myAgent.sinks.mySink.hdfs.filePrefix = ${myConstant}
>
> It seems like the use of flume events header keys, but I
> don't want to add interceptors or other components just to
> have parameters in my config file.
>
> Thanks for your help.
>
> Best regards,
>
> Roberto
>
> --
>
> **
>
> **
>
> **
>
> **
>
> *Roberto Coluccio*
>
> Solution Designer
> Direzione Ricerca e Innovazione
> roberto.coluccio@eng.it <ma...@eng.it>
>
> Interno 604135 | Esterno +39 0510435135 <tel:+39%20051%20043%205135>
>
>
>
> *Engineering Ingegneria Informatica spa*
> Via Marconi, 10, 40122 Bologna
>
> www.eng.it <http://www.eng.it/>
>
--
**
**
**
**
**
*Roberto Coluccio*
Solution Designer
Direzione Ricerca e Innovazione
roberto.coluccio@eng.it <ma...@eng.it>
Interno 604135 | Esterno +39 0510435135
*Engineering Ingegneria Informatica spa*
Via Marconi, 10, 40122 Bologna
www.eng.it <http://www.eng.it/>
Re: Define and leverage custom constants in agent configuration
Posted by Bessenyei Balázs Donát <be...@apache.org>.
Hi Roberto,
Do you mean something like the proposed change at
https://issues.apache.org/jira/browse/FLUME-2993 ? Would that fit your
requirements?
Thank you,
Donat
On Dec 1, 2016 7:33 PM, "Shang Wu (CS)" <Sh...@symantec.com> wrote:
> Hi Robert,
>
> Looks like what you want is more like a global configuration, and based on
> my limited knowledge in Apache Flume, it is missing now.
>
> BR,
>
> Shang
>
> *From: *Roberto Coluccio <ro...@eng.it>
> *Reply-To: *"user@flume.apache.org" <us...@flume.apache.org>
> *Date: *Thursday, December 1, 2016 at 2:59 AM
> *To: *"user@flume.apache.org" <us...@flume.apache.org>
> *Subject: *Re: Define and leverage custom constants in agent configuration
>
>
>
> Hi Silvio,
>
> thanks for your suggestion. It gives me a workaround, but it's not exactly
> what I was hoping/asking about.
>
> I'm going to assume what I'd want to do being just not doable.
>
> Thank you,
>
> Roberto
>
>
>
> On 29/11/2016 09:26, Silvio Di gregorio wrote:
>
> with the "static interceptor" you can add key/value pairs in your flume
> event header:
>
>
>
> agentOne.sources.s1.interceptors = interOne
>
> agentOne.sources.s1.interceptors.interOne.type = static
>
> agentOne.sources.s1.interceptors.interOne.key = costante
>
> agentOne.sources.s1.interceptors.interOne.value = quello_che_vuoi
>
>
>
> I hope it is useful
>
> have nice day
>
> Silvio
>
>
>
> 2016-11-28 18:26 GMT+01:00 Roberto Coluccio <ro...@eng.it>:
>
> Hello folks,
>
> I was wondering if it's possible to define custom constants in a Flume
> agent configuration file, so to avoid errors and replicated stuff when not
> needed. An example to make my point clear:
>
> assume I want to launch an agent of type *myAgent* configured with file
> channel and hdfs sink.
>
> I want the file channel to persist data and checkpoint into
> "~/flume/myAgent1/data" and "~/flume/myAgent1/checkpoint"
>
> Also, I want the hdfs sink to write files with prefix "myAgent1"
>
> In this scenario, I should harcorde the "myAgent1" string 3 times, one for
> each configuration parameter. This is error prone and kinda frustrating.
>
> Is there a way to define a constant like "myConstant" in the configuration
> file, and leverage it as parameter in the other configuration lines?
>
> e.g.
>
> myConstant = myAgent1
> myAgent.channels.myChannel.dataDirs = ~/flume/${myConstant}/data
> myAgent.sinks.mySink.hdfs.filePrefix = ${myConstant}
>
> It seems like the use of flume events header keys, but I don't want to add
> interceptors or other components just to have parameters in my config file.
>
> Thanks for your help.
>
> Best regards,
>
> Roberto
>
>
>
>
>
> --
>
>
>
>
>
>
>
>
>
> *Roberto Coluccio*
>
> Solution Designer
> Direzione Ricerca e Innovazione
> roberto.coluccio@eng.it
>
> Interno 604135 | Esterno +39 0510435135 <+39%20051%20043%205135>
>
>
>
> *Engineering Ingegneria Informatica spa*
> Via Marconi, 10, 40122 Bologna
>
> www.eng.it
>
>
>
Re: Define and leverage custom constants in agent configuration
Posted by "Shang Wu (CS)" <Sh...@symantec.com>.
Hi Robert,
Looks like what you want is more like a global configuration, and based on my limited knowledge in Apache Flume, it is missing now.
BR,
Shang
From: Roberto Coluccio <ro...@eng.it>
Reply-To: "user@flume.apache.org" <us...@flume.apache.org>
Date: Thursday, December 1, 2016 at 2:59 AM
To: "user@flume.apache.org" <us...@flume.apache.org>
Subject: Re: Define and leverage custom constants in agent configuration
Hi Silvio,
thanks for your suggestion. It gives me a workaround, but it's not exactly what I was hoping/asking about.
I'm going to assume what I'd want to do being just not doable.
Thank you,
Roberto
On 29/11/2016 09:26, Silvio Di gregorio wrote:
with the "static interceptor" you can add key/value pairs in your flume event header:
agentOne.sources.s1.interceptors = interOne
agentOne.sources.s1.interceptors.interOne.type = static
agentOne.sources.s1.interceptors.interOne.key = costante
agentOne.sources.s1.interceptors.interOne.value = quello_che_vuoi
I hope it is useful
have nice day
Silvio
2016-11-28 18:26 GMT+01:00 Roberto Coluccio <ro...@eng.it>>:
Hello folks,
I was wondering if it's possible to define custom constants in a Flume agent configuration file, so to avoid errors and replicated stuff when not needed. An example to make my point clear:
assume I want to launch an agent of type myAgent configured with file channel and hdfs sink.
I want the file channel to persist data and checkpoint into "~/flume/myAgent1/data" and "~/flume/myAgent1/checkpoint"
Also, I want the hdfs sink to write files with prefix "myAgent1"
In this scenario, I should harcorde the "myAgent1" string 3 times, one for each configuration parameter. This is error prone and kinda frustrating.
Is there a way to define a constant like "myConstant" in the configuration file, and leverage it as parameter in the other configuration lines?
e.g.
myConstant = myAgent1
myAgent.channels.myChannel.dataDirs = ~/flume/${myConstant}/data
myAgent.sinks.mySink.hdfs.filePrefix = ${myConstant}
It seems like the use of flume events header keys, but I don't want to add interceptors or other components just to have parameters in my config file.
Thanks for your help.
Best regards,
Roberto
--
[cid:image001.jpg@01D24BBD.CB7A2E90]
Roberto Coluccio
Solution Designer
Direzione Ricerca e Innovazione
roberto.coluccio@eng.it<ma...@eng.it>
Interno 604135 | Esterno +39 0510435135
Engineering Ingegneria Informatica spa
Via Marconi, 10, 40122 Bologna
www.eng.it<http://www.eng.it/>