You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flume.apache.org by R P <ha...@outlook.com> on 2015/12/03 03:45:31 UTC

Context/Configuration values not passed to custom serializer.

Hello All,

 I need to use some configuration values in my implementation of Serializer which extends only EventSerializer.

For customizing behavior I am trying to use Context provided via Builder's build method.


Although the configuration values or none of the Context properties are not passed to Serializer.


I am able to use this Serializer without configuration through flume and write data so I am certain that my implementation works correctly.


Any idea why Context is passed as empty to the Serializer?  Or any pointers on how I should go on debugging this issue?


Thanks for reading and appreciate your response.


Thanks,

R

Re: Context/Configuration values not passed to custom serializer.

Posted by R P <ha...@outlook.com>.

Thanks Gonzalo.
I was missing serializer in my configuration, that's why it didn't appear in context.

-R

________________________________
From: Gonzalo Herreros <gh...@gmail.com>
Sent: Thursday, December 3, 2015 1:54 AM
To: user
Subject: Re: Context/Configuration values not passed to custom serializer.

Not sure how you are using the serializer but I understand it's part of an hdfs sink

When you do this:
a1.sinks.k1.IgnoreNullKeyData=True

your property goes to the sink.
To make it apperar in the serializer context it should be like this:

a1.sinks.k1.serializer.IgnoreNullKeyData=True

How are you using the serializer in the configuration?

Gonzalo

On 3 December 2015 at 09:49, R P <ha...@outlook.com>> wrote:

Thanks for the reply Gonzalo.

There is no configure method as Serializer doesn't extend Configurable.
Do I have to extend Configurable to access custom configuration values?

-R
P.S - Sent code sample and config in separate email directly to you.

________________________________
From: Gonzalo Herreros <gh...@gmail.com>>
Sent: Thursday, December 3, 2015 12:32 AM
To: user
Subject: Re: Context/Configuration values not passed to custom serializer.

It might be a bug in the sink you are using.
For instance, I have a serializer for the HbaseSink so I added two custom properties.

tier1.sinks.hbase-sink-1.serializer.numberBuckets=20
tier1.sinks.hbase-sink-1.serializer.customKey=timestamp,type,resource,hostname

Then in the configure method the context has the properties "numberBuckets" and "customKey" (notice all the prefix of the property is removed and not passed to the serializer)
If you paste your configuration and code maybe we can spot where is your issue.

Regards,
Gonzalo

On 3 December 2015 at 02:45, R P <ha...@outlook.com>> wrote:

Hello All,

 I need to use some configuration values in my implementation of Serializer which extends only EventSerializer.

For customizing behavior I am trying to use Context provided via Builder's build method.

Although the configuration values or none of the Context properties are not passed to Serializer.

I am able to use this Serializer without configuration through flume and write data so I am certain that my implementation works correctly.

Any idea why Context is passed as empty to the Serializer?  Or any pointers on how I should go on debugging this issue?

Thanks for reading and appreciate your response.

Thanks,

R

Re: Context/Configuration values not passed to custom serializer.

Posted by Gonzalo Herreros <gh...@gmail.com>.

Not sure how you are using the serializer but I understand it's part of an
hdfs sink

When you do this:
a1.sinks.k1.IgnoreNullKeyData=True

your property goes to the sink.
To make it apperar in the serializer context it should be like this:

a1.sinks.k1.serializer.IgnoreNullKeyData=True

How are you using the serializer in the configuration?

Gonzalo


On 3 December 2015 at 09:49, R P <ha...@outlook.com> wrote:

> Thanks for the reply Gonzalo.
>
> There is no configure method as Serializer doesn't extend Configurable.
> Do I have to extend Configurable to access custom configuration values?
>
> -R
> P.S - Sent code sample and config in separate email directly to you.
>
> ------------------------------
> *From:* Gonzalo Herreros <gh...@gmail.com>
> *Sent:* Thursday, December 3, 2015 12:32 AM
> *To:* user
> *Subject:* Re: Context/Configuration values not passed to custom
> serializer.
>
> It might be a bug in the sink you are using.
> For instance, I have a serializer for the HbaseSink so I added two custom
> properties.
>
> tier1.sinks.hbase-sink-1.serializer.numberBuckets=20
>
> tier1.sinks.hbase-sink-1.serializer.customKey=timestamp,type,resource,hostname
>
>
> Then in the configure method the context has the properties
> "numberBuckets" and "customKey" (notice all the prefix of the property is
> removed and not passed to the serializer)
> If you paste your configuration and code maybe we can spot where is your
> issue.
>
> Regards,
> Gonzalo
>
>
> On 3 December 2015 at 02:45, R P <ha...@outlook.com> wrote:
>
>> Hello All,
>>
>>  I need to use some configuration values in my implementation of
>> Serializer which extends only EventSerializer.
>>
>> For customizing behavior I am trying to use Context provided via
>> Builder's build method.
>>
>>
>> Although the configuration values or none of the Context properties are
>> not passed to Serializer.
>>
>>
>> I am able to use this Serializer without configuration through flume and
>> write data so I am certain that my implementation works correctly.
>>
>>
>> Any idea why Context is passed as empty to the Serializer?  Or any
>> pointers on how I should go on debugging this issue?
>>
>>
>> Thanks for reading and appreciate your response.
>>
>>
>> Thanks,
>>
>> R
>>
>>
>>
>>
>

Re: Context/Configuration values not passed to custom serializer.

Posted by R P <ha...@outlook.com>.

Thanks for the reply Gonzalo.

There is no configure method as Serializer doesn't extend Configurable.
Do I have to extend Configurable to access custom configuration values?

-R
P.S - Sent code sample and config in separate email directly to you.

________________________________
From: Gonzalo Herreros <gh...@gmail.com>
Sent: Thursday, December 3, 2015 12:32 AM
To: user
Subject: Re: Context/Configuration values not passed to custom serializer.

It might be a bug in the sink you are using.
For instance, I have a serializer for the HbaseSink so I added two custom properties.

tier1.sinks.hbase-sink-1.serializer.numberBuckets=20
tier1.sinks.hbase-sink-1.serializer.customKey=timestamp,type,resource,hostname


Then in the configure method the context has the properties "numberBuckets" and "customKey" (notice all the prefix of the property is removed and not passed to the serializer)
If you paste your configuration and code maybe we can spot where is your issue.

Regards,
Gonzalo


On 3 December 2015 at 02:45, R P <ha...@outlook.com>> wrote:

Hello All,

 I need to use some configuration values in my implementation of Serializer which extends only EventSerializer.

For customizing behavior I am trying to use Context provided via Builder's build method.


Although the configuration values or none of the Context properties are not passed to Serializer.


I am able to use this Serializer without configuration through flume and write data so I am certain that my implementation works correctly.


Any idea why Context is passed as empty to the Serializer?  Or any pointers on how I should go on debugging this issue?


Thanks for reading and appreciate your response.


Thanks,

R

Re: Context/Configuration values not passed to custom serializer.

Posted by Gonzalo Herreros <gh...@gmail.com>.

It might be a bug in the sink you are using.
For instance, I have a serializer for the HbaseSink so I added two custom
properties.

tier1.sinks.hbase-sink-1.serializer.numberBuckets=20
tier1.sinks.hbase-sink-1.serializer.customKey=timestamp,type,resource,hostname

Then in the configure method the context has the properties "numberBuckets"
and "customKey" (notice all the prefix of the property is removed and not
passed to the serializer)
If you paste your configuration and code maybe we can spot where is your
issue.

Regards,
Gonzalo

On 3 December 2015 at 02:45, R P <ha...@outlook.com> wrote:

> Hello All,
>
>  I need to use some configuration values in my implementation of
> Serializer which extends only EventSerializer.
>
> For customizing behavior I am trying to use Context provided via Builder's
> build method.
>
>
> Although the configuration values or none of the Context properties are
> not passed to Serializer.
>
>
> I am able to use this Serializer without configuration through flume and
> write data so I am certain that my implementation works correctly.
>
>
> Any idea why Context is passed as empty to the Serializer?  Or any
> pointers on how I should go on debugging this issue?
>
>
> Thanks for reading and appreciate your response.
>
>
> Thanks,
>
> R
>
>
>
>