Interceptor, multiplexing Events to two differents path in HDFS dynamically,

[Guillermo Ortiz <ko...@gmail.com>]

Hello,


I want to create an interceptor to validate and multiplexing the Events, if
they don't get validated to send them a different path in HDFS.

Is it possible to do it?
I thought.. to have others interceptor and the HDFS path is built on base
of these static interceptors, so I could modify them on my validator
interceptor. Is it fine?

Re: Interceptor, multiplexing Events to two differents path in HDFS dynamically,

[Guillermo Ortiz <ko...@gmail.com>]

I think that it's exactly what I need, thanks.


2014-07-30 17:46 GMT+02:00 Matt Tenenbaum <ma...@rockyou.com>:

> Hi Guillermo --
>
> If I understand correctly, you want to create two paths. Events that pass
> the validating interceptor get directed to one path in HDFS and events that
> fail get directed to another. If so, you could define your interceptor to
> set a 'valid' header to 'yes' or 'no' depending on its internal logic. Then
> you would be able to use the ordinary 'selector.mapping' configuration
> behavior to route to separate channels and attach your separate
> destinations as sinks to each of those channels. I'm omitting all but the
> relevant configurations:
>
>
> multi.sources = s1
> multi.sinks = valid invalid
> multi.channels = cpos cneg
>
> multi.sources.s1.selector.type = multiplexing
> # Route based on the value of the 'valid' header set by the interceptor
> multi.sources.s1.selector.header = valid
> # Send events with valid == yes to 'cpos' channel
> multi.sources.s1.selector.mapping.yes = cpos
> # Send events with valid == no to 'cneg' channel
> multi.sources.s1.selector.mapping.no = cneg
>
> multi.sinks.valid.type = hdfs
> multi.sinks.valid.channel = cpos
> multi.sinks.valid.hdfs.path = hdfs://nn/path/for/valid/data
> # Other configurations for valid data...
>
> multi.sinks.invalid.type = hdfs
> multi.sinks.invalid.channel = cneg
> multi.sinks.invalid.hdfs.path = hdfs://nn/path/for/invalid/data
> # Other configuration for data that fails validation...
>
> Would that do what you want?
>
> Cheers
> -mt
>
>
> On Wed, Jul 30, 2014 at 6:02 AM, Guillermo Ortiz <ko...@gmail.com>
> wrote:
>
>> Hello,
>>
>>
>> I want to create an interceptor to validate and multiplexing the Events,
>> if they don't get validated to send them a different path in HDFS.
>>
>> Is it possible to do it?
>> I thought.. to have others interceptor and the HDFS path is built on base
>> of these static interceptors, so I could modify them on my validator
>> interceptor. Is it fine?
>>
>
>

Re: Interceptor, multiplexing Events to two differents path in HDFS dynamically,

[Matt Tenenbaum <ma...@rockyou.com>]

Hi Guillermo --

If I understand correctly, you want to create two paths. Events that pass
the validating interceptor get directed to one path in HDFS and events that
fail get directed to another. If so, you could define your interceptor to
set a 'valid' header to 'yes' or 'no' depending on its internal logic. Then
you would be able to use the ordinary 'selector.mapping' configuration
behavior to route to separate channels and attach your separate
destinations as sinks to each of those channels. I'm omitting all but the
relevant configurations:


multi.sources = s1
multi.sinks = valid invalid
multi.channels = cpos cneg

multi.sources.s1.selector.type = multiplexing
# Route based on the value of the 'valid' header set by the interceptor
multi.sources.s1.selector.header = valid
# Send events with valid == yes to 'cpos' channel
multi.sources.s1.selector.mapping.yes = cpos
# Send events with valid == no to 'cneg' channel
multi.sources.s1.selector.mapping.no = cneg

multi.sinks.valid.type = hdfs
multi.sinks.valid.channel = cpos
multi.sinks.valid.hdfs.path = hdfs://nn/path/for/valid/data
# Other configurations for valid data...

multi.sinks.invalid.type = hdfs
multi.sinks.invalid.channel = cneg
multi.sinks.invalid.hdfs.path = hdfs://nn/path/for/invalid/data
# Other configuration for data that fails validation...

Would that do what you want?

Cheers
-mt


On Wed, Jul 30, 2014 at 6:02 AM, Guillermo Ortiz <ko...@gmail.com>
wrote:

> Hello,
>
>
> I want to create an interceptor to validate and multiplexing the Events,
> if they don't get validated to send them a different path in HDFS.
>
> Is it possible to do it?
> I thought.. to have others interceptor and the HDFS path is built on base
> of these static interceptors, so I could modify them on my validator
> interceptor. Is it fine?
>