You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by "Guyle M. Taber" <gu...@gmtech.net> on 2019/06/13 15:42:23 UTC

Flume - Interceptor, regex mapping and two default selectors

I've got three channels and three sinks. I'm using a flume regex interceptor and sending only matching data to a specific sink, but I want all data going to the other two channels/sinks. I may be doing this wrong, but it seems like I might be able to use multiple channels for the "selector.default" to send ALL data to the other two sinks. (one in HDFS and the other in S3).

So what I have below is regex matching data going to the GCS channel. I want EVERYTHING going to the other two channels (HDFS and S3).

agent.sources.apache.interceptors.ievent.type = regex_extractor
agent.sources.apache.interceptors.ievent.regex = (product_page)
agent.sources.apache.interceptors.ievent.serializers = to_gcs
agent.sources.apache.interceptors.ievent.serializers.to_gcs.name = event_type
agent.sources.apache.selector.type = multiplexing
agent.sources.apache.selector.header = event_type
agent.sources.apache.selector.mapping.product_page = GCSChannel AvroChannel S3Channel
agent.sources.apache.selector.default = AvroChannel S3Channel
Is this an approach that would work? "agent.sources.apache.selector.default = AvroChannel S3Channel"