Architecture question

[robert <te...@yahoo.com>]

I need to grab avro data from a kafka topic and write to the local file
system

Inside the avro record there is a date time field. From that field I need to
name the file accordingly. (20180103) as an example


I was thinking of using flink to read, unpack this generic record then put
to a sink that will sort to make sure it goes into the right file.

Does anyhow have a high-level approach for this ? 

The bucketing sink look promising. Any examples of this type of problem for
flink to solve ?

Thanks



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Architecture question

[Fabian Hueske <fh...@gmail.com>]

Hi,

What you are looking for is a BucketingSink that works on event time (the
timestamp is encoded in your data).
AFAIK, Flink's BucketingSink has been designed to work in processing time,
but you can implement a Bucketer that creates buckets based on a timestamp
in the data.
You might need to play around with the parameters for closing open buckets
for a good behavior (similar to watermark tuning).

Best, Fabian

2018-02-14 22:18 GMT+01:00 robert <te...@yahoo.com>:

> I need to grab avro data from a kafka topic and write to the local file
> system
>
> Inside the avro record there is a date time field. From that field I need
> to
> name the file accordingly. (20180103) as an example
>
>
> I was thinking of using flink to read, unpack this generic record then put
> to a sink that will sort to make sure it goes into the right file.
>
> Does anyhow have a high-level approach for this ?
>
> The bucketing sink look promising. Any examples of this type of problem for
> flink to solve ?
>
> Thanks
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/
>