You are viewing a plain text version of this content. The canonical link for it is here.
Posted to community@flink.apache.org by Alex Ott <al...@gmail.com> on 2017/03/04 09:36:28 UTC
Re: Syslog source connector for Flink
Why not to send syslog data to Kafka, via rsyslog, or something like? From
my point of view this could be more flexible solution
On Sat, Mar 4, 2017 at 1:39 AM, trang anh (Joey) <an...@gmail.com>
wrote:
> Hi,
>
> I am looking for the Flink syslog source connector which can source data
> directly from syslog event as input data stream to Flink
>
> I have checked the source connectors list, but seems there no such thing
> available.
>
> Appreciate any advise?
>
> Regards
>
> Joey
--
With best wishes, Alex Ott
http://alexott.net/
Twitter: alexott_en (English), alexott (Russian)
Skype: alex.ott
Re: Syslog source connector for Flink
Posted by trang anh <an...@gmail.com>.
Hi Alex,
That is one of solutions we are looking right now, but the problem with us
is the our system will be processing the huge data, few TB daily basis,
So that we don’t want store the data in our KAFKA cluster, we just want get
event data, do some processing and emitting to downstream systems
Please look at this link for more detail of our challenges and our current
approach for that
Any advice would be grateful
On 4 Mar 2017, at 5:36 PM, Alex Ott <al...@gmail.com> wrote:
Why not to send syslog data to Kafka, via rsyslog, or something like? From
my point of view this could be more flexible solution
On Sat, Mar 4, 2017 at 1:39 AM, trang anh (Joey) <an...@gmail.com>
wrote:
Hi,
I am looking for the Flink syslog source connector which can source data
directly from syslog event as input data stream to Flink
I have checked the source connectors list, but seems there no such thing
available.
Appreciate any advise?
Regards
Joey
--
With best wishes, Alex Ott
http://alexott.net/
Twitter: alexott_en (English), alexott (Russian)
Skype: alex.ott
Re: Syslog source connector for Flink
Posted by Robert Metzger <rm...@apache.org>.
I agree with Alex, that a very short retention is probably a good solution,
if you want to minimize data loss when Flink is restarting.
If some small degree of data loss is okay, you could also consider
implementing a systlog connector for Flink.
I think that it is probably quite easy to create a Flink data source that
implements the syslog protocol. Flink's worker nodes would just look like a
cluster of syslog servers you can send data to.
On Sat, Mar 4, 2017 at 11:00 PM, Alex Ott <al...@gmail.com> wrote:
> Why not to use Kafka with very short retention time?
>
> On Sat, 4 Mar 2017 at 14:16, trang anh (Joey) <an...@gmail.com>
> wrote:
>
> > Hi Alex,
> >
> > That is one of solutions we are looking right now, but the problem with
> us
> > is the our system will be processing the huge data, few TB daily basis,
> >
> > So that we don’t want store the data in our KAFKA cluster, we just want
> > get event data, do some processing and emitting to downstream systems
> >
> > Please look at this link for more detail of our challenges and our
> current
> > approach for that
> >
> > Any advice would be grateful
> >
> >
> > http://stackoverflow.com/questions/42525680/best-
> practice-for-flink-streaming-processing
> > <
> > http://stackoverflow.com/questions/42525680/best-
> practice-for-flink-streaming-processing
> > >
> >
> >
> >
> >
> > > On 4 Mar 2017, at 5:36 PM, Alex Ott <al...@gmail.com> wrote:
> > >
> > > Why not to send syslog data to Kafka, via rsyslog, or something like?
> > From
> > > my point of view this could be more flexible solution
> > >
> > > On Sat, Mar 4, 2017 at 1:39 AM, trang anh (Joey) <
> anhtv010888@gmail.com>
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> I am looking for the Flink syslog source connector which can source
> > data
> > >> directly from syslog event as input data stream to Flink
> > >>
> > >> I have checked the source connectors list, but seems there no such
> thing
> > >> available.
> > >>
> > >> Appreciate any advise?
> > >>
> > >> Regards
> > >>
> > >> Joey
> > >
> > >
> > >
> > >
> > > --
> > > With best wishes, Alex Ott
> > > http://alexott.net/
> > > Twitter: alexott_en (English), alexott (Russian)
> > > Skype: alex.ott
> >
> > --
> With best wishes, Alex Ott
> http://alexott.net/
> Twitter: alexott_en (English), alexott (Russian)
> Skype: alex.ott
>
Re: Syslog source connector for Flink
Posted by Alex Ott <al...@gmail.com>.
Why not to use Kafka with very short retention time?
On Sat, 4 Mar 2017 at 14:16, trang anh (Joey) <an...@gmail.com> wrote:
> Hi Alex,
>
> That is one of solutions we are looking right now, but the problem with us
> is the our system will be processing the huge data, few TB daily basis,
>
> So that we don’t want store the data in our KAFKA cluster, we just want
> get event data, do some processing and emitting to downstream systems
>
> Please look at this link for more detail of our challenges and our current
> approach for that
>
> Any advice would be grateful
>
>
> http://stackoverflow.com/questions/42525680/best-practice-for-flink-streaming-processing
> <
> http://stackoverflow.com/questions/42525680/best-practice-for-flink-streaming-processing
> >
>
>
>
>
> > On 4 Mar 2017, at 5:36 PM, Alex Ott <al...@gmail.com> wrote:
> >
> > Why not to send syslog data to Kafka, via rsyslog, or something like?
> From
> > my point of view this could be more flexible solution
> >
> > On Sat, Mar 4, 2017 at 1:39 AM, trang anh (Joey) <an...@gmail.com>
> > wrote:
> >
> >> Hi,
> >>
> >> I am looking for the Flink syslog source connector which can source
> data
> >> directly from syslog event as input data stream to Flink
> >>
> >> I have checked the source connectors list, but seems there no such thing
> >> available.
> >>
> >> Appreciate any advise?
> >>
> >> Regards
> >>
> >> Joey
> >
> >
> >
> >
> > --
> > With best wishes, Alex Ott
> > http://alexott.net/
> > Twitter: alexott_en (English), alexott (Russian)
> > Skype: alex.ott
>
> --
With best wishes, Alex Ott
http://alexott.net/
Twitter: alexott_en (English), alexott (Russian)
Skype: alex.ott
Re: Syslog source connector for Flink
Posted by "trang anh (Joey)" <an...@gmail.com>.
Hi Alex,
That is one of solutions we are looking right now, but the problem with us is the our system will be processing the huge data, few TB daily basis,
So that we don’t want store the data in our KAFKA cluster, we just want get event data, do some processing and emitting to downstream systems
Please look at this link for more detail of our challenges and our current approach for that
Any advice would be grateful
http://stackoverflow.com/questions/42525680/best-practice-for-flink-streaming-processing <http://stackoverflow.com/questions/42525680/best-practice-for-flink-streaming-processing>
> On 4 Mar 2017, at 5:36 PM, Alex Ott <al...@gmail.com> wrote:
>
> Why not to send syslog data to Kafka, via rsyslog, or something like? From
> my point of view this could be more flexible solution
>
> On Sat, Mar 4, 2017 at 1:39 AM, trang anh (Joey) <an...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am looking for the Flink syslog source connector which can source data
>> directly from syslog event as input data stream to Flink
>>
>> I have checked the source connectors list, but seems there no such thing
>> available.
>>
>> Appreciate any advise?
>>
>> Regards
>>
>> Joey
>
>
>
>
> --
> With best wishes, Alex Ott
> http://alexott.net/
> Twitter: alexott_en (English), alexott (Russian)
> Skype: alex.ott