You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Erik Bertrand <Er...@bertpc.com> on 2013/02/17 05:50:42 UTC
Sinking data to a Node.js server/listener
I'm looking to get Flume data transferred to a Node.js server listening on
a specific port. I'm not interested in actually storing the data anywhere,
as it's simply using that data to display "current activity" and doesn't
need to be persist it anywhere. I'm transferring a very small amount of
data - just a series of IP addresses, actually. So I'd like to keep the
architecture simple, too.
At first I was thinking I could use the Avro sink to send the data directly
to Node.js using a dnode <https://npmjs.org/package/dnode> server object
(i.e. RPC), but there seems to be more to it than the basic setup. I'm
just not sure how to configure the Node.js side to understand the Avro sink
RPC request (or if that's even possible).
I've been looking at creating a custom sink to do this; I've not written
one before, much less written anything in Java, so that'd be new to me.
Any pointers?
Erik
Re: Sinking data to a Node.js server/listener
Posted by Bertrand Dechoux <de...@gmail.com>.
In the end a syslog event/message on tcp is only a tcp message, so indeed
it should be simple with a tcp server (like nodejs). BUT I did answer too
quickly. I have more experience with flume (og) than flume-ng. And it does
indeed look like there is no syslog sink for the new generation. So I am
afraid that writing a custom sink is the way to go...
Bertrand
On Thu, Feb 21, 2013 at 2:41 PM, Erik Bertrand <Er...@bertpc.com> wrote:
> Hello Bertrand. Thanks very much for the reply! (I love that you we
> share a common first name/last name!)
>
> I was not aware that there is a syslog sink. I see there's an HDFS sink
> and a "File Roll" sink, but those do not seem to utilize syslog (I'm very
> new to this, so I may be misguided). How would I go about setting up such
> a sync - are you saying I would be able to post the data to an IP/port that
> Node.js is listening on?
>
> Again, many thanks for the assistance!
>
> Erik
>
>
>
> On Sun, Feb 17, 2013 at 5:43 PM, Bertrand Dechoux <de...@gmail.com>wrote:
>
>> Should it be assumed that structured data are transferred using flume and
>> that's why Avro is mentioned? One easy solution I can see would be to use a
>> syslog output that would actually point to your nodejs instance (and not a
>> real syslog endpoint).
>>
>> Regards
>>
>> Bertrand
>>
>>
>> On Sun, Feb 17, 2013 at 5:50 AM, Erik Bertrand <Er...@bertpc.com> wrote:
>>
>>> I'm looking to get Flume data transferred to a Node.js server listening
>>> on a specific port. I'm not interested in actually storing the data
>>> anywhere, as it's simply using that data to display "current activity" and
>>> doesn't need to be persist it anywhere. I'm transferring a very small
>>> amount of data - just a series of IP addresses, actually. So I'd like to
>>> keep the architecture simple, too.
>>>
>>> At first I was thinking I could use the Avro sink to send the data
>>> directly to Node.js using a dnode <https://npmjs.org/package/dnode> server
>>> object (i.e. RPC), but there seems to be more to it than the basic setup.
>>> I'm just not sure how to configure the Node.js side to understand the Avro
>>> sink RPC request (or if that's even possible).
>>>
>>> I've been looking at creating a custom sink to do this; I've not written
>>> one before, much less written anything in Java, so that'd be new to me.
>>> Any pointers?
>>>
>>> Erik
>>>
>>>
>>
>>
>> --
>> Bertrand Dechoux
>
>
>
Re: Sinking data to a Node.js server/listener
Posted by Erik Bertrand <Er...@bertpc.com>.
Hello Bertrand. Thanks very much for the reply! (I love that you we share
a common first name/last name!)
I was not aware that there is a syslog sink. I see there's an HDFS sink
and a "File Roll" sink, but those do not seem to utilize syslog (I'm very
new to this, so I may be misguided). How would I go about setting up such
a sync - are you saying I would be able to post the data to an IP/port that
Node.js is listening on?
Again, many thanks for the assistance!
Erik
On Sun, Feb 17, 2013 at 5:43 PM, Bertrand Dechoux <de...@gmail.com>wrote:
> Should it be assumed that structured data are transferred using flume and
> that's why Avro is mentioned? One easy solution I can see would be to use a
> syslog output that would actually point to your nodejs instance (and not a
> real syslog endpoint).
>
> Regards
>
> Bertrand
>
>
> On Sun, Feb 17, 2013 at 5:50 AM, Erik Bertrand <Er...@bertpc.com> wrote:
>
>> I'm looking to get Flume data transferred to a Node.js server listening
>> on a specific port. I'm not interested in actually storing the data
>> anywhere, as it's simply using that data to display "current activity" and
>> doesn't need to be persist it anywhere. I'm transferring a very small
>> amount of data - just a series of IP addresses, actually. So I'd like to
>> keep the architecture simple, too.
>>
>> At first I was thinking I could use the Avro sink to send the data
>> directly to Node.js using a dnode <https://npmjs.org/package/dnode> server
>> object (i.e. RPC), but there seems to be more to it than the basic setup.
>> I'm just not sure how to configure the Node.js side to understand the Avro
>> sink RPC request (or if that's even possible).
>>
>> I've been looking at creating a custom sink to do this; I've not written
>> one before, much less written anything in Java, so that'd be new to me.
>> Any pointers?
>>
>> Erik
>>
>>
>
>
> --
> Bertrand Dechoux
Re: Sinking data to a Node.js server/listener
Posted by Bertrand Dechoux <de...@gmail.com>.
Should it be assumed that structured data are transferred using flume and
that's why Avro is mentioned? One easy solution I can see would be to use a
syslog output that would actually point to your nodejs instance (and not a
real syslog endpoint).
Regards
Bertrand
On Sun, Feb 17, 2013 at 5:50 AM, Erik Bertrand <Er...@bertpc.com> wrote:
> I'm looking to get Flume data transferred to a Node.js server listening on
> a specific port. I'm not interested in actually storing the data anywhere,
> as it's simply using that data to display "current activity" and doesn't
> need to be persist it anywhere. I'm transferring a very small amount of
> data - just a series of IP addresses, actually. So I'd like to keep the
> architecture simple, too.
>
> At first I was thinking I could use the Avro sink to send the data
> directly to Node.js using a dnode <https://npmjs.org/package/dnode> server
> object (i.e. RPC), but there seems to be more to it than the basic setup.
> I'm just not sure how to configure the Node.js side to understand the Avro
> sink RPC request (or if that's even possible).
>
> I've been looking at creating a custom sink to do this; I've not written
> one before, much less written anything in Java, so that'd be new to me.
> Any pointers?
>
> Erik
>
>
--
Bertrand Dechoux