You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Neil Collier <nc...@valueclick.com> on 2013/04/25 01:22:43 UTC
Flume C Client
I have setup flume in our organization using Java clients with the SDK. There is a need for us to stream data to flume from our existing C code into our flume avro source.
Is there an available C client or documentation somewhere? Note: I am not a C programmer.
Thanks,
Neil
This email and any files included with it may contain privileged,
proprietary and/or confidential information that is for the sole use
of the intended recipient(s). Any disclosure, copying, distribution,
posting, or use of the information contained in or attached to this
email is prohibited unless permitted by the sender. If you have
received this email in error, please immediately notify the sender
via return email, telephone, or fax and destroy this original transmission
and its included files without reading or saving it in any manner.
Thank you.
Re: Flume C Client
Posted by Hari Shreedharan <hs...@cloudera.com>.
Neil,
There is no current C client available out of the box. You can use Flume's thrift source together with a thrift client which can use the API generated by thrift from Flume's IDL file, which is available here: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob;f=flume-ng-sdk/src/main/thrift/flume.thrift
Hari
--
Hari Shreedharan
On Wednesday, April 24, 2013 at 4:22 PM, Neil Collier wrote:
> I have setup flume in our organization using Java clients with the SDK. There is a need for us to stream data to flume from our existing C code into our flume avro source.
>
> Is there an available C client or documentation somewhere? Note: I am not a C programmer.
>
> Thanks,
>
> Neil
>
>
>
>
> This email and any files included with it may contain privileged,
> proprietary and/or confidential information that is for the sole use
> of the intended recipient(s). Any disclosure, copying, distribution,
> posting, or use of the information contained in or attached to this
> email is prohibited unless permitted by the sender. If you have
> received this email in error, please immediately notify the sender
> via return email, telephone, or fax and destroy this original transmission
> and its included files without reading or saving it in any manner.
> Thank you.
>
>
Re: Flume C Client
Posted by Hari Shreedharan <hs...@cloudera.com>.
Actually, the Avro C API is not likely to be compatible with Flume, so you should use Thrift or HTTP. Thrift is a mechanism designed for IPC, so it is likely to perform better than HTTP.
Hari
--
Hari Shreedharan
On Thursday, April 25, 2013 at 4:25 AM, Israel Ekpo wrote:
> Flume does not have any of its components or plugins implemented in C.
>
> So there is no C client for it. It is not really designed in a
> client-server architecture.
>
> It was designed intentionally for collecting, aggregating, and transporting
> large amounts of data as events within its internal set of components.
>
> Nevertheless, there are components within this architecture (such as
> sources, channels and sinks) that interact with external endpoints in a
> client-server manner.
>
> Be that as it may, since your objective is to stream data to Flume, you can
> implement your own custom event-driven source for Flume and then connect to
> this source from your code to bridge in and move data from your system into
> flume.
>
> You could set up HTTPSource as the source and then use libcurl within your
> C code to send events to the HTTPSource for Flume.
>
> http://flume.apache.org/FlumeUserGuide.html#http-source
> http://curl.haxx.se/libcurl/c/
>
> Apache Avro has a C api documented here
> http://avro.apache.org/docs/current/api/c/index.html
>
> I hope this helps.
>
> On 24 April 2013 19:22, Neil Collier <ncollier@valueclick.com (mailto:ncollier@valueclick.com)> wrote:
>
> > I have setup flume in our organization using Java clients with the SDK.
> > There is a need for us to stream data to flume from our existing C code
> > into our flume avro source.
> >
> > Is there an available C client or documentation somewhere? Note: I am not
> > a C programmer.
> >
> > Thanks,
> >
> > Neil
> >
> >
> >
> >
> > This email and any files included with it may contain privileged,
> > proprietary and/or confidential information that is for the sole use
> > of the intended recipient(s). Any disclosure, copying, distribution,
> > posting, or use of the information contained in or attached to this
> > email is prohibited unless permitted by the sender. If you have
> > received this email in error, please immediately notify the sender
> > via return email, telephone, or fax and destroy this original transmission
> > and its included files without reading or saving it in any manner.
> > Thank you.
> >
>
>
>
Re: Flume C Client
Posted by Israel Ekpo <is...@aicer.org>.
Flume does not have any of its components or plugins implemented in C.
So there is no C client for it. It is not really designed in a
client-server architecture.
It was designed intentionally for collecting, aggregating, and transporting
large amounts of data as events within its internal set of components.
Nevertheless, there are components within this architecture (such as
sources, channels and sinks) that interact with external endpoints in a
client-server manner.
Be that as it may, since your objective is to stream data to Flume, you can
implement your own custom event-driven source for Flume and then connect to
this source from your code to bridge in and move data from your system into
flume.
You could set up HTTPSource as the source and then use libcurl within your
C code to send events to the HTTPSource for Flume.
http://flume.apache.org/FlumeUserGuide.html#http-source
http://curl.haxx.se/libcurl/c/
Apache Avro has a C api documented here
http://avro.apache.org/docs/current/api/c/index.html
I hope this helps.
On 24 April 2013 19:22, Neil Collier <nc...@valueclick.com> wrote:
> I have setup flume in our organization using Java clients with the SDK.
> There is a need for us to stream data to flume from our existing C code
> into our flume avro source.
>
> Is there an available C client or documentation somewhere? Note: I am not
> a C programmer.
>
> Thanks,
>
> Neil
>
>
>
>
> This email and any files included with it may contain privileged,
> proprietary and/or confidential information that is for the sole use
> of the intended recipient(s). Any disclosure, copying, distribution,
> posting, or use of the information contained in or attached to this
> email is prohibited unless permitted by the sender. If you have
> received this email in error, please immediately notify the sender
> via return email, telephone, or fax and destroy this original transmission
> and its included files without reading or saving it in any manner.
> Thank you.
>