You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Aijas Mohammed <Ai...@infotech-enterprises.com> on 2013/12/09 06:58:56 UTC
Streaming Sensor Data using Flume
Hi all,
This is Aijas.
Presently we are getting Sensor data from Cloud in a .csv file format. We have been given access to a URL were user is providing From Date & To Date to get Sensor Data .Now I want to STREAM that data using FLUME into HDFS.
One Line: I want to Stream Sensor Data into HDFS using FLUME.
How can we do that ? Any suggestions.
Thanks & Regards,
Aijas Mohammed
+91-9642933393
________________________________
DISCLAIMER:
This email may contain confidential information and is intended only for the use of the specific individual(s) to which it is addressed. If you are not the intended recipient of this email, you are hereby notified that any unauthorized use, dissemination or copying of this email or the information contained in it or attached to it is strictly prohibited. If you received this message in error, please immediately notify the sender at Infotech and delete the original message.
Re: Streaming Sensor Data using Flume
Posted by Peyman Mohajerian <mo...@gmail.com>.
Your flume source is a 'custom source', you will put the code for
connecting to the source data there and then link the flume source with the
channel and hdfs sink. There is a twitter example in the cloudera blog, a
three part blog that explains this in nice details.
On Sun, Dec 8, 2013 at 9:58 PM, Aijas Mohammed <
Aijas.Mohammed@infotech-enterprises.com> wrote:
> Hi all,
>
>
>
> This is Aijas.
>
> Presently we are getting Sensor data from Cloud in a .csv file format. We
> have been given access to a URL were user is providing From Date & To Date
> to get Sensor Data .Now I want to STREAM that data using FLUME into HDFS.
>
>
>
> *One Line: * I want to Stream Sensor Data into HDFS using FLUME.
>
>
>
> How can we do that ? Any suggestions.
>
>
>
> Thanks & Regards,
>
> Aijas Mohammed
>
> +91-9642933393
>
> ------------------------------
>
> DISCLAIMER:
>
> This email may contain confidential information and is intended only for
> the use of the specific individual(s) to which it is addressed. If you are
> not the intended recipient of this email, you are hereby notified that any
> unauthorized use, dissemination or copying of this email or the information
> contained in it or attached to it is strictly prohibited. If you received
> this message in error, please immediately notify the sender at Infotech and
> delete the original message.
>
Re: Streaming Sensor Data using Flume
Posted by Peyman Mohajerian <mo...@gmail.com>.
Your flume source is a 'custom source', you will put the code for
connecting to the source data there and then link the flume source with the
channel and hdfs sink. There is a twitter example in the cloudera blog, a
three part blog that explains this in nice details.
On Sun, Dec 8, 2013 at 9:58 PM, Aijas Mohammed <
Aijas.Mohammed@infotech-enterprises.com> wrote:
> Hi all,
>
>
>
> This is Aijas.
>
> Presently we are getting Sensor data from Cloud in a .csv file format. We
> have been given access to a URL were user is providing From Date & To Date
> to get Sensor Data .Now I want to STREAM that data using FLUME into HDFS.
>
>
>
> *One Line: * I want to Stream Sensor Data into HDFS using FLUME.
>
>
>
> How can we do that ? Any suggestions.
>
>
>
> Thanks & Regards,
>
> Aijas Mohammed
>
> +91-9642933393
>
> ------------------------------
>
> DISCLAIMER:
>
> This email may contain confidential information and is intended only for
> the use of the specific individual(s) to which it is addressed. If you are
> not the intended recipient of this email, you are hereby notified that any
> unauthorized use, dissemination or copying of this email or the information
> contained in it or attached to it is strictly prohibited. If you received
> this message in error, please immediately notify the sender at Infotech and
> delete the original message.
>
Re: Streaming Sensor Data using Flume
Posted by Peyman Mohajerian <mo...@gmail.com>.
Your flume source is a 'custom source', you will put the code for
connecting to the source data there and then link the flume source with the
channel and hdfs sink. There is a twitter example in the cloudera blog, a
three part blog that explains this in nice details.
On Sun, Dec 8, 2013 at 9:58 PM, Aijas Mohammed <
Aijas.Mohammed@infotech-enterprises.com> wrote:
> Hi all,
>
>
>
> This is Aijas.
>
> Presently we are getting Sensor data from Cloud in a .csv file format. We
> have been given access to a URL were user is providing From Date & To Date
> to get Sensor Data .Now I want to STREAM that data using FLUME into HDFS.
>
>
>
> *One Line: * I want to Stream Sensor Data into HDFS using FLUME.
>
>
>
> How can we do that ? Any suggestions.
>
>
>
> Thanks & Regards,
>
> Aijas Mohammed
>
> +91-9642933393
>
> ------------------------------
>
> DISCLAIMER:
>
> This email may contain confidential information and is intended only for
> the use of the specific individual(s) to which it is addressed. If you are
> not the intended recipient of this email, you are hereby notified that any
> unauthorized use, dissemination or copying of this email or the information
> contained in it or attached to it is strictly prohibited. If you received
> this message in error, please immediately notify the sender at Infotech and
> delete the original message.
>
Re: Streaming Sensor Data using Flume
Posted by Peyman Mohajerian <mo...@gmail.com>.
Your flume source is a 'custom source', you will put the code for
connecting to the source data there and then link the flume source with the
channel and hdfs sink. There is a twitter example in the cloudera blog, a
three part blog that explains this in nice details.
On Sun, Dec 8, 2013 at 9:58 PM, Aijas Mohammed <
Aijas.Mohammed@infotech-enterprises.com> wrote:
> Hi all,
>
>
>
> This is Aijas.
>
> Presently we are getting Sensor data from Cloud in a .csv file format. We
> have been given access to a URL were user is providing From Date & To Date
> to get Sensor Data .Now I want to STREAM that data using FLUME into HDFS.
>
>
>
> *One Line: * I want to Stream Sensor Data into HDFS using FLUME.
>
>
>
> How can we do that ? Any suggestions.
>
>
>
> Thanks & Regards,
>
> Aijas Mohammed
>
> +91-9642933393
>
> ------------------------------
>
> DISCLAIMER:
>
> This email may contain confidential information and is intended only for
> the use of the specific individual(s) to which it is addressed. If you are
> not the intended recipient of this email, you are hereby notified that any
> unauthorized use, dissemination or copying of this email or the information
> contained in it or attached to it is strictly prohibited. If you received
> this message in error, please immediately notify the sender at Infotech and
> delete the original message.
>