You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Bertrand Dechoux <de...@gmail.com> on 2014/05/09 15:46:24 UTC
Re: Realtime sensor's tcpip data to hadoop
Flume is indeed something you should look into. 'Log files' is a
simplification. Flume really handles events and yes logs are a common kind
of event but not the only one.
Regards
Bertrand
On Wed, May 7, 2014 at 4:48 AM, Alex Lee <el...@hotmail.com> wrote:
> Sensors' may send tcpip data to server. Each sensor may send tcpip data
> like a stream to the server, the quatity of the sensors and the data rate
> of the data is high.
>
> Firstly, how the data from tcpip can be put into hadoop. It need to do
> some process and store in hbase. Does it need through save to data files
> and put into hadoop or can be done in some direct ways from tcpip. Is there
> any software module can take care of this. Searched that Ganglia Nagios and
> Flume may do it. But when looking into details, ganglia and nagios are
> more for monitoring hadoop cluster itself. Flume is for log files.
>
> Secondly, if the total network traffic from sensors are over the limit of
> one lan port, how to share the loads, is there any component in hadoop to
> make this done automatically.
>
> Any suggestions, thanks.
>
RE: Realtime sensor's tcpip data to hadoop
Posted by Alex Lee <el...@hotmail.com>.
There are so many choice
OpenTSDB, flume, ActiveMQ, Lustre, Splunk,
Ganglia or Nagios. Not very sure which one is more fiitted.
Date: Fri, 9 May 2014 15:46:24 +0200
Subject: Re: Realtime sensor's tcpip data to hadoop
From: dechouxb@gmail.com
To: user@hadoop.apache.org; user@flume.apache.org
Flume is indeed something you should look into. 'Log files' is a simplification. Flume really handles events and yes logs are a common kind of event but not the only one.
Regards
Bertrand
On Wed, May 7, 2014 at 4:48 AM, Alex Lee <el...@hotmail.com> wrote:
Sensors' may send tcpip data to server. Each sensor may send tcpip data like a stream to the server, the quatity of the sensors and the data rate of the data is high.
Firstly, how the data from tcpip can be put into hadoop. It need to do some process and store in hbase. Does it need through save to data files and put into hadoop or can be done in some direct ways from tcpip. Is there any software module can take care of this. Searched that Ganglia Nagios and Flume may do it. But when looking into details, ganglia and nagios are more for monitoring hadoop cluster itself. Flume is for log files.
Secondly, if the total network traffic from sensors are over the limit of one lan port, how to share the loads, is there any component in hadoop to make this done automatically.
Any suggestions, thanks.
RE: Realtime sensor's tcpip data to hadoop
Posted by Alex Lee <el...@hotmail.com>.
There are so many choice
OpenTSDB, flume, ActiveMQ, Lustre, Splunk,
Ganglia or Nagios. Not very sure which one is more fiitted.
Date: Fri, 9 May 2014 15:46:24 +0200
Subject: Re: Realtime sensor's tcpip data to hadoop
From: dechouxb@gmail.com
To: user@hadoop.apache.org; user@flume.apache.org
Flume is indeed something you should look into. 'Log files' is a simplification. Flume really handles events and yes logs are a common kind of event but not the only one.
Regards
Bertrand
On Wed, May 7, 2014 at 4:48 AM, Alex Lee <el...@hotmail.com> wrote:
Sensors' may send tcpip data to server. Each sensor may send tcpip data like a stream to the server, the quatity of the sensors and the data rate of the data is high.
Firstly, how the data from tcpip can be put into hadoop. It need to do some process and store in hbase. Does it need through save to data files and put into hadoop or can be done in some direct ways from tcpip. Is there any software module can take care of this. Searched that Ganglia Nagios and Flume may do it. But when looking into details, ganglia and nagios are more for monitoring hadoop cluster itself. Flume is for log files.
Secondly, if the total network traffic from sensors are over the limit of one lan port, how to share the loads, is there any component in hadoop to make this done automatically.
Any suggestions, thanks.
RE: Realtime sensor's tcpip data to hadoop
Posted by Alex Lee <el...@hotmail.com>.
There are so many choice
OpenTSDB, flume, ActiveMQ, Lustre, Splunk,
Ganglia or Nagios. Not very sure which one is more fiitted.
Date: Fri, 9 May 2014 15:46:24 +0200
Subject: Re: Realtime sensor's tcpip data to hadoop
From: dechouxb@gmail.com
To: user@hadoop.apache.org; user@flume.apache.org
Flume is indeed something you should look into. 'Log files' is a simplification. Flume really handles events and yes logs are a common kind of event but not the only one.
Regards
Bertrand
On Wed, May 7, 2014 at 4:48 AM, Alex Lee <el...@hotmail.com> wrote:
Sensors' may send tcpip data to server. Each sensor may send tcpip data like a stream to the server, the quatity of the sensors and the data rate of the data is high.
Firstly, how the data from tcpip can be put into hadoop. It need to do some process and store in hbase. Does it need through save to data files and put into hadoop or can be done in some direct ways from tcpip. Is there any software module can take care of this. Searched that Ganglia Nagios and Flume may do it. But when looking into details, ganglia and nagios are more for monitoring hadoop cluster itself. Flume is for log files.
Secondly, if the total network traffic from sensors are over the limit of one lan port, how to share the loads, is there any component in hadoop to make this done automatically.
Any suggestions, thanks.
RE: Realtime sensor's tcpip data to hadoop
Posted by Alex Lee <el...@hotmail.com>.
There are so many choice
OpenTSDB, flume, ActiveMQ, Lustre, Splunk,
Ganglia or Nagios. Not very sure which one is more fiitted.
Date: Fri, 9 May 2014 15:46:24 +0200
Subject: Re: Realtime sensor's tcpip data to hadoop
From: dechouxb@gmail.com
To: user@hadoop.apache.org; user@flume.apache.org
Flume is indeed something you should look into. 'Log files' is a simplification. Flume really handles events and yes logs are a common kind of event but not the only one.
Regards
Bertrand
On Wed, May 7, 2014 at 4:48 AM, Alex Lee <el...@hotmail.com> wrote:
Sensors' may send tcpip data to server. Each sensor may send tcpip data like a stream to the server, the quatity of the sensors and the data rate of the data is high.
Firstly, how the data from tcpip can be put into hadoop. It need to do some process and store in hbase. Does it need through save to data files and put into hadoop or can be done in some direct ways from tcpip. Is there any software module can take care of this. Searched that Ganglia Nagios and Flume may do it. But when looking into details, ganglia and nagios are more for monitoring hadoop cluster itself. Flume is for log files.
Secondly, if the total network traffic from sensors are over the limit of one lan port, how to share the loads, is there any component in hadoop to make this done automatically.
Any suggestions, thanks.