You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by "Arthur.hk.chan@gmail.com" <ar...@gmail.com> on 2014/09/02 10:21:20 UTC
Use Flume to Stream Real Time Video into HDFS
Hi,
Is there any sample or tutorial about streaming Real Time Videos into HDFS?
I am trying to use net cat but it might not be a good choice (I am new to flume).
Regards
Arthur
(below is my test configuration)
### A single-node Flume configuration
### Name the components on this agent
a3.sources = r1
a3.channels = c1
a3.sinks = k1
### Describe/configure the source
a3.sources.r1.type = netcat
a3.sources.r1.bind = localhost
a3.sources.r1.port = 44444
a3.sources.r1.channels = c1
### Describe the sink
### Use a channel which buffers events in memory
a3.channels.c1.type = memory
a3.channels.c1.capacity = 10000
a3.channels.c1.transactionCapacity = 10000
### Bind the source and sink to the channel
a3.sinks.k1.type = hdfs
a3.sinks.k1.channel = c1
a3.sinks.k1.hdfs.path = hdfs://hdfs/flume_test
a3.sinks.k1.hdfs.filePrefix = events-
a3.sinks.k1.hdfs.round = true
a3.sinks.k1.hdfs.roundValue = 10
a3.sinks.k1.hdfs.roundUnit = minute
Re: Use Flume to Stream Real Time Video into HDFS
Posted by Sandeep Khurana <sk...@gmail.com>.
You may try writing a custom source which can use a gstreamer pipeline
(which you will build) and send data to channel etc .
On Tue, Sep 2, 2014 at 1:51 PM, Arthur.hk.chan@gmail.com <
arthur.hk.chan@gmail.com> wrote:
> Hi,
>
> Is there any sample or tutorial about streaming Real Time Videos into
> HDFS?
>
> I am trying to use net cat but it might not be a good choice (I am new to
> flume).
>
> Regards
> Arthur
>
> (below is my test configuration)
> ### A single-node Flume configuration
> ### Name the components on this agent
> a3.sources = r1
> a3.channels = c1
> a3.sinks = k1
> ### Describe/configure the source
> a3.sources.r1.type = netcat
> a3.sources.r1.bind = localhost
> a3.sources.r1.port = 44444
> a3.sources.r1.channels = c1
> ### Describe the sink
> ### Use a channel which buffers events in memory
> a3.channels.c1.type = memory
> a3.channels.c1.capacity = 10000
> a3.channels.c1.transactionCapacity = 10000
> ### Bind the source and sink to the channel
> a3.sinks.k1.type = hdfs
> a3.sinks.k1.channel = c1
> a3.sinks.k1.hdfs.path = hdfs://hdfs/flume_test
> a3.sinks.k1.hdfs.filePrefix = events-
> a3.sinks.k1.hdfs.round = true
> a3.sinks.k1.hdfs.roundValue = 10
> a3.sinks.k1.hdfs.roundUnit = minute
>
>
>
--
Thanks and regards
Sandeep Khurana