You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by kishore alajangi <al...@gmail.com> on 2014/06/16 11:19:38 UTC
flume-hdfssink error
Hi Flummers,
I copied this code from on-line to do experiments with flume,
a1.sources = r1
a1.sinks = k1 k2
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.shell = /bin/bash -c
a1.sources.r1.command = cat /Users/david/flume/testdata.txt
a1.sources.r1.interceptors = a
a1.sources.r1.interceptors.a.type =
org.apache.flume.interceptor.TimestampInterceptor$Builder
# Describe the sinks
a1.sinks.k1.type = logger
a1.sinks.k2.type = hdfs
a1.sinks.k2.channel = c1
a1.sinks.k2.hdfs.fileType = DataStream
a1.sinks.k2.hdfs.batchSize = 2000
a1.sinks.k2.hdfs.rollCount = 5000
a1.sinks.k2.hdfs.rollSize = 1000000
a1.sinks.k2.hdfs.rollInterval = 10
a1.sinks.k2.hdfs.writeFormat = Text
# a1.sinks.k2.hdfs.path = /flume
a1.sinks.k2.hdfs.path = /flume/trades/%y-%m-%d/%H%M/%S
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c1
the logger sink is working fine but for the hdfs sink, it gives the
following error
process failed
This is supposed to be overridden by subclasses,
what could be the problem, please help me.
--
Thanks,
Kishore.
Re: flume-hdfssink error
Posted by kishore alajangi <al...@gmail.com>.
where the error logs stored? I installed flume 1.5.0, with tar.gz.
On Mon, Jun 16, 2014 at 1:25 PM, Nitin Pawar <ni...@gmail.com>
wrote:
> can you share full error log ?
>
>
> On Mon, Jun 16, 2014 at 2:49 PM, kishore alajangi <
> alajangikishore@gmail.com> wrote:
>
>> Hi Flummers,
>>
>> I copied this code from on-line to do experiments with flume,
>>
>> a1.sources = r1
>> a1.sinks = k1 k2
>> a1.channels = c1
>>
>> # Describe/configure the source
>>
>> a1.sources.r1.type = exec
>> a1.sources.r1.shell = /bin/bash -c
>> a1.sources.r1.command = cat /Users/david/flume/testdata.txt
>> a1.sources.r1.interceptors = a
>> a1.sources.r1.interceptors.a.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
>>
>> # Describe the sinks
>> a1.sinks.k1.type = logger
>>
>> a1.sinks.k2.type = hdfs
>> a1.sinks.k2.channel = c1
>> a1.sinks.k2.hdfs.fileType = DataStream
>> a1.sinks.k2.hdfs.batchSize = 2000
>> a1.sinks.k2.hdfs.rollCount = 5000
>> a1.sinks.k2.hdfs.rollSize = 1000000
>> a1.sinks.k2.hdfs.rollInterval = 10
>> a1.sinks.k2.hdfs.writeFormat = Text
>> # a1.sinks.k2.hdfs.path = /flume
>> a1.sinks.k2.hdfs.path = /flume/trades/%y-%m-%d/%H%M/%S
>>
>> # Use a channel which buffers events in memory
>> a1.channels.c1.type = memory
>> a1.channels.c1.capacity = 1000
>> a1.channels.c1.transactionCapacity = 100
>>
>> # Bind the source and sink to the channel
>> a1.sources.r1.channels = c1
>> a1.sinks.k1.channel = c1
>> a1.sinks.k2.channel = c1
>>
>>
>>
>> the logger sink is working fine but for the hdfs sink, it gives the
>> following error
>>
>> process failed
>> This is supposed to be overridden by subclasses,
>>
>> what could be the problem, please help me.
>>
>> --
>> Thanks,
>> Kishore.
>>
>
>
>
> --
> Nitin Pawar
>
--
Thanks,
Kishore.
Re: flume-hdfssink error
Posted by Nitin Pawar <ni...@gmail.com>.
can you share full error log ?
On Mon, Jun 16, 2014 at 2:49 PM, kishore alajangi <alajangikishore@gmail.com
> wrote:
> Hi Flummers,
>
> I copied this code from on-line to do experiments with flume,
>
> a1.sources = r1
> a1.sinks = k1 k2
> a1.channels = c1
>
> # Describe/configure the source
>
> a1.sources.r1.type = exec
> a1.sources.r1.shell = /bin/bash -c
> a1.sources.r1.command = cat /Users/david/flume/testdata.txt
> a1.sources.r1.interceptors = a
> a1.sources.r1.interceptors.a.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
>
> # Describe the sinks
> a1.sinks.k1.type = logger
>
> a1.sinks.k2.type = hdfs
> a1.sinks.k2.channel = c1
> a1.sinks.k2.hdfs.fileType = DataStream
> a1.sinks.k2.hdfs.batchSize = 2000
> a1.sinks.k2.hdfs.rollCount = 5000
> a1.sinks.k2.hdfs.rollSize = 1000000
> a1.sinks.k2.hdfs.rollInterval = 10
> a1.sinks.k2.hdfs.writeFormat = Text
> # a1.sinks.k2.hdfs.path = /flume
> a1.sinks.k2.hdfs.path = /flume/trades/%y-%m-%d/%H%M/%S
>
> # Use a channel which buffers events in memory
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 1000
> a1.channels.c1.transactionCapacity = 100
>
> # Bind the source and sink to the channel
> a1.sources.r1.channels = c1
> a1.sinks.k1.channel = c1
> a1.sinks.k2.channel = c1
>
>
>
> the logger sink is working fine but for the hdfs sink, it gives the
> following error
>
> process failed
> This is supposed to be overridden by subclasses,
>
> what could be the problem, please help me.
>
> --
> Thanks,
> Kishore.
>
--
Nitin Pawar