You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Rahul Patodi <pa...@gmail.com> on 2012/06/21 09:53:23 UTC
Flume-NG HBase sink
Hi,
I am trying to use HBase sink
Data is not getting copy into HBase
Console is showing (stuck):
12/06/21 13:16:58 INFO sink.DefaultSinkFactory: Creating instance of sink
sink1 typeorg.apache.flume.sink.hbase.HBaseSink
I think I am missing some thing, can any one help me in this
I am using
HBase 0.90.4
Flume 1.1.0
My configurations are:
host1.sources = src1
host1.sinks = sink1
host1.channels = ch1
host1.sources.src1.type = exec
host1.sources.src1.bind = pseudo
host1.sources.src1.command = tail -F /tmp/test05
host1.sources.src1.channels = ch1
host1.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
host1.sinks.sink1.channel = ch1
host1.sinks.sink1.table = t001
host1.sinks.sink1.columnFamily = cf
host1.sinks.sink1.column = foo
host1.sinks.sink1.serializer =
org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
host1.sinks.sink1.serializer.payloadColumn = pcol
host1.sinks.sink1.serializer.incrementColumn = icol
Re: Flume-NG HBase sink
Posted by Rahul Patodi <pa...@gmail.com>.
Thanks Hari
It got working with HBase 0.90.4 version
On Thu, Jun 21, 2012 at 3:46 PM, Rahul Patodi
<pa...@gmail.com>wrote:
> Hi Hari,
> Thanks for the response, I have HBase and Hadoop in my classpath,
> but console is getting stuck after:
> ....
> ....
> 12/06/21 15:42:13 INFO conf.FlumeConfiguration: Post-validation flume
> configuration contains configuration for agents: [hbase-agent]
> 12/06/21 15:42:13 INFO properties.PropertiesFileConfigurationProvider:
> Creating channels
> 12/06/21 15:42:13 INFO properties.PropertiesFileConfigurationProvider:
> created channel ch1
> 12/06/21 15:42:13 INFO sink.DefaultSinkFactory: Creating instance of sink
> sink1 typeorg.apache.flume.sink.hbase.HBaseSink
> 12/06/21 15:42:14 INFO hbase.HBaseSink: No serializer defined, Will use
> default
> 12/06/21 15:42:14 INFO nodemanager.DefaultLogicalNodeManager: Starting new
> configuration:{ sourceRunners:{tail=EventDrivenSourceRunner: {
> source:org.apache.flume.source.ExecSource@19632847 }}
> sinkRunners:{sink1=SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@3ce95a56 counterGroup:{
> name:null counters:{} } }}
> channels:{ch1=org.apache.flume.channel.MemoryChannel@7c29e357} }
> 12/06/21 15:42:14 INFO nodemanager.DefaultLogicalNodeManager: Starting
> Channel ch1
> 12/06/21 15:42:14 INFO nodemanager.DefaultLogicalNodeManager: Waiting for
> channel: ch1 to start. Sleeping for 500 ms
> 12/06/21 15:42:14 INFO nodemanager.DefaultLogicalNodeManager: Starting
> Sink sink1
> 12/06/21 15:42:14 INFO nodemanager.DefaultLogicalNodeManager: Starting
> Source tail
> 12/06/21 15:42:14 INFO source.ExecSource: Exec source starting with
> command:tail -F /tmp/test05
>
> Can anyone help resolving this issue ?
>
>
>
> On Thu, Jun 21, 2012 at 1:27 PM, Hari Shreedharan <
> hshreedharan@cloudera.com> wrote:
>
>> Hi Rahul,
>>
>> This is most likely because you don't have either Hbase or Hadoop in your
>> class path. Also the sink is written to work against Hbase 0.92. Please
>> sure these are on your class path, and try again.
>>
>> Thanks
>> Hari
>>
>> --
>> Hari Shreedharan
>>
>> On Thursday, June 21, 2012 at 12:53 AM, Rahul Patodi wrote:
>>
>> Hi,
>> I am trying to use HBase sink
>> Data is not getting copy into HBase
>> Console is showing (stuck):
>> 12/06/21 13:16:58 INFO sink.DefaultSinkFactory: Creating instance of sink
>> sink1 typeorg.apache.flume.sink.hbase.HBaseSink
>>
>> I think I am missing some thing, can any one help me in this
>>
>> I am using
>> HBase 0.90.4
>> Flume 1.1.0
>>
>> My configurations are:
>> host1.sources = src1
>> host1.sinks = sink1
>> host1.channels = ch1
>> host1.sources.src1.type = exec
>> host1.sources.src1.bind = pseudo
>> host1.sources.src1.command = tail -F /tmp/test05
>> host1.sources.src1.channels = ch1
>>
>> host1.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
>> host1.sinks.sink1.channel = ch1
>> host1.sinks.sink1.table = t001
>> host1.sinks.sink1.columnFamily = cf
>> host1.sinks.sink1.column = foo
>> host1.sinks.sink1.serializer =
>> org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
>> host1.sinks.sink1.serializer.payloadColumn = pcol
>> host1.sinks.sink1.serializer.incrementColumn = icol
>>
>>
>>
>>
>
>
> --
> *Regards*,
> Rahul Patodi
>
>
>
--
*Regards*,
Rahul Patodi
Re: Flume-NG HBase sink
Posted by Rahul Patodi <pa...@gmail.com>.
Hi Hari,
Thanks for the response, I have HBase and Hadoop in my classpath,
but console is getting stuck after:
....
....
12/06/21 15:42:13 INFO conf.FlumeConfiguration: Post-validation flume
configuration contains configuration for agents: [hbase-agent]
12/06/21 15:42:13 INFO properties.PropertiesFileConfigurationProvider:
Creating channels
12/06/21 15:42:13 INFO properties.PropertiesFileConfigurationProvider:
created channel ch1
12/06/21 15:42:13 INFO sink.DefaultSinkFactory: Creating instance of sink
sink1 typeorg.apache.flume.sink.hbase.HBaseSink
12/06/21 15:42:14 INFO hbase.HBaseSink: No serializer defined, Will use
default
12/06/21 15:42:14 INFO nodemanager.DefaultLogicalNodeManager: Starting new
configuration:{ sourceRunners:{tail=EventDrivenSourceRunner: {
source:org.apache.flume.source.ExecSource@19632847 }}
sinkRunners:{sink1=SinkRunner: {
policy:org.apache.flume.sink.DefaultSinkProcessor@3ce95a56 counterGroup:{
name:null counters:{} } }}
channels:{ch1=org.apache.flume.channel.MemoryChannel@7c29e357} }
12/06/21 15:42:14 INFO nodemanager.DefaultLogicalNodeManager: Starting
Channel ch1
12/06/21 15:42:14 INFO nodemanager.DefaultLogicalNodeManager: Waiting for
channel: ch1 to start. Sleeping for 500 ms
12/06/21 15:42:14 INFO nodemanager.DefaultLogicalNodeManager: Starting Sink
sink1
12/06/21 15:42:14 INFO nodemanager.DefaultLogicalNodeManager: Starting
Source tail
12/06/21 15:42:14 INFO source.ExecSource: Exec source starting with
command:tail -F /tmp/test05
Can anyone help resolving this issue ?
On Thu, Jun 21, 2012 at 1:27 PM, Hari Shreedharan <hshreedharan@cloudera.com
> wrote:
> Hi Rahul,
>
> This is most likely because you don't have either Hbase or Hadoop in your
> class path. Also the sink is written to work against Hbase 0.92. Please
> sure these are on your class path, and try again.
>
> Thanks
> Hari
>
> --
> Hari Shreedharan
>
> On Thursday, June 21, 2012 at 12:53 AM, Rahul Patodi wrote:
>
> Hi,
> I am trying to use HBase sink
> Data is not getting copy into HBase
> Console is showing (stuck):
> 12/06/21 13:16:58 INFO sink.DefaultSinkFactory: Creating instance of sink
> sink1 typeorg.apache.flume.sink.hbase.HBaseSink
>
> I think I am missing some thing, can any one help me in this
>
> I am using
> HBase 0.90.4
> Flume 1.1.0
>
> My configurations are:
> host1.sources = src1
> host1.sinks = sink1
> host1.channels = ch1
> host1.sources.src1.type = exec
> host1.sources.src1.bind = pseudo
> host1.sources.src1.command = tail -F /tmp/test05
> host1.sources.src1.channels = ch1
>
> host1.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
> host1.sinks.sink1.channel = ch1
> host1.sinks.sink1.table = t001
> host1.sinks.sink1.columnFamily = cf
> host1.sinks.sink1.column = foo
> host1.sinks.sink1.serializer =
> org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
> host1.sinks.sink1.serializer.payloadColumn = pcol
> host1.sinks.sink1.serializer.incrementColumn = icol
>
>
>
>
--
*Regards*,
Rahul Patodi
Re: Flume-NG HBase sink
Posted by Hari Shreedharan <hs...@cloudera.com>.
Hi Rahul,
This is most likely because you don't have either Hbase or Hadoop in your class path. Also the sink is written to work against Hbase 0.92. Please sure these are on your class path, and try again.
Thanks
Hari
--
Hari Shreedharan
On Thursday, June 21, 2012 at 12:53 AM, Rahul Patodi wrote:
> Hi,
> I am trying to use HBase sink
> Data is not getting copy into HBase
> Console is showing (stuck):
> 12/06/21 13:16:58 INFO sink.DefaultSinkFactory: Creating instance of sink sink1 typeorg.apache.flume.sink.hbase.HBaseSink
>
> I think I am missing some thing, can any one help me in this
>
> I am using
> HBase 0.90.4
> Flume 1.1.0
>
> My configurations are:
> host1.sources = src1
> host1.sinks = sink1
> host1.channels = ch1
> host1.sources.src1.type = exec
> host1.sources.src1.bind = pseudo
> host1.sources.src1.command = tail -F /tmp/test05
> host1.sources.src1.channels = ch1
>
> host1.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
> host1.sinks.sink1.channel = ch1
> host1.sinks.sink1.table = t001
> host1.sinks.sink1.columnFamily = cf
> host1.sinks.sink1.column = foo
> host1.sinks.sink1.serializer = org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
> host1.sinks.sink1.serializer.payloadColumn = pcol
> host1.sinks.sink1.serializer.incrementColumn = icol
>
>