You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Rahul Patodi <pa...@gmail.com> on 2012/06/21 14:09:23 UTC
Zookeeper in Flume NG log
Hi,
I am trying to put data into HBase using Flume-NG (configure flume-NG Hbase
sink)
When I start flume NG (1.1)
patodirahul@hadoopPseudo:/opt/hes/flume-1.1.0-cdh4.0.0$ bin/flume-ng agent
-n hbase-agent -f conf/hbase.cnf
Having configuration:
hbase-agent.sources=tail
hbase-agent.sinks=sink1
hbase-agent.channels=ch1
hbase-agent.sources.tail.type=exec
hbase-agent.sources.tail.command=tail -F /tmp/test05
hbase-agent.sources.tail.channels=ch1
hbase-agent.sinks.sink1.type=org.apache.flume.sink.hbase.HBaseSink
hbase-agent.sinks.sink1.channel=ch1
hbase-agent.sinks.sink1.table=t002
hbase-agent.sinks.sink1.columnFamily=cf
hbase-agent.sinks.sink1.column=foo
hbase-agent.sinks.sink1.serializer=org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
hbase-agent.sinks.sink1.serializer.payloadColumn=col1
hbase-agent.sinks.sink1.serializer.incrementColumn=col1
hbase-agent.sinks.sink1.serializer.rowPrefix=1+
hbase-agent.sinks.sink1.serializer.suffix=timestamp
hbase-agent.channels.ch1.type=memory
its showing zookeeper in logs (on console) as mentioned below
...
...
12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client
environment:java.library.path=:/opt/hes/hadoop-0.20.2-cdh3u3//lib/native/Linux-amd64-64:/opt/hes/hadoop-0.20.2-cdh3u3//lib/native/Linux-amd64-64:/opt/hes/hbase-0.90.4-cdh3u3//lib/native/Linux-amd64-64
12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client
environment:os.version=3.0.0-12-server
12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:user.name
=patodirahul
12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client
environment:user.home=/home/patodirahul
12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client
environment:user.dir=/opt/hes/flume-1.1.0-cdh4.0.0
12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Initiating client connection,
connectString=localhost:2181 sessionTimeout=180000 watcher=hconnection
12/06/21 17:28:25 INFO zookeeper.ClientCnxn: Opening socket connection to
server localhost/127.0.0.1:2181
12/06/21 17:28:25 INFO zookeeper.ClientCnxn: Socket connection established
to localhost/127.0.0.1:2181, initiating session
12/06/21 17:28:25 INFO zookeeper.ClientCnxn: Session establishment complete
on server localhost/127.0.0.1:2181, sessionid = 0x137bb31af040068,
negotiated timeout = 40000
as far as I know there is no requirement of zookeeper in Flume NG
Can Any One help me in this issue
Re: Zookeeper in Flume NG log
Posted by Hari Shreedharan <hs...@cloudera.com>.
Exactly. The Hbase client API requires ZK to find -ROOT-, and .META. ZK is a dependency introduced by Hbase client API not Flume itself.
Thanks
Hari
--
Hari Shreedharan
On Thursday, June 21, 2012 at 9:37 AM, Mubarak Seyed wrote:
> Hbase client (HTable) requires ZK calls to find the location of -ROOT- and then location of .META. to find the Region server which host the region for the given key range. This lookup will happen for first time and then it will be cached in HBase client side until region location changed.
>
>
>
> On Jun 21, 2012, at 5:30 AM, Will McQueen <will@cloudera.com (mailto:will@cloudera.com)> wrote:
>
> > >>as far as I know there is no requirement of zookeeper in Flume NG
> >
> > I believe that hbase has a dependency on zookeeper lib. I don't believe that Flume has a direct dependency on zookeeper, though. Can someone please confirm?
> >
> > Cheers,
> > Will
> >
> > On Thu, Jun 21, 2012 at 5:09 AM, Rahul Patodi <patodirahul.hadoop@gmail.com (mailto:patodirahul.hadoop@gmail.com)> wrote:
> > > Hi,
> > > I am trying to put data into HBase using Flume-NG (configure flume-NG Hbase sink)
> > >
> > > When I start flume NG (1.1)
> > > patodirahul@hadoopPseudo:/opt/hes/flume-1.1.0-cdh4.0.0$ bin/flume-ng agent -n hbase-agent -f conf/hbase.cnf
> > > Having configuration:
> > >
> > > hbase-agent.sources=tail
> > > hbase-agent.sinks=sink1
> > > hbase-agent.channels=ch1
> > >
> > > hbase-agent.sources.tail.type=exec
> > > hbase-agent.sources.tail.command=tail -F /tmp/test05
> > > hbase-agent.sources.tail.channels=ch1
> > >
> > > hbase-agent.sinks.sink1.type=org.apache.flume.sink.hbase.HBaseSink
> > > hbase-agent.sinks.sink1.channel=ch1
> > > hbase-agent.sinks.sink1.table=t002
> > > hbase-agent.sinks.sink1.columnFamily=cf
> > > hbase-agent.sinks.sink1.column=foo
> > > hbase-agent.sinks.sink1.serializer=org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
> > > hbase-agent.sinks.sink1.serializer.payloadColumn=col1
> > > hbase-agent.sinks.sink1.serializer.incrementColumn=col1
> > > hbase-agent.sinks.sink1.serializer.rowPrefix=1+
> > > hbase-agent.sinks.sink1.serializer.suffix=timestamp
> > >
> > > hbase-agent.channels.ch1.type=memory
> > >
> > > its showing zookeeper in logs (on console) as mentioned below
> > >
> > > ...
> > > ...
> > >
> > > 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:java.library.path=:/opt/hes/hadoop-0.20.2-cdh3u3//lib/native/Linux-amd64-64:/opt/hes/hadoop-0.20.2-cdh3u3//lib/native/Linux-amd64-64:/opt/hes/hbase-0.90.4-cdh3u3//lib/native/Linux-amd64-64
> > >
> > >
> > > 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
> > >
> > >
> > > 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
> > >
> > >
> > > 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:os.name (http://os.name)=Linux
> > >
> > >
> > > 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
> > >
> > >
> > > 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:os.version=3.0.0-12-server
> > >
> > >
> > > 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:user.name (http://user.name)=patodirahul
> > >
> > >
> > > 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/patodirahul
> > >
> > >
> > > 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:user.dir=/opt/hes/flume-1.1.0-cdh4.0.0
> > >
> > >
> > > 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=180000 watcher=hconnection
> > >
> > >
> > > 12/06/21 17:28:25 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181 (http://127.0.0.1:2181)
> > >
> > >
> > > 12/06/21 17:28:25 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181 (http://127.0.0.1:2181), initiating session
> > >
> > >
> > > 12/06/21 17:28:25 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181 (http://127.0.0.1:2181), sessionid = 0x137bb31af040068, negotiated timeout = 40000
> > >
> > >
> > > as far as I know there is no requirement of zookeeper in Flume NG
> > > Can Any One help me in this issue
> > >
> > >
> >
Re: Zookeeper in Flume NG log
Posted by Mubarak Seyed <se...@apple.com>.
Hbase client (HTable) requires ZK calls to find the location of -ROOT- and then location of .META. to find the Region server which host the region for the given key range. This lookup will happen for first time and then it will be cached in HBase client side until region location changed.
On Jun 21, 2012, at 5:30 AM, Will McQueen <wi...@cloudera.com> wrote:
> >>as far as I know there is no requirement of zookeeper in Flume NG
>
> I believe that hbase has a dependency on zookeeper lib. I don't believe that Flume has a direct dependency on zookeeper, though. Can someone please confirm?
>
> Cheers,
> Will
>
> On Thu, Jun 21, 2012 at 5:09 AM, Rahul Patodi <pa...@gmail.com> wrote:
> Hi,
> I am trying to put data into HBase using Flume-NG (configure flume-NG Hbase sink)
>
> When I start flume NG (1.1)
> patodirahul@hadoopPseudo:/opt/hes/flume-1.1.0-cdh4.0.0$ bin/flume-ng agent -n hbase-agent -f conf/hbase.cnf
> Having configuration:
>
> hbase-agent.sources=tail
> hbase-agent.sinks=sink1
> hbase-agent.channels=ch1
>
> hbase-agent.sources.tail.type=exec
> hbase-agent.sources.tail.command=tail -F /tmp/test05
> hbase-agent.sources.tail.channels=ch1
>
> hbase-agent.sinks.sink1.type=org.apache.flume.sink.hbase.HBaseSink
> hbase-agent.sinks.sink1.channel=ch1
> hbase-agent.sinks.sink1.table=t002
> hbase-agent.sinks.sink1.columnFamily=cf
> hbase-agent.sinks.sink1.column=foo
> hbase-agent.sinks.sink1.serializer=org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
> hbase-agent.sinks.sink1.serializer.payloadColumn=col1
> hbase-agent.sinks.sink1.serializer.incrementColumn=col1
> hbase-agent.sinks.sink1.serializer.rowPrefix=1+
> hbase-agent.sinks.sink1.serializer.suffix=timestamp
>
> hbase-agent.channels.ch1.type=memory
>
> its showing zookeeper in logs (on console) as mentioned below
>
> ...
> ...
> 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:java.library.path=:/opt/hes/hadoop-0.20.2-cdh3u3//lib/native/Linux-amd64-64:/opt/hes/hadoop-0.20.2-cdh3u3//lib/native/Linux-amd64-64:/opt/hes/hbase-0.90.4-cdh3u3//lib/native/Linux-amd64-64
>
> 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
>
> 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
>
> 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
>
> 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
>
> 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:os.version=3.0.0-12-server
>
> 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:user.name=patodirahul
>
> 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/patodirahul
>
> 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:user.dir=/opt/hes/flume-1.1.0-cdh4.0.0
>
> 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=180000 watcher=hconnection
>
> 12/06/21 17:28:25 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181
>
> 12/06/21 17:28:25 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
>
> 12/06/21 17:28:25 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x137bb31af040068, negotiated timeout = 40000
>
>
> as far as I know there is no requirement of zookeeper in Flume NG
> Can Any One help me in this issue
>
>
>
Re: Zookeeper in Flume NG log
Posted by Will McQueen <wi...@cloudera.com>.
>>as far as I know there is no requirement of zookeeper in Flume NG
I believe that hbase has a dependency on zookeeper lib. I don't believe
that Flume has a direct dependency on zookeeper, though. Can someone please
confirm?
Cheers,
Will
On Thu, Jun 21, 2012 at 5:09 AM, Rahul Patodi
<pa...@gmail.com>wrote:
> Hi,
> I am trying to put data into HBase using Flume-NG (configure flume-NG
> Hbase sink)
>
> When I start flume NG (1.1)
> patodirahul@hadoopPseudo:/opt/hes/flume-1.1.0-cdh4.0.0$ bin/flume-ng
> agent -n hbase-agent -f conf/hbase.cnf
> Having configuration:
>
> hbase-agent.sources=tail
> hbase-agent.sinks=sink1
> hbase-agent.channels=ch1
>
> hbase-agent.sources.tail.type=exec
> hbase-agent.sources.tail.command=tail -F /tmp/test05
> hbase-agent.sources.tail.channels=ch1
>
> hbase-agent.sinks.sink1.type=org.apache.flume.sink.hbase.HBaseSink
> hbase-agent.sinks.sink1.channel=ch1
> hbase-agent.sinks.sink1.table=t002
> hbase-agent.sinks.sink1.columnFamily=cf
> hbase-agent.sinks.sink1.column=foo
>
> hbase-agent.sinks.sink1.serializer=org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
> hbase-agent.sinks.sink1.serializer.payloadColumn=col1
> hbase-agent.sinks.sink1.serializer.incrementColumn=col1
> hbase-agent.sinks.sink1.serializer.rowPrefix=1+
> hbase-agent.sinks.sink1.serializer.suffix=timestamp
>
> hbase-agent.channels.ch1.type=memory
>
> its showing zookeeper in logs (on console) as mentioned below
>
> ...
> ...
>
> 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client
> environment:java.library.path=:/opt/hes/hadoop-0.20.2-cdh3u3//lib/native/Linux-amd64-64:/opt/hes/hadoop-0.20.2-cdh3u3//lib/native/Linux-amd64-64:/opt/hes/hbase-0.90.4-cdh3u3//lib/native/Linux-amd64-64
>
> 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/tmp
>
> 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
>
> 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:os.name
> =Linux
>
> 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client
> environment:os.arch=amd64
>
> 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client
> environment:os.version=3.0.0-12-server
>
> 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client environment:user.name
> =patodirahul
>
> 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client
> environment:user.home=/home/patodirahul
>
> 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Client
> environment:user.dir=/opt/hes/flume-1.1.0-cdh4.0.0
>
> 12/06/21 17:28:25 INFO zookeeper.ZooKeeper: Initiating client connection,
> connectString=localhost:2181 sessionTimeout=180000 watcher=hconnection
>
> 12/06/21 17:28:25 INFO zookeeper.ClientCnxn: Opening socket connection to
> server localhost/127.0.0.1:2181
>
> 12/06/21 17:28:25 INFO zookeeper.ClientCnxn: Socket connection established
> to localhost/127.0.0.1:2181, initiating session
>
> 12/06/21 17:28:25 INFO zookeeper.ClientCnxn: Session establishment
> complete on server localhost/127.0.0.1:2181, sessionid =
> 0x137bb31af040068, negotiated timeout = 40000
>
> as far as I know there is no requirement of zookeeper in Flume NG
> Can Any One help me in this issue
>
>
>