You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Gaurav Khanna <kh...@yahoo.com> on 2012/02/02 19:54:49 UTC
HBase and Flume
Hi,
Am using flume Flume 0.9.4-cdh3u2 and wanted to use the HBase Sink.
Entered this as the sink: hbase("testtable", "%{nanos}", "colfam1", "q1", "%{body}")
and got the following in the master logs (now I know that I am missing some configuration steps - please can anybody point me to where I can get the Hbase related plugin and how to configure that?):
2012-02-02 12:41:36,916 [exec-thread] INFO master.CommandManager: Executing command: config [POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com, tail("/app/log-file-gen/out/server.log"), hbase("testtable", "%{nanos}", "colfam1", "q1", "%{body}")]
2012-02-02 12:41:36,917 [exec-thread] ERROR master.CommandManager: Unexpected exception during cmdid:0 config [POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com, tail("/app/log-file-gen/out/server.log"), hbase("testtable", "%{nanos}", "colfam1", "q1", "%{body}")] : Attempted to write an invalid sink/source: Invalid sink: hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
java.lang.IllegalArgumentException: Attempted to write an invalid sink/source: Invalid sink: hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
at com.cloudera.flume.master.ConfigManager.setConfig(ConfigManager.java:96)
at com.cloudera.flume.master.flows.FlowConfigManager.setConfig(FlowConfigManager.java:285)
at com.cloudera.flume.master.TranslatingConfigurationManager.setConfig(TranslatingConfigurationManager.java:101)
at com.cloudera.flume.master.ConfigCommand$1.exec(ConfigCommand.java:87)
at com.cloudera.flume.master.CommandManager.exec(CommandManager.java:266)
at com.cloudera.flume.master.CommandManager.handleCommand(CommandManager.java:205)
at com.cloudera.flume.master.CommandManager$ExecThread.run(CommandManager.java:236)
Caused by: com.cloudera.flume.conf.FlumeIdException: Invalid sink: hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
at com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:503)
at com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:531)
at com.cloudera.flume.conf.FlumeBuilder.buildSink(FlumeBuilder.java:302)
at com.cloudera.flume.master.ConfigManager.setConfig(ConfigManager.java:93)
Re: HBase and Flume
Posted by Gaurav Khanna <kh...@yahoo.com>.
Hi Alex,
I got that working. I had to define the FLUME_CLASSPATHto point to the HBase config.
________________________________
From: Alex Baranau <al...@gmail.com>
To: flume-user@incubator.apache.org; Gaurav Khanna <kh...@yahoo.com>
Sent: Monday, February 6, 2012 10:15 AM
Subject: Re: HBase and Flume
Hi,
Hm. Looks like there are too many open connections to the ZK from that host. How many flume nodes which write to Hbase did you start?
Also, make sure your ZK allows proper number of connections from one host (I remember there was some issue with having default less than expected(30) in one of ZK versions). You can add this to zoo.cfg (and restart ZK) manually to be sure about how many is allowed:
maxClientCnxns=30
Alex Baranau
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase
On Thu, Feb 2, 2012 at 2:59 PM, Gaurav Khanna <kh...@yahoo.com> wrote:
>
>Got the jar for the hbase plugin and updated the flume-site.xml with the plugin configuration. Now I can see the plugin and use it as well. But here is the problem:
>
>
>This is on a client box that is not part of the hbase or hadoop cluster and am getting the following exception in the flume node. These are the env variables:
># HADOOP RELATED STUFF
>export HADOOP_CONF_DIR=/app/hadoop/hadoop-conf
>export HBASE_CONF_DIR=/app/hadoop/hbase-conf
>export JAVA_HOME=/app/downloads/cloudera/jdk1.6.0_25
>export PATH=/app/downloads/cloudera/jdk1.6.0_25/bin:$PATH
>export ZOOKEEPER_CONF=$HBASE_CONF_DIR
>export
HIVE_AUX_JARS_PATH=/usr/lib/hbase/hbase-0.90.4-cdh3u3.jar:/usr/lib/hbase/lib/zookeeper.jar:/usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u2.jar:/usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u2.jar
>
>
>
>And this is the exception that I get:
>
>
>012-02-02 13:52:08,721 [logicalNode POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22] ERROR connector.DirectDriver: Closing down due to exception on open calls
>2012-02-02 13:52:08,722 [logicalNode POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22] INFO connector.DirectDriver: Connector logicalNode POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22 exited with error: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more
information.
>org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:160)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1209)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:511)
> at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:502)
> at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:172)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:175)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:153)
> at com.cloudera.flume.hbase.HBaseSink.open(HBaseSink.java:144)
> at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:88)
>Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>
>
>
>
>________________________________
> From: Gaurav Khanna <kh...@yahoo.com>
>To: "flume-user@incubator.apache.org" <fl...@incubator.apache.org>; Gaurav Khanna <kh...@yahoo.com>
>Sent: Thursday, February 2, 2012 10:57 AM
>Subject: Re: HBase and Flume
>
>
>
>Got the information at:
>https://github.com/cloudera/flume/tree/master/plugins/flume-plugin-hbasesink
>
>
>Thanks
>Gaurav
>
>
>
>________________________________
> From: Gaurav Khanna <kh...@yahoo.com>
>To: "flume-user@incubator.apache.org" <fl...@incubator.apache.org>
>Sent: Thursday, February 2, 2012 10:54 AM
>Subject: HBase and Flume
>
>
>Hi,
>Am using flume Flume 0.9.4-cdh3u2 and wanted to use the HBase Sink.
>
>
>Entered this as the sink: hbase("testtable", "%{nanos}", "colfam1", "q1", "%{body}")
>
>
>and got the following in the master logs (now I know that I am missing some configuration steps - please can anybody point me to where I can get the Hbase related plugin and how to configure that?):
>
>
>2012-02-02 12:41:36,916 [exec-thread] INFO master.CommandManager: Executing command: config [POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com, tail("/app/log-file-gen/out/server.log"), hbase("testtable", "%{nanos}", "colfam1", "q1", "%{body}")]
>2012-02-02 12:41:36,917 [exec-thread] ERROR master.CommandManager: Unexpected exception during cmdid:0 config
[POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com, tail("/app/log-file-gen/out/server.log"), hbase("testtable", "%{nanos}", "colfam1", "q1", "%{body}")] : Attempted to write an invalid sink/source: Invalid sink: hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
>java.lang.IllegalArgumentException: Attempted to write an invalid sink/source: Invalid sink: hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
> at com.cloudera.flume.master.ConfigManager.setConfig(ConfigManager.java:96)
> at com.cloudera.flume.master.flows.FlowConfigManager.setConfig(FlowConfigManager.java:285)
> at com.cloudera.flume.master.TranslatingConfigurationManager.setConfig(TranslatingConfigurationManager.java:101)
> at
com.cloudera.flume.master.ConfigCommand$1.exec(ConfigCommand.java:87)
> at com.cloudera.flume.master.CommandManager.exec(CommandManager.java:266)
> at com.cloudera.flume.master.CommandManager.handleCommand(CommandManager.java:205)
> at com.cloudera.flume.master.CommandManager$ExecThread.run(CommandManager.java:236)
>Caused by: com.cloudera.flume.conf.FlumeIdException: Invalid sink: hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
> at com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:503)
> at com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:531)
> at
com.cloudera.flume.conf.FlumeBuilder.buildSink(FlumeBuilder.java:302)
> at com.cloudera.flume.master.ConfigManager.setConfig(ConfigManager.java:93)
>
>
>
>
>
>
Re: HBase and Flume
Posted by Alex Baranau <al...@gmail.com>.
Hi,
Hm. Looks like there are too many open connections to the ZK from that
host. How many flume nodes which write to Hbase did you start?
Also, make sure your ZK allows proper number of connections from one host
(I remember there was some issue with having default less than expected(30)
in one of ZK versions). You can add this to zoo.cfg (and restart ZK)
manually to be sure about how many is allowed:
maxClientCnxns=30
Alex Baranau
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase
On Thu, Feb 2, 2012 at 2:59 PM, Gaurav Khanna <kh...@yahoo.com> wrote:
>
> Got the jar for the hbase plugin and updated the flume-site.xml with the
> plugin configuration. Now I can see the plugin and use it as well. But here
> is the problem:
>
> This is on a client box that is not part of the hbase or hadoop cluster
> and am getting the following exception in the flume node. These are the env
> variables:
> # HADOOP RELATED STUFF
> export HADOOP_CONF_DIR=/app/hadoop/hadoop-conf
> export HBASE_CONF_DIR=/app/hadoop/hbase-conf
> export JAVA_HOME=/app/downloads/cloudera/jdk1.6.0_25
> export PATH=/app/downloads/cloudera/jdk1.6.0_25/bin:$PATH
> export ZOOKEEPER_CONF=$HBASE_CONF_DIR
> export
> HIVE_AUX_JARS_PATH=/usr/lib/hbase/hbase-0.90.4-cdh3u3.jar:/usr/lib/hbase/lib/zookeeper.jar:/usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u2.jar:/usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u2.jar
>
> And this is the exception that I get:
>
> 012-02-02 13:52:08,721 [logicalNode
> POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22] ERROR
> connector.DirectDriver: Closing down due to exception on open calls
> 2012-02-02 13:52:08,722 [logicalNode
> POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22] INFO
> connector.DirectDriver: Connector logicalNode
> POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22 exited with error: HBase is
> able to connect to ZooKeeper but the connection closes immediately. This
> could be a sign that the server has too many connections (30 is the
> default). Consider inspecting your ZK server logs for that error and then
> make sure you are reusing HBaseConfiguration as often as you can. See
> HTable's javadoc for more information.
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could be a
> sign that the server has too many connections (30 is the default). Consider
> inspecting your ZK server logs for that error and then make sure you are
> reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> more information.
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:160)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1209)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:511)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:502)
> at
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:172)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:175)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:153)
> at com.cloudera.flume.hbase.HBaseSink.open(HBaseSink.java:144)
> at
> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:88)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>
> ------------------------------
> *From:* Gaurav Khanna <kh...@yahoo.com>
> *To:* "flume-user@incubator.apache.org" <fl...@incubator.apache.org>;
> Gaurav Khanna <kh...@yahoo.com>
> *Sent:* Thursday, February 2, 2012 10:57 AM
> *Subject:* Re: HBase and Flume
>
> Got the information at:
>
> https://github.com/cloudera/flume/tree/master/plugins/flume-plugin-hbasesink
>
> Thanks
> Gaurav
>
> ------------------------------
> *From:* Gaurav Khanna <kh...@yahoo.com>
> *To:* "flume-user@incubator.apache.org" <fl...@incubator.apache.org>
> *Sent:* Thursday, February 2, 2012 10:54 AM
> *Subject:* HBase and Flume
>
> Hi,
> Am using flume Flume 0.9.4-cdh3u2 and wanted to use the HBase Sink.
>
> Entered this as the sink: hbase("testtable", "%{nanos}", "colfam1", "q1",
> "%{body}")
>
> and got the following in the master logs (now I know that I am missing
> some configuration steps - please can anybody point me to where I can get
> the Hbase related plugin and how to configure that?):
>
> 2012-02-02 12:41:36,916 [exec-thread] INFO master.CommandManager:
> Executing command: config [POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com,
> tail("/app/log-file-gen/out/server.log"), hbase("testtable", "%{nanos}",
> "colfam1", "q1", "%{body}")]
> 2012-02-02 12:41:36,917 [exec-thread] ERROR master.CommandManager:
> Unexpected exception during cmdid:0 config [
> POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com,
> tail("/app/log-file-gen/out/server.log"), hbase("testtable", "%{nanos}",
> "colfam1", "q1", "%{body}")] : Attempted to write an invalid sink/source:
> Invalid sink: hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
> java.lang.IllegalArgumentException: Attempted to write an invalid
> sink/source: Invalid sink: hbase( "testtable", "%{nanos}", "colfam1", "q1",
> "%{body}" )
> at
> com.cloudera.flume.master.ConfigManager.setConfig(ConfigManager.java:96)
> at
> com.cloudera.flume.master.flows.FlowConfigManager.setConfig(FlowConfigManager.java:285)
> at
> com.cloudera.flume.master.TranslatingConfigurationManager.setConfig(TranslatingConfigurationManager.java:101)
> at
> com.cloudera.flume.master.ConfigCommand$1.exec(ConfigCommand.java:87)
> at
> com.cloudera.flume.master.CommandManager.exec(CommandManager.java:266)
> at
> com.cloudera.flume.master.CommandManager.handleCommand(CommandManager.java:205)
> at
> com.cloudera.flume.master.CommandManager$ExecThread.run(CommandManager.java:236)
> Caused by: com.cloudera.flume.conf.FlumeIdException: Invalid sink: hbase(
> "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
> at
> com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:503)
> at
> com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:531)
> at
> com.cloudera.flume.conf.FlumeBuilder.buildSink(FlumeBuilder.java:302)
> at
> com.cloudera.flume.master.ConfigManager.setConfig(ConfigManager.java:93)
>
>
>
>
>
>
Re: Is there a flume decorator that only allows events to pass through based on condition
Posted by Gaurav Khanna <kh...@yahoo.com>.
Guess what I am asking for is the equivalent for the grep here:
tail -f aa.log | grep "User not logged in" ....
Thanks
________________________________
From: Gaurav Khanna <kh...@yahoo.com>
To: "flume-user@incubator.apache.org" <fl...@incubator.apache.org>
Sent: Sunday, February 5, 2012 7:18 PM
Subject: Is there a flume decorator that only allows events to pass through based on condition
Hi,
Want to create a selection criteria that allows only certain lines in a log file that fulfill the criteria to be passed on to the sink. Is there such a decorator or sink?
For instance: if there is a line in the log that contains: "User login failed", then that line should be allowed to pass through the next sink in the line.
For instance: in the following config, I need to replace the text between "****" with a selection criteria that if true then allows the event to be written to HBase.
POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com : tail("/app/log-file-gen/out/server.log") | regex("some regex",1,"aaa") ****selection criteria**** hbase("testtable", "%{aaa}", "colfam1", "q1", "%{body}") ;
Thanks
Gaurav
Is there a flume decorator that only allows events to pass through based on condition
Posted by Gaurav Khanna <kh...@yahoo.com>.
Hi,
Want to create a selection criteria that allows only certain lines in a log file that fulfill the criteria to be passed on to the sink. Is there such a decorator or sink?
For instance: if there is a line in the log that contains: "User login failed", then that line should be allowed to pass through the next sink in the line.
For instance: in the following config, I need to replace the text between "****" with a selection criteria that if true then allows the event to be written to HBase.
POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com : tail("/app/log-file-gen/out/server.log") | regex("some regex",1,"aaa") ****selection criteria**** hbase("testtable", "%{aaa}", "colfam1", "q1", "%{body}") ;
Thanks
Gaurav
Re: HBase and Flume
Posted by Gaurav Khanna <kh...@yahoo.com>.
Got the jar for the hbase plugin and updated the flume-site.xml with the plugin configuration. Now I can see the plugin and use it as well. But here is the problem:
This is on a client box that is not part of the hbase or hadoop cluster and am getting the following exception in the flume node. These are the env variables:
# HADOOP RELATED STUFF
export HADOOP_CONF_DIR=/app/hadoop/hadoop-conf
export HBASE_CONF_DIR=/app/hadoop/hbase-conf
export JAVA_HOME=/app/downloads/cloudera/jdk1.6.0_25
export PATH=/app/downloads/cloudera/jdk1.6.0_25/bin:$PATH
export ZOOKEEPER_CONF=$HBASE_CONF_DIR
export HIVE_AUX_JARS_PATH=/usr/lib/hbase/hbase-0.90.4-cdh3u3.jar:/usr/lib/hbase/lib/zookeeper.jar:/usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u2.jar:/usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u2.jar
And this is the exception that I get:
012-02-02 13:52:08,721 [logicalNode POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22] ERROR connector.DirectDriver: Closing down due to exception on open calls
2012-02-02 13:52:08,722 [logicalNode POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22] INFO connector.DirectDriver: Connector logicalNode POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com-22 exited with error: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:160)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1209)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:511)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:502)
at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:172)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:175)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:153)
at com.cloudera.flume.hbase.HBaseSink.open(HBaseSink.java:144)
at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:88)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
________________________________
From: Gaurav Khanna <kh...@yahoo.com>
To: "flume-user@incubator.apache.org" <fl...@incubator.apache.org>; Gaurav Khanna <kh...@yahoo.com>
Sent: Thursday, February 2, 2012 10:57 AM
Subject: Re: HBase and Flume
Got the information at:
https://github.com/cloudera/flume/tree/master/plugins/flume-plugin-hbasesink
Thanks
Gaurav
________________________________
From: Gaurav Khanna <kh...@yahoo.com>
To: "flume-user@incubator.apache.org" <fl...@incubator.apache.org>
Sent: Thursday, February 2, 2012 10:54 AM
Subject: HBase and Flume
Hi,
Am using flume Flume 0.9.4-cdh3u2 and wanted to use the HBase Sink.
Entered this as the sink: hbase("testtable", "%{nanos}", "colfam1", "q1", "%{body}")
and got the following in the master logs (now I know that I am missing some configuration steps - please can anybody point me to where I can get the Hbase related plugin and how to configure that?):
2012-02-02 12:41:36,916 [exec-thread] INFO master.CommandManager: Executing command: config [POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com, tail("/app/log-file-gen/out/server.log"), hbase("testtable", "%{nanos}", "colfam1", "q1", "%{body}")]
2012-02-02 12:41:36,917 [exec-thread] ERROR master.CommandManager: Unexpected exception during cmdid:0 config
[POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com, tail("/app/log-file-gen/out/server.log"), hbase("testtable", "%{nanos}", "colfam1", "q1", "%{body}")] : Attempted to write an invalid sink/source: Invalid sink: hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
java.lang.IllegalArgumentException: Attempted to write an invalid sink/source: Invalid sink: hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
at com.cloudera.flume.master.ConfigManager.setConfig(ConfigManager.java:96)
at com.cloudera.flume.master.flows.FlowConfigManager.setConfig(FlowConfigManager.java:285)
at com.cloudera.flume.master.TranslatingConfigurationManager.setConfig(TranslatingConfigurationManager.java:101)
at
com.cloudera.flume.master.ConfigCommand$1.exec(ConfigCommand.java:87)
at com.cloudera.flume.master.CommandManager.exec(CommandManager.java:266)
at com.cloudera.flume.master.CommandManager.handleCommand(CommandManager.java:205)
at com.cloudera.flume.master.CommandManager$ExecThread.run(CommandManager.java:236)
Caused by: com.cloudera.flume.conf.FlumeIdException: Invalid sink: hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
at com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:503)
at com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:531)
at
com.cloudera.flume.conf.FlumeBuilder.buildSink(FlumeBuilder.java:302)
at com.cloudera.flume.master.ConfigManager.setConfig(ConfigManager.java:93)
Re: HBase and Flume
Posted by Gaurav Khanna <kh...@yahoo.com>.
Got the information at:
https://github.com/cloudera/flume/tree/master/plugins/flume-plugin-hbasesink
Thanks
Gaurav
________________________________
From: Gaurav Khanna <kh...@yahoo.com>
To: "flume-user@incubator.apache.org" <fl...@incubator.apache.org>
Sent: Thursday, February 2, 2012 10:54 AM
Subject: HBase and Flume
Hi,
Am using flume Flume 0.9.4-cdh3u2 and wanted to use the HBase Sink.
Entered this as the sink: hbase("testtable", "%{nanos}", "colfam1", "q1", "%{body}")
and got the following in the master logs (now I know that I am missing some configuration steps - please can anybody point me to where I can get the Hbase related plugin and how to configure that?):
2012-02-02 12:41:36,916 [exec-thread] INFO master.CommandManager: Executing command: config [POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com, tail("/app/log-file-gen/out/server.log"), hbase("testtable", "%{nanos}", "colfam1", "q1", "%{body}")]
2012-02-02 12:41:36,917 [exec-thread] ERROR master.CommandManager: Unexpected exception during cmdid:0 config
[POC_Hadoop_Client_1.ssodev-bo.mtv1.vrsn.com, tail("/app/log-file-gen/out/server.log"), hbase("testtable", "%{nanos}", "colfam1", "q1", "%{body}")] : Attempted to write an invalid sink/source: Invalid sink: hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
java.lang.IllegalArgumentException: Attempted to write an invalid sink/source: Invalid sink: hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
at com.cloudera.flume.master.ConfigManager.setConfig(ConfigManager.java:96)
at com.cloudera.flume.master.flows.FlowConfigManager.setConfig(FlowConfigManager.java:285)
at com.cloudera.flume.master.TranslatingConfigurationManager.setConfig(TranslatingConfigurationManager.java:101)
at
com.cloudera.flume.master.ConfigCommand$1.exec(ConfigCommand.java:87)
at com.cloudera.flume.master.CommandManager.exec(CommandManager.java:266)
at com.cloudera.flume.master.CommandManager.handleCommand(CommandManager.java:205)
at com.cloudera.flume.master.CommandManager$ExecThread.run(CommandManager.java:236)
Caused by: com.cloudera.flume.conf.FlumeIdException: Invalid sink: hbase( "testtable", "%{nanos}", "colfam1", "q1", "%{body}" )
at com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:503)
at com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:531)
at
com.cloudera.flume.conf.FlumeBuilder.buildSink(FlumeBuilder.java:302)
at com.cloudera.flume.master.ConfigManager.setConfig(ConfigManager.java:93)