You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by "Xu (Simon) Chen" <xc...@gmail.com> on 2013/01/09 05:19:56 UTC
weird hbase sink behavior
Hi folks,
I am using flume to sink into hbase. I had the set up working
yesterday, but when I restarted my flume agent, it for some reason
cannot dump to hbase anymore (too bad I don't have a version-control
for my config yet.)
Here is the configuration and execution output. The agent essentially
listens for objects via avro and dump to console for debugging and to
hbase. The last portion of the console output shows that an object is
indeed received, but for some reason "scan" on the hbase doesn't show
anything new. I tried Regex/Simple serializer and HBase and
AsyncHbase, none works now...
>From the log I don't see anything obviously wrong. I did a tcpdump on
the agent, and saw it established a couple of connections to the
regionservers, but closed them a few seconds later.
Any info on how to correctly configure this or troubleshoot would be
appreciated.
Thanks!
-Simon
root@flume-agent1:/usr/local/lib/flume# cat conf/test.conf
a1.channels = c1 c2
a1.sinks = k1 k2
a1.sources = r1
a1.sources.r1.type = avro
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 8080
a1.sinks.k1.type = org.apache.flume.sink.hbase.HBaseSink
a1.sinks.k1.table = foo_table
a1.sinks.k1.columnFamily = bar_cf
a1.sinks.k1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
a1.sinks.k2.type = logger
#a1.channels.c1.type = file
#a1.channels.c1.checkpointDir = /mnt/flume/checkpoint
#a1.channels.c1.dataDirs = /mnt/flume/data
a1.channels.c1.type = memory
a1.channels.c2.type = memory
a1.sources.r1.channels = c1 c2
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c2
root@flume-agent1:/usr/local/lib/flume# bin/flume-ng agent --name a1
--conf conf --conf-file conf/test.conf
-Dflume.root.logger=INFO,console
Info: Including Hadoop libraries found via
(/usr/local/hadoop/bin/hadoop) for HDFS access
Warning: $HADOOP_HOME is deprecated.
Warning: $HADOOP_HOME is deprecated.
Info: Excluding /usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar
from classpath
Info: Excluding
/usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
classpath
Info: Including HBASE libraries found via (/usr/local/hbase/bin/hbase)
for HBASE access
Info: Excluding /usr/local/hbase/lib/slf4j-api-1.4.3.jar from classpath
Info: Excluding /usr/local/hbase/lib/slf4j-log4j12-1.4.3.jar from classpath
Info: Excluding /usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar
from classpath
Info: Excluding
/usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
classpath
+ exec /usr/lib/jvm/default-java/bin/java -Xmx20m
-Dflume.root.logger=INFO,console -cp
'/usr/local/lib/flume/conf:/usr/local/lib/flume/lib/*:/usr/local/hadoop/libexec/../conf:/usr/lib/jvm/default-java/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.4.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/hbase/conf:/usr/lib/tools.jar:/usr/local/hbase:/usr/local/hbase/hbase-0.94.3.jar:/usr/local/hbase/hbase-0.94.3-tests.jar:/usr/local/hbase/lib/activation-1.1.jar:/usr/local/hbase/lib/asm-3.1.jar:/usr/local/hbase/lib/avro-1.5.3.jar:/usr/local/hbase/lib/avro-ipc-1.5.3.jar:/usr/local/hbase/lib/commons-beanutils-1.7.0.jar:/usr/local/hbase/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hbase/lib/commons-cli-1.2.jar:/usr/local/hbase/lib/commons-codec-1.4.jar:/usr/local/hbase/lib/commons-collections-3.2.1.jar:/usr/local/hbase/lib/commons-configuration-1.6.jar:/usr/local/hbase/lib/commons-digester-1.8.jar:/usr/local/hbase/lib/commons-el-1.0.jar:/usr/local/hbase/lib/commons-httpclient-3.1.jar:/usr/local/hbase/lib/commons-io-2.1.jar:/usr/local/hbase/lib/commons-lang-2.5.jar:/usr/local/hbase/lib/commons-logging-1.1.1.jar:/usr/local/hbase/lib/commons-math-2.1.jar:/usr/local/hbase/lib/commons-net-1.4.1.jar:/usr/local/hbase/lib/core-3.1.1.jar:/usr/local/hbase/lib/guava-11.0.2.jar:/usr/local/hbase/lib/hadoop-core-1.0.4.jar:/usr/local/hbase/lib/high-scale-lib-1.1.1.jar:/usr/local/hbase/lib/httpclient-4.1.2.jar:/usr/local/hbase/lib/httpcore-4.1.3.jar:/usr/local/hbase/lib/jackson-core-asl-1.8.8.jar:/usr/local/hbase/lib/jackson-jaxrs-1.8.8.jar:/usr/local/hbase/lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hbase/lib/jackson-xc-1.8.8.jar:/usr/local/hbase/lib/jamon-runtime-2.3.1.jar:/usr/local/hbase/lib/jasper-compiler-5.5.23.jar:/usr/local/hbase/lib/jasper-runtime-5.5.23.jar:/usr/local/hbase/lib/jaxb-api-2.1.jar:/usr/local/hbase/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hbase/lib/jersey-core-1.8.jar:/usr/local/hbase/lib/jersey-json-1.8.jar:/usr/local/hbase/lib/jersey-server-1.8.jar:/usr/local/hbase/lib/jettison-1.1.jar:/usr/local/hbase/lib/jetty-6.1.26.jar:/usr/local/hbase/lib/jetty-util-6.1.26.jar:/usr/local/hbase/lib/jruby-complete-1.6.5.jar:/usr/local/hbase/lib/jsp-2.1-6.1.14.jar:/usr/local/hbase/lib/jsp-api-2.1-6.1.14.jar:/usr/local/hbase/lib/jsr305-1.3.9.jar:/usr/local/hbase/lib/junit-4.10-HBASE-1.jar:/usr/local/hbase/lib/libthrift-0.8.0.jar:/usr/local/hbase/lib/log4j-1.2.16.jar:/usr/local/hbase/lib/metrics-core-2.1.2.jar:/usr/local/hbase/lib/netty-3.2.4.Final.jar:/usr/local/hbase/lib/protobuf-java-2.4.0a.jar:/usr/local/hbase/lib/servlet-api-2.5-6.1.14.jar:/usr/local/hbase/lib/snappy-java-1.0.3.2.jar:/usr/local/hbase/lib/stax-api-1.0.1.jar:/usr/local/hbase/lib/velocity-1.7.jar:/usr/local/hbase/lib/xmlenc-0.52.jar:/usr/local/hbase/lib/zookeeper-3.4.3.jar:/usr/local/hadoop/libexec/../conf:/usr/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.4.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/hbase/conf'
-Djava.library.path=:/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64:/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64:/usr/local/hbase/lib/native/Linux-amd64-64
org.apache.flume.node.Application --name a1 --conf-file conf/test.conf
2013-01-09 04:11:15,871 (main) [INFO -
org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
Starting lifecycle supervisor 1
2013-01-09 04:11:15,874 (main) [INFO -
org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node
starting - a1
2013-01-09 04:11:15,876 (lifecycleSupervisor-1-0) [INFO -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:203)]
Node manager starting
2013-01-09 04:11:15,878 (lifecycleSupervisor-1-0) [INFO -
org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
Starting lifecycle supervisor 9
2013-01-09 04:11:15,877 (lifecycleSupervisor-1-2) [INFO -
org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]
Configuration provider starting
2013-01-09 04:11:15,880 (conf-file-poller-0) [INFO -
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)]
Reloading configuration file:conf/test.conf
2013-01-09 04:11:15,884 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
Processing:k1
2013-01-09 04:11:15,884 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
Processing:k1
2013-01-09 04:11:15,884 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
Processing:k1
2013-01-09 04:11:15,884 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:912)]
Added sinks: k1 k2 Agent: a1
2013-01-09 04:11:15,884 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
Processing:k2
2013-01-09 04:11:15,885 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
Processing:k1
2013-01-09 04:11:15,885 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
Processing:k1
2013-01-09 04:11:15,885 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
Processing:k2
2013-01-09 04:11:15,905 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)]
Post-validation flume configuration contains configuration for
agents: [a1]
2013-01-09 04:11:15,905 (conf-file-poller-0) [INFO -
org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)]
Creating channels
2013-01-09 04:11:15,920 (conf-file-poller-0) [INFO -
org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)]
created channel c1
2013-01-09 04:11:15,921 (conf-file-poller-0) [INFO -
org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)]
created channel c2
2013-01-09 04:11:15,941 (conf-file-poller-0) [INFO -
org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)]
Creating instance of sink: k2, type: logger
2013-01-09 04:11:15,943 (conf-file-poller-0) [INFO -
org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)]
Creating instance of sink: k1, type:
org.apache.flume.sink.hbase.HBaseSink
2013-01-09 04:11:16,046 (conf-file-poller-0) [INFO -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:106)]
Starting new configuration:{
sourceRunners:{r1=EventDrivenSourceRunner: { source:Avro source r1: {
bindAddress: 0.0.0.0, port: 8080 } }} sinkRunners:{k1=SinkRunner: {
policy:org.apache.flume.sink.DefaultSinkProcessor@6baa9f99
counterGroup:{ name:null counters:{} } }, k2=SinkRunner: {
policy:org.apache.flume.sink.DefaultSinkProcessor@ed42d08
counterGroup:{ name:null counters:{} } }}
channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1},
c2=org.apache.flume.channel.MemoryChannel{name: c2}} }
2013-01-09 04:11:16,046 (conf-file-poller-0) [INFO -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:113)]
Starting Channel c1
2013-01-09 04:11:16,046 (conf-file-poller-0) [INFO -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:113)]
Starting Channel c2
2013-01-09 04:11:16,081 (lifecycleSupervisor-1-0) [INFO -
org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:89)]
Monitoried counter group for type: CHANNEL, name: c1, registered
successfully.
2013-01-09 04:11:16,081 (lifecycleSupervisor-1-1) [INFO -
org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:89)]
Monitoried counter group for type: CHANNEL, name: c2, registered
successfully.
2013-01-09 04:11:16,082 (lifecycleSupervisor-1-1) [INFO -
org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:73)]
Component type: CHANNEL, name: c2 started
2013-01-09 04:11:16,082 (lifecycleSupervisor-1-0) [INFO -
org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:73)]
Component type: CHANNEL, name: c1 started
2013-01-09 04:11:16,082 (conf-file-poller-0) [INFO -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:141)]
Starting Sink k1
2013-01-09 04:11:16,083 (conf-file-poller-0) [INFO -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:141)]
Starting Sink k2
2013-01-09 04:11:16,083 (conf-file-poller-0) [INFO -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:152)]
Starting Source r1
2013-01-09 04:11:16,084 (lifecycleSupervisor-1-2) [INFO -
org.apache.flume.source.AvroSource.start(AvroSource.java:142)]
Starting Avro source r1: { bindAddress: 0.0.0.0, port: 8080 }...
2013-01-09 04:11:16,526 (lifecycleSupervisor-1-2) [INFO -
org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:89)]
Monitoried counter group for type: SOURCE, name: r1, registered
successfully.
2013-01-09 04:11:16,527 (lifecycleSupervisor-1-2) [INFO -
org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:73)]
Component type: SOURCE, name: r1 started
2013-01-09 04:11:16,535 (lifecycleSupervisor-1-2) [INFO -
org.apache.flume.source.AvroSource.start(AvroSource.java:168)] Avro
source r1 started.
2013-01-09 04:11:16,633 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52
GMT
2013-01-09 04:11:16,634 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:host.name=flume-agent1.dfw2
2013-01-09 04:11:16,634 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.version=1.6.0_37
2013-01-09 04:11:16,634 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.vendor=Sun Microsystems Inc.
2013-01-09 04:11:16,634 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.home=/usr/lib/jvm/jdk1.6.0_37/jre
2013-01-09 04:11:16,634 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.class.path=/usr/local/lib/flume/conf:/usr/local/lib/flume/lib/zookeeper-3.4.5.jar:/usr/local/lib/flume/lib/flume-recoverable-memory-channel-1.3.1.jar:/usr/local/lib/flume/lib/flume-scribe-source-1.3.1.jar:/usr/local/lib/flume/lib/irclib-1.10.jar:/usr/local/lib/flume/lib/flume-file-channel-1.3.1.jar:/usr/local/lib/flume/lib/libthrift-0.6.1.jar:/usr/local/lib/flume/lib/flume-ng-elasticsearch-sink-1.3.1.jar:/usr/local/lib/flume/lib/jetty-util-6.1.26.jar:/usr/local/lib/flume/lib/derby-10.8.2.2.jar:/usr/local/lib/flume/lib/jackson-core-asl-1.9.3.jar:/usr/local/lib/flume/lib/jetty-6.1.26.jar:/usr/local/lib/flume/lib/flume-irc-sink-1.3.1.jar:/usr/local/lib/flume/lib/netty-3.4.0.Final.jar:/usr/local/lib/flume/lib/slf4j-api-1.6.1.jar:/usr/local/lib/flume/lib/flume-ng-sdk-1.3.1.jar:/usr/local/lib/flume/lib/flume-jdbc-channel-1.3.1.jar:/usr/local/lib/flume/lib/joda-time-2.1.jar:/usr/local/lib/flume/lib/flume-ng-core-1.3.1.jar:/usr/local/lib/flume/lib/async-1.3.1.jar:/usr/local/lib/flume/lib/flume-ng-log4jappender-1.3.1.jar:/usr/local/lib/flume/lib/flume-ng-node-1.3.1.jar:/usr/local/lib/flume/lib/asynchbase-1.2.0.jar:/usr/local/lib/flume/lib/log4j-1.2.16.jar:/usr/local/lib/flume/lib/flume-ng-hbase-sink-1.3.1.jar:/usr/local/lib/flume/lib/slf4j-log4j12-1.6.1.jar:/usr/local/lib/flume/lib/commons-lang-2.5.jar:/usr/local/lib/flume/lib/velocity-1.7.jar:/usr/local/lib/flume/lib/guava-10.0.1.jar:/usr/local/lib/flume/lib/servlet-api-2.5-20110124.jar:/usr/local/lib/flume/lib/commons-dbcp-1.4.jar:/usr/local/lib/flume/lib/commons-collections-3.2.1.jar:/usr/local/lib/flume/lib/flume-thrift-source-1.3.1.jar:/usr/local/lib/flume/lib/paranamer-2.3.jar:/usr/local/lib/flume/lib/gson-2.2.2.jar:/usr/local/lib/flume/lib/commons-pool-1.5.4.jar:/usr/local/lib/flume/lib/jcl-over-slf4j-1.7.2.jar:/usr/local/lib/flume/lib/commons-cli-1.2.jar:/usr/local/lib/flume/lib/avro-1.7.2.jar:/usr/local/lib/flume/lib/jackson-mapper-asl-1.9.3.jar:/usr/local/lib/flume/lib/protobuf-java-2.4.1.jar:/usr/local/lib/flume/lib/flume-hdfs-sink-1.3.1.jar:/usr/local/lib/flume/lib/mina-core-2.0.4.jar:/usr/local/lib/flume/lib/snappy-java-1.0.4.1.jar:/usr/local/lib/flume/lib/flume-ng-configuration-1.3.1.jar:/usr/local/lib/flume/lib/commons-io-2.1.jar:/usr/local/lib/flume/lib/flume-avro-source-1.3.1.jar:/usr/local/lib/flume/lib/jsr305-1.3.9.jar:/usr/local/lib/flume/lib/avro-ipc-1.7.2.jar:/usr/local/hadoop/libexec/../conf:/usr/lib/jvm/default-java/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.4.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/hbase/conf:/usr/lib/tools.jar:/usr/local/hbase:/usr/local/hbase/hbase-0.94.3.jar:/usr/local/hbase/hbase-0.94.3-tests.jar:/usr/local/hbase/lib/activation-1.1.jar:/usr/local/hbase/lib/asm-3.1.jar:/usr/local/hbase/lib/avro-1.5.3.jar:/usr/local/hbase/lib/avro-ipc-1.5.3.jar:/usr/local/hbase/lib/commons-beanutils-1.7.0.jar:/usr/local/hbase/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hbase/lib/commons-cli-1.2.jar:/usr/local/hbase/lib/commons-codec-1.4.jar:/usr/local/hbase/lib/commons-collections-3.2.1.jar:/usr/local/hbase/lib/commons-configuration-1.6.jar:/usr/local/hbase/lib/commons-digester-1.8.jar:/usr/local/hbase/lib/commons-el-1.0.jar:/usr/local/hbase/lib/commons-httpclient-3.1.jar:/usr/local/hbase/lib/commons-io-2.1.jar:/usr/local/hbase/lib/commons-lang-2.5.jar:/usr/local/hbase/lib/commons-logging-1.1.1.jar:/usr/local/hbase/lib/commons-math-2.1.jar:/usr/local/hbase/lib/commons-net-1.4.1.jar:/usr/local/hbase/lib/core-3.1.1.jar:/usr/local/hbase/lib/guava-11.0.2.jar:/usr/local/hbase/lib/hadoop-core-1.0.4.jar:/usr/local/hbase/lib/high-scale-lib-1.1.1.jar:/usr/local/hbase/lib/httpclient-4.1.2.jar:/usr/local/hbase/lib/httpcore-4.1.3.jar:/usr/local/hbase/lib/jackson-core-asl-1.8.8.jar:/usr/local/hbase/lib/jackson-jaxrs-1.8.8.jar:/usr/local/hbase/lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hbase/lib/jackson-xc-1.8.8.jar:/usr/local/hbase/lib/jamon-runtime-2.3.1.jar:/usr/local/hbase/lib/jasper-compiler-5.5.23.jar:/usr/local/hbase/lib/jasper-runtime-5.5.23.jar:/usr/local/hbase/lib/jaxb-api-2.1.jar:/usr/local/hbase/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hbase/lib/jersey-core-1.8.jar:/usr/local/hbase/lib/jersey-json-1.8.jar:/usr/local/hbase/lib/jersey-server-1.8.jar:/usr/local/hbase/lib/jettison-1.1.jar:/usr/local/hbase/lib/jetty-6.1.26.jar:/usr/local/hbase/lib/jetty-util-6.1.26.jar:/usr/local/hbase/lib/jruby-complete-1.6.5.jar:/usr/local/hbase/lib/jsp-2.1-6.1.14.jar:/usr/local/hbase/lib/jsp-api-2.1-6.1.14.jar:/usr/local/hbase/lib/jsr305-1.3.9.jar:/usr/local/hbase/lib/junit-4.10-HBASE-1.jar:/usr/local/hbase/lib/libthrift-0.8.0.jar:/usr/local/hbase/lib/log4j-1.2.16.jar:/usr/local/hbase/lib/metrics-core-2.1.2.jar:/usr/local/hbase/lib/netty-3.2.4.Final.jar:/usr/local/hbase/lib/protobuf-java-2.4.0a.jar:/usr/local/hbase/lib/servlet-api-2.5-6.1.14.jar:/usr/local/hbase/lib/snappy-java-1.0.3.2.jar:/usr/local/hbase/lib/stax-api-1.0.1.jar:/usr/local/hbase/lib/velocity-1.7.jar:/usr/local/hbase/lib/xmlenc-0.52.jar:/usr/local/hbase/lib/zookeeper-3.4.3.jar:/usr/local/hadoop/libexec/../conf:/usr/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.4.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/hbase/conf
2013-01-09 04:11:16,635 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.library.path=:/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64:/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64:/usr/local/hbase/lib/native/Linux-amd64-64
2013-01-09 04:11:16,635 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.io.tmpdir=/tmp
2013-01-09 04:11:16,635 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.compiler=<NA>
2013-01-09 04:11:16,635 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:os.name=Linux
2013-01-09 04:11:16,636 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:os.arch=amd64
2013-01-09 04:11:16,636 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:os.version=3.2.0-34-virtual
2013-01-09 04:11:16,636 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:user.name=root
2013-01-09 04:11:16,636 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:user.home=/root
2013-01-09 04:11:16,636 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:user.dir=/usr/local/lib/apache-flume-1.3.1-bin
2013-01-09 04:11:16,637 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:438)] Initiating
client connection,
connectString=flume-zk5:2181,flume-zk1:2181,flume-zk2:2181,flume-zk3:2181,flume-zk4:2181
sessionTimeout=60000 watcher=hconnection
2013-01-09 04:11:16,660 (lifecycleSupervisor-1-1) [INFO -
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:98)]
The identifier of this process is 28030@flume-agent1
2013-01-09 04:11:16,676
(lifecycleSupervisor-1-1-SendThread(flume-zk1.dfw2:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
Opening socket connection to server flume-zk1.dfw2/10.1.1.251:2181.
Will not attempt to authenticate using SASL (Unable to locate a login
configuration)
2013-01-09 04:11:16,681
(lifecycleSupervisor-1-1-SendThread(flume-zk1.dfw2:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
Socket connection established to flume-zk1.dfw2/10.1.1.251:2181,
initiating session
2013-01-09 04:11:16,689
(lifecycleSupervisor-1-1-SendThread(flume-zk1.dfw2:2181)) [WARN -
org.apache.zookeeper.ClientCnxnSocket.readConnectResult(ClientCnxnSocket.java:139)]
Connected to an old server; r-o mode will be unavailable
2013-01-09 04:11:16,689
(lifecycleSupervisor-1-1-SendThread(flume-zk1.dfw2:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
Session establishment complete on server
flume-zk1.dfw2/10.1.1.251:2181, sessionid = 0x33c065f29a10013,
negotiated timeout = 40000
2013-01-09 04:11:54,073 (pool-4-thread-1) [INFO -
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)]
[id: 0x29a7789c, /135.207.171.228:60711 => /10.1.1.233:8080] OPEN
2013-01-09 04:11:54,077 (pool-5-thread-1) [INFO -
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)]
[id: 0x29a7789c, /135.207.171.228:60711 => /10.1.1.233:8080] BOUND:
/10.1.1.233:8080
2013-01-09 04:11:54,077 (pool-5-thread-1) [INFO -
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)]
[id: 0x29a7789c, /135.207.171.228:60711 => /10.1.1.233:8080]
CONNECTED: /135.207.171.228:60711
2013-01-09 04:11:54,126
(SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: {
headers:{region=flp, timestamp=1357722708000, dataType=syslog,
Severity=5, host=switch1, Facility=20} body: 43 6C 69 3A 20 25 53 59
53 2D 35 2D 43 4F 4E 46 Cli: %SYS-5-CONF }
2013-01-09 04:11:54,126
(SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: {
headers:{region=flp, timestamp=1357722710000, dataType=syslog,
Severity=5, host=switch1, Facility=20} body: 43 6C 69 3A 20 25 53 59
53 2D 35 2D 43 4F 4E 46 Cli: %SYS-5-CONF }
Re: weird hbase sink behavior
Posted by "Xu (Simon) Chen" <xc...@gmail.com>.
Silly me... it works now, thanks a lot!
On Wed, Jan 9, 2013 at 12:10 AM, Brock Noland <br...@cloudera.com> wrote:
> incrementColumn column can be skipped. It's either payloadColumn or
> incrementColumn.
>
> On Tue, Jan 8, 2013 at 10:50 PM, Xu (Simon) Chen <xc...@gmail.com> wrote:
>> a1.sinks.k1.payloadColumn = test
>
> I think that should be:
>
> a1.sinks.k1.serializer.payloadColumn = test
>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: weird hbase sink behavior
Posted by Brock Noland <br...@cloudera.com>.
incrementColumn column can be skipped. It's either payloadColumn or
incrementColumn.
On Tue, Jan 8, 2013 at 10:50 PM, Xu (Simon) Chen <xc...@gmail.com> wrote:
> a1.sinks.k1.payloadColumn = test
I think that should be:
a1.sinks.k1.serializer.payloadColumn = test
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: weird hbase sink behavior
Posted by "Xu (Simon) Chen" <xc...@gmail.com>.
Tried this new config, still doesn't work...
root@flume-agent1:/usr/local/lib/flume# cat conf/test.conf
a1.channels = c1 c2
a1.sinks = k1 k2
a1.sources = r1
a1.sources.r1.type = avro
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 8080
a1.sinks.k1.type = org.apache.flume.sink.hbase.HBaseSink
a1.sinks.k1.table = foo_table
a1.sinks.k1.columnFamily = bar_cf
a1.sinks.k1.payloadColumn = test
a1.sinks.k1.incrementColumn = count
a1.sinks.k1.serializer = org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
a1.sinks.k2.type = logger
#a1.channels.c1.type = file
#a1.channels.c1.checkpointDir = /mnt/flume/checkpoint
#a1.channels.c1.dataDirs = /mnt/flume/data
a1.channels.c1.type = memory
a1.channels.c2.type = memory
a1.sources.r1.channels = c1 c2
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c2
On Tue, Jan 8, 2013 at 11:43 PM, Xu (Simon) Chen <xc...@gmail.com> wrote:
> Yeah, now I remembered, I was using Regex serializer...
>
> The comment in the code is actually confusing:
> https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/SimpleHbaseEventSerializer.java
>
> It gives me the impression that incrementColumn can be skipped...
>
> On Tue, Jan 8, 2013 at 11:39 PM, Brock Noland <br...@cloudera.com> wrote:
>> payloadColumn or incrementColumn must be specified. This should give
>> you a better message and the open JIRA is
>> https://issues.apache.org/jira/browse/FLUME-1757
>>
>> Brock
>>
>> On Tue, Jan 8, 2013 at 10:19 PM, Xu (Simon) Chen <xc...@gmail.com> wrote:
>>> Hi folks,
>>>
>>> I am using flume to sink into hbase. I had the set up working
>>> yesterday, but when I restarted my flume agent, it for some reason
>>> cannot dump to hbase anymore (too bad I don't have a version-control
>>> for my config yet.)
>>>
>>> Here is the configuration and execution output. The agent essentially
>>> listens for objects via avro and dump to console for debugging and to
>>> hbase. The last portion of the console output shows that an object is
>>> indeed received, but for some reason "scan" on the hbase doesn't show
>>> anything new. I tried Regex/Simple serializer and HBase and
>>> AsyncHbase, none works now...
>>>
>>> From the log I don't see anything obviously wrong. I did a tcpdump on
>>> the agent, and saw it established a couple of connections to the
>>> regionservers, but closed them a few seconds later.
>>>
>>> Any info on how to correctly configure this or troubleshoot would be
>>> appreciated.
>>>
>>> Thanks!
>>> -Simon
>>>
>>>
>>>
>>> root@flume-agent1:/usr/local/lib/flume# cat conf/test.conf
>>> a1.channels = c1 c2
>>> a1.sinks = k1 k2
>>> a1.sources = r1
>>>
>>> a1.sources.r1.type = avro
>>> a1.sources.r1.bind = 0.0.0.0
>>> a1.sources.r1.port = 8080
>>>
>>> a1.sinks.k1.type = org.apache.flume.sink.hbase.HBaseSink
>>> a1.sinks.k1.table = foo_table
>>> a1.sinks.k1.columnFamily = bar_cf
>>> a1.sinks.k1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
>>>
>>> a1.sinks.k2.type = logger
>>>
>>> #a1.channels.c1.type = file
>>> #a1.channels.c1.checkpointDir = /mnt/flume/checkpoint
>>> #a1.channels.c1.dataDirs = /mnt/flume/data
>>> a1.channels.c1.type = memory
>>> a1.channels.c2.type = memory
>>>
>>> a1.sources.r1.channels = c1 c2
>>> a1.sinks.k1.channel = c1
>>> a1.sinks.k2.channel = c2
>>>
>>> root@flume-agent1:/usr/local/lib/flume# bin/flume-ng agent --name a1
>>> --conf conf --conf-file conf/test.conf
>>> -Dflume.root.logger=INFO,console
>>> Info: Including Hadoop libraries found via
>>> (/usr/local/hadoop/bin/hadoop) for HDFS access
>>> Warning: $HADOOP_HOME is deprecated.
>>>
>>> Warning: $HADOOP_HOME is deprecated.
>>>
>>> Info: Excluding /usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar
>>> from classpath
>>> Info: Excluding
>>> /usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
>>> classpath
>>> Info: Including HBASE libraries found via (/usr/local/hbase/bin/hbase)
>>> for HBASE access
>>> Info: Excluding /usr/local/hbase/lib/slf4j-api-1.4.3.jar from classpath
>>> Info: Excluding /usr/local/hbase/lib/slf4j-log4j12-1.4.3.jar from classpath
>>> Info: Excluding /usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar
>>> from classpath
>>> Info: Excluding
>>> /usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
>>> classpath
>>> + exec /usr/lib/jvm/default-java/bin/java -Xmx20m
>>> -Dflume.root.logger=INFO,console -cp
>>> '/usr/local/lib/flume/conf:/usr/local/lib/flume/lib/*:/usr/local/hadoop/libexec/../conf:/usr/lib/jvm/default-java/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.4.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/hbase/conf:/usr/lib/tools.jar:/usr/local/hbase:/usr/local/hbase/hbase-0.94.3.jar:/usr/local/hbase/hbase-0.94.3-tests.jar:/usr/local/hbase/lib/activation-1.1.jar:/usr/local/hbase/lib/asm-3.1.jar:/usr/local/hbase/lib/avro-1.5.3.jar:/usr/local/hbase/lib/avro-ipc-1.5.3.jar:/usr/local/hbase/lib/commons-beanutils-1.7.0.jar:/usr/local/hbase/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hbase/lib/commons-cli-1.2.jar:/usr/local/hbase/lib/commons-codec-1.4.jar:/usr/local/hbase/lib/commons-collections-3.2.1.jar:/usr/local/hbase/lib/commons-configuration-1.6.jar:/usr/local/hbase/lib/commons-digester-1.8.jar:/usr/local/hbase/lib/commons-el-1.0.jar:/usr/local/hbase/lib/commons-httpclient-3.1.jar:/usr/local/hbase/lib/commons-io-2.1.jar:/usr/local/hbase/lib/commons-lang-2.5.jar:/usr/local/hbase/lib/commons-logging-1.1.1.jar:/usr/local/hbase/lib/commons-math-2.1.jar:/usr/local/hbase/lib/commons-net-1.4.1.jar:/usr/local/hbase/lib/core-3.1.1.jar:/usr/local/hbase/lib/guava-11.0.2.jar:/usr/local/hbase/lib/hadoop-core-1.0.4.jar:/usr/local/hbase/lib/high-scale-lib-1.1.1.jar:/usr/local/hbase/lib/httpclient-4.1.2.jar:/usr/local/hbase/lib/httpcore-4.1.3.jar:/usr/local/hbase/lib/jackson-core-asl-1.8.8.jar:/usr/local/hbase/lib/jackson-jaxrs-1.8.8.jar:/usr/local/hbase/lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hbase/lib/jackson-xc-1.8.8.jar:/usr/local/hbase/lib/jamon-runtime-2.3.1.jar:/usr/local/hbase/lib/jasper-compiler-5.5.23.jar:/usr/local/hbase/lib/jasper-runtime-5.5.23.jar:/usr/local/hbase/lib/jaxb-api-2.1.jar:/usr/local/hbase/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hbase/lib/jersey-core-1.8.jar:/usr/local/hbase/lib/jersey-json-1.8.jar:/usr/local/hbase/lib/jersey-server-1.8.jar:/usr/local/hbase/lib/jettison-1.1.jar:/usr/local/hbase/lib/jetty-6.1.26.jar:/usr/local/hbase/lib/jetty-util-6.1.26.jar:/usr/local/hbase/lib/jruby-complete-1.6.5.jar:/usr/local/hbase/lib/jsp-2.1-6.1.14.jar:/usr/local/hbase/lib/jsp-api-2.1-6.1.14.jar:/usr/local/hbase/lib/jsr305-1.3.9.jar:/usr/local/hbase/lib/junit-4.10-HBASE-1.jar:/usr/local/hbase/lib/libthrift-0.8.0.jar:/usr/local/hbase/lib/log4j-1.2.16.jar:/usr/local/hbase/lib/metrics-core-2.1.2.jar:/usr/local/hbase/lib/netty-3.2.4.Final.jar:/usr/local/hbase/lib/protobuf-java-2.4.0a.jar:/usr/local/hbase/lib/servlet-api-2.5-6.1.14.jar:/usr/local/hbase/lib/snappy-java-1.0.3.2.jar:/usr/local/hbase/lib/stax-api-1.0.1.jar:/usr/local/hbase/lib/velocity-1.7.jar:/usr/local/hbase/lib/xmlenc-0.52.jar:/usr/local/hbase/lib/zookeeper-3.4.3.jar:/usr/local/hadoop/libexec/../conf:/usr/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.4.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/hbase/conf'
>>> -Djava.library.path=:/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64:/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64:/usr/local/hbase/lib/native/Linux-amd64-64
>>> org.apache.flume.node.Application --name a1 --conf-file conf/test.conf
>>> 2013-01-09 04:11:15,871 (main) [INFO -
>>> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
>>> Starting lifecycle supervisor 1
>>> 2013-01-09 04:11:15,874 (main) [INFO -
>>> org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node
>>> starting - a1
>>> 2013-01-09 04:11:15,876 (lifecycleSupervisor-1-0) [INFO -
>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:203)]
>>> Node manager starting
>>> 2013-01-09 04:11:15,878 (lifecycleSupervisor-1-0) [INFO -
>>> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
>>> Starting lifecycle supervisor 9
>>> 2013-01-09 04:11:15,877 (lifecycleSupervisor-1-2) [INFO -
>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]
>>> Configuration provider starting
>>> 2013-01-09 04:11:15,880 (conf-file-poller-0) [INFO -
>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)]
>>> Reloading configuration file:conf/test.conf
>>> 2013-01-09 04:11:15,884 (conf-file-poller-0) [INFO -
>>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
>>> Processing:k1
>>> 2013-01-09 04:11:15,884 (conf-file-poller-0) [INFO -
>>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
>>> Processing:k1
>>> 2013-01-09 04:11:15,884 (conf-file-poller-0) [INFO -
>>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
>>> Processing:k1
>>> 2013-01-09 04:11:15,884 (conf-file-poller-0) [INFO -
>>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:912)]
>>> Added sinks: k1 k2 Agent: a1
>>> 2013-01-09 04:11:15,884 (conf-file-poller-0) [INFO -
>>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
>>> Processing:k2
>>> 2013-01-09 04:11:15,885 (conf-file-poller-0) [INFO -
>>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
>>> Processing:k1
>>> 2013-01-09 04:11:15,885 (conf-file-poller-0) [INFO -
>>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
>>> Processing:k1
>>> 2013-01-09 04:11:15,885 (conf-file-poller-0) [INFO -
>>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
>>> Processing:k2
>>> 2013-01-09 04:11:15,905 (conf-file-poller-0) [INFO -
>>> org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)]
>>> Post-validation flume configuration contains configuration for
>>> agents: [a1]
>>> 2013-01-09 04:11:15,905 (conf-file-poller-0) [INFO -
>>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)]
>>> Creating channels
>>> 2013-01-09 04:11:15,920 (conf-file-poller-0) [INFO -
>>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)]
>>> created channel c1
>>> 2013-01-09 04:11:15,921 (conf-file-poller-0) [INFO -
>>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)]
>>> created channel c2
>>> 2013-01-09 04:11:15,941 (conf-file-poller-0) [INFO -
>>> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)]
>>> Creating instance of sink: k2, type: logger
>>> 2013-01-09 04:11:15,943 (conf-file-poller-0) [INFO -
>>> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)]
>>> Creating instance of sink: k1, type:
>>> org.apache.flume.sink.hbase.HBaseSink
>>> 2013-01-09 04:11:16,046 (conf-file-poller-0) [INFO -
>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:106)]
>>> Starting new configuration:{
>>> sourceRunners:{r1=EventDrivenSourceRunner: { source:Avro source r1: {
>>> bindAddress: 0.0.0.0, port: 8080 } }} sinkRunners:{k1=SinkRunner: {
>>> policy:org.apache.flume.sink.DefaultSinkProcessor@6baa9f99
>>> counterGroup:{ name:null counters:{} } }, k2=SinkRunner: {
>>> policy:org.apache.flume.sink.DefaultSinkProcessor@ed42d08
>>> counterGroup:{ name:null counters:{} } }}
>>> channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1},
>>> c2=org.apache.flume.channel.MemoryChannel{name: c2}} }
>>> 2013-01-09 04:11:16,046 (conf-file-poller-0) [INFO -
>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:113)]
>>> Starting Channel c1
>>> 2013-01-09 04:11:16,046 (conf-file-poller-0) [INFO -
>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:113)]
>>> Starting Channel c2
>>> 2013-01-09 04:11:16,081 (lifecycleSupervisor-1-0) [INFO -
>>> org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:89)]
>>> Monitoried counter group for type: CHANNEL, name: c1, registered
>>> successfully.
>>> 2013-01-09 04:11:16,081 (lifecycleSupervisor-1-1) [INFO -
>>> org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:89)]
>>> Monitoried counter group for type: CHANNEL, name: c2, registered
>>> successfully.
>>> 2013-01-09 04:11:16,082 (lifecycleSupervisor-1-1) [INFO -
>>> org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:73)]
>>> Component type: CHANNEL, name: c2 started
>>> 2013-01-09 04:11:16,082 (lifecycleSupervisor-1-0) [INFO -
>>> org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:73)]
>>> Component type: CHANNEL, name: c1 started
>>> 2013-01-09 04:11:16,082 (conf-file-poller-0) [INFO -
>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:141)]
>>> Starting Sink k1
>>> 2013-01-09 04:11:16,083 (conf-file-poller-0) [INFO -
>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:141)]
>>> Starting Sink k2
>>> 2013-01-09 04:11:16,083 (conf-file-poller-0) [INFO -
>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:152)]
>>> Starting Source r1
>>> 2013-01-09 04:11:16,084 (lifecycleSupervisor-1-2) [INFO -
>>> org.apache.flume.source.AvroSource.start(AvroSource.java:142)]
>>> Starting Avro source r1: { bindAddress: 0.0.0.0, port: 8080 }...
>>> 2013-01-09 04:11:16,526 (lifecycleSupervisor-1-2) [INFO -
>>> org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:89)]
>>> Monitoried counter group for type: SOURCE, name: r1, registered
>>> successfully.
>>> 2013-01-09 04:11:16,527 (lifecycleSupervisor-1-2) [INFO -
>>> org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:73)]
>>> Component type: SOURCE, name: r1 started
>>> 2013-01-09 04:11:16,535 (lifecycleSupervisor-1-2) [INFO -
>>> org.apache.flume.source.AvroSource.start(AvroSource.java:168)] Avro
>>> source r1 started.
>>> 2013-01-09 04:11:16,633 (lifecycleSupervisor-1-1) [INFO -
>>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>>> environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52
>>> GMT
>>> 2013-01-09 04:11:16,634 (lifecycleSupervisor-1-1) [INFO -
>>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>>> environment:host.name=flume-agent1.dfw2
>>> 2013-01-09 04:11:16,634 (lifecycleSupervisor-1-1) [INFO -
>>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>>> environment:java.version=1.6.0_37
>>> 2013-01-09 04:11:16,634 (lifecycleSupervisor-1-1) [INFO -
>>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>>> environment:java.vendor=Sun Microsystems Inc.
>>> 2013-01-09 04:11:16,634 (lifecycleSupervisor-1-1) [INFO -
>>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>>> environment:java.home=/usr/lib/jvm/jdk1.6.0_37/jre
>>> 2013-01-09 04:11:16,634 (lifecycleSupervisor-1-1) [INFO -
>>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>>> environment:java.class.path=/usr/local/lib/flume/conf:/usr/local/lib/flume/lib/zookeeper-3.4.5.jar:/usr/local/lib/flume/lib/flume-recoverable-memory-channel-1.3.1.jar:/usr/local/lib/flume/lib/flume-scribe-source-1.3.1.jar:/usr/local/lib/flume/lib/irclib-1.10.jar:/usr/local/lib/flume/lib/flume-file-channel-1.3.1.jar:/usr/local/lib/flume/lib/libthrift-0.6.1.jar:/usr/local/lib/flume/lib/flume-ng-elasticsearch-sink-1.3.1.jar:/usr/local/lib/flume/lib/jetty-util-6.1.26.jar:/usr/local/lib/flume/lib/derby-10.8.2.2.jar:/usr/local/lib/flume/lib/jackson-core-asl-1.9.3.jar:/usr/local/lib/flume/lib/jetty-6.1.26.jar:/usr/local/lib/flume/lib/flume-irc-sink-1.3.1.jar:/usr/local/lib/flume/lib/netty-3.4.0.Final.jar:/usr/local/lib/flume/lib/slf4j-api-1.6.1.jar:/usr/local/lib/flume/lib/flume-ng-sdk-1.3.1.jar:/usr/local/lib/flume/lib/flume-jdbc-channel-1.3.1.jar:/usr/local/lib/flume/lib/joda-time-2.1.jar:/usr/local/lib/flume/lib/flume-ng-core-1.3.1.jar:/usr/local/lib/flume/lib/async-1.3.1.jar:/usr/local/lib/flume/lib/flume-ng-log4jappender-1.3.1.jar:/usr/local/lib/flume/lib/flume-ng-node-1.3.1.jar:/usr/local/lib/flume/lib/asynchbase-1.2.0.jar:/usr/local/lib/flume/lib/log4j-1.2.16.jar:/usr/local/lib/flume/lib/flume-ng-hbase-sink-1.3.1.jar:/usr/local/lib/flume/lib/slf4j-log4j12-1.6.1.jar:/usr/local/lib/flume/lib/commons-lang-2.5.jar:/usr/local/lib/flume/lib/velocity-1.7.jar:/usr/local/lib/flume/lib/guava-10.0.1.jar:/usr/local/lib/flume/lib/servlet-api-2.5-20110124.jar:/usr/local/lib/flume/lib/commons-dbcp-1.4.jar:/usr/local/lib/flume/lib/commons-collections-3.2.1.jar:/usr/local/lib/flume/lib/flume-thrift-source-1.3.1.jar:/usr/local/lib/flume/lib/paranamer-2.3.jar:/usr/local/lib/flume/lib/gson-2.2.2.jar:/usr/local/lib/flume/lib/commons-pool-1.5.4.jar:/usr/local/lib/flume/lib/jcl-over-slf4j-1.7.2.jar:/usr/local/lib/flume/lib/commons-cli-1.2.jar:/usr/local/lib/flume/lib/avro-1.7.2.jar:/usr/local/lib/flume/lib/jackson-mapper-asl-1.9.3.jar:/usr/local/lib/flume/lib/protobuf-java-2.4.1.jar:/usr/local/lib/flume/lib/flume-hdfs-sink-1.3.1.jar:/usr/local/lib/flume/lib/mina-core-2.0.4.jar:/usr/local/lib/flume/lib/snappy-java-1.0.4.1.jar:/usr/local/lib/flume/lib/flume-ng-configuration-1.3.1.jar:/usr/local/lib/flume/lib/commons-io-2.1.jar:/usr/local/lib/flume/lib/flume-avro-source-1.3.1.jar:/usr/local/lib/flume/lib/jsr305-1.3.9.jar:/usr/local/lib/flume/lib/avro-ipc-1.7.2.jar:/usr/local/hadoop/libexec/../conf:/usr/lib/jvm/default-java/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.4.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/hbase/conf:/usr/lib/tools.jar:/usr/local/hbase:/usr/local/hbase/hbase-0.94.3.jar:/usr/local/hbase/hbase-0.94.3-tests.jar:/usr/local/hbase/lib/activation-1.1.jar:/usr/local/hbase/lib/asm-3.1.jar:/usr/local/hbase/lib/avro-1.5.3.jar:/usr/local/hbase/lib/avro-ipc-1.5.3.jar:/usr/local/hbase/lib/commons-beanutils-1.7.0.jar:/usr/local/hbase/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hbase/lib/commons-cli-1.2.jar:/usr/local/hbase/lib/commons-codec-1.4.jar:/usr/local/hbase/lib/commons-collections-3.2.1.jar:/usr/local/hbase/lib/commons-configuration-1.6.jar:/usr/local/hbase/lib/commons-digester-1.8.jar:/usr/local/hbase/lib/commons-el-1.0.jar:/usr/local/hbase/lib/commons-httpclient-3.1.jar:/usr/local/hbase/lib/commons-io-2.1.jar:/usr/local/hbase/lib/commons-lang-2.5.jar:/usr/local/hbase/lib/commons-logging-1.1.1.jar:/usr/local/hbase/lib/commons-math-2.1.jar:/usr/local/hbase/lib/commons-net-1.4.1.jar:/usr/local/hbase/lib/core-3.1.1.jar:/usr/local/hbase/lib/guava-11.0.2.jar:/usr/local/hbase/lib/hadoop-core-1.0.4.jar:/usr/local/hbase/lib/high-scale-lib-1.1.1.jar:/usr/local/hbase/lib/httpclient-4.1.2.jar:/usr/local/hbase/lib/httpcore-4.1.3.jar:/usr/local/hbase/lib/jackson-core-asl-1.8.8.jar:/usr/local/hbase/lib/jackson-jaxrs-1.8.8.jar:/usr/local/hbase/lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hbase/lib/jackson-xc-1.8.8.jar:/usr/local/hbase/lib/jamon-runtime-2.3.1.jar:/usr/local/hbase/lib/jasper-compiler-5.5.23.jar:/usr/local/hbase/lib/jasper-runtime-5.5.23.jar:/usr/local/hbase/lib/jaxb-api-2.1.jar:/usr/local/hbase/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hbase/lib/jersey-core-1.8.jar:/usr/local/hbase/lib/jersey-json-1.8.jar:/usr/local/hbase/lib/jersey-server-1.8.jar:/usr/local/hbase/lib/jettison-1.1.jar:/usr/local/hbase/lib/jetty-6.1.26.jar:/usr/local/hbase/lib/jetty-util-6.1.26.jar:/usr/local/hbase/lib/jruby-complete-1.6.5.jar:/usr/local/hbase/lib/jsp-2.1-6.1.14.jar:/usr/local/hbase/lib/jsp-api-2.1-6.1.14.jar:/usr/local/hbase/lib/jsr305-1.3.9.jar:/usr/local/hbase/lib/junit-4.10-HBASE-1.jar:/usr/local/hbase/lib/libthrift-0.8.0.jar:/usr/local/hbase/lib/log4j-1.2.16.jar:/usr/local/hbase/lib/metrics-core-2.1.2.jar:/usr/local/hbase/lib/netty-3.2.4.Final.jar:/usr/local/hbase/lib/protobuf-java-2.4.0a.jar:/usr/local/hbase/lib/servlet-api-2.5-6.1.14.jar:/usr/local/hbase/lib/snappy-java-1.0.3.2.jar:/usr/local/hbase/lib/stax-api-1.0.1.jar:/usr/local/hbase/lib/velocity-1.7.jar:/usr/local/hbase/lib/xmlenc-0.52.jar:/usr/local/hbase/lib/zookeeper-3.4.3.jar:/usr/local/hadoop/libexec/../conf:/usr/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.4.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/hbase/conf
>>> 2013-01-09 04:11:16,635 (lifecycleSupervisor-1-1) [INFO -
>>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>>> environment:java.library.path=:/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64:/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64:/usr/local/hbase/lib/native/Linux-amd64-64
>>> 2013-01-09 04:11:16,635 (lifecycleSupervisor-1-1) [INFO -
>>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>>> environment:java.io.tmpdir=/tmp
>>> 2013-01-09 04:11:16,635 (lifecycleSupervisor-1-1) [INFO -
>>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>>> environment:java.compiler=<NA>
>>> 2013-01-09 04:11:16,635 (lifecycleSupervisor-1-1) [INFO -
>>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>>> environment:os.name=Linux
>>> 2013-01-09 04:11:16,636 (lifecycleSupervisor-1-1) [INFO -
>>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>>> environment:os.arch=amd64
>>> 2013-01-09 04:11:16,636 (lifecycleSupervisor-1-1) [INFO -
>>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>>> environment:os.version=3.2.0-34-virtual
>>> 2013-01-09 04:11:16,636 (lifecycleSupervisor-1-1) [INFO -
>>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>>> environment:user.name=root
>>> 2013-01-09 04:11:16,636 (lifecycleSupervisor-1-1) [INFO -
>>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>>> environment:user.home=/root
>>> 2013-01-09 04:11:16,636 (lifecycleSupervisor-1-1) [INFO -
>>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>>> environment:user.dir=/usr/local/lib/apache-flume-1.3.1-bin
>>> 2013-01-09 04:11:16,637 (lifecycleSupervisor-1-1) [INFO -
>>> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:438)] Initiating
>>> client connection,
>>> connectString=flume-zk5:2181,flume-zk1:2181,flume-zk2:2181,flume-zk3:2181,flume-zk4:2181
>>> sessionTimeout=60000 watcher=hconnection
>>> 2013-01-09 04:11:16,660 (lifecycleSupervisor-1-1) [INFO -
>>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:98)]
>>> The identifier of this process is 28030@flume-agent1
>>> 2013-01-09 04:11:16,676
>>> (lifecycleSupervisor-1-1-SendThread(flume-zk1.dfw2:2181)) [INFO -
>>> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
>>> Opening socket connection to server flume-zk1.dfw2/10.1.1.251:2181.
>>> Will not attempt to authenticate using SASL (Unable to locate a login
>>> configuration)
>>> 2013-01-09 04:11:16,681
>>> (lifecycleSupervisor-1-1-SendThread(flume-zk1.dfw2:2181)) [INFO -
>>> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
>>> Socket connection established to flume-zk1.dfw2/10.1.1.251:2181,
>>> initiating session
>>> 2013-01-09 04:11:16,689
>>> (lifecycleSupervisor-1-1-SendThread(flume-zk1.dfw2:2181)) [WARN -
>>> org.apache.zookeeper.ClientCnxnSocket.readConnectResult(ClientCnxnSocket.java:139)]
>>> Connected to an old server; r-o mode will be unavailable
>>> 2013-01-09 04:11:16,689
>>> (lifecycleSupervisor-1-1-SendThread(flume-zk1.dfw2:2181)) [INFO -
>>> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
>>> Session establishment complete on server
>>> flume-zk1.dfw2/10.1.1.251:2181, sessionid = 0x33c065f29a10013,
>>> negotiated timeout = 40000
>>>
>>>
>>> 2013-01-09 04:11:54,073 (pool-4-thread-1) [INFO -
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)]
>>> [id: 0x29a7789c, /135.207.171.228:60711 => /10.1.1.233:8080] OPEN
>>> 2013-01-09 04:11:54,077 (pool-5-thread-1) [INFO -
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)]
>>> [id: 0x29a7789c, /135.207.171.228:60711 => /10.1.1.233:8080] BOUND:
>>> /10.1.1.233:8080
>>> 2013-01-09 04:11:54,077 (pool-5-thread-1) [INFO -
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)]
>>> [id: 0x29a7789c, /135.207.171.228:60711 => /10.1.1.233:8080]
>>> CONNECTED: /135.207.171.228:60711
>>> 2013-01-09 04:11:54,126
>>> (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
>>> org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: {
>>> headers:{region=flp, timestamp=1357722708000, dataType=syslog,
>>> Severity=5, host=switch1, Facility=20} body: 43 6C 69 3A 20 25 53 59
>>> 53 2D 35 2D 43 4F 4E 46 Cli: %SYS-5-CONF }
>>> 2013-01-09 04:11:54,126
>>> (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
>>> org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: {
>>> headers:{region=flp, timestamp=1357722710000, dataType=syslog,
>>> Severity=5, host=switch1, Facility=20} body: 43 6C 69 3A 20 25 53 59
>>> 53 2D 35 2D 43 4F 4E 46 Cli: %SYS-5-CONF }
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: weird hbase sink behavior
Posted by "Xu (Simon) Chen" <xc...@gmail.com>.
Yeah, now I remembered, I was using Regex serializer...
The comment in the code is actually confusing:
https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/SimpleHbaseEventSerializer.java
It gives me the impression that incrementColumn can be skipped...
On Tue, Jan 8, 2013 at 11:39 PM, Brock Noland <br...@cloudera.com> wrote:
> payloadColumn or incrementColumn must be specified. This should give
> you a better message and the open JIRA is
> https://issues.apache.org/jira/browse/FLUME-1757
>
> Brock
>
> On Tue, Jan 8, 2013 at 10:19 PM, Xu (Simon) Chen <xc...@gmail.com> wrote:
>> Hi folks,
>>
>> I am using flume to sink into hbase. I had the set up working
>> yesterday, but when I restarted my flume agent, it for some reason
>> cannot dump to hbase anymore (too bad I don't have a version-control
>> for my config yet.)
>>
>> Here is the configuration and execution output. The agent essentially
>> listens for objects via avro and dump to console for debugging and to
>> hbase. The last portion of the console output shows that an object is
>> indeed received, but for some reason "scan" on the hbase doesn't show
>> anything new. I tried Regex/Simple serializer and HBase and
>> AsyncHbase, none works now...
>>
>> From the log I don't see anything obviously wrong. I did a tcpdump on
>> the agent, and saw it established a couple of connections to the
>> regionservers, but closed them a few seconds later.
>>
>> Any info on how to correctly configure this or troubleshoot would be
>> appreciated.
>>
>> Thanks!
>> -Simon
>>
>>
>>
>> root@flume-agent1:/usr/local/lib/flume# cat conf/test.conf
>> a1.channels = c1 c2
>> a1.sinks = k1 k2
>> a1.sources = r1
>>
>> a1.sources.r1.type = avro
>> a1.sources.r1.bind = 0.0.0.0
>> a1.sources.r1.port = 8080
>>
>> a1.sinks.k1.type = org.apache.flume.sink.hbase.HBaseSink
>> a1.sinks.k1.table = foo_table
>> a1.sinks.k1.columnFamily = bar_cf
>> a1.sinks.k1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
>>
>> a1.sinks.k2.type = logger
>>
>> #a1.channels.c1.type = file
>> #a1.channels.c1.checkpointDir = /mnt/flume/checkpoint
>> #a1.channels.c1.dataDirs = /mnt/flume/data
>> a1.channels.c1.type = memory
>> a1.channels.c2.type = memory
>>
>> a1.sources.r1.channels = c1 c2
>> a1.sinks.k1.channel = c1
>> a1.sinks.k2.channel = c2
>>
>> root@flume-agent1:/usr/local/lib/flume# bin/flume-ng agent --name a1
>> --conf conf --conf-file conf/test.conf
>> -Dflume.root.logger=INFO,console
>> Info: Including Hadoop libraries found via
>> (/usr/local/hadoop/bin/hadoop) for HDFS access
>> Warning: $HADOOP_HOME is deprecated.
>>
>> Warning: $HADOOP_HOME is deprecated.
>>
>> Info: Excluding /usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar
>> from classpath
>> Info: Excluding
>> /usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
>> classpath
>> Info: Including HBASE libraries found via (/usr/local/hbase/bin/hbase)
>> for HBASE access
>> Info: Excluding /usr/local/hbase/lib/slf4j-api-1.4.3.jar from classpath
>> Info: Excluding /usr/local/hbase/lib/slf4j-log4j12-1.4.3.jar from classpath
>> Info: Excluding /usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar
>> from classpath
>> Info: Excluding
>> /usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
>> classpath
>> + exec /usr/lib/jvm/default-java/bin/java -Xmx20m
>> -Dflume.root.logger=INFO,console -cp
>> '/usr/local/lib/flume/conf:/usr/local/lib/flume/lib/*:/usr/local/hadoop/libexec/../conf:/usr/lib/jvm/default-java/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.4.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/hbase/conf:/usr/lib/tools.jar:/usr/local/hbase:/usr/local/hbase/hbase-0.94.3.jar:/usr/local/hbase/hbase-0.94.3-tests.jar:/usr/local/hbase/lib/activation-1.1.jar:/usr/local/hbase/lib/asm-3.1.jar:/usr/local/hbase/lib/avro-1.5.3.jar:/usr/local/hbase/lib/avro-ipc-1.5.3.jar:/usr/local/hbase/lib/commons-beanutils-1.7.0.jar:/usr/local/hbase/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hbase/lib/commons-cli-1.2.jar:/usr/local/hbase/lib/commons-codec-1.4.jar:/usr/local/hbase/lib/commons-collections-3.2.1.jar:/usr/local/hbase/lib/commons-configuration-1.6.jar:/usr/local/hbase/lib/commons-digester-1.8.jar:/usr/local/hbase/lib/commons-el-1.0.jar:/usr/local/hbase/lib/commons-httpclient-3.1.jar:/usr/local/hbase/lib/commons-io-2.1.jar:/usr/local/hbase/lib/commons-lang-2.5.jar:/usr/local/hbase/lib/commons-logging-1.1.1.jar:/usr/local/hbase/lib/commons-math-2.1.jar:/usr/local/hbase/lib/commons-net-1.4.1.jar:/usr/local/hbase/lib/core-3.1.1.jar:/usr/local/hbase/lib/guava-11.0.2.jar:/usr/local/hbase/lib/hadoop-core-1.0.4.jar:/usr/local/hbase/lib/high-scale-lib-1.1.1.jar:/usr/local/hbase/lib/httpclient-4.1.2.jar:/usr/local/hbase/lib/httpcore-4.1.3.jar:/usr/local/hbase/lib/jackson-core-asl-1.8.8.jar:/usr/local/hbase/lib/jackson-jaxrs-1.8.8.jar:/usr/local/hbase/lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hbase/lib/jackson-xc-1.8.8.jar:/usr/local/hbase/lib/jamon-runtime-2.3.1.jar:/usr/local/hbase/lib/jasper-compiler-5.5.23.jar:/usr/local/hbase/lib/jasper-runtime-5.5.23.jar:/usr/local/hbase/lib/jaxb-api-2.1.jar:/usr/local/hbase/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hbase/lib/jersey-core-1.8.jar:/usr/local/hbase/lib/jersey-json-1.8.jar:/usr/local/hbase/lib/jersey-server-1.8.jar:/usr/local/hbase/lib/jettison-1.1.jar:/usr/local/hbase/lib/jetty-6.1.26.jar:/usr/local/hbase/lib/jetty-util-6.1.26.jar:/usr/local/hbase/lib/jruby-complete-1.6.5.jar:/usr/local/hbase/lib/jsp-2.1-6.1.14.jar:/usr/local/hbase/lib/jsp-api-2.1-6.1.14.jar:/usr/local/hbase/lib/jsr305-1.3.9.jar:/usr/local/hbase/lib/junit-4.10-HBASE-1.jar:/usr/local/hbase/lib/libthrift-0.8.0.jar:/usr/local/hbase/lib/log4j-1.2.16.jar:/usr/local/hbase/lib/metrics-core-2.1.2.jar:/usr/local/hbase/lib/netty-3.2.4.Final.jar:/usr/local/hbase/lib/protobuf-java-2.4.0a.jar:/usr/local/hbase/lib/servlet-api-2.5-6.1.14.jar:/usr/local/hbase/lib/snappy-java-1.0.3.2.jar:/usr/local/hbase/lib/stax-api-1.0.1.jar:/usr/local/hbase/lib/velocity-1.7.jar:/usr/local/hbase/lib/xmlenc-0.52.jar:/usr/local/hbase/lib/zookeeper-3.4.3.jar:/usr/local/hadoop/libexec/../conf:/usr/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.4.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/hbase/conf'
>> -Djava.library.path=:/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64:/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64:/usr/local/hbase/lib/native/Linux-amd64-64
>> org.apache.flume.node.Application --name a1 --conf-file conf/test.conf
>> 2013-01-09 04:11:15,871 (main) [INFO -
>> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
>> Starting lifecycle supervisor 1
>> 2013-01-09 04:11:15,874 (main) [INFO -
>> org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node
>> starting - a1
>> 2013-01-09 04:11:15,876 (lifecycleSupervisor-1-0) [INFO -
>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:203)]
>> Node manager starting
>> 2013-01-09 04:11:15,878 (lifecycleSupervisor-1-0) [INFO -
>> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
>> Starting lifecycle supervisor 9
>> 2013-01-09 04:11:15,877 (lifecycleSupervisor-1-2) [INFO -
>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]
>> Configuration provider starting
>> 2013-01-09 04:11:15,880 (conf-file-poller-0) [INFO -
>> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)]
>> Reloading configuration file:conf/test.conf
>> 2013-01-09 04:11:15,884 (conf-file-poller-0) [INFO -
>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
>> Processing:k1
>> 2013-01-09 04:11:15,884 (conf-file-poller-0) [INFO -
>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
>> Processing:k1
>> 2013-01-09 04:11:15,884 (conf-file-poller-0) [INFO -
>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
>> Processing:k1
>> 2013-01-09 04:11:15,884 (conf-file-poller-0) [INFO -
>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:912)]
>> Added sinks: k1 k2 Agent: a1
>> 2013-01-09 04:11:15,884 (conf-file-poller-0) [INFO -
>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
>> Processing:k2
>> 2013-01-09 04:11:15,885 (conf-file-poller-0) [INFO -
>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
>> Processing:k1
>> 2013-01-09 04:11:15,885 (conf-file-poller-0) [INFO -
>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
>> Processing:k1
>> 2013-01-09 04:11:15,885 (conf-file-poller-0) [INFO -
>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
>> Processing:k2
>> 2013-01-09 04:11:15,905 (conf-file-poller-0) [INFO -
>> org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)]
>> Post-validation flume configuration contains configuration for
>> agents: [a1]
>> 2013-01-09 04:11:15,905 (conf-file-poller-0) [INFO -
>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)]
>> Creating channels
>> 2013-01-09 04:11:15,920 (conf-file-poller-0) [INFO -
>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)]
>> created channel c1
>> 2013-01-09 04:11:15,921 (conf-file-poller-0) [INFO -
>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)]
>> created channel c2
>> 2013-01-09 04:11:15,941 (conf-file-poller-0) [INFO -
>> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)]
>> Creating instance of sink: k2, type: logger
>> 2013-01-09 04:11:15,943 (conf-file-poller-0) [INFO -
>> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)]
>> Creating instance of sink: k1, type:
>> org.apache.flume.sink.hbase.HBaseSink
>> 2013-01-09 04:11:16,046 (conf-file-poller-0) [INFO -
>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:106)]
>> Starting new configuration:{
>> sourceRunners:{r1=EventDrivenSourceRunner: { source:Avro source r1: {
>> bindAddress: 0.0.0.0, port: 8080 } }} sinkRunners:{k1=SinkRunner: {
>> policy:org.apache.flume.sink.DefaultSinkProcessor@6baa9f99
>> counterGroup:{ name:null counters:{} } }, k2=SinkRunner: {
>> policy:org.apache.flume.sink.DefaultSinkProcessor@ed42d08
>> counterGroup:{ name:null counters:{} } }}
>> channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1},
>> c2=org.apache.flume.channel.MemoryChannel{name: c2}} }
>> 2013-01-09 04:11:16,046 (conf-file-poller-0) [INFO -
>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:113)]
>> Starting Channel c1
>> 2013-01-09 04:11:16,046 (conf-file-poller-0) [INFO -
>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:113)]
>> Starting Channel c2
>> 2013-01-09 04:11:16,081 (lifecycleSupervisor-1-0) [INFO -
>> org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:89)]
>> Monitoried counter group for type: CHANNEL, name: c1, registered
>> successfully.
>> 2013-01-09 04:11:16,081 (lifecycleSupervisor-1-1) [INFO -
>> org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:89)]
>> Monitoried counter group for type: CHANNEL, name: c2, registered
>> successfully.
>> 2013-01-09 04:11:16,082 (lifecycleSupervisor-1-1) [INFO -
>> org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:73)]
>> Component type: CHANNEL, name: c2 started
>> 2013-01-09 04:11:16,082 (lifecycleSupervisor-1-0) [INFO -
>> org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:73)]
>> Component type: CHANNEL, name: c1 started
>> 2013-01-09 04:11:16,082 (conf-file-poller-0) [INFO -
>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:141)]
>> Starting Sink k1
>> 2013-01-09 04:11:16,083 (conf-file-poller-0) [INFO -
>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:141)]
>> Starting Sink k2
>> 2013-01-09 04:11:16,083 (conf-file-poller-0) [INFO -
>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:152)]
>> Starting Source r1
>> 2013-01-09 04:11:16,084 (lifecycleSupervisor-1-2) [INFO -
>> org.apache.flume.source.AvroSource.start(AvroSource.java:142)]
>> Starting Avro source r1: { bindAddress: 0.0.0.0, port: 8080 }...
>> 2013-01-09 04:11:16,526 (lifecycleSupervisor-1-2) [INFO -
>> org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:89)]
>> Monitoried counter group for type: SOURCE, name: r1, registered
>> successfully.
>> 2013-01-09 04:11:16,527 (lifecycleSupervisor-1-2) [INFO -
>> org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:73)]
>> Component type: SOURCE, name: r1 started
>> 2013-01-09 04:11:16,535 (lifecycleSupervisor-1-2) [INFO -
>> org.apache.flume.source.AvroSource.start(AvroSource.java:168)] Avro
>> source r1 started.
>> 2013-01-09 04:11:16,633 (lifecycleSupervisor-1-1) [INFO -
>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>> environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52
>> GMT
>> 2013-01-09 04:11:16,634 (lifecycleSupervisor-1-1) [INFO -
>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>> environment:host.name=flume-agent1.dfw2
>> 2013-01-09 04:11:16,634 (lifecycleSupervisor-1-1) [INFO -
>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>> environment:java.version=1.6.0_37
>> 2013-01-09 04:11:16,634 (lifecycleSupervisor-1-1) [INFO -
>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>> environment:java.vendor=Sun Microsystems Inc.
>> 2013-01-09 04:11:16,634 (lifecycleSupervisor-1-1) [INFO -
>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>> environment:java.home=/usr/lib/jvm/jdk1.6.0_37/jre
>> 2013-01-09 04:11:16,634 (lifecycleSupervisor-1-1) [INFO -
>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>> environment:java.class.path=/usr/local/lib/flume/conf:/usr/local/lib/flume/lib/zookeeper-3.4.5.jar:/usr/local/lib/flume/lib/flume-recoverable-memory-channel-1.3.1.jar:/usr/local/lib/flume/lib/flume-scribe-source-1.3.1.jar:/usr/local/lib/flume/lib/irclib-1.10.jar:/usr/local/lib/flume/lib/flume-file-channel-1.3.1.jar:/usr/local/lib/flume/lib/libthrift-0.6.1.jar:/usr/local/lib/flume/lib/flume-ng-elasticsearch-sink-1.3.1.jar:/usr/local/lib/flume/lib/jetty-util-6.1.26.jar:/usr/local/lib/flume/lib/derby-10.8.2.2.jar:/usr/local/lib/flume/lib/jackson-core-asl-1.9.3.jar:/usr/local/lib/flume/lib/jetty-6.1.26.jar:/usr/local/lib/flume/lib/flume-irc-sink-1.3.1.jar:/usr/local/lib/flume/lib/netty-3.4.0.Final.jar:/usr/local/lib/flume/lib/slf4j-api-1.6.1.jar:/usr/local/lib/flume/lib/flume-ng-sdk-1.3.1.jar:/usr/local/lib/flume/lib/flume-jdbc-channel-1.3.1.jar:/usr/local/lib/flume/lib/joda-time-2.1.jar:/usr/local/lib/flume/lib/flume-ng-core-1.3.1.jar:/usr/local/lib/flume/lib/async-1.3.1.jar:/usr/local/lib/flume/lib/flume-ng-log4jappender-1.3.1.jar:/usr/local/lib/flume/lib/flume-ng-node-1.3.1.jar:/usr/local/lib/flume/lib/asynchbase-1.2.0.jar:/usr/local/lib/flume/lib/log4j-1.2.16.jar:/usr/local/lib/flume/lib/flume-ng-hbase-sink-1.3.1.jar:/usr/local/lib/flume/lib/slf4j-log4j12-1.6.1.jar:/usr/local/lib/flume/lib/commons-lang-2.5.jar:/usr/local/lib/flume/lib/velocity-1.7.jar:/usr/local/lib/flume/lib/guava-10.0.1.jar:/usr/local/lib/flume/lib/servlet-api-2.5-20110124.jar:/usr/local/lib/flume/lib/commons-dbcp-1.4.jar:/usr/local/lib/flume/lib/commons-collections-3.2.1.jar:/usr/local/lib/flume/lib/flume-thrift-source-1.3.1.jar:/usr/local/lib/flume/lib/paranamer-2.3.jar:/usr/local/lib/flume/lib/gson-2.2.2.jar:/usr/local/lib/flume/lib/commons-pool-1.5.4.jar:/usr/local/lib/flume/lib/jcl-over-slf4j-1.7.2.jar:/usr/local/lib/flume/lib/commons-cli-1.2.jar:/usr/local/lib/flume/lib/avro-1.7.2.jar:/usr/local/lib/flume/lib/jackson-mapper-asl-1.9.3.jar:/usr/local/lib/flume/lib/protobuf-java-2.4.1.jar:/usr/local/lib/flume/lib/flume-hdfs-sink-1.3.1.jar:/usr/local/lib/flume/lib/mina-core-2.0.4.jar:/usr/local/lib/flume/lib/snappy-java-1.0.4.1.jar:/usr/local/lib/flume/lib/flume-ng-configuration-1.3.1.jar:/usr/local/lib/flume/lib/commons-io-2.1.jar:/usr/local/lib/flume/lib/flume-avro-source-1.3.1.jar:/usr/local/lib/flume/lib/jsr305-1.3.9.jar:/usr/local/lib/flume/lib/avro-ipc-1.7.2.jar:/usr/local/hadoop/libexec/../conf:/usr/lib/jvm/default-java/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.4.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/hbase/conf:/usr/lib/tools.jar:/usr/local/hbase:/usr/local/hbase/hbase-0.94.3.jar:/usr/local/hbase/hbase-0.94.3-tests.jar:/usr/local/hbase/lib/activation-1.1.jar:/usr/local/hbase/lib/asm-3.1.jar:/usr/local/hbase/lib/avro-1.5.3.jar:/usr/local/hbase/lib/avro-ipc-1.5.3.jar:/usr/local/hbase/lib/commons-beanutils-1.7.0.jar:/usr/local/hbase/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hbase/lib/commons-cli-1.2.jar:/usr/local/hbase/lib/commons-codec-1.4.jar:/usr/local/hbase/lib/commons-collections-3.2.1.jar:/usr/local/hbase/lib/commons-configuration-1.6.jar:/usr/local/hbase/lib/commons-digester-1.8.jar:/usr/local/hbase/lib/commons-el-1.0.jar:/usr/local/hbase/lib/commons-httpclient-3.1.jar:/usr/local/hbase/lib/commons-io-2.1.jar:/usr/local/hbase/lib/commons-lang-2.5.jar:/usr/local/hbase/lib/commons-logging-1.1.1.jar:/usr/local/hbase/lib/commons-math-2.1.jar:/usr/local/hbase/lib/commons-net-1.4.1.jar:/usr/local/hbase/lib/core-3.1.1.jar:/usr/local/hbase/lib/guava-11.0.2.jar:/usr/local/hbase/lib/hadoop-core-1.0.4.jar:/usr/local/hbase/lib/high-scale-lib-1.1.1.jar:/usr/local/hbase/lib/httpclient-4.1.2.jar:/usr/local/hbase/lib/httpcore-4.1.3.jar:/usr/local/hbase/lib/jackson-core-asl-1.8.8.jar:/usr/local/hbase/lib/jackson-jaxrs-1.8.8.jar:/usr/local/hbase/lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hbase/lib/jackson-xc-1.8.8.jar:/usr/local/hbase/lib/jamon-runtime-2.3.1.jar:/usr/local/hbase/lib/jasper-compiler-5.5.23.jar:/usr/local/hbase/lib/jasper-runtime-5.5.23.jar:/usr/local/hbase/lib/jaxb-api-2.1.jar:/usr/local/hbase/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hbase/lib/jersey-core-1.8.jar:/usr/local/hbase/lib/jersey-json-1.8.jar:/usr/local/hbase/lib/jersey-server-1.8.jar:/usr/local/hbase/lib/jettison-1.1.jar:/usr/local/hbase/lib/jetty-6.1.26.jar:/usr/local/hbase/lib/jetty-util-6.1.26.jar:/usr/local/hbase/lib/jruby-complete-1.6.5.jar:/usr/local/hbase/lib/jsp-2.1-6.1.14.jar:/usr/local/hbase/lib/jsp-api-2.1-6.1.14.jar:/usr/local/hbase/lib/jsr305-1.3.9.jar:/usr/local/hbase/lib/junit-4.10-HBASE-1.jar:/usr/local/hbase/lib/libthrift-0.8.0.jar:/usr/local/hbase/lib/log4j-1.2.16.jar:/usr/local/hbase/lib/metrics-core-2.1.2.jar:/usr/local/hbase/lib/netty-3.2.4.Final.jar:/usr/local/hbase/lib/protobuf-java-2.4.0a.jar:/usr/local/hbase/lib/servlet-api-2.5-6.1.14.jar:/usr/local/hbase/lib/snappy-java-1.0.3.2.jar:/usr/local/hbase/lib/stax-api-1.0.1.jar:/usr/local/hbase/lib/velocity-1.7.jar:/usr/local/hbase/lib/xmlenc-0.52.jar:/usr/local/hbase/lib/zookeeper-3.4.3.jar:/usr/local/hadoop/libexec/../conf:/usr/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.4.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/hbase/conf
>> 2013-01-09 04:11:16,635 (lifecycleSupervisor-1-1) [INFO -
>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>> environment:java.library.path=:/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64:/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64:/usr/local/hbase/lib/native/Linux-amd64-64
>> 2013-01-09 04:11:16,635 (lifecycleSupervisor-1-1) [INFO -
>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>> environment:java.io.tmpdir=/tmp
>> 2013-01-09 04:11:16,635 (lifecycleSupervisor-1-1) [INFO -
>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>> environment:java.compiler=<NA>
>> 2013-01-09 04:11:16,635 (lifecycleSupervisor-1-1) [INFO -
>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>> environment:os.name=Linux
>> 2013-01-09 04:11:16,636 (lifecycleSupervisor-1-1) [INFO -
>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>> environment:os.arch=amd64
>> 2013-01-09 04:11:16,636 (lifecycleSupervisor-1-1) [INFO -
>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>> environment:os.version=3.2.0-34-virtual
>> 2013-01-09 04:11:16,636 (lifecycleSupervisor-1-1) [INFO -
>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>> environment:user.name=root
>> 2013-01-09 04:11:16,636 (lifecycleSupervisor-1-1) [INFO -
>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>> environment:user.home=/root
>> 2013-01-09 04:11:16,636 (lifecycleSupervisor-1-1) [INFO -
>> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
>> environment:user.dir=/usr/local/lib/apache-flume-1.3.1-bin
>> 2013-01-09 04:11:16,637 (lifecycleSupervisor-1-1) [INFO -
>> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:438)] Initiating
>> client connection,
>> connectString=flume-zk5:2181,flume-zk1:2181,flume-zk2:2181,flume-zk3:2181,flume-zk4:2181
>> sessionTimeout=60000 watcher=hconnection
>> 2013-01-09 04:11:16,660 (lifecycleSupervisor-1-1) [INFO -
>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:98)]
>> The identifier of this process is 28030@flume-agent1
>> 2013-01-09 04:11:16,676
>> (lifecycleSupervisor-1-1-SendThread(flume-zk1.dfw2:2181)) [INFO -
>> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
>> Opening socket connection to server flume-zk1.dfw2/10.1.1.251:2181.
>> Will not attempt to authenticate using SASL (Unable to locate a login
>> configuration)
>> 2013-01-09 04:11:16,681
>> (lifecycleSupervisor-1-1-SendThread(flume-zk1.dfw2:2181)) [INFO -
>> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
>> Socket connection established to flume-zk1.dfw2/10.1.1.251:2181,
>> initiating session
>> 2013-01-09 04:11:16,689
>> (lifecycleSupervisor-1-1-SendThread(flume-zk1.dfw2:2181)) [WARN -
>> org.apache.zookeeper.ClientCnxnSocket.readConnectResult(ClientCnxnSocket.java:139)]
>> Connected to an old server; r-o mode will be unavailable
>> 2013-01-09 04:11:16,689
>> (lifecycleSupervisor-1-1-SendThread(flume-zk1.dfw2:2181)) [INFO -
>> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
>> Session establishment complete on server
>> flume-zk1.dfw2/10.1.1.251:2181, sessionid = 0x33c065f29a10013,
>> negotiated timeout = 40000
>>
>>
>> 2013-01-09 04:11:54,073 (pool-4-thread-1) [INFO -
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)]
>> [id: 0x29a7789c, /135.207.171.228:60711 => /10.1.1.233:8080] OPEN
>> 2013-01-09 04:11:54,077 (pool-5-thread-1) [INFO -
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)]
>> [id: 0x29a7789c, /135.207.171.228:60711 => /10.1.1.233:8080] BOUND:
>> /10.1.1.233:8080
>> 2013-01-09 04:11:54,077 (pool-5-thread-1) [INFO -
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)]
>> [id: 0x29a7789c, /135.207.171.228:60711 => /10.1.1.233:8080]
>> CONNECTED: /135.207.171.228:60711
>> 2013-01-09 04:11:54,126
>> (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
>> org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: {
>> headers:{region=flp, timestamp=1357722708000, dataType=syslog,
>> Severity=5, host=switch1, Facility=20} body: 43 6C 69 3A 20 25 53 59
>> 53 2D 35 2D 43 4F 4E 46 Cli: %SYS-5-CONF }
>> 2013-01-09 04:11:54,126
>> (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
>> org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: {
>> headers:{region=flp, timestamp=1357722710000, dataType=syslog,
>> Severity=5, host=switch1, Facility=20} body: 43 6C 69 3A 20 25 53 59
>> 53 2D 35 2D 43 4F 4E 46 Cli: %SYS-5-CONF }
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: weird hbase sink behavior
Posted by Brock Noland <br...@cloudera.com>.
payloadColumn or incrementColumn must be specified. This should give
you a better message and the open JIRA is
https://issues.apache.org/jira/browse/FLUME-1757
Brock
On Tue, Jan 8, 2013 at 10:19 PM, Xu (Simon) Chen <xc...@gmail.com> wrote:
> Hi folks,
>
> I am using flume to sink into hbase. I had the set up working
> yesterday, but when I restarted my flume agent, it for some reason
> cannot dump to hbase anymore (too bad I don't have a version-control
> for my config yet.)
>
> Here is the configuration and execution output. The agent essentially
> listens for objects via avro and dump to console for debugging and to
> hbase. The last portion of the console output shows that an object is
> indeed received, but for some reason "scan" on the hbase doesn't show
> anything new. I tried Regex/Simple serializer and HBase and
> AsyncHbase, none works now...
>
> From the log I don't see anything obviously wrong. I did a tcpdump on
> the agent, and saw it established a couple of connections to the
> regionservers, but closed them a few seconds later.
>
> Any info on how to correctly configure this or troubleshoot would be
> appreciated.
>
> Thanks!
> -Simon
>
>
>
> root@flume-agent1:/usr/local/lib/flume# cat conf/test.conf
> a1.channels = c1 c2
> a1.sinks = k1 k2
> a1.sources = r1
>
> a1.sources.r1.type = avro
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.port = 8080
>
> a1.sinks.k1.type = org.apache.flume.sink.hbase.HBaseSink
> a1.sinks.k1.table = foo_table
> a1.sinks.k1.columnFamily = bar_cf
> a1.sinks.k1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
>
> a1.sinks.k2.type = logger
>
> #a1.channels.c1.type = file
> #a1.channels.c1.checkpointDir = /mnt/flume/checkpoint
> #a1.channels.c1.dataDirs = /mnt/flume/data
> a1.channels.c1.type = memory
> a1.channels.c2.type = memory
>
> a1.sources.r1.channels = c1 c2
> a1.sinks.k1.channel = c1
> a1.sinks.k2.channel = c2
>
> root@flume-agent1:/usr/local/lib/flume# bin/flume-ng agent --name a1
> --conf conf --conf-file conf/test.conf
> -Dflume.root.logger=INFO,console
> Info: Including Hadoop libraries found via
> (/usr/local/hadoop/bin/hadoop) for HDFS access
> Warning: $HADOOP_HOME is deprecated.
>
> Warning: $HADOOP_HOME is deprecated.
>
> Info: Excluding /usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar
> from classpath
> Info: Excluding
> /usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
> classpath
> Info: Including HBASE libraries found via (/usr/local/hbase/bin/hbase)
> for HBASE access
> Info: Excluding /usr/local/hbase/lib/slf4j-api-1.4.3.jar from classpath
> Info: Excluding /usr/local/hbase/lib/slf4j-log4j12-1.4.3.jar from classpath
> Info: Excluding /usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar
> from classpath
> Info: Excluding
> /usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
> classpath
> + exec /usr/lib/jvm/default-java/bin/java -Xmx20m
> -Dflume.root.logger=INFO,console -cp
> '/usr/local/lib/flume/conf:/usr/local/lib/flume/lib/*:/usr/local/hadoop/libexec/../conf:/usr/lib/jvm/default-java/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.4.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/hbase/conf:/usr/lib/tools.jar:/usr/local/hbase:/usr/local/hbase/hbase-0.94.3.jar:/usr/local/hbase/hbase-0.94.3-tests.jar:/usr/local/hbase/lib/activation-1.1.jar:/usr/local/hbase/lib/asm-3.1.jar:/usr/local/hbase/lib/avro-1.5.3.jar:/usr/local/hbase/lib/avro-ipc-1.5.3.jar:/usr/local/hbase/lib/commons-beanutils-1.7.0.jar:/usr/local/hbase/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hbase/lib/commons-cli-1.2.jar:/usr/local/hbase/lib/commons-codec-1.4.jar:/usr/local/hbase/lib/commons-collections-3.2.1.jar:/usr/local/hbase/lib/commons-configuration-1.6.jar:/usr/local/hbase/lib/commons-digester-1.8.jar:/usr/local/hbase/lib/commons-el-1.0.jar:/usr/local/hbase/lib/commons-httpclient-3.1.jar:/usr/local/hbase/lib/commons-io-2.1.jar:/usr/local/hbase/lib/commons-lang-2.5.jar:/usr/local/hbase/lib/commons-logging-1.1.1.jar:/usr/local/hbase/lib/commons-math-2.1.jar:/usr/local/hbase/lib/commons-net-1.4.1.jar:/usr/local/hbase/lib/core-3.1.1.jar:/usr/local/hbase/lib/guava-11.0.2.jar:/usr/local/hbase/lib/hadoop-core-1.0.4.jar:/usr/local/hbase/lib/high-scale-lib-1.1.1.jar:/usr/local/hbase/lib/httpclient-4.1.2.jar:/usr/local/hbase/lib/httpcore-4.1.3.jar:/usr/local/hbase/lib/jackson-core-asl-1.8.8.jar:/usr/local/hbase/lib/jackson-jaxrs-1.8.8.jar:/usr/local/hbase/lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hbase/lib/jackson-xc-1.8.8.jar:/usr/local/hbase/lib/jamon-runtime-2.3.1.jar:/usr/local/hbase/lib/jasper-compiler-5.5.23.jar:/usr/local/hbase/lib/jasper-runtime-5.5.23.jar:/usr/local/hbase/lib/jaxb-api-2.1.jar:/usr/local/hbase/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hbase/lib/jersey-core-1.8.jar:/usr/local/hbase/lib/jersey-json-1.8.jar:/usr/local/hbase/lib/jersey-server-1.8.jar:/usr/local/hbase/lib/jettison-1.1.jar:/usr/local/hbase/lib/jetty-6.1.26.jar:/usr/local/hbase/lib/jetty-util-6.1.26.jar:/usr/local/hbase/lib/jruby-complete-1.6.5.jar:/usr/local/hbase/lib/jsp-2.1-6.1.14.jar:/usr/local/hbase/lib/jsp-api-2.1-6.1.14.jar:/usr/local/hbase/lib/jsr305-1.3.9.jar:/usr/local/hbase/lib/junit-4.10-HBASE-1.jar:/usr/local/hbase/lib/libthrift-0.8.0.jar:/usr/local/hbase/lib/log4j-1.2.16.jar:/usr/local/hbase/lib/metrics-core-2.1.2.jar:/usr/local/hbase/lib/netty-3.2.4.Final.jar:/usr/local/hbase/lib/protobuf-java-2.4.0a.jar:/usr/local/hbase/lib/servlet-api-2.5-6.1.14.jar:/usr/local/hbase/lib/snappy-java-1.0.3.2.jar:/usr/local/hbase/lib/stax-api-1.0.1.jar:/usr/local/hbase/lib/velocity-1.7.jar:/usr/local/hbase/lib/xmlenc-0.52.jar:/usr/local/hbase/lib/zookeeper-3.4.3.jar:/usr/local/hadoop/libexec/../conf:/usr/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.4.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/hbase/conf'
> -Djava.library.path=:/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64:/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64:/usr/local/hbase/lib/native/Linux-amd64-64
> org.apache.flume.node.Application --name a1 --conf-file conf/test.conf
> 2013-01-09 04:11:15,871 (main) [INFO -
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
> Starting lifecycle supervisor 1
> 2013-01-09 04:11:15,874 (main) [INFO -
> org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node
> starting - a1
> 2013-01-09 04:11:15,876 (lifecycleSupervisor-1-0) [INFO -
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:203)]
> Node manager starting
> 2013-01-09 04:11:15,878 (lifecycleSupervisor-1-0) [INFO -
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
> Starting lifecycle supervisor 9
> 2013-01-09 04:11:15,877 (lifecycleSupervisor-1-2) [INFO -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]
> Configuration provider starting
> 2013-01-09 04:11:15,880 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)]
> Reloading configuration file:conf/test.conf
> 2013-01-09 04:11:15,884 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
> Processing:k1
> 2013-01-09 04:11:15,884 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
> Processing:k1
> 2013-01-09 04:11:15,884 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
> Processing:k1
> 2013-01-09 04:11:15,884 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:912)]
> Added sinks: k1 k2 Agent: a1
> 2013-01-09 04:11:15,884 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
> Processing:k2
> 2013-01-09 04:11:15,885 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
> Processing:k1
> 2013-01-09 04:11:15,885 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
> Processing:k1
> 2013-01-09 04:11:15,885 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)]
> Processing:k2
> 2013-01-09 04:11:15,905 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)]
> Post-validation flume configuration contains configuration for
> agents: [a1]
> 2013-01-09 04:11:15,905 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)]
> Creating channels
> 2013-01-09 04:11:15,920 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)]
> created channel c1
> 2013-01-09 04:11:15,921 (conf-file-poller-0) [INFO -
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)]
> created channel c2
> 2013-01-09 04:11:15,941 (conf-file-poller-0) [INFO -
> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)]
> Creating instance of sink: k2, type: logger
> 2013-01-09 04:11:15,943 (conf-file-poller-0) [INFO -
> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)]
> Creating instance of sink: k1, type:
> org.apache.flume.sink.hbase.HBaseSink
> 2013-01-09 04:11:16,046 (conf-file-poller-0) [INFO -
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:106)]
> Starting new configuration:{
> sourceRunners:{r1=EventDrivenSourceRunner: { source:Avro source r1: {
> bindAddress: 0.0.0.0, port: 8080 } }} sinkRunners:{k1=SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@6baa9f99
> counterGroup:{ name:null counters:{} } }, k2=SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@ed42d08
> counterGroup:{ name:null counters:{} } }}
> channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1},
> c2=org.apache.flume.channel.MemoryChannel{name: c2}} }
> 2013-01-09 04:11:16,046 (conf-file-poller-0) [INFO -
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:113)]
> Starting Channel c1
> 2013-01-09 04:11:16,046 (conf-file-poller-0) [INFO -
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:113)]
> Starting Channel c2
> 2013-01-09 04:11:16,081 (lifecycleSupervisor-1-0) [INFO -
> org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:89)]
> Monitoried counter group for type: CHANNEL, name: c1, registered
> successfully.
> 2013-01-09 04:11:16,081 (lifecycleSupervisor-1-1) [INFO -
> org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:89)]
> Monitoried counter group for type: CHANNEL, name: c2, registered
> successfully.
> 2013-01-09 04:11:16,082 (lifecycleSupervisor-1-1) [INFO -
> org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:73)]
> Component type: CHANNEL, name: c2 started
> 2013-01-09 04:11:16,082 (lifecycleSupervisor-1-0) [INFO -
> org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:73)]
> Component type: CHANNEL, name: c1 started
> 2013-01-09 04:11:16,082 (conf-file-poller-0) [INFO -
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:141)]
> Starting Sink k1
> 2013-01-09 04:11:16,083 (conf-file-poller-0) [INFO -
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:141)]
> Starting Sink k2
> 2013-01-09 04:11:16,083 (conf-file-poller-0) [INFO -
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:152)]
> Starting Source r1
> 2013-01-09 04:11:16,084 (lifecycleSupervisor-1-2) [INFO -
> org.apache.flume.source.AvroSource.start(AvroSource.java:142)]
> Starting Avro source r1: { bindAddress: 0.0.0.0, port: 8080 }...
> 2013-01-09 04:11:16,526 (lifecycleSupervisor-1-2) [INFO -
> org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:89)]
> Monitoried counter group for type: SOURCE, name: r1, registered
> successfully.
> 2013-01-09 04:11:16,527 (lifecycleSupervisor-1-2) [INFO -
> org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:73)]
> Component type: SOURCE, name: r1 started
> 2013-01-09 04:11:16,535 (lifecycleSupervisor-1-2) [INFO -
> org.apache.flume.source.AvroSource.start(AvroSource.java:168)] Avro
> source r1 started.
> 2013-01-09 04:11:16,633 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52
> GMT
> 2013-01-09 04:11:16,634 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:host.name=flume-agent1.dfw2
> 2013-01-09 04:11:16,634 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:java.version=1.6.0_37
> 2013-01-09 04:11:16,634 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:java.vendor=Sun Microsystems Inc.
> 2013-01-09 04:11:16,634 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:java.home=/usr/lib/jvm/jdk1.6.0_37/jre
> 2013-01-09 04:11:16,634 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:java.class.path=/usr/local/lib/flume/conf:/usr/local/lib/flume/lib/zookeeper-3.4.5.jar:/usr/local/lib/flume/lib/flume-recoverable-memory-channel-1.3.1.jar:/usr/local/lib/flume/lib/flume-scribe-source-1.3.1.jar:/usr/local/lib/flume/lib/irclib-1.10.jar:/usr/local/lib/flume/lib/flume-file-channel-1.3.1.jar:/usr/local/lib/flume/lib/libthrift-0.6.1.jar:/usr/local/lib/flume/lib/flume-ng-elasticsearch-sink-1.3.1.jar:/usr/local/lib/flume/lib/jetty-util-6.1.26.jar:/usr/local/lib/flume/lib/derby-10.8.2.2.jar:/usr/local/lib/flume/lib/jackson-core-asl-1.9.3.jar:/usr/local/lib/flume/lib/jetty-6.1.26.jar:/usr/local/lib/flume/lib/flume-irc-sink-1.3.1.jar:/usr/local/lib/flume/lib/netty-3.4.0.Final.jar:/usr/local/lib/flume/lib/slf4j-api-1.6.1.jar:/usr/local/lib/flume/lib/flume-ng-sdk-1.3.1.jar:/usr/local/lib/flume/lib/flume-jdbc-channel-1.3.1.jar:/usr/local/lib/flume/lib/joda-time-2.1.jar:/usr/local/lib/flume/lib/flume-ng-core-1.3.1.jar:/usr/local/lib/flume/lib/async-1.3.1.jar:/usr/local/lib/flume/lib/flume-ng-log4jappender-1.3.1.jar:/usr/local/lib/flume/lib/flume-ng-node-1.3.1.jar:/usr/local/lib/flume/lib/asynchbase-1.2.0.jar:/usr/local/lib/flume/lib/log4j-1.2.16.jar:/usr/local/lib/flume/lib/flume-ng-hbase-sink-1.3.1.jar:/usr/local/lib/flume/lib/slf4j-log4j12-1.6.1.jar:/usr/local/lib/flume/lib/commons-lang-2.5.jar:/usr/local/lib/flume/lib/velocity-1.7.jar:/usr/local/lib/flume/lib/guava-10.0.1.jar:/usr/local/lib/flume/lib/servlet-api-2.5-20110124.jar:/usr/local/lib/flume/lib/commons-dbcp-1.4.jar:/usr/local/lib/flume/lib/commons-collections-3.2.1.jar:/usr/local/lib/flume/lib/flume-thrift-source-1.3.1.jar:/usr/local/lib/flume/lib/paranamer-2.3.jar:/usr/local/lib/flume/lib/gson-2.2.2.jar:/usr/local/lib/flume/lib/commons-pool-1.5.4.jar:/usr/local/lib/flume/lib/jcl-over-slf4j-1.7.2.jar:/usr/local/lib/flume/lib/commons-cli-1.2.jar:/usr/local/lib/flume/lib/avro-1.7.2.jar:/usr/local/lib/flume/lib/jackson-mapper-asl-1.9.3.jar:/usr/local/lib/flume/lib/protobuf-java-2.4.1.jar:/usr/local/lib/flume/lib/flume-hdfs-sink-1.3.1.jar:/usr/local/lib/flume/lib/mina-core-2.0.4.jar:/usr/local/lib/flume/lib/snappy-java-1.0.4.1.jar:/usr/local/lib/flume/lib/flume-ng-configuration-1.3.1.jar:/usr/local/lib/flume/lib/commons-io-2.1.jar:/usr/local/lib/flume/lib/flume-avro-source-1.3.1.jar:/usr/local/lib/flume/lib/jsr305-1.3.9.jar:/usr/local/lib/flume/lib/avro-ipc-1.7.2.jar:/usr/local/hadoop/libexec/../conf:/usr/lib/jvm/default-java/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.4.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/hbase/conf:/usr/lib/tools.jar:/usr/local/hbase:/usr/local/hbase/hbase-0.94.3.jar:/usr/local/hbase/hbase-0.94.3-tests.jar:/usr/local/hbase/lib/activation-1.1.jar:/usr/local/hbase/lib/asm-3.1.jar:/usr/local/hbase/lib/avro-1.5.3.jar:/usr/local/hbase/lib/avro-ipc-1.5.3.jar:/usr/local/hbase/lib/commons-beanutils-1.7.0.jar:/usr/local/hbase/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hbase/lib/commons-cli-1.2.jar:/usr/local/hbase/lib/commons-codec-1.4.jar:/usr/local/hbase/lib/commons-collections-3.2.1.jar:/usr/local/hbase/lib/commons-configuration-1.6.jar:/usr/local/hbase/lib/commons-digester-1.8.jar:/usr/local/hbase/lib/commons-el-1.0.jar:/usr/local/hbase/lib/commons-httpclient-3.1.jar:/usr/local/hbase/lib/commons-io-2.1.jar:/usr/local/hbase/lib/commons-lang-2.5.jar:/usr/local/hbase/lib/commons-logging-1.1.1.jar:/usr/local/hbase/lib/commons-math-2.1.jar:/usr/local/hbase/lib/commons-net-1.4.1.jar:/usr/local/hbase/lib/core-3.1.1.jar:/usr/local/hbase/lib/guava-11.0.2.jar:/usr/local/hbase/lib/hadoop-core-1.0.4.jar:/usr/local/hbase/lib/high-scale-lib-1.1.1.jar:/usr/local/hbase/lib/httpclient-4.1.2.jar:/usr/local/hbase/lib/httpcore-4.1.3.jar:/usr/local/hbase/lib/jackson-core-asl-1.8.8.jar:/usr/local/hbase/lib/jackson-jaxrs-1.8.8.jar:/usr/local/hbase/lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hbase/lib/jackson-xc-1.8.8.jar:/usr/local/hbase/lib/jamon-runtime-2.3.1.jar:/usr/local/hbase/lib/jasper-compiler-5.5.23.jar:/usr/local/hbase/lib/jasper-runtime-5.5.23.jar:/usr/local/hbase/lib/jaxb-api-2.1.jar:/usr/local/hbase/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hbase/lib/jersey-core-1.8.jar:/usr/local/hbase/lib/jersey-json-1.8.jar:/usr/local/hbase/lib/jersey-server-1.8.jar:/usr/local/hbase/lib/jettison-1.1.jar:/usr/local/hbase/lib/jetty-6.1.26.jar:/usr/local/hbase/lib/jetty-util-6.1.26.jar:/usr/local/hbase/lib/jruby-complete-1.6.5.jar:/usr/local/hbase/lib/jsp-2.1-6.1.14.jar:/usr/local/hbase/lib/jsp-api-2.1-6.1.14.jar:/usr/local/hbase/lib/jsr305-1.3.9.jar:/usr/local/hbase/lib/junit-4.10-HBASE-1.jar:/usr/local/hbase/lib/libthrift-0.8.0.jar:/usr/local/hbase/lib/log4j-1.2.16.jar:/usr/local/hbase/lib/metrics-core-2.1.2.jar:/usr/local/hbase/lib/netty-3.2.4.Final.jar:/usr/local/hbase/lib/protobuf-java-2.4.0a.jar:/usr/local/hbase/lib/servlet-api-2.5-6.1.14.jar:/usr/local/hbase/lib/snappy-java-1.0.3.2.jar:/usr/local/hbase/lib/stax-api-1.0.1.jar:/usr/local/hbase/lib/velocity-1.7.jar:/usr/local/hbase/lib/xmlenc-0.52.jar:/usr/local/hbase/lib/zookeeper-3.4.3.jar:/usr/local/hadoop/libexec/../conf:/usr/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.4.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/hbase/conf
> 2013-01-09 04:11:16,635 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:java.library.path=:/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64:/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64:/usr/local/hbase/lib/native/Linux-amd64-64
> 2013-01-09 04:11:16,635 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:java.io.tmpdir=/tmp
> 2013-01-09 04:11:16,635 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:java.compiler=<NA>
> 2013-01-09 04:11:16,635 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:os.name=Linux
> 2013-01-09 04:11:16,636 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:os.arch=amd64
> 2013-01-09 04:11:16,636 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:os.version=3.2.0-34-virtual
> 2013-01-09 04:11:16,636 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:user.name=root
> 2013-01-09 04:11:16,636 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:user.home=/root
> 2013-01-09 04:11:16,636 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:user.dir=/usr/local/lib/apache-flume-1.3.1-bin
> 2013-01-09 04:11:16,637 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:438)] Initiating
> client connection,
> connectString=flume-zk5:2181,flume-zk1:2181,flume-zk2:2181,flume-zk3:2181,flume-zk4:2181
> sessionTimeout=60000 watcher=hconnection
> 2013-01-09 04:11:16,660 (lifecycleSupervisor-1-1) [INFO -
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:98)]
> The identifier of this process is 28030@flume-agent1
> 2013-01-09 04:11:16,676
> (lifecycleSupervisor-1-1-SendThread(flume-zk1.dfw2:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> Opening socket connection to server flume-zk1.dfw2/10.1.1.251:2181.
> Will not attempt to authenticate using SASL (Unable to locate a login
> configuration)
> 2013-01-09 04:11:16,681
> (lifecycleSupervisor-1-1-SendThread(flume-zk1.dfw2:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> Socket connection established to flume-zk1.dfw2/10.1.1.251:2181,
> initiating session
> 2013-01-09 04:11:16,689
> (lifecycleSupervisor-1-1-SendThread(flume-zk1.dfw2:2181)) [WARN -
> org.apache.zookeeper.ClientCnxnSocket.readConnectResult(ClientCnxnSocket.java:139)]
> Connected to an old server; r-o mode will be unavailable
> 2013-01-09 04:11:16,689
> (lifecycleSupervisor-1-1-SendThread(flume-zk1.dfw2:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> Session establishment complete on server
> flume-zk1.dfw2/10.1.1.251:2181, sessionid = 0x33c065f29a10013,
> negotiated timeout = 40000
>
>
> 2013-01-09 04:11:54,073 (pool-4-thread-1) [INFO -
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)]
> [id: 0x29a7789c, /135.207.171.228:60711 => /10.1.1.233:8080] OPEN
> 2013-01-09 04:11:54,077 (pool-5-thread-1) [INFO -
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)]
> [id: 0x29a7789c, /135.207.171.228:60711 => /10.1.1.233:8080] BOUND:
> /10.1.1.233:8080
> 2013-01-09 04:11:54,077 (pool-5-thread-1) [INFO -
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)]
> [id: 0x29a7789c, /135.207.171.228:60711 => /10.1.1.233:8080]
> CONNECTED: /135.207.171.228:60711
> 2013-01-09 04:11:54,126
> (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
> org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: {
> headers:{region=flp, timestamp=1357722708000, dataType=syslog,
> Severity=5, host=switch1, Facility=20} body: 43 6C 69 3A 20 25 53 59
> 53 2D 35 2D 43 4F 4E 46 Cli: %SYS-5-CONF }
> 2013-01-09 04:11:54,126
> (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
> org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: {
> headers:{region=flp, timestamp=1357722710000, dataType=syslog,
> Severity=5, host=switch1, Facility=20} body: 43 6C 69 3A 20 25 53 59
> 53 2D 35 2D 43 4F 4E 46 Cli: %SYS-5-CONF }
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/