You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Kris Ogirri <ka...@gmail.com> on 2014/03/12 20:23:24 UTC
Re: Issue with HBase Sink in Flume ( 1.3.0)
Hello Guys,
Just to revert back to the team... It seems that the upgrade to Flume
version 1.4.0 resolved this problem and items are currently being written
from the file Channel to the pCol CF in my specified Hbase table.
I hope there is a way to make this mail trail show as 'FIXED' so people
that have this problem in the future can refer back to this mail trail.
Thanks for all the support.
On 18 February 2014 20:53, Hari Shreedharan <hs...@cloudera.com>wrote:
> The performance issues in that version were mainly due to the file
> channel, which could essentially end up not working in some situations. I
> am not sure if that is the case, but there is nothing very obvious in your
> logs that suggest any other issues. Since this is a virtual appliance, you
> might want to contact the vendor's support channels who would have more
> info about what exactly is packaged in your specific version.
>
>
> Thanks,
> Hari
>
> On Tuesday, February 18, 2014 at 11:47 AM, Kris Ogirri wrote:
>
> Hello Hari,
>
> No I have no reason NOT to use a newer version but since I am working with
> a pre-packaged virtual appliance, I ideally would not want to perform any
> updates on any of the Hadoop components ( as I would then need to update
> the entire Virtual appliance to reflect the changes) without being sure
> that the update would solve this problem.
>
> What are your thoughts? Were the performance problems demonstrated related
> to Hbase Sinking? I am thinking this could be an issue between my Zookeeper
> deployment and my Hbase setup but I am open to suggestions.
>
> Thanks again for all the help.
>
>
>
> On 18 February 2014 20:41, Hari Shreedharan <hs...@cloudera.com>wrote:
>
> Looks like you are using Flume 1.3.0. Is there a reason for not using a
> newer version? Flume 1.4.0 is not almost 6 months old. 1.3.0 did have a
> known performance issue which was the reason 1.3.1 was release almost
> immediately after.
>
>
> Thanks,
> Hari
>
> On Tuesday, February 18, 2014 at 11:23 AM, Kris Ogirri wrote:
>
> Hello Hari,
>
> I didn't know it was a holiday in the US.
>
> Please see version information below:
>
> Hbase:
> HBase Shell; enter 'help<RETURN>' for list of supported commands.
> Type "exit<RETURN>" to leave the HBase Shell
> Version 0.94.3, rab548827f0c52211c1d67437484fcba635072767, Wed Jul 31
> 18:13:25 PDT 2013
>
>
> Flume:
> [biadmin@bivm bin]$ ./flume-ng version
> Flume 1.3.0
> Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
> Revision: abbccbd2ff14dd6fed2a8a3891eb51aff985e9f5
> Compiled by jenkins on Wed Jun 12 19:16:33 PDT 2013
> From source with checksum dce204011600e67e1455971266d3da07
>
>
> Thanks for all the assistance.
>
> BR,
>
>
>
> On 18 February 2014 20:14, Hari Shreedharan <hs...@cloudera.com>wrote:
>
> Hi Kris,
>
> Please realize that people usually work on their own time on these mailing
> lists and since your first message was sent on a Monday early morning on a
> long weekend in the US, others may not have seen your message either.
>
> Are you running Apache Flume and Apache HBase? If yes, what versions
> (output of flume-ng version and hbase version)?
>
>
> Thanks,
> Hari
>
> On Tuesday, February 18, 2014 at 10:22 AM, Kris Ogirri wrote:
>
> Hi,
>
> Cant anybody help with this? I am thinking its a small issue because
> everything seems to work fine but the data from the Channel never gets
> persisted into Hbase?
>
> I have added the description of the Hbase tables:
>
> hbase(main):005:0> describe 'telco_cdr_rec'
> DESCRIPTION
> ENABLED
> {NAME => 'telco_cdr_rec', FAMILIES => [{NAME => 'co
> true
> lfam', REPLICATION_SCOPE => '0',
> KEEP_DELETED_CELLS
> => 'false', COMPRESSION => 'NONE',
> ENCODE_ON_DISK
> => 'true', BLOCKCACHE => 'true', MIN_VERSIONS =>
> '0
> ', DATA_BLOCK_ENCODING => 'NONE', IN_MEMORY =>
> 'fal
> se', BLOOMFILTER => 'NONE', TTL => '2147483647',
> VE
> RSIONS => '3', BLOCKSIZE =>
> '65536'}]}
> 1 row(s) in 0.1600 seconds
>
>
> If no one can help with the problem, can anyone provide a link to the
> Flume -> Zookeeper -> Hbase Internal documentation so I can trace where the
> error lies.
>
> Are there Zookeeper log files where I can analyse whether Flume actually
> sends the Txns to Hbase via Zookeeper?
>
>
>
> On 17 February 2014 16:38, Kris Ogirri <ka...@gmail.com> wrote:
>
> Hello Jeff,
>
> Please find below requested logs.. Initiation part of the logs were
> unfortunately not included. I can run these again if necessary but the
> Zookeeper connection is included in the logs.
>
>
> 14/02/17 10:26:12 INFO properties.PropertiesFileConfigurationProvider:
> created channel ch2
> 14/02/17 10:26:13 INFO sink.DefaultSinkFactory: Creating instance of sink:
> hbase-sink, type: org.apache.flume.sink.hbase.HBaseSink
> 14/02/17 10:26:13 INFO sink.DefaultSinkFactory: Creating instance of sink:
> hdfs-sink, type: hdfs
> 14/02/17 10:26:14 INFO hdfs.HDFSEventSink: Hadoop Security enabled: false
> 14/02/17 10:26:14 INFO nodemanager.DefaultLogicalNodeManager: Starting new
> configuration:{ sourceRunners:{exec-source=EventDrivenSourceRunner: {
> source:org.apache.flume.source.ExecSource{name:exec-source,state:IDLE} }}
> sinkRunners:{hbase-sink=SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@4c004c counterGroup:{
> name:null counters:{} } }, hdfs-sink=SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@7b017b01 counterGroup:{
> name:null counters:{} } }} channels:{ch1=FileChannel ch1 { dataDirs:
> [/home/biadmin/.flume/file-channel/data] }, ch2=FileChannel ch2 { dataDirs:
> [/home/biadmin/.flume/file-channel2/data] }} }
> 14/02/17 10:26:14 INFO nodemanager.DefaultLogicalNodeManager: Starting
> Channel ch1
> 14/02/17 10:26:14 INFO file.FileChannel: Starting FileChannel ch1 {
> dataDirs: [/home/biadmin/.flume/file-channel/data] }...
> 14/02/17 10:26:14 INFO nodemanager.DefaultLogicalNodeManager: Starting
> Channel ch2
> 14/02/17 10:26:14 INFO file.FileChannel: Starting FileChannel ch2 {
> dataDirs: [/home/biadmin/.flume/file-channel2/data] }...
> 14/02/17 10:26:14 INFO file.Log: Encryption is not enabled
> 14/02/17 10:26:14 INFO file.Log: Replay started
> 14/02/17 10:26:14 INFO file.Log: Encryption is not enabled
> 14/02/17 10:26:14 INFO file.Log: Replay started
> 14/02/17 10:26:14 INFO file.Log: Found NextFileID 7, from
> [/home/biadmin/.flume/file-channel/data/log-7,
> /home/biadmin/.flume/file-channel/data/log-6]
> 14/02/17 10:26:14 INFO file.Log: Found NextFileID 6, from
> [/home/biadmin/.flume/file-channel2/data/log-6,
> /home/biadmin/.flume/file-channel2/data/log-4,
> /home/biadmin/.flume/file-channel2/data/log-5]
> 14/02/17 10:26:14 INFO file.EventQueueBackingStoreFileV3: Starting up with
> /home/biadmin/.flume/file-channel2/checkpoint/checkpoint and
> /home/biadmin/.flume/file-channel2/checkpoint/checkpoint.meta
> 14/02/17 10:26:14 INFO file.EventQueueBackingStoreFileV3: Reading
> checkpoint metadata from
> /home/biadmin/.flume/file-channel2/checkpoint/checkpoint.meta
> 14/02/17 10:26:14 INFO file.EventQueueBackingStoreFileV3: Starting up with
> /home/biadmin/.flume/file-channel/checkpoint/checkpoint and
> /home/biadmin/.flume/file-channel/checkpoint/checkpoint.meta
> 14/02/17 10:26:14 INFO file.EventQueueBackingStoreFileV3: Reading
> checkpoint metadata from
> /home/biadmin/.flume/file-channel/checkpoint/checkpoint.meta
> 14/02/17 10:26:14 INFO file.Log: Last Checkpoint Mon Feb 17 10:21:35 EST
> 2014, queue depth = 0
> 14/02/17 10:26:14 INFO file.Log: Last Checkpoint Mon Feb 17 10:21:31 EST
> 2014, queue depth = 0
> 14/02/17 10:26:14 INFO file.Log: Replaying logs with v2 replay logic
> 14/02/17 10:26:14 INFO file.Log: Replaying logs with v2 replay logic
> 14/02/17 10:26:14 INFO file.ReplayHandler: Starting replay of
> [/home/biadmin/.flume/file-channel/data/log-6,
> /home/biadmin/.flume/file-channel/data/log-7]
> 14/02/17 10:26:14 INFO file.ReplayHandler: Starting replay of
> [/home/biadmin/.flume/file-channel2/data/log-4,
> /home/biadmin/.flume/file-channel2/data/log-5,
> /home/biadmin/.flume/file-channel2/data/log-6]
> 14/02/17 10:26:14 INFO file.ReplayHandler: Replaying
> /home/biadmin/.flume/file-channel/data/log-6
> 14/02/17 10:26:14 INFO file.ReplayHandler: Replaying
> /home/biadmin/.flume/file-channel2/data/log-4
> 14/02/17 10:26:14 INFO tools.DirectMemoryUtils: Unable to get
> maxDirectMemory from VM: NoSuchMethodException:
> sun.misc.VM.maxDirectMemory(null)
> 14/02/17 10:26:14 INFO tools.DirectMemoryUtils: Direct Memory Allocation:
> Allocation = 1048576, Allocated = 0, MaxDirectMemorySize = 20971520,
> Remaining = 20971520
> 14/02/17 10:26:16 INFO file.LogFile: fast-forward to checkpoint position:
> 32040
> 14/02/17 10:26:16 INFO file.ReplayHandler: Replaying
> /home/biadmin/.flume/file-channel/data/log-7
> 14/02/17 10:26:16 INFO file.LogFile: fast-forward to checkpoint position:
> 2496
> 14/02/17 10:26:16 WARN file.LogFile: Checkpoint for
> file(/home/biadmin/.flume/file-channel2/data/log-4) is: 1392407375821,
> which is beyond the requested checkpoint time: 1392650490155 and position 0
> 14/02/17 10:26:16 INFO file.ReplayHandler: Replaying
> /home/biadmin/.flume/file-channel2/data/log-5
> 14/02/17 10:26:16 INFO file.LogFile: fast-forward to checkpoint position:
> 22843
> 14/02/17 10:26:16 INFO file.LogFile: Encountered EOF at 22843 in
> /home/biadmin/.flume/file-channel2/data/log-5
> 14/02/17 10:26:16 INFO file.ReplayHandler: Replaying
> /home/biadmin/.flume/file-channel2/data/log-6
> 14/02/17 10:26:16 WARN file.LogFile: Checkpoint for
> file(/home/biadmin/.flume/file-channel2/data/log-6) is: 1392650490155,
> which is beyond the requested checkpoint time: 1392650490155 and position 0
> 14/02/17 10:26:16 INFO file.ReplayHandler: read: 0, put: 0, take: 0,
> rollback: 0, commit: 0, skip: 0, eventCount:0
> 14/02/17 10:26:16 INFO file.Log: Rolling
> /home/biadmin/.flume/file-channel2/data
> 14/02/17 10:26:16 INFO file.Log: Roll start
> /home/biadmin/.flume/file-channel2/data
> 14/02/17 10:26:16 INFO file.LogFile: Opened
> /home/biadmin/.flume/file-channel2/data/log-7
> 14/02/17 10:26:16 INFO file.LogFile: Encountered EOF at 2496 in
> /home/biadmin/.flume/file-channel/data/log-7
> 14/02/17 10:26:16 INFO file.LogFile: Encountered EOF at 32071 in
> /home/biadmin/.flume/file-channel/data/log-6
> 14/02/17 10:26:16 INFO file.ReplayHandler: read: 1, put: 0, take: 0,
> rollback: 0, commit: 0, skip: 1, eventCount:0
> 14/02/17 10:26:16 INFO file.Log: Rolling
> /home/biadmin/.flume/file-channel/data
> 14/02/17 10:26:16 INFO file.Log: Roll start
> /home/biadmin/.flume/file-channel/data
> 14/02/17 10:26:16 INFO file.LogFile: Opened
> /home/biadmin/.flume/file-channel/data/log-8
> 14/02/17 10:26:16 INFO file.Log: Roll end
> 14/02/17 10:26:16 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 0
> 14/02/17 10:26:16 INFO file.Log: Roll end
> 14/02/17 10:26:16 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 0
> 14/02/17 10:26:16 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650774387, queueSize: 0,
> queueHead: 10516
> 14/02/17 10:26:16 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650774388, queueSize: 0,
> queueHead: 223682
> 14/02/17 10:26:16 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 0, logWriteOrderID = 1392650774387
> 14/02/17 10:26:16 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 0, logWriteOrderID = 1392650774388
> 14/02/17 10:26:16 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 0 logWriteOrderID:
> 1392650774387
> 14/02/17 10:26:16 INFO file.FileChannel: Queue Size after replay: 0
> [channel=ch2]
> 14/02/17 10:26:17 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 0 logWriteOrderID:
> 1392650774388
> 14/02/17 10:26:17 INFO file.FileChannel: Queue Size after replay: 0
> [channel=ch1]
> 14/02/17 10:26:17 INFO instrumentation.MonitoredCounterGroup: Monitoried
> counter group for type: CHANNEL, name: ch2, registered successfully.
> 14/02/17 10:26:17 INFO instrumentation.MonitoredCounterGroup: Component
> type: CHANNEL, name: ch2 started
> 14/02/17 10:26:17 INFO instrumentation.MonitoredCounterGroup: Monitoried
> counter group for type: CHANNEL, name: ch1, registered successfully.
> 14/02/17 10:26:17 INFO instrumentation.MonitoredCounterGroup: Component
> type: CHANNEL, name: ch1 started
> 14/02/17 10:26:17 INFO nodemanager.DefaultLogicalNodeManager: Starting
> Sink hbase-sink
> 14/02/17 10:26:17 INFO nodemanager.DefaultLogicalNodeManager: Starting
> Sink hdfs-sink
> 14/02/17 10:26:17 INFO nodemanager.DefaultLogicalNodeManager: Starting
> Source exec-source
> 14/02/17 10:26:17 INFO source.ExecSource: Exec source starting with
> command:tail -F /home/biadmin/bigdemo/data/rec_telco.cdr
> 14/02/17 10:26:17 INFO instrumentation.MonitoredCounterGroup: Monitoried
> counter group for type: SINK, name: hdfs-sink, registered successfully.
> 14/02/17 10:26:17 INFO instrumentation.MonitoredCounterGroup: Component
> type: SINK, name: hdfs-sink started
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.4.5--1, built on 01/23/2013 14:29 GMT
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:host.name
> =bivm
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:java.version=1.6.0
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:java.vendor=IBM Corporation
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:java.home=/opt/ibm/biginsights/jdk/jre
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:java.class.path=conf:/opt/ibm/biginsights/flume/lib/snappy-java-1.0.4.1.jar:/opt/ibm/biginsights/flume/lib/jetty-util-6.1.26.jar:/opt/ibm/biginsights/flume/lib/jackson-mapper-asl-1.9.3.jar:/opt/ibm/biginsights/flume/lib/flume-avro-source-1.3.0.jar:/opt/ibm/biginsights/flume/lib/flume-jdbc-channel-1.3.0.jar:/opt/ibm/biginsights/flume/lib/velocity-1.7.jar:/opt/ibm/biginsights/flume/lib/zookeeper-3.4.5.jar:/opt/ibm/biginsights/flume/lib/flume-ng-node-1.3.0.jar:/opt/ibm/biginsights/flume/lib/commons-dbcp-1.4.jar:/opt/ibm/biginsights/flume/lib/log4j-1.2.16.jar:/opt/ibm/biginsights/flume/lib/flume-hdfs-sink-1.3.0.jar:/opt/ibm/biginsights/flume/lib/asynchbase-1.2.0.jar:/opt/ibm/biginsights/flume/lib/flume-recoverable-memory-channel-1.3.0.jar:/opt/ibm/biginsights/flume/lib/async-1.3.1.jar:/opt/ibm/biginsights/flume/lib/slf4j-log4j12-1.6.1.jar:/opt/ibm/biginsights/flume/lib/flume-thrift-source-1.3.0.jar:/opt/ibm/biginsights/flume/lib/flume-file-channel-1.3.0.jar:/opt/ibm/biginsights/flume/lib/libthrift-0.6.1.jar:/opt/ibm/biginsights/flume/lib/avro-1.7.2.jar:/opt/ibm/biginsights/flume/lib/jetty-6.1.26.jar:/opt/ibm/biginsights/flume/lib/jackson-core-asl-1.9.3.jar:/opt/ibm/biginsights/flume/lib/servlet-api-2.5-20110124.jar:/opt/ibm/biginsights/flume/lib/flume-ng-elasticsearch-sink-1.3.0.jar:/opt/ibm/biginsights/flume/lib/flume-ng-configuration-1.3.0.jar:/opt/ibm/biginsights/flume/lib/jsr305-1.3.9.jar:/opt/ibm/biginsights/flume/lib/irclib-1.10.jar:/opt/ibm/biginsights/flume/lib/commons-cli-1.2.jar:/opt/ibm/biginsights/flume/lib/derby-10.8.3.1.jar:/opt/ibm/biginsights/flume/lib/flume-ng-log4jappender-1.3.0.jar:/opt/ibm/biginsights/flume/lib/netty-3.4.0.Final.jar:/opt/ibm/biginsights/flume/lib/flume-irc-sink-1.3.0.jar:/opt/ibm/biginsights/flume/lib/jcl-over-slf4j-1.7.2.jar:/opt/ibm/biginsights/flume/lib/slf4j-api-1.6.1.jar:/opt/ibm/biginsights/flume/lib/joda-time-2.1.jar:/opt/ibm/biginsights/flume/lib/commons-lang-2.5.jar:/opt/ibm/biginsights/flume/lib/commons-io-2.1.jar:/opt/ibm/biginsights/flume/lib/commons-collections-3.2.1.jar:/opt/ibm/biginsights/flume/lib/mina-core-2.0.4.jar:/opt/ibm/biginsights/flume/lib/commons-pool-1.5.4.jar:/opt/ibm/biginsights/flume/lib/flume-ng-hbase-sink-1.3.0.jar:/opt/ibm/biginsights/flume/lib/protobuf-java-2.4.1.jar:/opt/ibm/biginsights/flume/lib/flume-scribe-source-1.3.0.jar:/opt/ibm/biginsights/flume/lib/flume-ng-core-1.3.0.jar:/opt/ibm/biginsights/flume/lib/gson-2.2.2.jar:/opt/ibm/biginsights/flume/lib/flume-ng-sdk-1.3.0.jar:/opt/ibm/biginsights/flume/lib/avro-ipc-1.7.2.jar:/opt/ibm/biginsights/flume/lib/guava-10.0.1.jar:/opt/ibm/biginsights/flume/lib/paranamer-2.3.jar:/opt/ibm/biginsights/hadoop-conf:/opt/ibm/biginsights/jdk/lib/tools.jar:/opt/ibm/biginsights/IHC/libexec/..:/opt/ibm/biginsights/IHC/libexec/../hadoop-core-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/adaptive-mr.jar:/opt/ibm/biginsights/IHC/libexec/../lib/asm-3.2.jar:/opt/ibm/biginsights/IHC/libexec/../lib/aspectjrt-1.6.11.jar:/opt/ibm/biginsights/IHC/libexec/../lib/aspectjtools-1.6.11.jar:/opt/ibm/biginsights/IHC/libexec/../lib/biginsights-sftpfs-1.0.0.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-beanutils-1.8.0.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-cli-1.2.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-codec-1.4.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-collections-3.2.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-configuration-1.6.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-daemon-1.0.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-digester-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-el-1.0.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-httpclient-3.0.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-io-2.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-lang-2.4.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-logging-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-logging-api-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-math-2.2.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-net-3.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/core-3.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/ftplet-api-1.0.6.jar:/opt/ibm/biginsights/IHC/libexec/../lib/ftpserver-core-1.0.6.jar:/opt/ibm/biginsights/IHC/libexec/../lib/guardium-proxy.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hadoop-capacity-scheduler-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hadoop-fairscheduler-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hadoop-thriftfs-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hsqldb-1.8.0.10.jar:/opt/ibm/biginsights/IHC/libexec/../lib/ibm-compression.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jackson-core-asl-1.8.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jasper-compiler-5.5.12.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jasper-runtime-5.5.12.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jdeb-0.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jersey-core-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jersey-json-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jersey-server-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jets3t-0.6.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jetty-6.1.26.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jetty-util-6.1.26.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsch-0.1.42.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsch-0.1.43.jar:/opt/ibm/biginsights/IHC/libexec/../lib/junit-4.5.jar:/opt/ibm/biginsights/IHC/libexec/../lib/log4j-1.2.16.jar:/opt/ibm/biginsights/IHC/libexec/../lib/mina-core-2.0.4.jar:/opt/ibm/biginsights/IHC/libexec/../lib/mockito-all-1.8.5.jar:/opt/ibm/biginsights/IHC/libexec/../lib/oro-2.0.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/servlet-api-2.5-20081211.jar:/opt/ibm/biginsights/IHC/libexec/../lib/workflowScheduler.jar:/opt/ibm/biginsights/IHC/libexec/../lib/xmlenc-0.52.jar:/opt/ibm/biginsights/IHC/libexec/../lib/zookeeper-3.4.5.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsp-2.1/jsp-2.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/opt/ibm/biginsights/hbase/conf:/opt/ibm/biginsights/IHC/:/opt/ibm/biginsights/IHC/:/opt/ibm/biginsights/hadoop-conf:/opt/ibm/biginsights/hbase/conf:/opt/ibm/biginsights/IHC/lib/biginsights-gpfs-1.1.1.jar:/opt/ibm/biginsights/IHC/hadoop-core.jar:/opt/ibm/biginsights/IHC/lib/biginsights-gpfs-1.1.1.jar:/opt/ibm/biginsights/IHC/hadoop-core.jar:/home/biadmin/twitter4j/lib/twitter4j-media-support-3.0.3.jar:/home/biadmin/twitter4j/lib/twitter4j-core-3.0.3.jar:home/biadmin/twitter4j/lib/twitter4j-async-3.0.3.jar:/home/biadmin/twitter4j/lib/twitter4j-stream-3.0.3.jar:/home/biadmin/twitter4j/lib/twitter4j-media-support-3.0.3.jar:/home/biadmin/twitter4j/lib/twitter4j-core-3.0.3.jar:home/biadmin/twitter4j/lib/twitter4j-async-3.0.3.jar:/home/biadmin/twitter4j/lib/twitter4j-stream-3.0.3.jar:/opt/ibm/biginsights/jdk/lib/tools.jar:/opt/ibm/biginsights/hbase:/opt/ibm/biginsights/hbase/hbase-0.94.3-security.jar:/opt/ibm/biginsights/hbase/hbase-0.94.3-security-tests.jar:/opt/ibm/biginsights/hbase/hbase.jar:/opt/ibm/biginsights/hbase/lib/activation-1.1.jar:/opt/ibm/biginsights/hbase/lib/asm-3.1.jar:/opt/ibm/biginsights/hbase/lib/avro-1.7.2.jar:/opt/ibm/biginsights/hbase/lib/avro-ipc-1.7.2.jar:/opt/ibm/biginsights/hbase/lib/commons-beanutils-1.8.0.jar:/opt/ibm/biginsights/hbase/lib/commons-cli-1.2.jar:/opt/ibm/biginsights/hbase/lib/commons-codec-1.4.jar:/opt/ibm/biginsights/hbase/lib/commons-collections-3.2.1.jar:/opt/ibm/biginsights/hbase/lib/commons-configuration-1.6.jar:/opt/ibm/biginsights/hbase/lib/commons-digester-1.8.jar:/opt/ibm/biginsights/hbase/lib/commons-el-1.0.jar:/opt/ibm/biginsights/hbase/lib/commons-httpclient-3.1.jar:/opt/ibm/biginsights/hbase/lib/commons-io-2.1.jar:/opt/ibm/biginsights/hbase/lib/commons-lang-2.5.jar:/opt/ibm/biginsights/hbase/lib/commons-logging-1.1.1.jar:/opt/ibm/biginsights/hbase/lib/commons-math-2.2.jar:/opt/ibm/biginsights/hbase/lib/commons-net-3.1.jar:/opt/ibm/biginsights/hbase/lib/core-3.1.1.jar:/opt/ibm/biginsights/hbase/lib/guardium-proxy.jar:/opt/ibm/biginsights/hbase/lib/guava-11.0.2.jar:/opt/ibm/biginsights/hbase/lib/hadoop-core.jar:/opt/ibm/biginsights/hbase/lib/hadoop-tools-1.1.1.jar:/opt/ibm/biginsights/hbase/lib/high-scale-lib-1.1.1.jar:/opt/ibm/biginsights/hbase/lib/httpclient-4.1.2.jar:/opt/ibm/biginsights/hbase/lib/httpcore-4.1.3.jar:/opt/ibm/biginsights/hbase/lib/jackson-core-asl-1.8.8.jar:/opt/ibm/biginsights/hbase/lib/jackson-jaxrs-1.8.8.jar:/opt/ibm/biginsights/hbase/lib/jackson-mapper-asl-1.8.8.jar:/opt/ibm/biginsights/hbase/lib/jackson-xc-1.8.8.jar:/opt/ibm/biginsights/hbase/lib/jamon-runtime-2.3.1.jar:/opt/ibm/biginsights/hbase/lib/jasper-compiler-5.5.23.jar:/opt/ibm/biginsights/hbase/lib/jasper-runtime-5.5.23.jar:/opt/ibm/biginsights/hbase/lib/jaxb-api-2.1.jar:/opt/ibm/biginsights/hbase/lib/jaxb-impl-2.2.3-1.jar:/opt/ibm/biginsights/hbase/lib/jersey-core-1.8.jar:/opt/ibm/biginsights/hbase/lib/jersey-json-1.8.jar:/opt/ibm/biginsights/hbase/lib/jersey-server-1.8.jar:/opt/ibm/biginsights/hbase/lib/jettison-1.1.jar:/opt/ibm/biginsights/hbase/lib/jetty-6.1.26.jar:/opt/ibm/biginsights/hbase/lib/jetty-util-6.1.26.jar:/opt/ibm/biginsights/hbase/lib/jruby-complete-1.6.5.1.jar:/opt/ibm/biginsights/hbase/lib/jsp-2.1-6.1.14.jar:/opt/ibm/biginsights/hbase/lib/jsp-api-2.1-6.1.14.jar:/opt/ibm/biginsights/hbase/lib/jsp-api-2.1.jar:/opt/ibm/biginsights/hbase/lib/jsr305-1.3.9.jar:/opt/ibm/biginsights/hbase/lib/junit-4.10-HBASE-1.jar:/opt/ibm/biginsights/hbase/lib/libthrift-0.8.0.jar:/opt/ibm/biginsights/hbase/lib/log4j-1.2.16.jar:/opt/ibm/biginsights/hbase/lib/metrics-core-2.1.2.jar:/opt/ibm/biginsights/hbase/lib/netty-3.2.4.Final.jar:/opt/ibm/biginsights/hbase/lib/netty-3.4.0.Final.jar:/opt/ibm/biginsights/hbase/lib/protobuf-java-2.4.0a.jar:/opt/ibm/biginsights/hbase/lib/servlet-api-2.5-20081211.jar:/opt/ibm/biginsights/hbase/lib/servlet-api-2.5-6.1.14.jar:/opt/ibm/biginsights/hbase/lib/snappy-java-1.0.4.1.jar:/opt/ibm/biginsights/hbase/lib/stax-api-1.0.1.jar:/opt/ibm/biginsights/hbase/lib/velocity-1.7.jar:/opt/ibm/biginsights/hbase/lib/xmlenc-0.52.jar:/opt/ibm/biginsights/hbase/lib/xml-ibm.jar:/opt/ibm/biginsights/hbase/lib/zookeeper-3.4.5.jar:/opt/ibm/biginsights/hbase/lib/zookeeper.jar:/opt/ibm/biginsights/hadoop-conf:/opt/ibm/biginsights/jdk/lib/tools.jar:/opt/ibm/biginsights/IHC/libexec/..:/opt/ibm/biginsights/IHC/libexec/../hadoop-core-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/adaptive-mr.jar:/opt/ibm/biginsights/IHC/libexec/../lib/asm-3.2.jar:/opt/ibm/biginsights/IHC/libexec/../lib/aspectjrt-1.6.11.jar:/opt/ibm/biginsights/IHC/libexec/../lib/aspectjtools-1.6.11.jar:/opt/ibm/biginsights/IHC/libexec/../lib/biginsights-sftpfs-1.0.0.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-beanutils-1.8.0.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-cli-1.2.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-codec-1.4.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-collections-3.2.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-configuration-1.6.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-daemon-1.0.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-digester-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-el-1.0.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-httpclient-3.0.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-io-2.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-lang-2.4.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-logging-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-logging-api-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-math-2.2.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-net-3.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/core-3.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/ftplet-api-1.0.6.jar:/opt/ibm/biginsights/IHC/libexec/../lib/ftpserver-core-1.0.6.jar:/opt/ibm/biginsights/IHC/libexec/../lib/guardium-proxy.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hadoop-capacity-scheduler-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hadoop-fairscheduler-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hadoop-thriftfs-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hsqldb-1.8.0.10.jar:/opt/ibm/biginsights/IHC/libexec/../lib/ibm-compression.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jackson-core-asl-1.8.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jasper-compiler-5.5.12.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jasper-runtime-5.5.12.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jdeb-0.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jersey-core-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jersey-json-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jersey-server-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jets3t-0.6.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jetty-6.1.26.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jetty-util-6.1.26.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsch-0.1.42.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsch-0.1.43.jar:/opt/ibm/biginsights/IHC/libexec/../lib/junit-4.5.jar:/opt/ibm/biginsights/IHC/libexec/../lib/log4j-1.2.16.jar:/opt/ibm/biginsights/IHC/libexec/../lib/mina-core-2.0.4.jar:/opt/ibm/biginsights/IHC/libexec/../lib/mockito-all-1.8.5.jar:/opt/ibm/biginsights/IHC/libexec/../lib/oro-2.0.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/servlet-api-2.5-20081211.jar:/opt/ibm/biginsights/IHC/libexec/../lib/workflowScheduler.jar:/opt/ibm/biginsights/IHC/libexec/../lib/xmlenc-0.52.jar:/opt/ibm/biginsights/IHC/libexec/../lib/zookeeper-3.4.5.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsp-2.1/jsp-2.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/opt/ibm/biginsights/hbase/conf:/opt/ibm/biginsights/hbase/conf
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:java.library.path=:/opt/ibm/biginsights/IHC/libexec/../lib/native/Linux-amd64-64:/opt/ibm/biginsights/IHC/libexec/../lib/native/Linux-amd64-64
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/tmp
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:java.compiler=j9jit24
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:os.name
> =Linux
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:os.arch=amd64
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:os.version=2.6.18-194.17.4.el5
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:user.name
> =biadmin
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:user.home=/home/biadmin
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:user.dir=/opt/ibm/biginsights/flume/bin
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Initiating client connection,
> connectString=bivm:2181 sessionTimeout=180000 watcher=hconnection
> 14/02/17 10:26:17 INFO zookeeper.RecoverableZooKeeper: The identifier of
> this process is 20984@bivm
> 14/02/17 10:26:17 INFO zookeeper.ClientCnxn: Opening socket connection to
> server bivm/192.168.37.128:2181. Will not attempt to authenticate using
> SASL (Unable to locate a login configuration)
> 14/02/17 10:26:17 INFO zookeeper.ClientCnxn: Socket connection established
> to bivm/192.168.37.128:2181, initiating session
> 14/02/17 10:26:17 INFO zookeeper.ClientCnxn: Session establishment
> complete on server bivm/192.168.37.128:2181, sessionid =
> 0x144401355b4001d, negotiated timeout = 60000
> 14/02/17 10:29:56 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 60
> 14/02/17 10:29:56 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650774536, queueSize: 60,
> queueHead: 10514
> 14/02/17 10:29:56 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 32036, logWriteOrderID = 1392650774536
> 14/02/17 10:29:57 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 32036
> logWriteOrderID: 1392650774536
> 14/02/17 10:29:57 INFO file.LogFile: Closing RandomReader
> /home/biadmin/.flume/file-channel2/data/log-4
> 14/02/17 10:29:57 INFO file.Log: Removing old log
> /home/biadmin/.flume/file-channel2/data/log-4, result = true, minFileID 7
> 14/02/17 10:29:57 INFO file.LogFile: Closing RandomReader
> /home/biadmin/.flume/file-channel2/data/log-5
> 14/02/17 10:29:57 INFO file.Log: Removing old log
> /home/biadmin/.flume/file-channel2/data/log-5, result = true, minFileID 7
> 14/02/17 10:29:58 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 460
> 14/02/17 10:29:58 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650775504, queueSize: 520,
> queueHead: 10514
> 14/02/17 10:29:58 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 277565, logWriteOrderID = 1392650775504
> 14/02/17 10:29:58 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 277565
> logWriteOrderID: 1392650775504
> 14/02/17 10:29:58 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 540
> 14/02/17 10:29:59 INFO hdfs.BucketWriter: Creating
> hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998182.tmp
> 14/02/17 10:29:59 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 423
> 14/02/17 10:30:00 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650775933, queueSize: 137,
> queueHead: 10917
> 14/02/17 10:30:00 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650775934, queueSize: 539,
> queueHead: 223681
> 14/02/17 10:30:01 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 304892, logWriteOrderID = 1392650775933
> 14/02/17 10:30:01 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 304892
> logWriteOrderID: 1392650775933
> 14/02/17 10:30:02 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 137
> 14/02/17 10:30:02 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 288266, logWriteOrderID = 1392650775934
> 14/02/17 10:30:02 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776074, queueSize: 0,
> queueHead: 11054
> 14/02/17 10:30:04 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 288266
> logWriteOrderID: 1392650775934
> 14/02/17 10:30:04 INFO file.LogFile: Closing RandomReader
> /home/biadmin/.flume/file-channel/data/log-6
> 14/02/17 10:30:04 INFO file.Log: Removing old log
> /home/biadmin/.flume/file-channel/data/log-6, result = true, minFileID 8
> 14/02/17 10:30:05 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 29
> 14/02/17 10:30:06 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 310581, logWriteOrderID = 1392650776074
> 14/02/17 10:30:13 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776105, queueSize: 550,
> queueHead: 223690
> 14/02/17 10:30:19 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 310581
> logWriteOrderID: 1392650776074
> 14/02/17 10:30:21 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 20
> 14/02/17 10:30:29 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776127, queueSize: 20,
> queueHead: 11052
> 14/02/17 10:30:29 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 299362, logWriteOrderID = 1392650776105
> 14/02/17 10:30:30 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 321308, logWriteOrderID = 1392650776127
> 14/02/17 10:30:30 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 299362
> logWriteOrderID: 1392650776105
> 14/02/17 10:30:30 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 321308
> logWriteOrderID: 1392650776127
> 14/02/17 10:30:31 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 21
> 14/02/17 10:30:32 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 38
> 14/02/17 10:30:34 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776192, queueSize: 569,
> queueHead: 223691
> 14/02/17 10:30:34 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776193, queueSize: 20,
> queueHead: 11070
> 14/02/17 10:30:34 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 310040, logWriteOrderID = 1392650776192
> 14/02/17 10:30:34 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 332801, logWriteOrderID = 1392650776193
> 14/02/17 10:30:34 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 310040
> logWriteOrderID: 1392650776192
> 14/02/17 10:30:35 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 332801
> logWriteOrderID: 1392650776193
> 14/02/17 10:30:37 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 20
> 14/02/17 10:30:39 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 20
> 14/02/17 10:30:39 INFO hdfs.BucketWriter: Renaming
> hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998182.tmp to
> hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998182
> 14/02/17 10:30:40 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776236, queueSize: 0,
> queueHead: 11090
> 14/02/17 10:30:40 INFO hdfs.BucketWriter: Creating
> hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998183.tmp
> 14/02/17 10:30:42 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776237, queueSize: 589,
> queueHead: 223691
> 14/02/17 10:30:58 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 333657, logWriteOrderID = 1392650776236
> 14/02/17 10:30:59 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 320738, logWriteOrderID = 1392650776237
> 14/02/17 10:31:01 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 333657
> logWriteOrderID: 1392650776236
> 14/02/17 10:31:03 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 320738
> logWriteOrderID: 1392650776237
> 14/02/17 10:31:04 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 125
> 14/02/17 10:31:05 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 20
> 14/02/17 10:31:07 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776384, queueSize: 464,
> queueHead: 223816
> 14/02/17 10:31:07 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776385, queueSize: 20,
> queueHead: 11088
> 14/02/17 10:31:19 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 344355, logWriteOrderID = 1392650776385
> 14/02/17 10:31:19 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 325863, logWriteOrderID = 1392650776384
> 14/02/17 10:31:20 INFO hdfs.BucketWriter: Renaming
> hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998183.tmp to
> hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998183
> 14/02/17 10:31:22 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 325863
> logWriteOrderID: 1392650776384
> 14/02/17 10:31:22 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 344355
> logWriteOrderID: 1392650776385
> 14/02/17 10:31:23 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 20
> 14/02/17 10:31:23 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 1
> 14/02/17 10:31:23 INFO hdfs.BucketWriter: Creating
> hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998184.tmp
> 14/02/17 10:31:24 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776427, queueSize: 0,
> queueHead: 11108
> 14/02/17 10:31:24 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776428, queueSize: 463,
> queueHead: 223817
> 14/02/17 10:31:25 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 335946, logWriteOrderID = 1392650776428
> 14/02/17 10:31:26 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 345211, logWriteOrderID = 1392650776427
> 14/02/17 10:31:26 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 345211
> logWriteOrderID: 1392650776427
> 14/02/17 10:31:26 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 335946
> logWriteOrderID: 1392650776428
> 14/02/17 10:31:27 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 40
> 14/02/17 10:31:28 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 70
> 14/02/17 10:31:28 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776540, queueSize: 473,
> queueHead: 223847
> 14/02/17 10:31:28 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776541, queueSize: 40,
> queueHead: 11106
> 14/02/17 10:31:28 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 356818, logWriteOrderID = 1392650776540
> 14/02/17 10:31:28 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 356818
> logWriteOrderID: 1392650776540
> 14/02/17 10:31:28 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 366536, logWriteOrderID = 1392650776541
> 14/02/17 10:31:30 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 366536
> logWriteOrderID: 1392650776541
> 14/02/17 10:31:31 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 493
> 14/02/17 10:31:32 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 40
> 14/02/17 10:31:34 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650777082, queueSize: 0,
> queueHead: 11146
> 14/02/17 10:31:35 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650777083, queueSize: 0,
> queueHead: 224340
> 14/02/17 10:31:38 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 368733, logWriteOrderID = 1392650777082
> 14/02/17 10:31:38 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 379163, logWriteOrderID = 1392650777083
> 14/02/17 10:31:38 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 368733
> logWriteOrderID: 1392650777082
> 14/02/17 10:31:38 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 379163
> logWriteOrderID: 1392650777083
> 14/02/17 10:31:39 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 920
> 14/02/17 10:31:39 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 900
> 14/02/17 10:31:40 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650778995, queueSize: 900,
> queueHead: 224338
> 14/02/17 10:31:40 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650778996, queueSize: 920,
> queueHead: 11144
> 14/02/17 10:31:49 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 859009, logWriteOrderID = 1392650778996
> 14/02/17 10:31:49 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 859505, logWriteOrderID = 1392650778995
> 14/02/17 10:31:49 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 859009
> logWriteOrderID: 1392650778996
> 14/02/17 10:31:50 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 920
> 14/02/17 10:31:53 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 859505
> logWriteOrderID: 1392650778995
> 14/02/17 10:31:53 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650779929, queueSize: 0,
> queueHead: 12064
> 14/02/17 10:31:54 INFO hdfs.BucketWriter: Renaming
> hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998184.tmp to
> hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998184
> 14/02/17 10:31:54 INFO hdfs.BucketWriter: Creating
> hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998185.tmp
> 14/02/17 10:31:54 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 22
> 14/02/17 10:31:55 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650779951, queueSize: 918,
> queueHead: 224340
> 14/02/17 10:31:56 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 897089, logWriteOrderID = 1392650779929
> 14/02/17 10:31:56 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 870220, logWriteOrderID = 1392650779951
> 14/02/17 10:31:56 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 870220
> logWriteOrderID: 1392650779951
> 14/02/17 10:31:56 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 897089
> logWriteOrderID: 1392650779929
> 14/02/17 10:31:57 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 300
> 14/02/17 10:32:00 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650781760, queueSize: 300,
> queueHead: 12062
> 14/02/17 10:32:00 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 1198
> 14/02/17 10:32:01 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650781761, queueSize: 0,
> queueHead: 225538
> 14/02/17 10:32:02 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 1057180, logWriteOrderID = 1392650781760
> 14/02/17 10:32:03 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 1057180
> logWriteOrderID: 1392650781760
> 14/02/17 10:32:03 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 1068832, logWriteOrderID = 1392650781761
> 14/02/17 10:32:03 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 1068832
> logWriteOrderID: 1392650781761
> 14/02/17 10:32:04 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 798
> 14/02/17 10:32:07 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650783137, queueSize: 500,
> queueHead: 12360
> 14/02/17 10:32:07 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 520
> 14/02/17 10:32:08 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650783138, queueSize: 519,
> queueHead: 225537
> 14/02/17 10:32:12 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 1336479, logWriteOrderID = 1392650783137
> 14/02/17 10:32:14 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 1346456, logWriteOrderID = 1392650783138
> 14/02/17 10:32:14 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 1336479
> logWriteOrderID: 1392650783137
> 14/02/17 10:32:15 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 100
> 14/02/17 10:32:16 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 1346456
> logWriteOrderID: 1392650783138
> 14/02/17 10:32:17 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650783761, queueSize: 400,
> queueHead: 12460
> 14/02/17 10:32:17 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 519
> 14/02/17 10:32:20 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650783762, queueSize: 0,
> queueHead: 226056
> 14/02/17 10:32:21 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 1341143, logWriteOrderID = 1392650783761
> 14/02/17 10:32:23 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 1367771, logWriteOrderID = 1392650783762
> 14/02/17 10:32:23 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 1341143
> logWriteOrderID: 1392650783761
> 14/02/17 10:32:24 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 1367771
> logWriteOrderID: 1392650783762
> 14/02/17 10:32:24 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 300
> 14/02/17 10:32:25 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 100
> 14/02/17 10:32:25 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650784174, queueSize: 300,
> queueHead: 12660
> 14/02/17 10:32:25 INFO hdfs.BucketWriter: Renaming
> hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998185.tmp to
> hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998185
> 14/02/17 10:32:25 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650784175, queueSize: 100,
> queueHead: 226054
> 14/02/17 10:32:25 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 1402287, logWriteOrderID = 1392650784174
> 14/02/17 10:32:26 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 1402287
> logWriteOrderID: 1392650784174
> 14/02/17 10:32:26 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 1421128, logWriteOrderID = 1392650784175
> 14/02/17 10:32:26 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 1421128
> logWriteOrderID: 1392650784175
> 14/02/17 10:32:27 INFO hdfs.BucketWriter: Creating
> hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998186.tmp
> 14/02/17 10:32:27 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 480
> 14/02/17 10:32:28 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 278
> 14/02/17 10:32:28 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650785222, queueSize: 98,
> queueHead: 13042
> 14/02/17 10:32:32 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650785223, queueSize: 0,
> queueHead: 226332
> 14/02/17 10:32:33 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 1514767, logWriteOrderID = 1392650785222
> 14/02/17 10:32:34 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 1514767
> logWriteOrderID: 1392650785222
> 14/02/17 10:32:35 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 118
> 14/02/17 10:32:38 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 1528845, logWriteOrderID = 1392650785223
> 14/02/17 10:32:38 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650785364, queueSize: 0,
> queueHead: 13160
> 14/02/17 10:32:40 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 1528845
> logWriteOrderID: 1392650785223
> 14/02/17 10:32:41 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 1529781, logWriteOrderID = 1392650785364
> 14/02/17 10:32:42 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 1529781
> logWriteOrderID: 1392650785364
> 14/02/17 10:32:43 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 500
> 14/02/17 10:32:44 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 500
> 14/02/17 10:32:45 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650786415, queueSize: 500,
> queueHead: 13158
> 14/02/17 10:32:47 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650786416, queueSize: 500,
> queueHead: 226330
> 14/02/17 10:32:53 INFO node.FlumeNode: Flume node stopping - agent
> 14/02/17 10:32:53 INFO lifecycle.LifecycleSupervisor: Stopping lifecycle
> supervisor 9
> 14/02/17 10:32:53 INFO properties.PropertiesFileConfigurationProvider:
> Configuration provider stopping
> 14/02/17 10:32:53 INFO nodemanager.DefaultLogicalNodeManager: Node manager
> stopping
> 14/02/17 10:32:53 INFO nodemanager.DefaultLogicalNodeManager: Shutting
> down configuration: { sourceRunners:{exec-source=EventDrivenSourceRunner: {
> source:org.apache.flume.source.ExecSource{name:exec-source,state:START} }}
> sinkRunners:{hbase-sink=SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@4c004c counterGroup:{
> name:null counters:{runner.backoffs.consecutive=2, runner.backoffs=59} } },
> hdfs-sink=SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@7b017b01 counterGroup:{
> name:null counters:{runner.backoffs.consecutive=3, runner.backoffs=53} } }}
> channels:{ch1=FileChannel ch1 { dataDirs:
> [/home/biadmin/.flume/file-channel/data] }, ch2=FileChannel ch2 { dataDirs:
> [/home/biadmin/.flume/file-channel2/data] }} }
> 14/02/17 10:32:53 INFO nodemanager.DefaultLogicalNodeManager: Stopping
> Source exec-source
> 14/02/17 10:32:53 INFO lifecycle.LifecycleSupervisor: Stopping component:
> EventDrivenSourceRunner: {
> source:org.apache.flume.source.ExecSource{name:exec-source,state:START} }
> 14/02/17 10:32:53 INFO source.ExecSource: Stopping exec source with
> command:tail -F /home/biadmin/bigdemo/data/rec_telco.cdr
> 14/02/17 10:32:54 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 1795949, logWriteOrderID = 1392650786416
> 14/02/17 10:32:54 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 1796885, logWriteOrderID = 1392650786415
> 14/02/17 10:32:57 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 1795949
> logWriteOrderID: 1392650786416
> 14/02/17 10:32:57 ERROR source.ExecSource: Failed while running command:
> tail -F /home/biadmin/bigdemo/data/rec_telco.cdr
> java.io.IOException: Pipe closed
> at java.io.PipedInputStream.read(PipedInputStream.java:302)
> at java.lang.ProcessPipedInputStream.read(UNIXProcess.java:412)
> at java.io.PipedInputStream.read(PipedInputStream.java:372)
> at java.lang.ProcessInputStream.read(UNIXProcess.java:471)
> at
> sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:464)
> at
> sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:506)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:234)
> at java.io.InputStreamReader.read(InputStreamReader.java:188)
> at java.io.BufferedReader.fill(BufferedReader.java:147)
> at java.io.BufferedReader.readLine(BufferedReader.java:310)
> at java.io.BufferedReader.readLine(BufferedReader.java:373)
> at
> org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:272)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:452)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
> at java.util.concurrent.FutureTask.run(FutureTask.java:149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
> at java.lang.Thread.run(Thread.java:738)
> 14/02/17 10:32:58 INFO source.ExecSource: Command [tail -F
> /home/biadmin/bigdemo/data/rec_telco.cdr] exited with 130
> 14/02/17 10:32:58 INFO nodemanager.DefaultLogicalNodeManager: Stopping
> Sink hbase-sink
> 14/02/17 10:32:58 INFO lifecycle.LifecycleSupervisor: Stopping component:
> SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@4c004ccounterGroup:{ name:null counters:{runner.backoffs.consecutive=2,
> runner.backoffs=59} } }
> 14/02/17 10:32:58 INFO lifecycle.LifecycleSupervisor: Component has
> already been stopped EventDrivenSourceRunner: {
> source:org.apache.flume.source.ExecSource{name:exec-source,state:STOP} }
> 14/02/17 10:32:58 WARN file.Log: Interrupted while waiting for log shared
> lock
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1035)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1314)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:839)
> at org.apache.flume.channel.file.Log.tryLockShared(Log.java:599)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:446)
> at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> at
> org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:190)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:738)
> 14/02/17 10:32:58 ERROR flume.SinkRunner: Unable to deliver event.
> Exception follows.
> org.apache.flume.ChannelException: Failed to obtain lock for writing to
> the log. Try increasing the log write timeout value. [channel=ch2]
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:447)
> at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> at
> org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:190)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:738)
> 14/02/17 10:32:58 INFO
> client.HConnectionManager$HConnectionImplementation: Closed zookeeper
> sessionid=0x144401355b4001d
> 14/02/17 10:32:58 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 1796885
> logWriteOrderID: 1392650786415
> 14/02/17 10:32:57 WARN hdfs.BucketWriter: Caught IOException writing to
> HDFSWriter (Filesystem closed). Closing file
> (hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998186.tmp)
> and rethrowing exception.
> 14/02/17 10:32:58 WARN hdfs.BucketWriter: Caught IOException while closing
> file
> (hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998186.tmp).
> Exception follows.
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:319)
> at
> org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1026)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:524)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768)
> at
> org.apache.flume.sink.hdfs.BucketWriter.renameBucket(BucketWriter.java:426)
> at
> org.apache.flume.sink.hdfs.BucketWriter.doClose(BucketWriter.java:298)
> at
> org.apache.flume.sink.hdfs.BucketWriter.access$400(BucketWriter.java:53)
> at
> org.apache.flume.sink.hdfs.BucketWriter$3.run(BucketWriter.java:260)
> at
> org.apache.flume.sink.hdfs.BucketWriter$3.run(BucketWriter.java:258)
> at
> org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:143)
> at
> org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:258)
> at
> org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:382)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink$2.call(HDFSEventSink.java:729)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink$2.call(HDFSEventSink.java:727)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
> at java.util.concurrent.FutureTask.run(FutureTask.java:149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
> at java.lang.Thread.run(Thread.java:738)
> 14/02/17 10:32:58 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 1
> 14/02/17 10:32:58 INFO hdfs.BucketWriter: HDFSWriter is already closed:
> hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998186.tmp
> 14/02/17 10:32:58 ERROR hdfs.BucketWriter: Unexpected error
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:319)
> at
> org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1026)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:524)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768)
> at
> org.apache.flume.sink.hdfs.BucketWriter.renameBucket(BucketWriter.java:426)
> at
> org.apache.flume.sink.hdfs.BucketWriter.doClose(BucketWriter.java:298)
> at
> org.apache.flume.sink.hdfs.BucketWriter.access$400(BucketWriter.java:53)
> at
> org.apache.flume.sink.hdfs.BucketWriter$3.run(BucketWriter.java:260)
> at
> org.apache.flume.sink.hdfs.BucketWriter$3.run(BucketWriter.java:258)
> at
> org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:143)
> at
> org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:258)
> at
> org.apache.flume.sink.hdfs.BucketWriter$2.call(BucketWriter.java:237)
> at
> org.apache.flume.sink.hdfs.BucketWriter$2.call(BucketWriter.java:232)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
> at java.util.concurrent.FutureTask.run(FutureTask.java:149)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:109)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:217)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
> at java.lang.Thread.run(Thread.java:738)
> 14/02/17 10:32:58 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650786418, queueSize: 499,
> queueHead: 226331
> 14/02/17 10:32:58 INFO zookeeper.ZooKeeper: Session: 0x144401355b4001d
> closed
> 14/02/17 10:32:58 INFO nodemanager.DefaultLogicalNodeManager: Stopping
> Sink hdfs-sink
> 14/02/17 10:32:58 INFO lifecycle.LifecycleSupervisor: Stopping component:
> SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@7b017b01counterGroup:{ name:null counters:{runner.backoffs.consecutive=3,
> runner.backoffs=53} } }
> 14/02/17 10:32:58 WARN file.Log: Interrupted while waiting for log shared
> lock
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1035)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1314)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:839)
> at org.apache.flume.channel.file.Log.tryLockShared(Log.java:599)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:536)
> at
> org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:455)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:738)
> 14/02/17 10:32:58 ERROR flume.SinkRunner: Unable to deliver event.
> Exception follows.
> org.apache.flume.ChannelException: Failed to obtain lock for writing to
> the log. Try increasing the log write timeout value. [channel=ch1]
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:539)
> at
> org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:455)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:738)
> 14/02/17 10:32:58 INFO hdfs.HDFSEventSink: Closing
> hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec
> 14/02/17 10:32:58 INFO zookeeper.ClientCnxn: EventThread shut down
> 14/02/17 10:32:58 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 1795990, logWriteOrderID = 1392650786418
> 14/02/17 10:32:58 INFO hdfs.BucketWriter: HDFSWriter is already closed:
> hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998186.tmp
> 14/02/17 10:32:58 WARN hdfs.HDFSEventSink: Exception while closing
> hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec. Exception follows.
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:319)
> at
> org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1026)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:524)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768)
> at
> org.apache.flume.sink.hdfs.BucketWriter.renameBucket(BucketWriter.java:426)
> at
> org.apache.flume.sink.hdfs.BucketWriter.doClose(BucketWriter.java:298)
> at
> org.apache.flume.sink.hdfs.BucketWriter.access$400(BucketWriter.java:53)
> at
> org.apache.flume.sink.hdfs.BucketWriter$3.run(BucketWriter.java:260)
> at
> org.apache.flume.sink.hdfs.BucketWriter$3.run(BucketWriter.java:258)
> at
> org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:143)
> at
> org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:258)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink$4.call(HDFSEventSink.java:757)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink$4.call(HDFSEventSink.java:755)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
> at java.util.concurrent.FutureTask.run(FutureTask.java:149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
> at java.lang.Thread.run(Thread.java:738)
> 14/02/17 10:32:58 INFO instrumentation.MonitoredCounterGroup: Component
> type: SINK, name: hdfs-sink stopped
> 14/02/17 10:32:58 INFO nodemanager.DefaultLogicalNodeManager: Stopping
> Channel ch1
> 14/02/17 10:32:58 INFO lifecycle.LifecycleSupervisor: Stopping component:
> FileChannel ch1 { dataDirs: [/home/biadmin/.flume/file-channel/data] }
> 14/02/17 10:32:58 INFO file.FileChannel: Stopping FileChannel ch1 {
> dataDirs: [/home/biadmin/.flume/file-channel/data] }...
> 14/02/17 10:32:58 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 1795990
> logWriteOrderID: 1392650786418
> 14/02/17 10:32:58 INFO file.LogFile: Closing
> /home/biadmin/.flume/file-channel/data/log-8
> 14/02/17 10:32:58 INFO file.LogFile: Closing RandomReader
> /home/biadmin/.flume/file-channel/data/log-7
> 14/02/17 10:32:58 INFO file.LogFile: Closing RandomReader
> /home/biadmin/.flume/file-channel/data/log-8
> 14/02/17 10:32:58 INFO instrumentation.MonitoredCounterGroup: Component
> type: CHANNEL, name: ch1 stopped
> 14/02/17 10:32:58 INFO nodemanager.DefaultLogicalNodeManager: Stopping
> Channel ch2
> 14/02/17 10:32:58 INFO lifecycle.LifecycleSupervisor: Stopping component:
> FileChannel ch2 { dataDirs: [/home/biadmin/.flume/file-channel2/data] }
> 14/02/17 10:32:58 INFO file.FileChannel: Stopping FileChannel ch2 {
> dataDirs: [/home/biadmin/.flume/file-channel2/data] }...
> 14/02/17 10:32:58 INFO file.LogFile: Closing
> /home/biadmin/.flume/file-channel2/data/log-7
> 14/02/17 10:32:58 INFO file.LogFile: Closing RandomReader
> /home/biadmin/.flume/file-channel2/data/log-6
> 14/02/17 10:32:58 INFO file.LogFile: Closing RandomReader
> /home/biadmin/.flume/file-channel2/data/log-7
> 14/02/17 10:32:58 INFO instrumentation.MonitoredCounterGroup: Component
> type: CHANNEL, name: ch2 stopped
> 14/02/17 10:32:58 INFO lifecycle.LifecycleSupervisor: Stopping lifecycle
> supervisor 9
>
>
>
> On 17 February 2014 16:38, Kris Ogirri <ka...@gmail.com> wrote:
>
> Hello Jeff,
>
> Please find below requested logs.. Initiation part of the logs were
> unfortunately not included. I can run these again if necessary but the
> Zookeeper connection is included in the logs.
>
>
>
> On 17 February 2014 16:05, Jeff Lord <jl...@cloudera.com> wrote:
>
> Logs ?
>
> On Mon, Feb 17, 2014 at 5:51 AM, Kris Ogirri <ka...@gmail.com> wrote:
> > Dear Mailing Group,
> >
> > I am currently having issues with the Hbase sink function. I have
> developed
> > an agent with a fanout channel setup ( single source, multiple channels,
> > multiple sinks) sinking to a HDFS cluster and Hbase deployment.
> >
> > The issue is that although the HDFS is working well, the Hbase flow is
> > simply not working. There are no errors being reported by Flume for the
> > Hbase channel but there are never any records being written to the HBase
> > store. The Hbase table as stipulated in the config always remains empty.
> > Studying the Flume startup logs I observe that the session connection to
> > Zookeeper is always successfully established
> >
> > Are there any special configurations I am missing out?
> >
> > I am using the Async Event Serializer to persist the txns.
> >
> > Any assistance will be greatly appreciated.
> >
> >
> > Please see below for the flume configuration:
> >
> > [biadmin@bivm bin]$ cat flume-conf.properties.bigdemo
> > agent.sources=exec-source
> > agent.sinks=hdfs-sink hbase-sink
> > agent.channels=ch1 ch2
> >
> > agent.sources.exec-source.type=exec
> > agent.sources.exec-source.command=tail -F
> > /home/biadmin/bigdemo/data/rec_telco.cdr
> >
> > agent.sinks.hdfs-sink.type=hdfs
> > agent.sinks.hdfs-sink.hdfs.path=hdfs://XXXX:9000/user/biadmin/bigdemo/
> > agent.sinks.hdfs-sink.hdfs.filePrefix=telco_cdr_rec
> > # File size to trigger roll, in bytes (0: never roll based on file size)
> > agent.sinks.hdfs-sink.hdfs.rollSize = 134217728
> > agent.sinks.hdfs-sink.hdfs.rollCount = 0
> > # number of events written to file before it flushed to HDFS
> > agent.sinks.hdfs-sink.hdfs.batchSize = 10000
> > agent.sinks.hdfs-sink.hdfs.txnEventMax = 40000
> >
> >
> > agent.sinks.hbase-sink.type=org.apache.flume.sink.hbase.AsyncHBaseSink
> >
> agent.sinks.hbase-sink.serializer=org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer
> > agent.sinks.hbase-sink.table=telco_cdr_rec
> > agent.sinks.hbase-sink.columnFamily = colfam
> > agent.sinks.hbase-sink.channels = ch2
> > #agent.sinks.hbase-sink.hdfs.batchSize = 10000
> > #agent.sinks.hbase-sink.hdfs.txnEventMax = 40000
> >
> >
> > agent.channels.ch1.type=file
> > agent.channels.ch1.checkpointInterval=3000
> > agent.channels.ch1.transactionCapacity=10000
> >
> agent.channels.ch1.checkpointDir=/home/BDadmin/.flume/file-channel/checkpoint
> > agent.channels.ch1.dataDirs=/home/BDadmin/.flume/file-channel/data
> > agent.channels.ch1.write-timeout=30
> > agent.channels.ch1.keep-alive=30
> > #agent.channels.ch1.capacity=1000
> >
> > agent.channels.ch2.type=file
> > agent.channels.ch2.checkpointInterval=300
> > agent.channels.ch2.transactionCapacity=10000
> >
> agent.channels.ch2.checkpointDir=/home/BDadmin/.flume/file-channel2/checkpoint
> > agent.channels.ch2.dataDirs=/home/BDadmin/.flume/file-channel2/data
> > agent.channels.ch2.write-timeout=30
> > agent.channels.ch2.keep-alive=30
> > #agent.channels.ch2.capacity=1000
> >
> >
> > agent.sources.exec-source.channels=ch1 ch2
> > agent.sinks.hdfs-sink.channel=ch1
> > agent.sinks.hbase-sink.channel=ch2
> >
>
>
>
>
>
>
>
>
>
>