You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "hailin (JIRA)" <ji...@apache.org> on 2016/09/24 10:15:20 UTC
[jira] [Comment Edited] (FLUME-2996) Flume process shutdown
automaticly
[ https://issues.apache.org/jira/browse/FLUME-2996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15518810#comment-15518810 ]
hailin edited comment on FLUME-2996 at 9/24/16 10:14 AM:
---------------------------------------------------------
hi bellow is the configure file content:
a1.sources = r1
a1.sinks = sink_etl_product sink_kinesis_stream
a1.channels = channel_etl_product channel_kinesis_stream
#source setting
#a1.sources.r1.type = http
a1.sources.r1.type = flume.source.HTTPSource
a1.sources.r1.port = 6558
a1.sources.r1.dataSize = 1
a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.header = event_type
a1.sources.r1.selector.mapping.etl_product = channel_etl_product channel_kinesis_stream
#handler setting
a1.sources.r1.handler = flume.source.BlobExtHandler
a1.sources.r1.handler.gzip = true
a1.sources.r1.handler.maxBlobLength = 100000
a1.channels.channel_etl_product.type = file
a1.channels.channel_etl_product.checkpointDir = ./checkpoint/product
a1.channels.channel_etl_product.dataDirs = ./data/product
a1.channels.channel_etl_product.checkpointInterval = 10
#Kinesis Firehose
a1.sinks.sink_etl_mix.type = flume.sink.KinesisFirehoseSink
a1.sinks.sink_etl_mix.channel = channel_etl_product
a1.sinks.sink_etl_mix.streamName = product_flumeFirehoseStream
a1.sinks.sink_etl_mix.endpoint = {firehose_endpoint}
flume.source.HTTPSource,flume.source.BlobExtHandler,flume.sink.KinesisFirehoseSink, these 3 Java class contains our bussiness logic.
what I want is that the flume will transfer data into AWS Kinesis Firehose.
The application works fine on AWS EC2 linux instance. What I meet is flume shutdown automaticly after several hours , the process disappear when run "ps -ef | grep flume".
I check the flume log, no errors found, seems it shutdown normally:
24 Sep 2016 05:53:59,391 INFO [agent-shutdown-hook] (org.apache.flume.lifecycle.LifecycleSupervisor.stop:79) - Stopping lifecycle supervisor 10
24 Sep 2016 05:53:59,394 INFO [agent-shutdown-hook] (org.apache.flume.channel.file.FileChannel.stop:321) - Stopping FileChannel channel_etl_product { dataDirs: [./data/product] }...
24 Sep 2016 05:53:59,395 INFO [agent-shutdown-hook] (org.apache.flume.channel.file.EventQueueBackingStoreFile.beginCheckpoint:230) - Start checkpoint for ./checkpoint/product/checkpoint, elements to sync = 0
24 Sep 2016 05:53:59,401 INFO [agent-shutdown-hook] (org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:255) - Updating checkpoint metadata: logWriteOrderID: 1474685940834, queueSize: 0, queueHead: 0
24 Sep 2016 05:53:59,407 INFO [agent-shutdown-hook] (org.apache.flume.channel.file.Log.writeCheckpoint:1034) - Updated checkpoint for file: ./data/product/log-3 position: 6674 logWriteOrderID: 1474685940834
24 Sep 2016 05:53:59,407 INFO [agent-shutdown-hook] (org.apache.flume.channel.file.Log.shutdownWorker:857) - Attempting to shutdown background worker.
24 Sep 2016 05:53:59,409 INFO [agent-shutdown-hook] (org.apache.flume.channel.file.LogFile$Writer.close:370) - Closing ./data/product/log-3
24 Sep 2016 05:53:59,409 INFO [agent-shutdown-hook] (org.apache.flume.channel.file.LogFile$RandomReader.close:504) - Closing RandomReader ./data/product/log-2
24 Sep 2016 05:53:59,414 INFO [agent-shutdown-hook] (org.apache.flume.channel.file.LogFile$RandomReader.close:504) - Closing RandomReader ./data/product/log-3
24 Sep 2016 05:53:59,420 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:150) - Component type: CHANNEL, name: channel_etl_product stopped
please let me know if need more information.
thanks
was (Author: hehailin):
hi bellow is the configure file content:
a1.sources = r1
a1.sinks = sink_etl_product sink_kinesis_stream
a1.channels = channel_etl_product channel_kinesis_stream
#source setting
#a1.sources.r1.type = http
a1.sources.r1.type = flume.source.HTTPSource
a1.sources.r1.port = 6558
a1.sources.r1.dataSize = 1
a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.header = event_type
a1.sources.r1.selector.mapping.etl_product = channel_etl_product channel_kinesis_stream
#handler setting
a1.sources.r1.handler = flume.source.BlobExtHandler
a1.sources.r1.handler.gzip = true
a1.sources.r1.handler.maxBlobLength = 100000
a1.channels.channel_etl_product.type = file
a1.channels.channel_etl_product.checkpointDir = ./checkpoint/product
a1.channels.channel_etl_product.dataDirs = ./data/product
a1.channels.channel_etl_product.checkpointInterval = 10
#Kinesis Firehose
a1.sinks.sink_etl_mix.type = flume.sink.KinesisFirehoseSink
a1.sinks.sink_etl_mix.channel = channel_etl_product
a1.sinks.sink_etl_mix.streamName = product_flumeFirehoseStream
a1.sinks.sink_etl_mix.endpoint = {firehose_endpoint}
flume.source.HTTPSource,flume.source.BlobExtHandler,flume.sink.KinesisFirehoseSink, these 3 Java class contains our bussiness logic.
what I want is that the flume will transfer data into AWS Kinesis Firehose.
The application works fine on AWS EC2 linux instance. What I meet is flume shutdown automaticly after several hours , the process disappear when run "ps -ef | grep flume".
please let me know if need more information.
thanks
> Flume process shutdown automaticly
> ----------------------------------
>
> Key: FLUME-2996
> URL: https://issues.apache.org/jira/browse/FLUME-2996
> Project: Flume
> Issue Type: Bug
> Environment: Amazon Linux
> Reporter: hailin
>
> hi guys,
> I install flume from apache-flume-1.6.0-bin.tar.gz, and startup flume with some my custom configure file with cmd line like this:
> bin/flume-ng agent -Dflume.monitoring.type=http -Dflume.monitoring.port=41414 -c conf -f conf/flume.conf --name a1 -Dflume.root.logger=INFO,DAILY
> it works fine without any issue, but after several hours when I check the process, I found it was shutdown automaticly. Bellow is the flume log information:
> .....
> 24 Sep 2016 05:53:59,391 INFO [agent-shutdown-hook] (org.apache.flume.lifecycle.LifecycleSupervisor.stop:79) - Stopping lifecycle supervisor 10
> 24 Sep 2016 05:53:59,394 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:150) - Component type: SINK, name: sink_kinesis_stream stopped
> 24 Sep 2016 05:53:59,394 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:156) - Shutdown Metric for type: SINK, name: sink_kinesis_stream. sink.start.time == 1474685921906
> ....
> Can anyone help me fix it? How can I do to stop this issue as our bussiness already block because of this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)