You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flume.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/01/16 09:21:00 UTC
[jira] [Commented] (FLUME-2956) hive sink not sending heartbeat
correctly
[ https://issues.apache.org/jira/browse/FLUME-2956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16326919#comment-16326919 ]
ASF GitHub Bot commented on FLUME-2956:
---------------------------------------
Github user hejiang2000 closed the pull request at:
https://github.com/apache/flume/pull/190
> hive sink not sending heartbeat correctly
> -----------------------------------------
>
> Key: FLUME-2956
> URL: https://issues.apache.org/jira/browse/FLUME-2956
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: 1.5.2
> Environment: linux CentOS 6.6
> Reporter: Olivier brobecker
> Priority: Major
> Attachments: flume-hive-sink-1.8.0.jar, hive-sink-patch.txt
>
>
> Flume has been configured in order to populate a hive table as follow :
> agentSCDR.sinks.hive1.type = hive
> agentSCDR.sinks.hive1.channel = channel1
> agentSCDR.sinks.hive1.hive.metastore = thrift://myserver:9083
> agentSCDR.sinks.hive1.hive.txnsPerBatchAsk = 10
> agentSCDR.sinks.hive1.hive.database = myDatabase
> agentSCDR.sinks.hive1.hive.table = my_table
> agentSCDR.sinks.hive1.hive.partition = %Y
> agentSCDR.sinks.hive1.heartBeatInterval = 60
> agentSCDR.sinks.hive1.serializer = DELIMITED
> agentSCDR.sinks.hive1.serializer.delimiter = ;
> agentSCDR.sinks.hive1.serializer.serdeSeparator = ;
> agentSCDR.sinks.hive1.serializer.fieldnames = field1,field2,field3...
> My dataflow is irregular at best and I can have more than 30 minutes of inactivity, so I have a heartbeatInterval at 60s in order to keep my txn alive.
> The issue is that this heartBeat is only sent when flume is trying to inject some datas into hive instead of having one every 60s.
> # grep -i heartbeat flume-agentSCDR.log
> 15 juil. 2016 13:40:43,008 INFO [hive-hive1-call-runner-0] (org.apache.flume.sink.hive.HiveWriter$2.call:238) - Sending heartbeat on batch TxnIds=[3755...3764] on endPoint = {metaStoreUri=...
> 15 juil. 2016 14:12:21,001 INFO [hive-hive1-call-runner-0] (org.apache.flume.sink.hive.HiveWriter$2.call:231) - Sending heartbeat on batch TxnIds=[3785...3794] on endPoint = {metaStoreUri=...
> 15 juil. 2016 14:27:56,963 INFO [hive-hive1-call-runner-0] (org.apache.flume.sink.hive.HiveWriter$2.call:231) - Sending heartbeat on batch TxnIds=[3795...3804] on endPoint = {metaStoreUri=...
> ...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org