You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "Shaofeng SHI (JIRA)" <ji...@apache.org> on 2017/01/05 06:31:58 UTC
[jira] [Closed] (KYLIN-2281) Pending on the first step(save data
from kafka) when building streaming cube
[ https://issues.apache.org/jira/browse/KYLIN-2281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shaofeng SHI closed KYLIN-2281.
-------------------------------
Resolution: Pending Closed
> Pending on the first step(save data from kafka) when building streaming cube
> ----------------------------------------------------------------------------
>
> Key: KYLIN-2281
> URL: https://issues.apache.org/jira/browse/KYLIN-2281
> Project: Kylin
> Issue Type: Test
> Components: Client - CLI
> Affects Versions: v1.6.0
> Environment: CDH5.8 + Kylin1.6 + Kafka0.10
> Reporter: QiLiFei
> Priority: Critical
>
> Hi engineer,
> 1.
> I tried several times to build the streaming cube based on kafka source . However , it will always pending on the first step(already take about 160 mins ) to save data from kafka. I checked the data from kafka . There are only about one thousand data . Which is confused for me . Anyone who can help me ? I'm not sure what to do next . Is there configuration for kafka or kylin need to reset or not ?
> 2.Command
> curl -X PUT --user ADMIN:KYLIN -H "Content-Type: application/json;charset=utf-8" -d '{ "sourceOffsetStart": 0, "sourceOffsetEnd": 9223372036854775807, "buildType": "BUILD"}' http://172.31.18.62:7070/kylin/api/cubes/StreamingCube27/build2
> 3. config and log
> I set the available mb for kylin 16G of 20G. I can see the continously log info by command tail -f kylin.log as below . Currenty I've stop the kafka producer for this topic.
> 2016-12-15 11:19:41,457 DEBUG [pool-9-thread-1] dao.ExecutableDao:210 : updating job output, id: dfb72486-50b8-44c1-bee5-4ad0393ef66e-00
> 2016-12-15 11:19:41,462 DEBUG [pool-9-thread-1] hbase.HBaseResourceStore:262 : Update row /execute_output/dfb72486-50b8-44c1-bee5-4ad0393ef66e-00 from oldTs: 1481771971443, to newTs: 1481771981457, operation result: true
> 2016-12-15 11:19:51,505 DEBUG [pool-9-thread-1] dao.ExecutableDao:210 : updating job output, id: dfb72486-50b8-44c1-bee5-4ad0393ef66e-00
> 2016-12-15 11:19:51,510 DEBUG [pool-9-thread-1] hbase.HBaseResourceStore:262 : Update row /execute_output/dfb72486-50b8-44c1-bee5-4ad0393ef66e-00 from oldTs: 1481771981457, to newTs: 1481771991505, operation result: true
> 4.Kafka souce data count （sourceOffset & endOffset）
> phadoopdn1:/opt/kafka_2.11-0.10.1.0 # bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 172.31.18.62:9092 --topic kylin_demo30 --time -1
> kylin_demo30: 0:70572
> phadoopdn1:/opt/kafka_2.11-0.10.1.0 #
> Anyone who can help me to figure out this issue ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)