You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "Kun Liu (Jira)" <ji...@apache.org> on 2021/07/02 05:32:00 UTC
[jira] [Commented] (KYLIN-5024) stream receiver NPE: when restart
the receiver server
[ https://issues.apache.org/jira/browse/KYLIN-5024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17373237#comment-17373237 ]
Kun Liu commented on KYLIN-5024:
--------------------------------
When one partition is empty, there is no partition offset stored in the zk.
> stream receiver NPE: when restart the receiver server
> -----------------------------------------------------
>
> Key: KYLIN-5024
> URL: https://issues.apache.org/jira/browse/KYLIN-5024
> Project: Kylin
> Issue Type: Bug
> Reporter: Kun Liu
> Assignee: Kun Liu
> Priority: Major
>
> {code:java}
> 2021-06-22 14:13:08,299 INFO [qtp1551350415-80] rheos.RheosSource:109 : Create RheosStreamingConnector for Cube msg_tracker_streaming_v_cube, assignedPartitions [Partition [partitionId=32, partitionInfo=null], Partition [partitionId=35, partitionInfo=null], Partition [partitionId=17, partitionInfo=null], Partition [partitionId=26, partitionInfo=null], Partition [partitionId=29, partitionInfo=null], Partition [partitionId=38, partitionInfo=null], Partition [partitionId=2, partitionInfo=null], Partition [partitionId=14, partitionInfo=null], Partition [partitionId=40, partitionInfo=null], Partition [partitionId=13, partitionInfo=null], Partition [partitionId=22, partitionInfo=null], Partition [partitionId=16, partitionInfo=null], Partition [partitionId=43, partitionInfo=null], Partition [partitionId=34, partitionInfo=null], Partition [partitionId=37, partitionInfo=null], Partition [partitionId=28, partitionInfo=null], Partition [partitionId=45, partitionInfo=null], Partition [partitionId=27, partitionInfo=null], Partition [partitionId=9, partitionInfo=null], Partition [partitionId=48, partitionInfo=null], Partition [partitionId=3, partitionInfo=null], Partition [partitionId=39, partitionInfo=null], Partition [partitionId=42, partitionInfo=null], Partition [partitionId=24, partitionInfo=null], Partition [partitionId=0, partitionInfo=null]], startProtocol null
> 2021-06-22 14:13:08,308 INFO [qtp1551350415-80] kafka.KafkaSource:142 : Create StreamingConnector for Cube msg_tracker_streaming_v_cube, assignedPartitions [Partition [partitionId=32, partitionInfo=null], Partition [partitionId=35, partitionInfo=null], Partition [partitionId=17, partitionInfo=null], Partition [partitionId=26, partitionInfo=null], Partition [partitionId=29, partitionInfo=null], Partition [partitionId=38, partitionInfo=null], Partition [partitionId=2, partitionInfo=null], Partition [partitionId=14, partitionInfo=null], Partition [partitionId=40, partitionInfo=null], Partition [partitionId=13, partitionInfo=null], Partition [partitionId=22, partitionInfo=null], Partition [partitionId=16, partitionInfo=null], Partition [partitionId=43, partitionInfo=null], Partition [partitionId=34, partitionInfo=null], Partition [partitionId=37, partitionInfo=null], Partition [partitionId=28, partitionInfo=null], Partition [partitionId=45, partitionInfo=null], Partition [partitionId=27, partitionInfo=null], Partition [partitionId=9, partitionInfo=null], Partition [partitionId=48, partitionInfo=null], Partition [partitionId=3, partitionInfo=null], Partition [partitionId=39, partitionInfo=null], Partition [partitionId=42, partitionInfo=null], Partition [partitionId=24, partitionInfo=null], Partition [partitionId=0, partitionInfo=null]], startProtocol null
> 2021-06-22 13:47:46,490 INFO [main] kafka.KafkaSource:194 : localConsumeStats from local checkpoint null, remoteConsumeStats from remote checkpoint {"7":2361745719,"10":2361745724,"11":2361745695,"12":2361745778,"13":2361745763,"14":2361745673,"15":2361745433,"16":2361745759,"17":2361745772,"18":2361745744,"24":2361745716,"26":2361745715,"27":2361745762}
> 2021-06-22 13:47:46,491 INFO [main] kafka.KafkaSource:241 : start kafka connector from specified position:KafkaPosition{partitionOffsetMap={7=2361745719, 10=2361745724, 11=2361745695, 12=2361745778, 13=2361745763, 14=2361745673, 15=2361745433, 16=2361745759, 17=2361745772, 18=2361745744, 24=2361745716, 26=2361745715, 27=2361745762}}
> 2021-06-22 13:47:46,492 ERROR [main] server.StreamingServer:382 : consumer start fail for cube:xxxxx
> java.lang.NullPointerException
> at org.apache.kylin.stream.source.kafka.consumer.KafkaConnector.open(KafkaConnector.java:89)
> at org.apache.kylin.stream.core.consumer.StreamingConsumerChannel.start(StreamingConsumerChannel.java:87)
> at org.apache.kylin.stream.server.StreamingServer.startConsumer(StreamingServer.java:380)
> at org.apache.kylin.stream.server.StreamingServer.startConsumers(StreamingServer.java:352)
> at org.apache.kylin.stream.server.StreamingServer.addToReplicaSet(StreamingServer.java:462)
> at org.apache.kylin.stream.server.StreamingServer.start(StreamingServer.java:160)
> at org.apache.kylin.stream.server.StreamingReceiver.startStreamingServer(StreamingReceiver.java:67)
> at org.apache.kylin.stream.server.StreamingReceiver.start(StreamingReceiver.java:61)
> at org.apache.kylin.stream.server.StreamingReceiver.main(StreamingReceiver.java:51)
> {code}
> However, assignments for cube msg_tracker_streaming_v_cube in zookeeper is
> {code:java}
> zk: get stream/cubes/msg_tracker_streaming_v_cube/assignment {"cube_name":"xxxx","assignments":{ "0":[{"partition_id":23,"partition_info":null},{"partition_id":41,"partition_info":null},{"partition_id":44,"partition_info":null},{"partition_id":8,"partition_info":null},{"partition_id":11,"partition_info":null},{"partition_id":47,"partition_info":null},{"partition_id":20,"partition_info":null},{"partition_id":5,"partition_info":null},{"partition_id":46,"partition_info":null},{"partition_id":49,"partition_info":null},{"partition_id":4,"partition_info":null},{"partition_id":31,"partition_info":null},{"partition_id":7,"partition_info":null},{"partition_id":25,"partition_info":null},{"partition_id":10,"partition_info":null},{"partition_id":1,"partition_info":null},{"partition_id":19,"partition_info":null},{"partition_id":36,"partition_info":null},{"partition_id":18,"partition_info":null},{"partition_id":21,"partition_info":null},{"partition_id":12,"partition_info":null},{"partition_id":30,"partition_info":null},{"partition_id":15,"partition_info":null},{"partition_id":33,"partition_info":null},{"partition_id":6,"partition_info":null}], "1":[{"partition_id":32,"partition_info":null},{"partition_id":35,"partition_info":null},{"partition_id":17,"partition_info":null},{"partition_id":26,"partition_info":null},{"partition_id":29,"partition_info":null},{"partition_id":38,"partition_info":null},{"partition_id":2,"partition_info":null},{"partition_id":14,"partition_info":null},{"partition_id":40,"partition_info":null},{"partition_id":13,"partition_info":null},{"partition_id":22,"partition_info":null},{"partition_id":16,"partition_info":null},{"partition_id":43,"partition_info":null},{"partition_id":34,"partition_info":null},{"partition_id":37,"partition_info":null},{"partition_id":28,"partition_info":null},{"partition_id":45,"partition_info":null},{"partition_id":27,"partition_info":null},{"partition_id":9,"partition_info":null},{"partition_id":48,"partition_info":null},{"partition_id":3,"partition_info":null},{"partition_id":39,"partition_info":null},{"partition_id":42,"partition_info":null},{"partition_id":24,"partition_info":null},{"partition_id":0,"partition_info":null}]}}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)