You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by GitBox <gi...@apache.org> on 2020/08/06 10:35:28 UTC
[GitHub] [kylin] tianhui5 opened a new pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
tianhui5 opened a new pull request #1351:
URL: https://github.com/apache/kylin/pull/1351
## Proposed changes
JIRA：https://issues.apache.org/jira/browse/KYLIN-4683
## Types of changes
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Documentation Update (if none of the other choices apply)
## Checklist
- [x] I have create an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail
- [ ] Commit messages in my PR start with the related jira ID, like "KYLIN-0000 Make Kylin project open-source"
- [ ] Compiling and unit tests pass locally with my changes
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] If this change need a document change, I will prepare another pr against the `document` branch
- [ ] Any dependent changes have been merged
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-671064075
1. I will kill the producer.
2. check latest offset via Python
3. Resize partition count.
```sh
[root@cdh-client xiaoxiang.yu]# kafka-topics --alter --topic useraction_xxyu --zookeeper cdh-master:2181 --partitions 16
20/08/09 23:13:33 INFO utils.Log4jControllerRegistration$: Registered kafka:type=kafka.Log4jController MBean
20/08/09 23:13:34 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Initializing a new session to cdh-master:2181.
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-cdh5.14.2--1, built on 03/27/2018 20:39 GMT
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:host.name=cdh-client
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_171
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.8.0_171/jre
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-754.17.1.el6.x86_64
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:user.name=root
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:user.home=/root
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:user.dir=/root/xiaoxiang.yu
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=cdh-master:2181 sessionTimeout=30000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@6302bbb1
20/08/09 23:13:34 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Waiting until connected.
20/08/09 23:13:34 INFO zookeeper.ClientCnxn: Opening socket connection to server cdh-master/10.1.3.90:2181. Will not attempt to authenticate using SASL (unknown error)
20/08/09 23:13:34 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.1.3.91:54122, server: cdh-master/10.1.3.90:2181
20/08/09 23:13:34 INFO zookeeper.ClientCnxn: Session establishment complete on server cdh-master/10.1.3.90:2181, sessionid = 0x172bdedb56fcc33, negotiated timeout = 30000
20/08/09 23:13:34 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Connected.
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
20/08/09 23:13:34 INFO zk.AdminZkClient: Creating 6 partitions for 'useraction_xxyu' with the following replica assignment: Map(11 -> ArrayBuffer(115), 14 -> ArrayBuffer(115), 13 -> ArrayBuffer(114), 10 -> ArrayBuffer(114), 12 -> ArrayBuffer(116), 15 -> ArrayBuffer(116)).
20/08/09 23:13:34 INFO zk.AdminZkClient: Topic update Map(useraction_xxyu-8 -> Vector(115), useraction_xxyu-6 -> Vector(116), useraction_xxyu-0 -> Vector(116), useraction_xxyu-13 -> ArrayBuffer(114), useraction_xxyu-1 -> Vector(114), useraction_xxyu-9 -> Vector(116), useraction_xxyu-15 -> ArrayBuffer(116), useraction_xxyu-11 -> ArrayBuffer(115), useraction_xxyu-5 -> Vector(115), useraction_xxyu-2 -> Vector(115), useraction_xxyu-10 -> ArrayBuffer(114), useraction_xxyu-14 -> ArrayBuffer(115), useraction_xxyu-7 -> Vector(114), useraction_xxyu-12 -> ArrayBuffer(116), useraction_xxyu-4 -> Vector(114), useraction_xxyu-3 -> Vector(116))
Adding partitions succeeded!
20/08/09 23:13:34 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Closing.
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Session: 0x172bdedb56fcc33 closed
20/08/09 23:13:34 INFO zookeeper.ClientCnxn: EventThread shut down
20/08/09 23:13:34 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Closed.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-671065519
1. Go to web page and re-enable Cube, you will see some failure, please don't be in a panic.
```sh
2020-08-09 23:23:10,437 INFO [qtp193625667-35] controller.AdminController:58 : receive assign request:AssignRequest{cubeName='UserAnalysisCube', partitions=[Partition [partitionId=0, partitionInfo=null], Partition [partitionId=5, partitionInfo=null], Partition [partitionId=10, partitionInfo=null], Partition [partitionId=15, partitionInfo=null], Partition [partitionId=13, partitionInfo=null], Partition [partitionId=2, partitionInfo=null], Partition [partitionId=8, partitionInfo=null], Partition [partitionId=12, partitionInfo=null], Partition [partitionId=14, partitionInfo=null], Partition [partitionId=9, partitionInfo=null], Partition [partitionId=11, partitionInfo=null], Partition [partitionId=1, partitionInfo=null], Partition [partitionId=4, partitionInfo=null], Partition [partitionId=6, partitionInfo=null], Partition [partitionId=7, partitionInfo=null], Partition [partitionId=3, partitionInfo=null]], startConsumers=true}
2020-08-09 23:23:10,437 INFO [qtp193625667-35] server.StreamingServer:351 : create and start new consumer for cube:UserAnalysisCube
2020-08-09 23:23:10,468 INFO [qtp193625667-35] kafka.KafkaSource:141 : Create StreamingConnector for Cube UserAnalysisCube, assignedPartitions [Partition [partitionId=0, partitionInfo=null], Partition [partitionId=5, partitionInfo=null], Partition [partitionId=10, partitionInfo=null], Partition [partitionId=15, partitionInfo=null], Partition [partitionId=13, partitionInfo=null], Partition [partitionId=2, partitionInfo=null], Partition [partitionId=8, partitionInfo=null], Partition [partitionId=12, partitionInfo=null], Partition [partitionId=14, partitionInfo=null], Partition [partitionId=9, partitionInfo=null], Partition [partitionId=11, partitionInfo=null], Partition [partitionId=1, partitionInfo=null], Partition [partitionId=4, partitionInfo=null], Partition [partitionId=6, partitionInfo=null], Partition [partitionId=7, partitionInfo=null], Partition [partitionId=3, partitionInfo=null]], startProtocol null
2020-08-09 23:23:10,470 INFO [qtp193625667-35] kafka.TimedJsonStreamParser:98 : Using parser field mapping by {content_list=content_list, act_type=act_type, event_ts_2=event_ts_2, event_ts=event_ts, devide_type=user_detail.devide_type, location_city=user_detail.location.city, network_type=user_detail.network_type, video_id=video_id, event_date_2=event_date_2, str_minute=str_minute, video_type=video_type, play_times=play_times, pageview_id=pageview_id, active_minutes=active_minutes, device_brand=device_brand, str_minute_second=str_minute_second, play_duration=play_duration, event_date=event_date, page_id=page_id, str_second=str_second, uid=uid}
2020-08-09 23:23:10,470 INFO [qtp193625667-35] kafka.TimedJsonStreamParser:120 : TimedJsonStreamParser with formatTs false tsColName event_date
2020-08-09 23:23:10,470 INFO [qtp193625667-35] consumer.ConsumerConfig:279 : ConsumerConfig values:
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [cdh-master:9092, cdh-worker-1:9092, cdh-worker-2:9092]
check.crcs = true
client.dns.lookup = default
client.id =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id =
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
isolation.level = read_uncommitted
key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
session.timeout.ms = 20000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
2020-08-09 23:23:10,473 INFO [qtp193625667-35] utils.AppInfoParser:109 : Kafka version : 2.1.1
2020-08-09 23:23:10,473 INFO [qtp193625667-35] utils.AppInfoParser:110 : Kafka commitId : 21234bee31165527
2020-08-09 23:23:10,475 INFO [qtp193625667-35] kafka.KafkaSource:192 : localConsumeStats from local checkpoint null, remoteConsumeStats from remote checkpoint {"0":363171,"1":363198,"2":363249,"3":363171,"4":363199,"5":363250,"6":363170,"7":363196,"8":363250,"9":363170}
2020-08-09 23:23:10,475 INFO [qtp193625667-35] kafka.KafkaSource:239 : start kafka connector from specified position:KafkaPosition{partitionOffsetMap={0=363171, 1=363198, 2=363249, 3=363171, 4=363199, 5=363250, 6=363170, 7=363196, 8=363250, 9=363170}}
2020-08-09 23:23:10,476 ERROR [qtp193625667-35] server.StreamingServer:357 : consumer start fail for cube:UserAnalysisCube
java.lang.NullPointerException
at org.apache.kylin.stream.source.kafka.consumer.KafkaConnector.open(KafkaConnector.java:86)
at org.apache.kylin.stream.core.consumer.StreamingConsumerChannel.start(StreamingConsumerChannel.java:85)
at org.apache.kylin.stream.server.StreamingServer.startConsumer(StreamingServer.java:355)
at org.apache.kylin.stream.server.StreamingServer.startConsumers(StreamingServer.java:327)
at org.apache.kylin.stream.server.rest.controller.AdminController.assign(AdminController.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:849)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:760)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:547)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:748)
```
2. Let try doing a reassignment.
<img width="1517" alt="image" src="https://user-images.githubusercontent.com/14030549/89735825-bd5b1000-da97-11ea-9944-ae54331002aa.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-671064075
1. I will kill the producer.
2. check latest offset via Python
3. Resize topic partition.
```sh
[root@cdh-client xiaoxiang.yu]# kafka-topics --alter --topic useraction_xxyu --zookeeper cdh-master:2181 --partitions 16
20/08/09 23:13:33 INFO utils.Log4jControllerRegistration$: Registered kafka:type=kafka.Log4jController MBean
20/08/09 23:13:34 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Initializing a new session to cdh-master:2181.
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-cdh5.14.2--1, built on 03/27/2018 20:39 GMT
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:host.name=cdh-client
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_171
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.8.0_171/jre
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-754.17.1.el6.x86_64
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:user.name=root
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:user.home=/root
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:user.dir=/root/xiaoxiang.yu
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=cdh-master:2181 sessionTimeout=30000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@6302bbb1
20/08/09 23:13:34 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Waiting until connected.
20/08/09 23:13:34 INFO zookeeper.ClientCnxn: Opening socket connection to server cdh-master/10.1.3.90:2181. Will not attempt to authenticate using SASL (unknown error)
20/08/09 23:13:34 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.1.3.91:54122, server: cdh-master/10.1.3.90:2181
20/08/09 23:13:34 INFO zookeeper.ClientCnxn: Session establishment complete on server cdh-master/10.1.3.90:2181, sessionid = 0x172bdedb56fcc33, negotiated timeout = 30000
20/08/09 23:13:34 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Connected.
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
20/08/09 23:13:34 INFO zk.AdminZkClient: Creating 6 partitions for 'useraction_xxyu' with the following replica assignment: Map(11 -> ArrayBuffer(115), 14 -> ArrayBuffer(115), 13 -> ArrayBuffer(114), 10 -> ArrayBuffer(114), 12 -> ArrayBuffer(116), 15 -> ArrayBuffer(116)).
20/08/09 23:13:34 INFO zk.AdminZkClient: Topic update Map(useraction_xxyu-8 -> Vector(115), useraction_xxyu-6 -> Vector(116), useraction_xxyu-0 -> Vector(116), useraction_xxyu-13 -> ArrayBuffer(114), useraction_xxyu-1 -> Vector(114), useraction_xxyu-9 -> Vector(116), useraction_xxyu-15 -> ArrayBuffer(116), useraction_xxyu-11 -> ArrayBuffer(115), useraction_xxyu-5 -> Vector(115), useraction_xxyu-2 -> Vector(115), useraction_xxyu-10 -> ArrayBuffer(114), useraction_xxyu-14 -> ArrayBuffer(115), useraction_xxyu-7 -> Vector(114), useraction_xxyu-12 -> ArrayBuffer(116), useraction_xxyu-4 -> Vector(114), useraction_xxyu-3 -> Vector(116))
Adding partitions succeeded!
20/08/09 23:13:34 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Closing.
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Session: 0x172bdedb56fcc33 closed
20/08/09 23:13:34 INFO zookeeper.ClientCnxn: EventThread shut down
20/08/09 23:13:34 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Closed.
```
4. check partition again
<img width="801" alt="image" src="https://user-images.githubusercontent.com/14030549/89735618-77517c80-da96-11ea-950b-69fdcdf59cf7.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] tianhui5 commented on pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
tianhui5 commented on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-678068937
> > > Thank you for you effort that you spend on this issue, here is my suggestion, what do you think?
> > > If `offset` is null, please just set it to zero.
> >
> >
> > I wonder if there is any difference between setting offset to zero and seekToBeginning.
>
> Please correct me if I was wrong.
> Let take the above test by me as an example, after scale up, it should continue its consumptation from this position:
>
> ```json
> {
> "0":363171,
> "1":363198,
> "2":363249,
> "3":363171,
> "4":363199,
> "5":363250,
> "6":363170,
> "7":363196,
> "8":363250,
> "9":363170,
> "10":0,
> "11":0,
> "12":0,
> "13":0,
> "14":0,
> "15":0
> }
> ```
>
> not this position:
>
> ```json
> {
> "0":0,
> "1":0,
> "2":0,
> "3":0,
> "4":0,
> "5":0,
> "6":0,
> "7":0,
> "8":0,
> "9":0,
> "10":0,
> "11":0,
> "12":0,
> "13":0,
> "14":0,
> "15":0
> }
> ```
>
> So, if you are using `seekToBeginning` API, can you make sure the partition 0 to partition 9 consume at right position?
Actually, I just add new partitions to list which seekToBeging. So the old partitions' offset won't be effected. Here is my test when I scale kafka from 7 partitions to 10 partitions.
2020-08-21 14:18:02,634 INFO [qtp310350177-45] kafka.KafkaSource:192 : localConsumeStats from local checkpoint null, remoteConsumeStats from remote checkpoint {"0":1369,"1":1412,"2":1500,"3":1395,"4":1430,"5":0,"6":0}
2020-08-21 14:24:33,038 INFO [cost_test_channel] storage.StreamingSegmentManager:645 : Print check point for cube cost_test ,CheckPoint{sourceConsumePosition='{"0":1408,"4":1476,"9":0}', persistedIndexes={1597978800000=1, 1597986000000=1, 1597982400000=1, 1597975200000=1, 1597989600000=3}, longLatencyInfo=LongLatencyInfo{longLatencyEventCnts={}, totalLongLatencyEventCnt=0}, segmentSourceStartPosition={1597978800000={"0":1376,"4":1442}, 1597986000000={"0":1398,"4":1457}, 1597982400000={"0":1392,"4":1454}, 1597975200000={"0":1369,"4":1430}, 1597989600000={"0":1408,"4":1467,"9":0}}, checkPointTime=1597991073038, totalCount=88, checkPointCount=1}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-671059337
Before we scale up the topic partition , we should doing the following steps:
1. Disable the cube, thus all consumption task will be cacelled.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-678754154
Maybe you are right. Let me have a test and see if it is wroks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-671059037
Let's do a test to check what should we do when we need to scale up the topic partitions.
Give we have a topic `useraction_xxyu`, it has ten partitions.
<img width="1336" alt="image" src="https://user-images.githubusercontent.com/14030549/89734596-ae705f80-da8f-11ea-8777-afaad8556817.png">
```sh
[root@cdh-client xiaoxiang.yu]# kafka-topics --describe --topic useraction_xxyu --zookeeper cdh-master:2181
...
...
20/08/09 22:19:36 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Connected.
Topic:useraction_xxyu PartitionCount:10 ReplicationFactor:1 Configs:
Topic: useraction_xxyu Partition: 0 Leader: 116 Replicas: 116 Isr: 116
Topic: useraction_xxyu Partition: 1 Leader: 114 Replicas: 114 Isr: 114
Topic: useraction_xxyu Partition: 2 Leader: 115 Replicas: 115 Isr: 115
Topic: useraction_xxyu Partition: 3 Leader: 116 Replicas: 116 Isr: 116
Topic: useraction_xxyu Partition: 4 Leader: 114 Replicas: 114 Isr: 114
Topic: useraction_xxyu Partition: 5 Leader: 115 Replicas: 115 Isr: 115
Topic: useraction_xxyu Partition: 6 Leader: 116 Replicas: 116 Isr: 116
Topic: useraction_xxyu Partition: 7 Leader: 114 Replicas: 114 Isr: 114
Topic: useraction_xxyu Partition: 8 Leader: 115 Replicas: 115 Isr: 115
Topic: useraction_xxyu Partition: 9 Leader: 116 Replicas: 116 Isr: 116
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] tianhui5 commented on a change in pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
tianhui5 commented on a change in pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#discussion_r467656696
##########
File path: stream-source-kafka/src/main/java/org/apache/kylin/stream/source/kafka/consumer/KafkaConnector.java
##########
@@ -81,9 +81,15 @@ public void open() {
} else if (startMode == ConsumerStartMode.LATEST) {
kafkaConsumer.seekToEnd(topicPartitions);
} else {
+ List<TopicPartition> newTopicPartitions = Lists.newArrayList();
for (TopicPartition topicPartition : topicPartitions) {
Long offset = partitionOffsets.get(topicPartition.partition());
- kafkaConsumer.seek(topicPartition, offset);
+ if (offset != null) {
+ kafkaConsumer.seek(topicPartition, offset);
+ } else {
+ newTopicPartitions.add(topicPartition);
+ }
+ kafkaConsumer.seekToBeginning(newTopicPartitions);
Review comment:
My mistake, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on a change in pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on a change in pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#discussion_r467601221
##########
File path: stream-source-kafka/src/main/java/org/apache/kylin/stream/source/kafka/consumer/KafkaConnector.java
##########
@@ -81,9 +81,15 @@ public void open() {
} else if (startMode == ConsumerStartMode.LATEST) {
kafkaConsumer.seekToEnd(topicPartitions);
} else {
+ List<TopicPartition> newTopicPartitions = Lists.newArrayList();
for (TopicPartition topicPartition : topicPartitions) {
Long offset = partitionOffsets.get(topicPartition.partition());
- kafkaConsumer.seek(topicPartition, offset);
+ if (offset != null) {
+ kafkaConsumer.seek(topicPartition, offset);
+ } else {
+ newTopicPartitions.add(topicPartition);
+ }
+ kafkaConsumer.seekToBeginning(newTopicPartitions);
Review comment:
I think `kafkaConsumer.seekToBeginning(newTopicPartitions);` should not in **for-loop** .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-671065519
1. Go to web page and re-enable Cube, you will see some failure, please don't be in a panic.
```sh
2020-08-09 23:23:10,437 INFO [qtp193625667-35] controller.AdminController:58 : receive assign request:AssignRequest{cubeName='UserAnalysisCube', partitions=[Partition [partitionId=0, partitionInfo=null], Partition [partitionId=5, partitionInfo=null], Partition [partitionId=10, partitionInfo=null], Partition [partitionId=15, partitionInfo=null], Partition [partitionId=13, partitionInfo=null], Partition [partitionId=2, partitionInfo=null], Partition [partitionId=8, partitionInfo=null], Partition [partitionId=12, partitionInfo=null], Partition [partitionId=14, partitionInfo=null], Partition [partitionId=9, partitionInfo=null], Partition [partitionId=11, partitionInfo=null], Partition [partitionId=1, partitionInfo=null], Partition [partitionId=4, partitionInfo=null], Partition [partitionId=6, partitionInfo=null], Partition [partitionId=7, partitionInfo=null], Partition [partitionId=3, partitionInfo=null]], startConsumers=true}
2020-08-09 23:23:10,437 INFO [qtp193625667-35] server.StreamingServer:351 : create and start new consumer for cube:UserAnalysisCube
2020-08-09 23:23:10,468 INFO [qtp193625667-35] kafka.KafkaSource:141 : Create StreamingConnector for Cube UserAnalysisCube, assignedPartitions [Partition [partitionId=0, partitionInfo=null], Partition [partitionId=5, partitionInfo=null], Partition [partitionId=10, partitionInfo=null], Partition [partitionId=15, partitionInfo=null], Partition [partitionId=13, partitionInfo=null], Partition [partitionId=2, partitionInfo=null], Partition [partitionId=8, partitionInfo=null], Partition [partitionId=12, partitionInfo=null], Partition [partitionId=14, partitionInfo=null], Partition [partitionId=9, partitionInfo=null], Partition [partitionId=11, partitionInfo=null], Partition [partitionId=1, partitionInfo=null], Partition [partitionId=4, partitionInfo=null], Partition [partitionId=6, partitionInfo=null], Partition [partitionId=7, partitionInfo=null], Partition [partitionId=3, partitionInfo=null]], startProtocol null
2020-08-09 23:23:10,470 INFO [qtp193625667-35] kafka.TimedJsonStreamParser:98 : Using parser field mapping by {content_list=content_list, act_type=act_type, event_ts_2=event_ts_2, event_ts=event_ts, devide_type=user_detail.devide_type, location_city=user_detail.location.city, network_type=user_detail.network_type, video_id=video_id, event_date_2=event_date_2, str_minute=str_minute, video_type=video_type, play_times=play_times, pageview_id=pageview_id, active_minutes=active_minutes, device_brand=device_brand, str_minute_second=str_minute_second, play_duration=play_duration, event_date=event_date, page_id=page_id, str_second=str_second, uid=uid}
2020-08-09 23:23:10,470 INFO [qtp193625667-35] kafka.TimedJsonStreamParser:120 : TimedJsonStreamParser with formatTs false tsColName event_date
2020-08-09 23:23:10,470 INFO [qtp193625667-35] consumer.ConsumerConfig:279 : ConsumerConfig values:
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [cdh-master:9092, cdh-worker-1:9092, cdh-worker-2:9092]
check.crcs = true
client.dns.lookup = default
client.id =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id =
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
isolation.level = read_uncommitted
key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
session.timeout.ms = 20000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
2020-08-09 23:23:10,473 INFO [qtp193625667-35] utils.AppInfoParser:109 : Kafka version : 2.1.1
2020-08-09 23:23:10,473 INFO [qtp193625667-35] utils.AppInfoParser:110 : Kafka commitId : 21234bee31165527
2020-08-09 23:23:10,475 INFO [qtp193625667-35] kafka.KafkaSource:192 : localConsumeStats from local checkpoint null, remoteConsumeStats from remote checkpoint {"0":363171,"1":363198,"2":363249,"3":363171,"4":363199,"5":363250,"6":363170,"7":363196,"8":363250,"9":363170}
2020-08-09 23:23:10,475 INFO [qtp193625667-35] kafka.KafkaSource:239 : start kafka connector from specified position:KafkaPosition{partitionOffsetMap={0=363171, 1=363198, 2=363249, 3=363171, 4=363199, 5=363250, 6=363170, 7=363196, 8=363250, 9=363170}}
2020-08-09 23:23:10,476 ERROR [qtp193625667-35] server.StreamingServer:357 : consumer start fail for cube:UserAnalysisCube
java.lang.NullPointerException
at org.apache.kylin.stream.source.kafka.consumer.KafkaConnector.open(KafkaConnector.java:86)
at org.apache.kylin.stream.core.consumer.StreamingConsumerChannel.start(StreamingConsumerChannel.java:85)
at org.apache.kylin.stream.server.StreamingServer.startConsumer(StreamingServer.java:355)
at org.apache.kylin.stream.server.StreamingServer.startConsumers(StreamingServer.java:327)
at org.apache.kylin.stream.server.rest.controller.AdminController.assign(AdminController.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:849)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:760)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:547)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:748)
```
2. Let try doing a reassignment.
<img width="1517" alt="image" src="https://user-images.githubusercontent.com/14030549/89735825-bd5b1000-da97-11ea-9944-ae54331002aa.png">
3. I have to modify the source code and repackage. Here is my what I change:
```java
@Override
public void open() {
if (partitions == null || partitions.size() <= 0) {
throw new IllegalStateException("not assign partitions");
}
List<TopicPartition> topicPartitions = Lists.newArrayList();
for (Partition partition : partitions) {
topicPartitions.add(new TopicPartition(topic, partition.getPartitionId()));
}
kafkaConsumer.assign(topicPartitions);
if (startMode == ConsumerStartMode.EARLIEST) {
kafkaConsumer.seekToBeginning(topicPartitions);
} else if (startMode == ConsumerStartMode.LATEST) {
kafkaConsumer.seekToEnd(topicPartitions);
} else {
for (TopicPartition topicPartition : topicPartitions) {
Long offset = partitionOffsets.getOrDefault(topicPartition.partition(), 0L);
if (offset == 0L) {
logger.warn("Assume {} is a new added partition, let's seek it to 0", topicPartition);
}
kafkaConsumer.seek(topicPartition, offset);
}
}
}
```
```sh
mvn clean package -skipTests
...
scp stream-receiver/target/kylin-stream-receiver-3.1.1-SNAPSHOT-all.jar root@cdh-master:/root/xiaoxiang.yu/older_version/apache-kylin-3.1.0-bin-cdh57-receiver/lib/streaming
```
In receiver side, remove old kylin-stream-receiver-3.1.0-all.jar .
```
[root@cdh-master apache-kylin-3.1.0-bin-cdh57-receiver]# mv lib/streaming/kylin-stream-receiver-3.1.0-all.jar kylin-stream-receiver-3.1.0-all.jar.bak
```
4. Restart receiver and here is what we see:
```
2020-08-09 23:59:10,436 INFO [main] server.StreamingServer:725 : Using /root/xiaoxiang.yu/older_version/apache-kylin-3.1.0-bin-cdh57-receiver/stream_index to store local segment cache.
2020-08-09 23:59:10,437 INFO [main] server.StreamingServer:301 : register receiver: cdh-master:7514
2020-08-09 23:59:10,438 WARN [main] coordinator.ZookeeperStreamMetadataStore:488 : /stream/receivers/cdh-master_7514 exists.
2020-08-09 23:59:10,458 INFO [main] server.StreamingServer:421 : add the node to the replicaSet:0, join the group leader election.
2020-08-09 23:59:10,482 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.cube.CubeManager
2020-08-09 23:59:10,495 INFO [main] cube.CubeManager:136 : Initializing CubeManager with config REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:10,539 INFO [main] persistence.ResourceStore:90 : Using metadata url REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10 for resource store
2020-08-09 23:59:10,545 INFO [main] persistence.JDBCConnectionManager:92 : Connecting to Jdbc with url:jdbc:mysql://localhost:3306/NightlyBuild by user root
2020-08-09 23:59:10,559 INFO [Curator-LeaderSelector-0] server.ReplicaSetLeaderSelector:66 : become the leader of the replicaSet:0
2020-08-09 23:59:10,559 INFO [Curator-LeaderSelector-0] server.StreamingServer:575 : become leader of the replicaSet:0
2020-08-09 23:59:10,607 INFO [Curator-LeaderSelector-0] client.HttpCoordinatorClient:81 : send replicaSet lead change notification to coordinator
2020-08-09 23:59:11,194 INFO [main] persistence.JDBCConnectionManager:65 : Connected to MySQL 5.7.26
2020-08-09 23:59:11,212 INFO [main] persistence.JDBCResourceStore:115 : Table [REALTIME] already exists
2020-08-09 23:59:11,214 INFO [main] persistence.JDBCResourceStore:115 : Table [REALTIME_log] already exists
2020-08-09 23:59:11,217 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading CubeInstance from REALTIME(key='/cube')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,306 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.cube.CubeDescManager
2020-08-09 23:59:11,307 INFO [main] cube.CubeDescManager:91 : Initializing CubeDescManager with config REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,307 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading CubeDesc from REALTIME(key='/cube_desc')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,358 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.project.ProjectManager
2020-08-09 23:59:11,359 INFO [main] project.ProjectManager:85 : Initializing ProjectManager with metadata url REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,360 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading ProjectInstance from REALTIME(key='/project')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,369 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 1 ProjectInstance(s) out of 1 resource with 0 errors
2020-08-09 23:59:11,371 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.cachesync.Broadcaster
2020-08-09 23:59:11,373 DEBUG [main] cachesync.Broadcaster:102 : 1 nodes in the cluster: [localhost:7514]
2020-08-09 23:59:11,377 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.model.DataModelManager
2020-08-09 23:59:11,381 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.TableMetadataManager
2020-08-09 23:59:11,382 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading TableDesc from REALTIME(key='/table')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,400 INFO [main] measure.MeasureTypeFactory:121 : Checking custom measure types from kylin config
2020-08-09 23:59:11,401 INFO [main] measure.MeasureTypeFactory:150 : registering COUNT_DISTINCT(hllc), class org.apache.kylin.measure.hllc.HLLCMeasureType$Factory
2020-08-09 23:59:11,406 INFO [main] measure.MeasureTypeFactory:150 : registering COUNT_DISTINCT(bitmap), class org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory
2020-08-09 23:59:11,410 INFO [main] measure.MeasureTypeFactory:150 : registering TOP_N(topn), class org.apache.kylin.measure.topn.TopNMeasureType$Factory
2020-08-09 23:59:11,412 INFO [main] measure.MeasureTypeFactory:150 : registering RAW(raw), class org.apache.kylin.measure.raw.RawMeasureType$Factory
2020-08-09 23:59:11,414 INFO [main] measure.MeasureTypeFactory:150 : registering EXTENDED_COLUMN(extendedcolumn), class org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory
2020-08-09 23:59:11,415 INFO [main] measure.MeasureTypeFactory:150 : registering PERCENTILE_APPROX(percentile), class org.apache.kylin.measure.percentile.PercentileMeasureType$Factory
2020-08-09 23:59:11,416 INFO [main] measure.MeasureTypeFactory:150 : registering COUNT_DISTINCT(dim_dc), class org.apache.kylin.measure.dim.DimCountDistinctMeasureType$Factory
2020-08-09 23:59:11,418 INFO [main] measure.MeasureTypeFactory:150 : registering STDDEV_SUM(stddev_sum), class org.apache.kylin.measure.stddev.StdDevSumMeasureType$Factory
2020-08-09 23:59:11,431 INFO [main] measure.MeasureTypeFactory:150 : registering COUNT_DISTINCT(bitmap_map), class org.apache.kylin.measure.map.bitmap.BitmapMapMeasureType$Factory
2020-08-09 23:59:11,438 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 1 TableDesc(s) out of 1 resource with 0 errors
2020-08-09 23:59:11,439 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading TableExtDesc from REALTIME(key='/table_exd')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,442 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 0 TableExtDesc(s) out of 0 resource with 0 errors
2020-08-09 23:59:11,443 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading ExternalFilterDesc from REALTIME(key='/ext_filter')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,445 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 0 ExternalFilterDesc(s) out of 0 resource with 0 errors
2020-08-09 23:59:11,445 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading DataModelDesc from REALTIME(key='/model_desc')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,472 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 1 DataModelDesc(s) out of 1 resource with 0 errors
2020-08-09 23:59:11,494 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 1 CubeDesc(s) out of 1 resource with 0 errors
2020-08-09 23:59:11,495 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 1 CubeInstance(s) out of 1 resource with 0 errors
2020-08-09 23:59:11,518 INFO [main] storage.StreamingSegmentManager:239 : restore segments from local files for cube:UserAnalysisCube
2020-08-09 23:59:11,522 WARN [main] storage.StreamingSegmentManager:253 : no checkpoint for cube:UserAnalysisCube store state, remove all local segments folders
2020-08-09 23:59:11,522 INFO [main] server.StreamingServer:351 : create and start new consumer for cube:UserAnalysisCube
2020-08-09 23:59:11,550 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.cube.cuboid.CuboidManager
2020-08-09 23:59:11,566 INFO [main] kafka.KafkaSource:141 : Create StreamingConnector for Cube UserAnalysisCube, assignedPartitions [Partition [partitionId=0, partitionInfo=null], Partition [partitionId=5, partitionInfo=null], Partition [partitionId=10, partitionInfo=null], Partition [partitionId=15, partitionInfo=null], Partition [partitionId=13, partitionInfo=null], Partition [partitionId=2, partitionInfo=null], Partition [partitionId=8, partitionInfo=null], Partition [partitionId=12, partitionInfo=null], Partition [partitionId=14, partitionInfo=null], Partition [partitionId=9, partitionInfo=null], Partition [partitionId=11, partitionInfo=null], Partition [partitionId=1, partitionInfo=null], Partition [partitionId=4, partitionInfo=null], Partition [partitionId=6, partitionInfo=null], Partition [partitionId=7, partitionInfo=null], Partition [partitionId=3, partitionInfo=null]], startProtocol null
2020-08-09 23:59:11,571 INFO [main] kafka.TimedJsonStreamParser:98 : Using parser field mapping by {content_list=content_list, act_type=act_type, event_ts_2=event_ts_2, event_ts=event_ts, devide_type=user_detail.devide_type, location_city=user_detail.location.city, network_type=user_detail.network_type, video_id=video_id, event_date_2=event_date_2, str_minute=str_minute, video_type=video_type, play_times=play_times, pageview_id=pageview_id, active_minutes=active_minutes, device_brand=device_brand, str_minute_second=str_minute_second, play_duration=play_duration, event_date=event_date, page_id=page_id, str_second=str_second, uid=uid}
2020-08-09 23:59:11,584 INFO [main] kafka.TimedJsonStreamParser:120 : TimedJsonStreamParser with formatTs false tsColName event_date
2020-08-09 23:59:11,663 INFO [main] consumer.ConsumerConfig:279 : ConsumerConfig values:
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [cdh-master:9092, cdh-worker-1:9092, cdh-worker-2:9092]
check.crcs = true
client.dns.lookup = default
client.id =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id =
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
isolation.level = read_uncommitted
key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
session.timeout.ms = 20000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
2020-08-09 23:59:11,790 INFO [main] utils.AppInfoParser:109 : Kafka version : 2.1.1
2020-08-09 23:59:11,790 INFO [main] utils.AppInfoParser:110 : Kafka commitId : 21234bee31165527
2020-08-09 23:59:11,792 INFO [main] kafka.KafkaSource:192 : localConsumeStats from local checkpoint null, remoteConsumeStats from remote checkpoint {"0":363171,"1":363198,"2":363249,"3":363171,"4":363199,"5":363250,"6":363170,"7":363196,"8":363250,"9":363170}
2020-08-09 23:59:11,793 INFO [main] kafka.KafkaSource:239 : start kafka connector from specified position:KafkaPosition{partitionOffsetMap={0=363171, 1=363198, 2=363249, 3=363171, 4=363199, 5=363250, 6=363170, 7=363196, 8=363250, 9=363170}}
2020-08-09 23:59:11,798 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-10 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-15 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-13 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-12 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-14 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-11 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,801 INFO [main] metrics.StreamingMetrics:76 : Skip streaming metricRegistry because it is empty.
2020-08-09 23:59:11,819 INFO [main] util.log:169 : Logging initialized @3474ms to org.eclipse.jetty.util.log.Slf4jLog
2020-08-09 23:59:11,946 INFO [UserAnalysisCube_channel] clients.Metadata:285 : Cluster ID: W1YX_q9GRd-R0qwcwFJ9rg
2020-08-09 23:59:12,163 INFO [UserAnalysisCube_channel] storage.StreamingSegmentManager:189 : Create new segment:StreamingCubeSegment [cubeName=UserAnalysisCube, segmentName=20200809210000_20200809220000]
2020-08-09 23:59:12,826 INFO [UserAnalysisCube_channel] storage.StreamingSegmentManager:189 : Create new segment:StreamingCubeSegment [cubeName=UserAnalysisCube, segmentName=20200809230000_20200810000000]
2020-08-09 23:59:13,226 INFO [main] server.Server:359 : jetty-9.4.26.v20200117; built: 2020-01-17T12:35:33.676Z; git: 7b38981d25d14afb4a12ff1f2596756144edf695; jvm 1.8.0_171-b11
2020-08-09 23:59:13,260 INFO [main] ContextHandler.kylin:2223 : Initializing Spring FrameworkServlet 'org.springframework.web.servlet.DispatcherServlet-d2e6b0b'
2020-08-09 23:59:13,278 INFO [main] handler.ContextHandler:825 : Started o.e.j.s.ServletContextHandler@708eb310{/kylin,null,AVAILABLE}
2020-08-09 23:59:13,279 INFO [main] handler.ContextHandler:825 : Started o.e.j.s.h.ContextHandler@6ebc9573{/kylin/logs,null,AVAILABLE}
2020-08-09 23:59:13,294 INFO [main] server.AbstractConnector:330 : Started ServerConnector@369c9bb{HTTP/1.1,[http/1.1]}{0.0.0.0:7514}
2020-08-09 23:59:13,294 INFO [main] server.Server:399 : Started @4949ms
2020-08-09 23:59:15,572 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:106 : Start persist memory store for cube:UserAnalysisCube, segment:20200809210000_20200809220000, rowCnt:50000
2020-08-09 23:59:15,635 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:202 : cuboid-8191 transform to columnar, take 58 ms
2020-08-09 23:59:15,655 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '00000000000000003640', current '00000000000000003638'
2020-08-09 23:59:15,719 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 15962
2020-08-09 23:59:15,720 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:15,720 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,725 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 3640=>1219, 3638=>1218, 3630=>1212, 3631=>1213, 3632=>1214
2020-08-09 23:59:15,725 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : UID took : 88 ms
2020-08-09 23:59:15,738 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'page_19', current 'page_13'
2020-08-09 23:59:15,739 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 50
2020-08-09 23:59:15,739 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:15,740 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,740 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: page_19=>11, page_13=>5, page_14=>6, page_11=>3, page_12=>4
2020-08-09 23:59:15,740 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : PAGE_ID took : 13 ms
2020-08-09 23:59:15,751 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'stop', current 'pause'
2020-08-09 23:59:15,752 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 10
2020-08-09 23:59:15,752 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:15,752 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,752 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: stop=>9, pause=>5, play=>6, like=>4, start=>8
2020-08-09 23:59:15,752 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : ACT_TYPE took : 11 ms
2020-08-09 23:59:15,763 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 34
2020-08-09 23:59:15,763 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:15,763 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,764 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 0=>0, 1=>1, 2=>2, 3=>3, 4=>4
2020-08-09 23:59:15,764 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVIDE_TYPE took : 10 ms
2020-08-09 23:59:15,774 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'hongkong', current 'hangzhou'
2020-08-09 23:59:15,774 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 7
2020-08-09 23:59:15,774 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:15,775 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,775 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: hongkong=>3, hangzhou=>2, shenzhen=>6, berlin=>1, nanjing=>4
2020-08-09 23:59:15,775 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : LOCATION_CITY took : 10 ms
2020-08-09 23:59:15,785 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 4
2020-08-09 23:59:15,785 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:15,785 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,785 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 2=>0, 3=>1, 4=>2, 5=>3
2020-08-09 23:59:15,785 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : NETWORK_TYPE took : 10 ms
2020-08-09 23:59:15,794 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'xiaomi', current 'vivo'
2020-08-09 23:59:15,795 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 5
2020-08-09 23:59:15,795 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:15,795 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,795 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: huawei=>0, xiaomi=>4, vivo=>3, meizu=>2, iPhone=>1
2020-08-09 23:59:15,795 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVICE_BRAND took : 9 ms
2020-08-09 23:59:15,806 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '21_14_59', current '21_14_50'
2020-08-09 23:59:15,810 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 2032
2020-08-09 23:59:15,810 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:15,810 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,810 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 21_14_59=>355, 21_14_50=>346, 21_14_55=>351, 21_14_56=>352, 21_14_57=>353
2020-08-09 23:59:15,810 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : STR_MINUTE_SECOND took : 15 ms
2020-08-09 23:59:16,088 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:UID, took: 276ms
2020-08-09 23:59:16,171 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:PAGE_ID, took: 83ms
2020-08-09 23:59:16,254 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:ACT_TYPE, took: 82ms
2020-08-09 23:59:16,316 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVIDE_TYPE, took: 61ms
2020-08-09 23:59:16,391 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:LOCATION_CITY, took: 75ms
2020-08-09 23:59:16,461 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:NETWORK_TYPE, took: 69ms
2020-08-09 23:59:16,532 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVICE_BRAND, took: 70ms
2020-08-09 23:59:16,608 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:STR_MINUTE_SECOND, took: 75ms
2020-08-09 23:59:16,646 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MONTH_START, took: 38ms
2020-08-09 23:59:16,673 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:WEEK_START, took: 26ms
2020-08-09 23:59:16,703 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DAY_START, took: 29ms
2020-08-09 23:59:16,736 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:HOUR_START, took: 33ms
2020-08-09 23:59:16,779 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MINUTE_START, took: 41ms
2020-08-09 23:59:16,808 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:_COUNT_, took: 29ms
2020-08-09 23:59:16,871 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_TIMES, took: 63ms
2020-08-09 23:59:16,907 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_ACTIVE_MINUTES, took: 34ms
2020-08-09 23:59:16,932 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_DURATION, took: 25ms
2020-08-09 23:59:16,993 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_TIMES, took: 60ms
2020-08-09 23:59:17,027 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_ACTIVE_MINUTES, took: 33ms
2020-08-09 23:59:17,049 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_DURATION, took: 22ms
2020-08-09 23:59:17,081 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:PV_STAT, took: 31ms
2020-08-09 23:59:17,089 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:110 : Finish persist memory store for cube:UserAnalysisCube segment:20200809210000_20200809220000, take: 1517ms
2020-08-09 23:59:18,954 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:106 : Start persist memory store for cube:UserAnalysisCube, segment:20200809210000_20200809220000, rowCnt:50000
2020-08-09 23:59:18,980 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:202 : cuboid-8191 transform to columnar, take 26 ms
2020-08-09 23:59:18,989 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '00000000000000003640', current '00000000000000003638'
2020-08-09 23:59:19,008 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 16046
2020-08-09 23:59:19,008 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:19,008 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,008 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 3640=>1240, 3638=>1238, 3639=>1239, 3630=>1231, 3631=>1232
2020-08-09 23:59:19,008 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : UID took : 28 ms
2020-08-09 23:59:19,017 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'page_19', current 'page_13'
2020-08-09 23:59:19,017 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 50
2020-08-09 23:59:19,017 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:19,017 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,018 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: page_19=>11, page_13=>5, page_14=>6, page_11=>3, page_12=>4
2020-08-09 23:59:19,018 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : PAGE_ID took : 8 ms
2020-08-09 23:59:19,025 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'stop', current 'pause'
2020-08-09 23:59:19,026 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 10
2020-08-09 23:59:19,026 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:19,026 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,026 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: stop=>9, pause=>5, play=>6, like=>4, start=>8
2020-08-09 23:59:19,026 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : ACT_TYPE took : 8 ms
2020-08-09 23:59:19,035 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 44
2020-08-09 23:59:19,035 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:19,035 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,035 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 16=>0, 17=>1, 18=>2, 19=>3, 20=>4
2020-08-09 23:59:19,035 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVIDE_TYPE took : 8 ms
2020-08-09 23:59:19,043 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'hongkong', current 'hangzhou'
2020-08-09 23:59:19,043 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 7
2020-08-09 23:59:19,043 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:19,043 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,043 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: hongkong=>3, hangzhou=>2, shenzhen=>6, berlin=>1, nanjing=>4
2020-08-09 23:59:19,044 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : LOCATION_CITY took : 8 ms
2020-08-09 23:59:19,052 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 4
2020-08-09 23:59:19,052 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:19,052 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,052 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 2=>0, 3=>1, 4=>2, 5=>3
2020-08-09 23:59:19,052 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : NETWORK_TYPE took : 8 ms
2020-08-09 23:59:19,059 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'xiaomi', current 'vivo'
2020-08-09 23:59:19,060 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 5
2020-08-09 23:59:19,060 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:19,060 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,060 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: huawei=>0, xiaomi=>4, vivo=>3, meizu=>2, iPhone=>1
2020-08-09 23:59:19,060 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVICE_BRAND took : 7 ms
2020-08-09 23:59:19,069 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '21_55_9', current '21_55_0'
2020-08-09 23:59:19,077 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 2536
2020-08-09 23:59:19,077 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:19,077 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,077 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 21_55_7=>2341, 21_55_8=>2342, 21_55_9=>2343, 21_55_0=>2284, 21_55_1=>2285
2020-08-09 23:59:19,077 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : STR_MINUTE_SECOND took : 17 ms
2020-08-09 23:59:19,185 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:UID, took: 107ms
2020-08-09 23:59:19,235 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:PAGE_ID, took: 49ms
2020-08-09 23:59:19,284 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:ACT_TYPE, took: 48ms
2020-08-09 23:59:19,335 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVIDE_TYPE, took: 50ms
2020-08-09 23:59:19,396 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:LOCATION_CITY, took: 60ms
2020-08-09 23:59:19,455 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:NETWORK_TYPE, took: 59ms
2020-08-09 23:59:19,522 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVICE_BRAND, took: 66ms
2020-08-09 23:59:19,592 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:STR_MINUTE_SECOND, took: 69ms
2020-08-09 23:59:19,610 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MONTH_START, took: 17ms
2020-08-09 23:59:19,626 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:WEEK_START, took: 16ms
2020-08-09 23:59:19,642 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DAY_START, took: 15ms
2020-08-09 23:59:19,662 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:HOUR_START, took: 19ms
2020-08-09 23:59:19,691 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MINUTE_START, took: 28ms
2020-08-09 23:59:19,708 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:_COUNT_, took: 16ms
2020-08-09 23:59:19,766 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_TIMES, took: 57ms
2020-08-09 23:59:19,783 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_ACTIVE_MINUTES, took: 17ms
2020-08-09 23:59:19,801 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_DURATION, took: 17ms
2020-08-09 23:59:19,856 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_TIMES, took: 54ms
2020-08-09 23:59:19,873 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_ACTIVE_MINUTES, took: 17ms
2020-08-09 23:59:19,890 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_DURATION, took: 16ms
2020-08-09 23:59:19,908 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:PV_STAT, took: 18ms
2020-08-09 23:59:19,913 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:110 : Finish persist memory store for cube:UserAnalysisCube segment:20200809210000_20200809220000, take: 959ms
2020-08-09 23:59:20,175 INFO [UserAnalysisCube_channel] storage.StreamingSegmentManager:189 : Create new segment:StreamingCubeSegment [cubeName=UserAnalysisCube, segmentName=20200809220000_20200809230000]
2020-08-09 23:59:22,201 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:106 : Start persist memory store for cube:UserAnalysisCube, segment:20200809220000_20200809230000, rowCnt:50000
2020-08-09 23:59:22,235 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:202 : cuboid-8191 transform to columnar, take 33 ms
2020-08-09 23:59:22,245 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '00000000000000003640', current '00000000000000003638'
2020-08-09 23:59:22,258 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 15977
2020-08-09 23:59:22,258 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:22,258 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,258 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 3640=>1166, 3638=>1165, 3630=>1158, 3631=>1159, 3632=>1160
2020-08-09 23:59:22,258 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : UID took : 23 ms
2020-08-09 23:59:22,267 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'page_19', current 'page_13'
2020-08-09 23:59:22,268 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 50
2020-08-09 23:59:22,268 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:22,268 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,268 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: page_19=>11, page_13=>5, page_14=>6, page_11=>3, page_12=>4
2020-08-09 23:59:22,268 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : PAGE_ID took : 9 ms
2020-08-09 23:59:22,277 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'stop', current 'pause'
2020-08-09 23:59:22,277 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 10
2020-08-09 23:59:22,277 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:22,277 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,277 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: stop=>9, pause=>5, play=>6, like=>4, start=>8
2020-08-09 23:59:22,277 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : ACT_TYPE took : 9 ms
2020-08-09 23:59:22,285 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 34
2020-08-09 23:59:22,285 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:22,286 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,286 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 0=>0, 1=>1, 2=>2, 3=>3, 4=>4
2020-08-09 23:59:22,286 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVIDE_TYPE took : 8 ms
2020-08-09 23:59:22,294 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'hongkong', current 'hangzhou'
2020-08-09 23:59:22,294 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 7
2020-08-09 23:59:22,294 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:22,295 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,295 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: hongkong=>3, hangzhou=>2, shenzhen=>6, berlin=>1, nanjing=>4
2020-08-09 23:59:22,295 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : LOCATION_CITY took : 8 ms
2020-08-09 23:59:22,303 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 4
2020-08-09 23:59:22,303 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:22,303 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,303 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 2=>0, 3=>1, 4=>2, 5=>3
2020-08-09 23:59:22,303 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : NETWORK_TYPE took : 8 ms
2020-08-09 23:59:22,311 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'xiaomi', current 'vivo'
2020-08-09 23:59:22,311 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 5
2020-08-09 23:59:22,311 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:22,311 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,311 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: huawei=>0, xiaomi=>4, vivo=>3, meizu=>2, iPhone=>1
2020-08-09 23:59:22,311 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVICE_BRAND took : 8 ms
2020-08-09 23:59:22,321 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '22_14_59', current '22_14_41'
2020-08-09 23:59:22,323 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 1985
2020-08-09 23:59:22,324 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:22,324 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,324 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 22_14_58=>354, 22_14_59=>355, 22_14_41=>336, 22_14_42=>337, 22_14_40=>335
2020-08-09 23:59:22,324 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : STR_MINUTE_SECOND took : 12 ms
2020-08-09 23:59:22,426 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:UID, took: 102ms
2020-08-09 23:59:22,477 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:PAGE_ID, took: 50ms
2020-08-09 23:59:22,526 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:ACT_TYPE, took: 49ms
2020-08-09 23:59:22,574 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVIDE_TYPE, took: 47ms
2020-08-09 23:59:22,706 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:LOCATION_CITY, took: 61ms
2020-08-09 23:59:22,768 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:NETWORK_TYPE, took: 61ms
2020-08-09 23:59:22,834 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVICE_BRAND, took: 66ms
2020-08-09 23:59:22,903 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:STR_MINUTE_SECOND, took: 68ms
2020-08-09 23:59:22,920 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MONTH_START, took: 16ms
2020-08-09 23:59:22,937 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:WEEK_START, took: 16ms
2020-08-09 23:59:22,953 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DAY_START, took: 16ms
2020-08-09 23:59:22,971 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:HOUR_START, took: 17ms
2020-08-09 23:59:22,995 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MINUTE_START, took: 24ms
2020-08-09 23:59:23,008 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:_COUNT_, took: 12ms
2020-08-09 23:59:23,064 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_TIMES, took: 55ms
2020-08-09 23:59:23,082 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_ACTIVE_MINUTES, took: 17ms
2020-08-09 23:59:23,100 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_DURATION, took: 17ms
2020-08-09 23:59:23,157 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_TIMES, took: 56ms
2020-08-09 23:59:23,175 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_ACTIVE_MINUTES, took: 17ms
2020-08-09 23:59:23,193 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_DURATION, took: 18ms
2020-08-09 23:59:23,213 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:PV_STAT, took: 19ms
2020-08-09 23:59:23,214 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:110 : Finish persist memory store for cube:UserAnalysisCube segment:20200809220000_20200809230000, take: 1012ms
2020-08-09 23:59:25,467 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:106 : Start persist memory store for cube:UserAnalysisCube, segment:20200809230000_20200810000000, rowCnt:50000
2020-08-09 23:59:25,495 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:202 : cuboid-8191 transform to columnar, take 26 ms
2020-08-09 23:59:25,504 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '00000000000000003639', current '00000000000000003630'
2020-08-09 23:59:25,519 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 15936
2020-08-09 23:59:25,519 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:25,519 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,519 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 3638=>1084, 3639=>1085, 3630=>1077, 3631=>1078, 3632=>1079
2020-08-09 23:59:25,519 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : UID took : 24 ms
2020-08-09 23:59:25,532 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'page_19', current 'page_13'
2020-08-09 23:59:25,533 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 50
2020-08-09 23:59:25,533 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:25,533 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,533 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: page_19=>11, page_13=>5, page_14=>6, page_11=>3, page_12=>4
2020-08-09 23:59:25,533 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : PAGE_ID took : 13 ms
2020-08-09 23:59:25,542 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'stop', current 'pause'
2020-08-09 23:59:25,543 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 10
2020-08-09 23:59:25,543 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:25,543 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,543 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: stop=>9, pause=>5, play=>6, like=>4, start=>8
2020-08-09 23:59:25,543 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : ACT_TYPE took : 9 ms
2020-08-09 23:59:25,553 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 41
2020-08-09 23:59:25,553 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:25,553 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,553 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 19=>0, 20=>1, 21=>2, 22=>3, 23=>4
2020-08-09 23:59:25,553 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVIDE_TYPE took : 10 ms
2020-08-09 23:59:25,562 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'hongkong', current 'hangzhou'
2020-08-09 23:59:25,563 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 7
2020-08-09 23:59:25,563 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:25,563 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,563 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: hongkong=>3, hangzhou=>2, shenzhen=>6, berlin=>1, nanjing=>4
2020-08-09 23:59:25,563 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : LOCATION_CITY took : 9 ms
2020-08-09 23:59:25,572 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 4
2020-08-09 23:59:25,573 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:25,573 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,573 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 2=>0, 3=>1, 4=>2, 5=>3
2020-08-09 23:59:25,573 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : NETWORK_TYPE took : 9 ms
2020-08-09 23:59:25,582 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'xiaomi', current 'vivo'
2020-08-09 23:59:25,583 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 5
2020-08-09 23:59:25,583 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:25,583 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,583 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: huawei=>0, xiaomi=>4, vivo=>3, meizu=>2, iPhone=>1
2020-08-09 23:59:25,583 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVICE_BRAND took : 10 ms
2020-08-09 23:59:25,594 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '23_53_19', current '23_53_10'
2020-08-09 23:59:25,596 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 2412
2020-08-09 23:59:25,596 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:25,596 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,596 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 23_53_16=>2035, 23_53_17=>2036, 23_53_18=>2037, 23_53_19=>2038, 23_53_10=>2029
2020-08-09 23:59:25,596 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : STR_MINUTE_SECOND took : 12 ms
2020-08-09 23:59:25,702 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:UID, took: 105ms
2020-08-09 23:59:25,757 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:PAGE_ID, took: 54ms
2020-08-09 23:59:25,812 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:ACT_TYPE, took: 54ms
2020-08-09 23:59:25,866 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVIDE_TYPE, took: 54ms
2020-08-09 23:59:25,932 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:LOCATION_CITY, took: 65ms
2020-08-09 23:59:25,996 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:NETWORK_TYPE, took: 63ms
2020-08-09 23:59:26,065 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVICE_BRAND, took: 68ms
2020-08-09 23:59:26,138 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:STR_MINUTE_SECOND, took: 73ms
2020-08-09 23:59:26,157 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MONTH_START, took: 18ms
2020-08-09 23:59:26,175 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:WEEK_START, took: 18ms
2020-08-09 23:59:26,193 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DAY_START, took: 17ms
2020-08-09 23:59:26,210 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:HOUR_START, took: 17ms
2020-08-09 23:59:26,235 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MINUTE_START, took: 24ms
2020-08-09 23:59:26,256 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:_COUNT_, took: 20ms
2020-08-09 23:59:26,316 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_TIMES, took: 60ms
2020-08-09 23:59:26,336 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_ACTIVE_MINUTES, took: 19ms
2020-08-09 23:59:26,355 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_DURATION, took: 18ms
2020-08-09 23:59:26,412 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_TIMES, took: 57ms
2020-08-09 23:59:26,431 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_ACTIVE_MINUTES, took: 18ms
2020-08-09 23:59:26,450 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_DURATION, took: 18ms
2020-08-09 23:59:26,470 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:PV_STAT, took: 20ms
2020-08-09 23:59:26,471 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:110 : Finish persist memory store for cube:UserAnalysisCube segment:20200809230000_20200810000000, take: 1003ms
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-671064075
1. I will kill the producer.
2. check latest offset via Python
3. Resize topic partition.
```sh
[root@cdh-client xiaoxiang.yu]# kafka-topics --alter --topic useraction_xxyu --zookeeper cdh-master:2181 --partitions 16
20/08/09 23:13:33 INFO utils.Log4jControllerRegistration$: Registered kafka:type=kafka.Log4jController MBean
20/08/09 23:13:34 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Initializing a new session to cdh-master:2181.
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-cdh5.14.2--1, built on 03/27/2018 20:39 GMT
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:host.name=cdh-client
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_171
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.8.0_171/jre
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-754.17.1.el6.x86_64
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:user.name=root
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:user.home=/root
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Client environment:user.dir=/root/xiaoxiang.yu
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=cdh-master:2181 sessionTimeout=30000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@6302bbb1
20/08/09 23:13:34 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Waiting until connected.
20/08/09 23:13:34 INFO zookeeper.ClientCnxn: Opening socket connection to server cdh-master/10.1.3.90:2181. Will not attempt to authenticate using SASL (unknown error)
20/08/09 23:13:34 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.1.3.91:54122, server: cdh-master/10.1.3.90:2181
20/08/09 23:13:34 INFO zookeeper.ClientCnxn: Session establishment complete on server cdh-master/10.1.3.90:2181, sessionid = 0x172bdedb56fcc33, negotiated timeout = 30000
20/08/09 23:13:34 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Connected.
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
20/08/09 23:13:34 INFO zk.AdminZkClient: Creating 6 partitions for 'useraction_xxyu' with the following replica assignment: Map(11 -> ArrayBuffer(115), 14 -> ArrayBuffer(115), 13 -> ArrayBuffer(114), 10 -> ArrayBuffer(114), 12 -> ArrayBuffer(116), 15 -> ArrayBuffer(116)).
20/08/09 23:13:34 INFO zk.AdminZkClient: Topic update Map(useraction_xxyu-8 -> Vector(115), useraction_xxyu-6 -> Vector(116), useraction_xxyu-0 -> Vector(116), useraction_xxyu-13 -> ArrayBuffer(114), useraction_xxyu-1 -> Vector(114), useraction_xxyu-9 -> Vector(116), useraction_xxyu-15 -> ArrayBuffer(116), useraction_xxyu-11 -> ArrayBuffer(115), useraction_xxyu-5 -> Vector(115), useraction_xxyu-2 -> Vector(115), useraction_xxyu-10 -> ArrayBuffer(114), useraction_xxyu-14 -> ArrayBuffer(115), useraction_xxyu-7 -> Vector(114), useraction_xxyu-12 -> ArrayBuffer(116), useraction_xxyu-4 -> Vector(114), useraction_xxyu-3 -> Vector(116))
Adding partitions succeeded!
20/08/09 23:13:34 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Closing.
20/08/09 23:13:34 INFO zookeeper.ZooKeeper: Session: 0x172bdedb56fcc33 closed
20/08/09 23:13:34 INFO zookeeper.ClientCnxn: EventThread shut down
20/08/09 23:13:34 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Closed.
```
4. check partition again
<img width="801" alt="image" src="https://user-images.githubusercontent.com/14030549/89735618-77517c80-da96-11ea-950b-69fdcdf59cf7.png">
5. Restart producer and check offset again.
```sh
python fake.py | kafka-console-producer --topic useraction_xxyu --broker-list cdh-master:9092,cdh-worker-1:9092,cdh-worker-2:9092
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus merged pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
hit-lacus merged pull request #1351:
URL: https://github.com/apache/kylin/pull/1351
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-671133465
> > Thank you for you effort that you spend on this issue, here is my suggestion, what do you think?
> > If `offset` is null, please just set it to zero.
>
> I wonder if there is any difference between setting offset to zero and seekToBeginning.
Please correct me if I was wrong.
Let take the above test by me as an example, after scale up, it should continue its consumptation from this position:
```json
{
"0":363171,
"1":363198,
"2":363249,
"3":363171,
"4":363199,
"5":363250,
"6":363170,
"7":363196,
"8":363250,
"9":363170,
"10":0,
"11":0,
"12":0,
"13":0,
"14":0,
"15":0
}
```
not this position:
```json
{
"0":0,
"1":0,
"2":0,
"3":0,
"4":0,
"5":0,
"6":0,
"7":0,
"8":0,
"9":0,
"10":0,
"11":0,
"12":0,
"13":0,
"14":0,
"15":0
}
```
So, if you are using `seekToBeginning` API, can you make sure the partition 0 to partition 9 consume at right position?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-694084949
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-669972679
Thank you for your contributions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-671065519
1. Go to web page and re-enable Cube, you will see some failure, please don't be in a panic.
```sh
2020-08-09 23:23:10,437 INFO [qtp193625667-35] controller.AdminController:58 : receive assign request:AssignRequest{cubeName='UserAnalysisCube', partitions=[Partition [partitionId=0, partitionInfo=null], Partition [partitionId=5, partitionInfo=null], Partition [partitionId=10, partitionInfo=null], Partition [partitionId=15, partitionInfo=null], Partition [partitionId=13, partitionInfo=null], Partition [partitionId=2, partitionInfo=null], Partition [partitionId=8, partitionInfo=null], Partition [partitionId=12, partitionInfo=null], Partition [partitionId=14, partitionInfo=null], Partition [partitionId=9, partitionInfo=null], Partition [partitionId=11, partitionInfo=null], Partition [partitionId=1, partitionInfo=null], Partition [partitionId=4, partitionInfo=null], Partition [partitionId=6, partitionInfo=null], Partition [partitionId=7, partitionInfo=null], Partition [partitionId=3, partitionInfo=null]], startConsumers=true}
2020-08-09 23:23:10,437 INFO [qtp193625667-35] server.StreamingServer:351 : create and start new consumer for cube:UserAnalysisCube
2020-08-09 23:23:10,468 INFO [qtp193625667-35] kafka.KafkaSource:141 : Create StreamingConnector for Cube UserAnalysisCube, assignedPartitions [Partition [partitionId=0, partitionInfo=null], Partition [partitionId=5, partitionInfo=null], Partition [partitionId=10, partitionInfo=null], Partition [partitionId=15, partitionInfo=null], Partition [partitionId=13, partitionInfo=null], Partition [partitionId=2, partitionInfo=null], Partition [partitionId=8, partitionInfo=null], Partition [partitionId=12, partitionInfo=null], Partition [partitionId=14, partitionInfo=null], Partition [partitionId=9, partitionInfo=null], Partition [partitionId=11, partitionInfo=null], Partition [partitionId=1, partitionInfo=null], Partition [partitionId=4, partitionInfo=null], Partition [partitionId=6, partitionInfo=null], Partition [partitionId=7, partitionInfo=null], Partition [partitionId=3, partitionInfo=null]], startProtocol null
2020-08-09 23:23:10,470 INFO [qtp193625667-35] kafka.TimedJsonStreamParser:98 : Using parser field mapping by {content_list=content_list, act_type=act_type, event_ts_2=event_ts_2, event_ts=event_ts, devide_type=user_detail.devide_type, location_city=user_detail.location.city, network_type=user_detail.network_type, video_id=video_id, event_date_2=event_date_2, str_minute=str_minute, video_type=video_type, play_times=play_times, pageview_id=pageview_id, active_minutes=active_minutes, device_brand=device_brand, str_minute_second=str_minute_second, play_duration=play_duration, event_date=event_date, page_id=page_id, str_second=str_second, uid=uid}
2020-08-09 23:23:10,470 INFO [qtp193625667-35] kafka.TimedJsonStreamParser:120 : TimedJsonStreamParser with formatTs false tsColName event_date
2020-08-09 23:23:10,470 INFO [qtp193625667-35] consumer.ConsumerConfig:279 : ConsumerConfig values:
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [cdh-master:9092, cdh-worker-1:9092, cdh-worker-2:9092]
check.crcs = true
client.dns.lookup = default
client.id =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id =
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
isolation.level = read_uncommitted
key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
session.timeout.ms = 20000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
2020-08-09 23:23:10,473 INFO [qtp193625667-35] utils.AppInfoParser:109 : Kafka version : 2.1.1
2020-08-09 23:23:10,473 INFO [qtp193625667-35] utils.AppInfoParser:110 : Kafka commitId : 21234bee31165527
2020-08-09 23:23:10,475 INFO [qtp193625667-35] kafka.KafkaSource:192 : localConsumeStats from local checkpoint null, remoteConsumeStats from remote checkpoint {"0":363171,"1":363198,"2":363249,"3":363171,"4":363199,"5":363250,"6":363170,"7":363196,"8":363250,"9":363170}
2020-08-09 23:23:10,475 INFO [qtp193625667-35] kafka.KafkaSource:239 : start kafka connector from specified position:KafkaPosition{partitionOffsetMap={0=363171, 1=363198, 2=363249, 3=363171, 4=363199, 5=363250, 6=363170, 7=363196, 8=363250, 9=363170}}
2020-08-09 23:23:10,476 ERROR [qtp193625667-35] server.StreamingServer:357 : consumer start fail for cube:UserAnalysisCube
java.lang.NullPointerException
at org.apache.kylin.stream.source.kafka.consumer.KafkaConnector.open(KafkaConnector.java:86)
at org.apache.kylin.stream.core.consumer.StreamingConsumerChannel.start(StreamingConsumerChannel.java:85)
at org.apache.kylin.stream.server.StreamingServer.startConsumer(StreamingServer.java:355)
at org.apache.kylin.stream.server.StreamingServer.startConsumers(StreamingServer.java:327)
at org.apache.kylin.stream.server.rest.controller.AdminController.assign(AdminController.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:849)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:760)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:547)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:748)
```
2. Let try doing a reassignment.
<img width="1517" alt="image" src="https://user-images.githubusercontent.com/14030549/89735825-bd5b1000-da97-11ea-9944-ae54331002aa.png">
3. I have to modify the source code and repackage. Here is my what I change:
```java
@Override
public void open() {
if (partitions == null || partitions.size() <= 0) {
throw new IllegalStateException("not assign partitions");
}
List<TopicPartition> topicPartitions = Lists.newArrayList();
for (Partition partition : partitions) {
topicPartitions.add(new TopicPartition(topic, partition.getPartitionId()));
}
kafkaConsumer.assign(topicPartitions);
if (startMode == ConsumerStartMode.EARLIEST) {
kafkaConsumer.seekToBeginning(topicPartitions);
} else if (startMode == ConsumerStartMode.LATEST) {
kafkaConsumer.seekToEnd(topicPartitions);
} else {
for (TopicPartition topicPartition : topicPartitions) {
Long offset = partitionOffsets.getOrDefault(topicPartition.partition(), 0L);
if (offset == 0L) {
logger.warn("Assume {} is a new added partition, let's seek it to 0", topicPartition);
}
kafkaConsumer.seek(topicPartition, offset);
}
}
}
```
```sh
mvn clean package -skipTests
...
scp stream-receiver/target/kylin-stream-receiver-3.1.1-SNAPSHOT-all.jar root@cdh-master:/root/xiaoxiang.yu/older_version/apache-kylin-3.1.0-bin-cdh57-receiver/lib/streaming
```
In receiver side, remove old kylin-stream-receiver-3.1.0-all.jar .
```
[root@cdh-master apache-kylin-3.1.0-bin-cdh57-receiver]# mv lib/streaming/kylin-stream-receiver-3.1.0-all.jar kylin-stream-receiver-3.1.0-all.jar.bak
```
4. Restart receiver and here is what we see:
```
2020-08-09 23:59:10,436 INFO [main] server.StreamingServer:725 : Using /root/xiaoxiang.yu/older_version/apache-kylin-3.1.0-bin-cdh57-receiver/stream_index to store local segment cache.
2020-08-09 23:59:10,437 INFO [main] server.StreamingServer:301 : register receiver: cdh-master:7514
2020-08-09 23:59:10,438 WARN [main] coordinator.ZookeeperStreamMetadataStore:488 : /stream/receivers/cdh-master_7514 exists.
2020-08-09 23:59:10,458 INFO [main] server.StreamingServer:421 : add the node to the replicaSet:0, join the group leader election.
2020-08-09 23:59:10,482 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.cube.CubeManager
2020-08-09 23:59:10,495 INFO [main] cube.CubeManager:136 : Initializing CubeManager with config REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:10,539 INFO [main] persistence.ResourceStore:90 : Using metadata url REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10 for resource store
2020-08-09 23:59:10,545 INFO [main] persistence.JDBCConnectionManager:92 : Connecting to Jdbc with url:jdbc:mysql://localhost:3306/NightlyBuild by user root
2020-08-09 23:59:10,559 INFO [Curator-LeaderSelector-0] server.ReplicaSetLeaderSelector:66 : become the leader of the replicaSet:0
2020-08-09 23:59:10,559 INFO [Curator-LeaderSelector-0] server.StreamingServer:575 : become leader of the replicaSet:0
2020-08-09 23:59:10,607 INFO [Curator-LeaderSelector-0] client.HttpCoordinatorClient:81 : send replicaSet lead change notification to coordinator
2020-08-09 23:59:11,194 INFO [main] persistence.JDBCConnectionManager:65 : Connected to MySQL 5.7.26
2020-08-09 23:59:11,212 INFO [main] persistence.JDBCResourceStore:115 : Table [REALTIME] already exists
2020-08-09 23:59:11,214 INFO [main] persistence.JDBCResourceStore:115 : Table [REALTIME_log] already exists
2020-08-09 23:59:11,217 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading CubeInstance from REALTIME(key='/cube')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,306 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.cube.CubeDescManager
2020-08-09 23:59:11,307 INFO [main] cube.CubeDescManager:91 : Initializing CubeDescManager with config REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,307 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading CubeDesc from REALTIME(key='/cube_desc')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,358 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.project.ProjectManager
2020-08-09 23:59:11,359 INFO [main] project.ProjectManager:85 : Initializing ProjectManager with metadata url REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,360 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading ProjectInstance from REALTIME(key='/project')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,369 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 1 ProjectInstance(s) out of 1 resource with 0 errors
2020-08-09 23:59:11,371 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.cachesync.Broadcaster
2020-08-09 23:59:11,373 DEBUG [main] cachesync.Broadcaster:102 : 1 nodes in the cluster: [localhost:7514]
2020-08-09 23:59:11,377 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.model.DataModelManager
2020-08-09 23:59:11,381 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.TableMetadataManager
2020-08-09 23:59:11,382 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading TableDesc from REALTIME(key='/table')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,400 INFO [main] measure.MeasureTypeFactory:121 : Checking custom measure types from kylin config
2020-08-09 23:59:11,401 INFO [main] measure.MeasureTypeFactory:150 : registering COUNT_DISTINCT(hllc), class org.apache.kylin.measure.hllc.HLLCMeasureType$Factory
2020-08-09 23:59:11,406 INFO [main] measure.MeasureTypeFactory:150 : registering COUNT_DISTINCT(bitmap), class org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory
2020-08-09 23:59:11,410 INFO [main] measure.MeasureTypeFactory:150 : registering TOP_N(topn), class org.apache.kylin.measure.topn.TopNMeasureType$Factory
2020-08-09 23:59:11,412 INFO [main] measure.MeasureTypeFactory:150 : registering RAW(raw), class org.apache.kylin.measure.raw.RawMeasureType$Factory
2020-08-09 23:59:11,414 INFO [main] measure.MeasureTypeFactory:150 : registering EXTENDED_COLUMN(extendedcolumn), class org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory
2020-08-09 23:59:11,415 INFO [main] measure.MeasureTypeFactory:150 : registering PERCENTILE_APPROX(percentile), class org.apache.kylin.measure.percentile.PercentileMeasureType$Factory
2020-08-09 23:59:11,416 INFO [main] measure.MeasureTypeFactory:150 : registering COUNT_DISTINCT(dim_dc), class org.apache.kylin.measure.dim.DimCountDistinctMeasureType$Factory
2020-08-09 23:59:11,418 INFO [main] measure.MeasureTypeFactory:150 : registering STDDEV_SUM(stddev_sum), class org.apache.kylin.measure.stddev.StdDevSumMeasureType$Factory
2020-08-09 23:59:11,431 INFO [main] measure.MeasureTypeFactory:150 : registering COUNT_DISTINCT(bitmap_map), class org.apache.kylin.measure.map.bitmap.BitmapMapMeasureType$Factory
2020-08-09 23:59:11,438 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 1 TableDesc(s) out of 1 resource with 0 errors
2020-08-09 23:59:11,439 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading TableExtDesc from REALTIME(key='/table_exd')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,442 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 0 TableExtDesc(s) out of 0 resource with 0 errors
2020-08-09 23:59:11,443 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading ExternalFilterDesc from REALTIME(key='/ext_filter')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,445 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 0 ExternalFilterDesc(s) out of 0 resource with 0 errors
2020-08-09 23:59:11,445 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading DataModelDesc from REALTIME(key='/model_desc')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,472 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 1 DataModelDesc(s) out of 1 resource with 0 errors
2020-08-09 23:59:11,494 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 1 CubeDesc(s) out of 1 resource with 0 errors
2020-08-09 23:59:11,495 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 1 CubeInstance(s) out of 1 resource with 0 errors
2020-08-09 23:59:11,518 INFO [main] storage.StreamingSegmentManager:239 : restore segments from local files for cube:UserAnalysisCube
2020-08-09 23:59:11,522 WARN [main] storage.StreamingSegmentManager:253 : no checkpoint for cube:UserAnalysisCube store state, remove all local segments folders
2020-08-09 23:59:11,522 INFO [main] server.StreamingServer:351 : create and start new consumer for cube:UserAnalysisCube
2020-08-09 23:59:11,550 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.cube.cuboid.CuboidManager
2020-08-09 23:59:11,566 INFO [main] kafka.KafkaSource:141 : Create StreamingConnector for Cube UserAnalysisCube, assignedPartitions [Partition [partitionId=0, partitionInfo=null], Partition [partitionId=5, partitionInfo=null], Partition [partitionId=10, partitionInfo=null], Partition [partitionId=15, partitionInfo=null], Partition [partitionId=13, partitionInfo=null], Partition [partitionId=2, partitionInfo=null], Partition [partitionId=8, partitionInfo=null], Partition [partitionId=12, partitionInfo=null], Partition [partitionId=14, partitionInfo=null], Partition [partitionId=9, partitionInfo=null], Partition [partitionId=11, partitionInfo=null], Partition [partitionId=1, partitionInfo=null], Partition [partitionId=4, partitionInfo=null], Partition [partitionId=6, partitionInfo=null], Partition [partitionId=7, partitionInfo=null], Partition [partitionId=3, partitionInfo=null]], startProtocol null
2020-08-09 23:59:11,571 INFO [main] kafka.TimedJsonStreamParser:98 : Using parser field mapping by {content_list=content_list, act_type=act_type, event_ts_2=event_ts_2, event_ts=event_ts, devide_type=user_detail.devide_type, location_city=user_detail.location.city, network_type=user_detail.network_type, video_id=video_id, event_date_2=event_date_2, str_minute=str_minute, video_type=video_type, play_times=play_times, pageview_id=pageview_id, active_minutes=active_minutes, device_brand=device_brand, str_minute_second=str_minute_second, play_duration=play_duration, event_date=event_date, page_id=page_id, str_second=str_second, uid=uid}
2020-08-09 23:59:11,584 INFO [main] kafka.TimedJsonStreamParser:120 : TimedJsonStreamParser with formatTs false tsColName event_date
2020-08-09 23:59:11,663 INFO [main] consumer.ConsumerConfig:279 : ConsumerConfig values:
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [cdh-master:9092, cdh-worker-1:9092, cdh-worker-2:9092]
check.crcs = true
client.dns.lookup = default
client.id =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id =
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
isolation.level = read_uncommitted
key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
session.timeout.ms = 20000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
2020-08-09 23:59:11,790 INFO [main] utils.AppInfoParser:109 : Kafka version : 2.1.1
2020-08-09 23:59:11,790 INFO [main] utils.AppInfoParser:110 : Kafka commitId : 21234bee31165527
2020-08-09 23:59:11,792 INFO [main] kafka.KafkaSource:192 : localConsumeStats from local checkpoint null, remoteConsumeStats from remote checkpoint {"0":363171,"1":363198,"2":363249,"3":363171,"4":363199,"5":363250,"6":363170,"7":363196,"8":363250,"9":363170}
2020-08-09 23:59:11,793 INFO [main] kafka.KafkaSource:239 : start kafka connector from specified position:KafkaPosition{partitionOffsetMap={0=363171, 1=363198, 2=363249, 3=363171, 4=363199, 5=363250, 6=363170, 7=363196, 8=363250, 9=363170}}
2020-08-09 23:59:11,798 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-10 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-15 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-13 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-12 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-14 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-11 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,801 INFO [main] metrics.StreamingMetrics:76 : Skip streaming metricRegistry because it is empty.
2020-08-09 23:59:11,819 INFO [main] util.log:169 : Logging initialized @3474ms to org.eclipse.jetty.util.log.Slf4jLog
2020-08-09 23:59:11,946 INFO [UserAnalysisCube_channel] clients.Metadata:285 : Cluster ID: W1YX_q9GRd-R0qwcwFJ9rg
2020-08-09 23:59:12,163 INFO [UserAnalysisCube_channel] storage.StreamingSegmentManager:189 : Create new segment:StreamingCubeSegment [cubeName=UserAnalysisCube, segmentName=20200809210000_20200809220000]
2020-08-09 23:59:12,826 INFO [UserAnalysisCube_channel] storage.StreamingSegmentManager:189 : Create new segment:StreamingCubeSegment [cubeName=UserAnalysisCube, segmentName=20200809230000_20200810000000]
2020-08-09 23:59:13,226 INFO [main] server.Server:359 : jetty-9.4.26.v20200117; built: 2020-01-17T12:35:33.676Z; git: 7b38981d25d14afb4a12ff1f2596756144edf695; jvm 1.8.0_171-b11
2020-08-09 23:59:13,260 INFO [main] ContextHandler.kylin:2223 : Initializing Spring FrameworkServlet 'org.springframework.web.servlet.DispatcherServlet-d2e6b0b'
2020-08-09 23:59:13,278 INFO [main] handler.ContextHandler:825 : Started o.e.j.s.ServletContextHandler@708eb310{/kylin,null,AVAILABLE}
2020-08-09 23:59:13,279 INFO [main] handler.ContextHandler:825 : Started o.e.j.s.h.ContextHandler@6ebc9573{/kylin/logs,null,AVAILABLE}
2020-08-09 23:59:13,294 INFO [main] server.AbstractConnector:330 : Started ServerConnector@369c9bb{HTTP/1.1,[http/1.1]}{0.0.0.0:7514}
2020-08-09 23:59:13,294 INFO [main] server.Server:399 : Started @4949ms
2020-08-09 23:59:15,572 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:106 : Start persist memory store for cube:UserAnalysisCube, segment:20200809210000_20200809220000, rowCnt:50000
2020-08-09 23:59:15,635 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:202 : cuboid-8191 transform to columnar, take 58 ms
2020-08-09 23:59:15,655 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '00000000000000003640', current '00000000000000003638'
2020-08-09 23:59:15,719 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 15962
2020-08-09 23:59:15,720 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:15,720 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,725 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 3640=>1219, 3638=>1218, 3630=>1212, 3631=>1213, 3632=>1214
2020-08-09 23:59:15,725 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : UID took : 88 ms
2020-08-09 23:59:15,738 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'page_19', current 'page_13'
2020-08-09 23:59:15,739 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 50
2020-08-09 23:59:15,739 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:15,740 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,740 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: page_19=>11, page_13=>5, page_14=>6, page_11=>3, page_12=>4
2020-08-09 23:59:15,740 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : PAGE_ID took : 13 ms
2020-08-09 23:59:15,751 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'stop', current 'pause'
2020-08-09 23:59:15,752 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 10
2020-08-09 23:59:15,752 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:15,752 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,752 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: stop=>9, pause=>5, play=>6, like=>4, start=>8
2020-08-09 23:59:15,752 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : ACT_TYPE took : 11 ms
2020-08-09 23:59:15,763 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 34
2020-08-09 23:59:15,763 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:15,763 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,764 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 0=>0, 1=>1, 2=>2, 3=>3, 4=>4
2020-08-09 23:59:15,764 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVIDE_TYPE took : 10 ms
2020-08-09 23:59:15,774 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'hongkong', current 'hangzhou'
2020-08-09 23:59:15,774 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 7
2020-08-09 23:59:15,774 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:15,775 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,775 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: hongkong=>3, hangzhou=>2, shenzhen=>6, berlin=>1, nanjing=>4
2020-08-09 23:59:15,775 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : LOCATION_CITY took : 10 ms
2020-08-09 23:59:15,785 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 4
2020-08-09 23:59:15,785 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:15,785 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,785 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 2=>0, 3=>1, 4=>2, 5=>3
2020-08-09 23:59:15,785 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : NETWORK_TYPE took : 10 ms
2020-08-09 23:59:15,794 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'xiaomi', current 'vivo'
2020-08-09 23:59:15,795 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 5
2020-08-09 23:59:15,795 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:15,795 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,795 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: huawei=>0, xiaomi=>4, vivo=>3, meizu=>2, iPhone=>1
2020-08-09 23:59:15,795 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVICE_BRAND took : 9 ms
2020-08-09 23:59:15,806 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '21_14_59', current '21_14_50'
2020-08-09 23:59:15,810 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 2032
2020-08-09 23:59:15,810 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:15,810 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,810 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 21_14_59=>355, 21_14_50=>346, 21_14_55=>351, 21_14_56=>352, 21_14_57=>353
2020-08-09 23:59:15,810 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : STR_MINUTE_SECOND took : 15 ms
2020-08-09 23:59:16,088 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:UID, took: 276ms
2020-08-09 23:59:16,171 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:PAGE_ID, took: 83ms
2020-08-09 23:59:16,254 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:ACT_TYPE, took: 82ms
2020-08-09 23:59:16,316 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVIDE_TYPE, took: 61ms
2020-08-09 23:59:16,391 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:LOCATION_CITY, took: 75ms
2020-08-09 23:59:16,461 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:NETWORK_TYPE, took: 69ms
2020-08-09 23:59:16,532 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVICE_BRAND, took: 70ms
2020-08-09 23:59:16,608 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:STR_MINUTE_SECOND, took: 75ms
2020-08-09 23:59:16,646 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MONTH_START, took: 38ms
2020-08-09 23:59:16,673 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:WEEK_START, took: 26ms
2020-08-09 23:59:16,703 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DAY_START, took: 29ms
2020-08-09 23:59:16,736 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:HOUR_START, took: 33ms
2020-08-09 23:59:16,779 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MINUTE_START, took: 41ms
2020-08-09 23:59:16,808 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:_COUNT_, took: 29ms
2020-08-09 23:59:16,871 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_TIMES, took: 63ms
2020-08-09 23:59:16,907 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_ACTIVE_MINUTES, took: 34ms
2020-08-09 23:59:16,932 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_DURATION, took: 25ms
2020-08-09 23:59:16,993 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_TIMES, took: 60ms
2020-08-09 23:59:17,027 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_ACTIVE_MINUTES, took: 33ms
2020-08-09 23:59:17,049 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_DURATION, took: 22ms
2020-08-09 23:59:17,081 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:PV_STAT, took: 31ms
2020-08-09 23:59:17,089 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:110 : Finish persist memory store for cube:UserAnalysisCube segment:20200809210000_20200809220000, take: 1517ms
2020-08-09 23:59:18,954 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:106 : Start persist memory store for cube:UserAnalysisCube, segment:20200809210000_20200809220000, rowCnt:50000
2020-08-09 23:59:18,980 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:202 : cuboid-8191 transform to columnar, take 26 ms
2020-08-09 23:59:18,989 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '00000000000000003640', current '00000000000000003638'
2020-08-09 23:59:19,008 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 16046
2020-08-09 23:59:19,008 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:19,008 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,008 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 3640=>1240, 3638=>1238, 3639=>1239, 3630=>1231, 3631=>1232
2020-08-09 23:59:19,008 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : UID took : 28 ms
2020-08-09 23:59:19,017 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'page_19', current 'page_13'
2020-08-09 23:59:19,017 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 50
2020-08-09 23:59:19,017 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:19,017 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,018 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: page_19=>11, page_13=>5, page_14=>6, page_11=>3, page_12=>4
2020-08-09 23:59:19,018 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : PAGE_ID took : 8 ms
2020-08-09 23:59:19,025 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'stop', current 'pause'
2020-08-09 23:59:19,026 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 10
2020-08-09 23:59:19,026 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:19,026 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,026 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: stop=>9, pause=>5, play=>6, like=>4, start=>8
2020-08-09 23:59:19,026 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : ACT_TYPE took : 8 ms
2020-08-09 23:59:19,035 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 44
2020-08-09 23:59:19,035 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:19,035 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,035 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 16=>0, 17=>1, 18=>2, 19=>3, 20=>4
2020-08-09 23:59:19,035 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVIDE_TYPE took : 8 ms
2020-08-09 23:59:19,043 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'hongkong', current 'hangzhou'
2020-08-09 23:59:19,043 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 7
2020-08-09 23:59:19,043 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:19,043 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,043 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: hongkong=>3, hangzhou=>2, shenzhen=>6, berlin=>1, nanjing=>4
2020-08-09 23:59:19,044 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : LOCATION_CITY took : 8 ms
2020-08-09 23:59:19,052 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 4
2020-08-09 23:59:19,052 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:19,052 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,052 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 2=>0, 3=>1, 4=>2, 5=>3
2020-08-09 23:59:19,052 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : NETWORK_TYPE took : 8 ms
2020-08-09 23:59:19,059 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'xiaomi', current 'vivo'
2020-08-09 23:59:19,060 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 5
2020-08-09 23:59:19,060 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:19,060 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,060 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: huawei=>0, xiaomi=>4, vivo=>3, meizu=>2, iPhone=>1
2020-08-09 23:59:19,060 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVICE_BRAND took : 7 ms
2020-08-09 23:59:19,069 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '21_55_9', current '21_55_0'
2020-08-09 23:59:19,077 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 2536
2020-08-09 23:59:19,077 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:19,077 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,077 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 21_55_7=>2341, 21_55_8=>2342, 21_55_9=>2343, 21_55_0=>2284, 21_55_1=>2285
2020-08-09 23:59:19,077 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : STR_MINUTE_SECOND took : 17 ms
2020-08-09 23:59:19,185 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:UID, took: 107ms
2020-08-09 23:59:19,235 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:PAGE_ID, took: 49ms
2020-08-09 23:59:19,284 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:ACT_TYPE, took: 48ms
2020-08-09 23:59:19,335 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVIDE_TYPE, took: 50ms
2020-08-09 23:59:19,396 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:LOCATION_CITY, took: 60ms
2020-08-09 23:59:19,455 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:NETWORK_TYPE, took: 59ms
2020-08-09 23:59:19,522 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVICE_BRAND, took: 66ms
2020-08-09 23:59:19,592 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:STR_MINUTE_SECOND, took: 69ms
2020-08-09 23:59:19,610 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MONTH_START, took: 17ms
2020-08-09 23:59:19,626 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:WEEK_START, took: 16ms
2020-08-09 23:59:19,642 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DAY_START, took: 15ms
2020-08-09 23:59:19,662 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:HOUR_START, took: 19ms
2020-08-09 23:59:19,691 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MINUTE_START, took: 28ms
2020-08-09 23:59:19,708 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:_COUNT_, took: 16ms
2020-08-09 23:59:19,766 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_TIMES, took: 57ms
2020-08-09 23:59:19,783 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_ACTIVE_MINUTES, took: 17ms
2020-08-09 23:59:19,801 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_DURATION, took: 17ms
2020-08-09 23:59:19,856 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_TIMES, took: 54ms
2020-08-09 23:59:19,873 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_ACTIVE_MINUTES, took: 17ms
2020-08-09 23:59:19,890 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_DURATION, took: 16ms
2020-08-09 23:59:19,908 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:PV_STAT, took: 18ms
2020-08-09 23:59:19,913 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:110 : Finish persist memory store for cube:UserAnalysisCube segment:20200809210000_20200809220000, take: 959ms
2020-08-09 23:59:20,175 INFO [UserAnalysisCube_channel] storage.StreamingSegmentManager:189 : Create new segment:StreamingCubeSegment [cubeName=UserAnalysisCube, segmentName=20200809220000_20200809230000]
2020-08-09 23:59:22,201 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:106 : Start persist memory store for cube:UserAnalysisCube, segment:20200809220000_20200809230000, rowCnt:50000
2020-08-09 23:59:22,235 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:202 : cuboid-8191 transform to columnar, take 33 ms
2020-08-09 23:59:22,245 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '00000000000000003640', current '00000000000000003638'
2020-08-09 23:59:22,258 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 15977
2020-08-09 23:59:22,258 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:22,258 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,258 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 3640=>1166, 3638=>1165, 3630=>1158, 3631=>1159, 3632=>1160
2020-08-09 23:59:22,258 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : UID took : 23 ms
2020-08-09 23:59:22,267 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'page_19', current 'page_13'
2020-08-09 23:59:22,268 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 50
2020-08-09 23:59:22,268 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:22,268 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,268 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: page_19=>11, page_13=>5, page_14=>6, page_11=>3, page_12=>4
2020-08-09 23:59:22,268 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : PAGE_ID took : 9 ms
2020-08-09 23:59:22,277 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'stop', current 'pause'
2020-08-09 23:59:22,277 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 10
2020-08-09 23:59:22,277 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:22,277 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,277 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: stop=>9, pause=>5, play=>6, like=>4, start=>8
2020-08-09 23:59:22,277 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : ACT_TYPE took : 9 ms
2020-08-09 23:59:22,285 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 34
2020-08-09 23:59:22,285 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:22,286 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,286 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 0=>0, 1=>1, 2=>2, 3=>3, 4=>4
2020-08-09 23:59:22,286 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVIDE_TYPE took : 8 ms
2020-08-09 23:59:22,294 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'hongkong', current 'hangzhou'
2020-08-09 23:59:22,294 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 7
2020-08-09 23:59:22,294 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:22,295 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,295 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: hongkong=>3, hangzhou=>2, shenzhen=>6, berlin=>1, nanjing=>4
2020-08-09 23:59:22,295 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : LOCATION_CITY took : 8 ms
2020-08-09 23:59:22,303 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 4
2020-08-09 23:59:22,303 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:22,303 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,303 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 2=>0, 3=>1, 4=>2, 5=>3
2020-08-09 23:59:22,303 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : NETWORK_TYPE took : 8 ms
2020-08-09 23:59:22,311 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'xiaomi', current 'vivo'
2020-08-09 23:59:22,311 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 5
2020-08-09 23:59:22,311 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:22,311 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,311 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: huawei=>0, xiaomi=>4, vivo=>3, meizu=>2, iPhone=>1
2020-08-09 23:59:22,311 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVICE_BRAND took : 8 ms
2020-08-09 23:59:22,321 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '22_14_59', current '22_14_41'
2020-08-09 23:59:22,323 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 1985
2020-08-09 23:59:22,324 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:22,324 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,324 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 22_14_58=>354, 22_14_59=>355, 22_14_41=>336, 22_14_42=>337, 22_14_40=>335
2020-08-09 23:59:22,324 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : STR_MINUTE_SECOND took : 12 ms
2020-08-09 23:59:22,426 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:UID, took: 102ms
2020-08-09 23:59:22,477 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:PAGE_ID, took: 50ms
2020-08-09 23:59:22,526 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:ACT_TYPE, took: 49ms
2020-08-09 23:59:22,574 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVIDE_TYPE, took: 47ms
2020-08-09 23:59:22,706 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:LOCATION_CITY, took: 61ms
2020-08-09 23:59:22,768 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:NETWORK_TYPE, took: 61ms
2020-08-09 23:59:22,834 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVICE_BRAND, took: 66ms
2020-08-09 23:59:22,903 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:STR_MINUTE_SECOND, took: 68ms
2020-08-09 23:59:22,920 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MONTH_START, took: 16ms
2020-08-09 23:59:22,937 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:WEEK_START, took: 16ms
2020-08-09 23:59:22,953 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DAY_START, took: 16ms
2020-08-09 23:59:22,971 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:HOUR_START, took: 17ms
2020-08-09 23:59:22,995 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MINUTE_START, took: 24ms
2020-08-09 23:59:23,008 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:_COUNT_, took: 12ms
2020-08-09 23:59:23,064 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_TIMES, took: 55ms
2020-08-09 23:59:23,082 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_ACTIVE_MINUTES, took: 17ms
2020-08-09 23:59:23,100 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_DURATION, took: 17ms
2020-08-09 23:59:23,157 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_TIMES, took: 56ms
2020-08-09 23:59:23,175 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_ACTIVE_MINUTES, took: 17ms
2020-08-09 23:59:23,193 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_DURATION, took: 18ms
2020-08-09 23:59:23,213 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:PV_STAT, took: 19ms
2020-08-09 23:59:23,214 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:110 : Finish persist memory store for cube:UserAnalysisCube segment:20200809220000_20200809230000, take: 1012ms
2020-08-09 23:59:25,467 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:106 : Start persist memory store for cube:UserAnalysisCube, segment:20200809230000_20200810000000, rowCnt:50000
2020-08-09 23:59:25,495 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:202 : cuboid-8191 transform to columnar, take 26 ms
2020-08-09 23:59:25,504 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '00000000000000003639', current '00000000000000003630'
2020-08-09 23:59:25,519 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 15936
2020-08-09 23:59:25,519 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:25,519 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,519 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 3638=>1084, 3639=>1085, 3630=>1077, 3631=>1078, 3632=>1079
2020-08-09 23:59:25,519 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : UID took : 24 ms
2020-08-09 23:59:25,532 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'page_19', current 'page_13'
2020-08-09 23:59:25,533 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 50
2020-08-09 23:59:25,533 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:25,533 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,533 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: page_19=>11, page_13=>5, page_14=>6, page_11=>3, page_12=>4
2020-08-09 23:59:25,533 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : PAGE_ID took : 13 ms
2020-08-09 23:59:25,542 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'stop', current 'pause'
2020-08-09 23:59:25,543 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 10
2020-08-09 23:59:25,543 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:25,543 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,543 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: stop=>9, pause=>5, play=>6, like=>4, start=>8
2020-08-09 23:59:25,543 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : ACT_TYPE took : 9 ms
2020-08-09 23:59:25,553 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 41
2020-08-09 23:59:25,553 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:25,553 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,553 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 19=>0, 20=>1, 21=>2, 22=>3, 23=>4
2020-08-09 23:59:25,553 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVIDE_TYPE took : 10 ms
2020-08-09 23:59:25,562 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'hongkong', current 'hangzhou'
2020-08-09 23:59:25,563 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 7
2020-08-09 23:59:25,563 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:25,563 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,563 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: hongkong=>3, hangzhou=>2, shenzhen=>6, berlin=>1, nanjing=>4
2020-08-09 23:59:25,563 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : LOCATION_CITY took : 9 ms
2020-08-09 23:59:25,572 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 4
2020-08-09 23:59:25,573 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:25,573 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,573 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 2=>0, 3=>1, 4=>2, 5=>3
2020-08-09 23:59:25,573 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : NETWORK_TYPE took : 9 ms
2020-08-09 23:59:25,582 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'xiaomi', current 'vivo'
2020-08-09 23:59:25,583 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 5
2020-08-09 23:59:25,583 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:25,583 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,583 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: huawei=>0, xiaomi=>4, vivo=>3, meizu=>2, iPhone=>1
2020-08-09 23:59:25,583 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVICE_BRAND took : 10 ms
2020-08-09 23:59:25,594 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '23_53_19', current '23_53_10'
2020-08-09 23:59:25,596 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 2412
2020-08-09 23:59:25,596 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:25,596 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,596 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 23_53_16=>2035, 23_53_17=>2036, 23_53_18=>2037, 23_53_19=>2038, 23_53_10=>2029
2020-08-09 23:59:25,596 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : STR_MINUTE_SECOND took : 12 ms
2020-08-09 23:59:25,702 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:UID, took: 105ms
2020-08-09 23:59:25,757 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:PAGE_ID, took: 54ms
2020-08-09 23:59:25,812 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:ACT_TYPE, took: 54ms
2020-08-09 23:59:25,866 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVIDE_TYPE, took: 54ms
2020-08-09 23:59:25,932 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:LOCATION_CITY, took: 65ms
2020-08-09 23:59:25,996 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:NETWORK_TYPE, took: 63ms
2020-08-09 23:59:26,065 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVICE_BRAND, took: 68ms
2020-08-09 23:59:26,138 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:STR_MINUTE_SECOND, took: 73ms
2020-08-09 23:59:26,157 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MONTH_START, took: 18ms
2020-08-09 23:59:26,175 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:WEEK_START, took: 18ms
2020-08-09 23:59:26,193 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DAY_START, took: 17ms
2020-08-09 23:59:26,210 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:HOUR_START, took: 17ms
2020-08-09 23:59:26,235 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MINUTE_START, took: 24ms
2020-08-09 23:59:26,256 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:_COUNT_, took: 20ms
2020-08-09 23:59:26,316 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_TIMES, took: 60ms
2020-08-09 23:59:26,336 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_ACTIVE_MINUTES, took: 19ms
2020-08-09 23:59:26,355 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_DURATION, took: 18ms
2020-08-09 23:59:26,412 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_TIMES, took: 57ms
2020-08-09 23:59:26,431 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_ACTIVE_MINUTES, took: 18ms
2020-08-09 23:59:26,450 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_DURATION, took: 18ms
2020-08-09 23:59:26,470 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:PV_STAT, took: 20ms
2020-08-09 23:59:26,471 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:110 : Finish persist memory store for cube:UserAnalysisCube segment:20200809230000_20200810000000, take: 1003ms
```
The key part is the folowing:
```
2020-08-09 23:59:11,792 INFO [main] kafka.KafkaSource:192 : localConsumeStats from local checkpoint null, remoteConsumeStats from remote checkpoint {"0":363171,"1":363198,"2":363249,"3":363171,"4":363199,"5":363250,"6":363170,"7":363196,"8":363250,"9":363170}
2020-08-09 23:59:11,793 INFO [main] kafka.KafkaSource:239 : start kafka connector from specified position:KafkaPosition{partitionOffsetMap={0=363171, 1=363198, 2=363249, 3=363171, 4=363199, 5=363250, 6=363170, 7=363196, 8=363250, 9=363170}}
2020-08-09 23:59:11,798 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-10 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-15 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-13 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-12 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-14 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-11 is a new added partition, let's seek it to 0
```
It indicated taht receiver consume from position, which shoulf be the correct way:
```json
{
"0":363171,
"1":363198,
"2":363249,
"3":363171,
"4":363199,
"5":363250,
"6":363170,
"7":363196,
"8":363250,
"9":363170,
"10":0,
"11":0,
"12":0,
"13":0,
"14":0,
"15":0
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] tianhui5 commented on pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
tianhui5 commented on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-671132270
> Thank you for you effort that you spend on this issue, here is my suggestion, what do you think?
> If `offset` is null, please just set it to zero.
I wonder if there is any difference between setting offset to zero and seekToBeginning.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] coveralls edited a comment on pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-669893863
## Pull Request Test Coverage Report for [Build 6234](https://coveralls.io/builds/32636098)
* **0** of **5** **(0.0%)** changed or added relevant lines in **1** file are covered.
* **3** unchanged lines in **2** files lost coverage.
* Overall coverage decreased (**-0.001%**) to **28.027%**
---
| Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
| :-----|--------------|--------|---: |
| [stream-source-kafka/src/main/java/org/apache/kylin/stream/source/kafka/consumer/KafkaConnector.java](https://coveralls.io/builds/32636098/source?filename=stream-source-kafka%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fsource%2Fkafka%2Fconsumer%2FKafkaConnector.java#L84) | 0 | 5 | 0.0%
<!-- | **Total:** | **0** | **5** | **0.0%** | -->
| Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [stream-source-kafka/src/main/java/org/apache/kylin/stream/source/kafka/consumer/KafkaConnector.java](https://coveralls.io/builds/32636098/source?filename=stream-source-kafka%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fsource%2Fkafka%2Fconsumer%2FKafkaConnector.java#L92) | 1 | 0% |
| [core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java](https://coveralls.io/builds/32636098/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fthreadpool%2FDefaultScheduler.java#L194) | 2 | 80.23% |
<!-- | **Total:** | **3** | | -->
| Totals | [](https://coveralls.io/builds/32636098) |
| :-- | --: |
| Change from base [Build 6232](https://coveralls.io/builds/32631719): | -0.001% |
| Covered Lines: | 26236 |
| Relevant Lines: | 93609 |
---
##### 💛 - [Coveralls](https://coveralls.io)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-671059337
Before we scale up the topic partition , we should doing the following steps:
1. Disable the cube, thus all consumption task will be cacelled.
2. Use the REST API `http://${KYLIN_INSTANCE_IP}7236/kylin/cubes/view/${CUBE_NAME}/instancejson` to check the `CubeInstance.json`. You will find each READY segment has a property named `stream_source_checkpoint`. Here s part of its content:
```json
{
"uuid": "aab58181-29f7-0593-d7f8-ca9d9b97d49b",
"name": "20200809200000_20200809210000",
"storage_location_identifier": "APACHE:REALTIME_OLAP_ISH2P3HM30",
"date_range_start": 1597003200000,
"date_range_end": 1597006800000,
"source_offset_start": 0,
"source_offset_end": 0,
"status": "READY",
"size_kb": 102984,
"is_merged": false,
"estimate_ratio": null,
"input_records": 118216,
"input_records_size": 0,
"last_build_time": 1596982095554,
"last_build_job_id": "8f3660b8-72f7-7aac-e3bd-1e06ea30d8e2",
"create_time_utc": 1596981735548,
"cuboid_shard_nums": {},
"total_shards": 1,
"blackout_cuboids": [],
"binary_signature": null,
"dictionaries": {
"USERACTIONSTREAM.DEVICE_BRAND": "/dict/APACHE.USERACTIONSTREAM/DEVICE_BRAND/afb80e63-4fef-d575-3483-1e0314bf4bef.dict",
"USERACTIONSTREAM.DEVIDE_TYPE": "/dict/APACHE.USERACTIONSTREAM/DEVIDE_TYPE/61bf9051-3cdd-ec82-bc0f-1bb3226bb411.dict",
"USERACTIONSTREAM.LOCATION_CITY": "/dict/APACHE.USERACTIONSTREAM/LOCATION_CITY/292ce446-62a1-0b12-e4d0-13bc249b0dbe.dict",
"USERACTIONSTREAM.PAGE_ID": "/dict/APACHE.USERACTIONSTREAM/PAGE_ID/12fa188b-0f89-db3d-560a-8aea5b970349.dict",
"USERACTIONSTREAM.NETWORK_TYPE": "/dict/APACHE.USERACTIONSTREAM/NETWORK_TYPE/99d38dea-25ef-c03d-73b7-19a6fda2ce4c.dict",
"USERACTIONSTREAM.STR_MINUTE_SECOND": "/dict/APACHE.USERACTIONSTREAM/STR_MINUTE_SECOND/2b23e9aa-dee0-b88f-7e3e-0a3d74d89f89.dict",
"USERACTIONSTREAM.ACT_TYPE": "/dict/APACHE.USERACTIONSTREAM/ACT_TYPE/e77f008d-6bd8-bc1f-ba94-ff62764c3e14.dict",
"USERACTIONSTREAM.UID": "/dict/APACHE.USERACTIONSTREAM/UID/665546b1-424a-fc42-a35b-1a58fcd1fb5f.dict"
},
"snapshots": null,
"rowkey_stats": [
[
"ACT_TYPE",
10,
1
],
[
"NETWORK_TYPE",
4,
1
],
[
"LOCATION_CITY",
7,
1
],
[
"STR_MINUTE_SECOND",
3600,
2
],
[
"PAGE_ID",
50,
1
],
[
"DEVICE_BRAND",
5,
1
],
[
"DEVIDE_TYPE",
60,
1
],
[
"UID",
19543,
2
]
],
"stream_source_checkpoint": "{\"0\":363171,\"1\":363198,\"2\":363249,\"3\":363171,\"4\":363199,\"5\":363250,\"6\":363170,\"7\":363196,\"8\":363250,\"9\":363170}"
}
```
3. Check `${KYLIN_RECEIVER_HOME}/logs/kylin_streaming_receiver.log`, you can find the some output:
```
2020-08-09 22:34:05,381 INFO [UserAnalysisCube_channel] storage.StreamingSegmentManager:645 : Print check point for cube UserAnalysisCube ,CheckPoint{sourceConsumePosition='{"0":381733,"1":381763,"2":381820,"3":381733,"4":381764,"5":381820,"6":381732,"7":381760,"8":381821,"9":381733}', persistedIndexes={1597006800000=13, 1597010400000=8}, longLatencyInfo=LongLatencyInfo{longLatencyEventCnts={20200808000000_20200808010000=3, 20200808060000_20200808070000=2, 20200809000000_20200809010000=2, 20200809060000_20200809070000=2}, totalLongLatencyEventCnt=9}, segmentSourceStartPosition={1597006800000={"0":363171,"1":363198,"2":363249,"3":363171,"4":363199,"5":363250,"6":363170,"7":363196,"8":363250,"9":363170}, 1597010400000={"0":375013,"1":375040,"2":375099,"3":375013,"4":375041,"5":375099,"6":375012,"7":375038,"8":375100,"9":375013}}, checkPointTime=1596983645381, totalCount=3817689, checkPointCount=5801}
```
These logs indicated that the data ingetsed and indexed in receiver side is checkpointed at position :
```json
{
"0":375013,
"1":375040,
"2":375099,
"3":375013,
"4":375041,
"5":375099,
"6":375012,
"7":375038,
"8":375100,
"9":375013
}
```
4. When disable cube , data ingetsed and indexed in receiver side will be removed, so when scaled up, we expected receiver will continue its consumpution after following position:
```json
{
"0":363171,
"1":363198,
"2":363249,
"3":363171,
"4":363199,
"5":363250,
"6":363170,
"7":363196,
"8":363250,
"9":363170
}
```
5. So let's check if it is correct ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] tianhui5 commented on pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
tianhui5 commented on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-669855782
I think there is some network failure of CI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-671059337
Before we scale up the topic partition , we should doing the following steps:
1. Disable the cube, thus all consumption task will be cacelled.
2. Use the REST API `http://${KYLIN_INSTANCE_IP}:${KYLIN_INSTANCE_PORT}/kylin/cubes/view/${CUBE_NAME}/instancejson` to check the `CubeInstance.json`. You will find each READY segment has a property named `stream_source_checkpoint`. Here s part of its content:
```json
{
"uuid": "aab58181-29f7-0593-d7f8-ca9d9b97d49b",
"name": "20200809200000_20200809210000",
"storage_location_identifier": "APACHE:REALTIME_OLAP_ISH2P3HM30",
"date_range_start": 1597003200000,
"date_range_end": 1597006800000,
"source_offset_start": 0,
"source_offset_end": 0,
"status": "READY",
"size_kb": 102984,
"is_merged": false,
"estimate_ratio": null,
"input_records": 118216,
"input_records_size": 0,
"last_build_time": 1596982095554,
"last_build_job_id": "8f3660b8-72f7-7aac-e3bd-1e06ea30d8e2",
"create_time_utc": 1596981735548,
"cuboid_shard_nums": {},
"total_shards": 1,
"blackout_cuboids": [],
"binary_signature": null,
"dictionaries": {
"USERACTIONSTREAM.DEVICE_BRAND": "/dict/APACHE.USERACTIONSTREAM/DEVICE_BRAND/afb80e63-4fef-d575-3483-1e0314bf4bef.dict",
"USERACTIONSTREAM.DEVIDE_TYPE": "/dict/APACHE.USERACTIONSTREAM/DEVIDE_TYPE/61bf9051-3cdd-ec82-bc0f-1bb3226bb411.dict",
"USERACTIONSTREAM.LOCATION_CITY": "/dict/APACHE.USERACTIONSTREAM/LOCATION_CITY/292ce446-62a1-0b12-e4d0-13bc249b0dbe.dict",
"USERACTIONSTREAM.PAGE_ID": "/dict/APACHE.USERACTIONSTREAM/PAGE_ID/12fa188b-0f89-db3d-560a-8aea5b970349.dict",
"USERACTIONSTREAM.NETWORK_TYPE": "/dict/APACHE.USERACTIONSTREAM/NETWORK_TYPE/99d38dea-25ef-c03d-73b7-19a6fda2ce4c.dict",
"USERACTIONSTREAM.STR_MINUTE_SECOND": "/dict/APACHE.USERACTIONSTREAM/STR_MINUTE_SECOND/2b23e9aa-dee0-b88f-7e3e-0a3d74d89f89.dict",
"USERACTIONSTREAM.ACT_TYPE": "/dict/APACHE.USERACTIONSTREAM/ACT_TYPE/e77f008d-6bd8-bc1f-ba94-ff62764c3e14.dict",
"USERACTIONSTREAM.UID": "/dict/APACHE.USERACTIONSTREAM/UID/665546b1-424a-fc42-a35b-1a58fcd1fb5f.dict"
},
"snapshots": null,
"rowkey_stats": [
[
"ACT_TYPE",
10,
1
],
[
"NETWORK_TYPE",
4,
1
],
[
"LOCATION_CITY",
7,
1
],
[
"STR_MINUTE_SECOND",
3600,
2
],
[
"PAGE_ID",
50,
1
],
[
"DEVICE_BRAND",
5,
1
],
[
"DEVIDE_TYPE",
60,
1
],
[
"UID",
19543,
2
]
],
"stream_source_checkpoint": "{\"0\":363171,\"1\":363198,\"2\":363249,\"3\":363171,\"4\":363199,\"5\":363250,\"6\":363170,\"7\":363196,\"8\":363250,\"9\":363170}"
}
```
3. Check `${KYLIN_RECEIVER_HOME}/logs/kylin_streaming_receiver.log`, you can find the some output:
```
2020-08-09 22:34:05,381 INFO [UserAnalysisCube_channel] storage.StreamingSegmentManager:645 : Print check point for cube UserAnalysisCube ,CheckPoint{sourceConsumePosition='{"0":381733,"1":381763,"2":381820,"3":381733,"4":381764,"5":381820,"6":381732,"7":381760,"8":381821,"9":381733}', persistedIndexes={1597006800000=13, 1597010400000=8}, longLatencyInfo=LongLatencyInfo{longLatencyEventCnts={20200808000000_20200808010000=3, 20200808060000_20200808070000=2, 20200809000000_20200809010000=2, 20200809060000_20200809070000=2}, totalLongLatencyEventCnt=9}, segmentSourceStartPosition={1597006800000={"0":363171,"1":363198,"2":363249,"3":363171,"4":363199,"5":363250,"6":363170,"7":363196,"8":363250,"9":363170}, 1597010400000={"0":375013,"1":375040,"2":375099,"3":375013,"4":375041,"5":375099,"6":375012,"7":375038,"8":375100,"9":375013}}, checkPointTime=1596983645381, totalCount=3817689, checkPointCount=5801}
```
These logs indicated that the data ingetsed and indexed in receiver side is checkpointed at position :
```json
{
"0":375013,
"1":375040,
"2":375099,
"3":375013,
"4":375041,
"5":375099,
"6":375012,
"7":375038,
"8":375100,
"9":375013
}
```
4. When disable cube , data ingetsed and indexed in receiver side will be removed, so when scaled up, we expected receiver will continue its consumpution after following position:
```json
{
"0":363171,
"1":363198,
"2":363249,
"3":363171,
"4":363199,
"5":363250,
"6":363170,
"7":363196,
"8":363250,
"9":363170
}
```
5. So let's check if it is correct ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] coveralls commented on pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
coveralls commented on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-669893863
## Pull Request Test Coverage Report for [Build 6228](https://coveralls.io/builds/32580088)
* **0** of **5** **(0.0%)** changed or added relevant lines in **1** file are covered.
* **3** unchanged lines in **2** files lost coverage.
* Overall coverage increased (+**0.002%**) to **28.033%**
---
| Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
| :-----|--------------|--------|---: |
| [stream-source-kafka/src/main/java/org/apache/kylin/stream/source/kafka/consumer/KafkaConnector.java](https://coveralls.io/builds/32580088/source?filename=stream-source-kafka%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fsource%2Fkafka%2Fconsumer%2FKafkaConnector.java#L84) | 0 | 5 | 0.0%
<!-- | **Total:** | **0** | **5** | **0.0%** | -->
| Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [stream-source-kafka/src/main/java/org/apache/kylin/stream/source/kafka/consumer/KafkaConnector.java](https://coveralls.io/builds/32580088/source?filename=stream-source-kafka%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fsource%2Fkafka%2Fconsumer%2FKafkaConnector.java#L93) | 1 | 0% |
| [core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java](https://coveralls.io/builds/32580088/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fthreadpool%2FDefaultScheduler.java#L194) | 2 | 80.23% |
<!-- | **Total:** | **3** | | -->
| Totals | [](https://coveralls.io/builds/32580088) |
| :-- | --: |
| Change from base [Build 6212](https://coveralls.io/builds/32477996): | 0.002% |
| Covered Lines: | 26231 |
| Relevant Lines: | 93572 |
---
##### 💛 - [Coveralls](https://coveralls.io)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus edited a comment on pull request #1351: KYLIN-4683 Fail to consume kafka when partition number get larger
Posted by GitBox <gi...@apache.org>.
hit-lacus edited a comment on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-669972679
Thank you for your contributions. But I think we should have a more discussion after check this patch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org