You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by "Kumar, Manoj H" <ma...@jpmorgan.com> on 2017/10/12 06:29:21 UTC
Kafka Streaming data - Error while building the Cube
Building the Cube from Kylin UI - Although Messages are there in Kafka topic but Kylin is not able read the offset. Can someone help on this?
2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] utils.AppInfoParser:84 : Kafka commitId : unknown
2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:107 : The end offsets are {0=0}
2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8] controller.CubeController:305 : No new message comes, startOffset = endOffset:0
java.lang.IllegalArgumentException: No new message comes, startOffset = endOffset:0
at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:134)
at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:236)
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:208)
at org.apache.kylin.rest.service.JobService$$FastClassBySpringCGLIB$$83a44b2a.invoke(<generated>)
Regards,
Manoj
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
RE: Kafka Streaming data - Error while building the Cube
Posted by "Kumar, Manoj H" <ma...@jpmorgan.com>.
Excellent catch.. It was due to wrong Topic name. It ran fine when changed it.
Regards,
Manoj
From: Billy Liu [mailto:billyliu@apache.org]
Sent: Friday, October 13, 2017 12:05 PM
To: user
Subject: Re: Kafka Streaming data - Error while building the Cube
And in Kylin tutorial, the topic is kylindemo, in your sample, the topic is kylin_demo. Please double check the topic name.
2017-10-13 14:27 GMT+08:00 Billy Liu <bi...@apache.org>>:
If you could package the source code, please try to add more debug log when retrieving partition info by Kafka consumer. Check which topic and how many partitions you got.
2017-10-12 23:19 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>>:
I guess KafkaSource Class file where it has enrichSourcePartitionBeforeBuild() to get partition values. There its getting Error out. Do we know how can we test out to find why its coming as 0 for start & end offset?
Regards,
Manoj
From: Kumar, Manoj H
Sent: Thursday, October 12, 2017 3:35 PM
To: 'user@kylin.apache.org<ma...@kylin.apache.org>'
Subject: RE: Kafka Streaming data - Error while building the Cube
Yes its there.. I could see the messages..
Regards,
Manoj
From: Billy Liu [mailto:billyliu@apache.org]
Sent: Thursday, October 12, 2017 3:11 PM
To: user
Subject: Re: Kafka Streaming data - Error while building the Cube
STREAMING_SALES_TABLE table reads messages from Kafka topic kylin_demo，but got 0 message.
Could you check if the topic has incoming message: bin/kafka-console-consumer.sh --zookeeper localhost:2181 --bootstrap-server localhost:9092 --topic kylin_demo
2017-10-12 17:19 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>>:
Pls. find below information about consumer config from Kylin log file.
2017-10-11 02:11:43,787 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:12:13,783 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:12:40,734 INFO [http-bio-7070-exec-3] streaming.StreamingManager:222 : Reloading Streaming Metadata from folder kylin_metadata(key='/streaming')@kylin_metadata@hbase
2017-10-11 02:12:40,760 DEBUG [http-bio-7070-exec-3] streaming.StreamingManager:247 : Loaded 1 StreamingConfig(s)
2017-10-11 02:12:43,789 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:13:13,788 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:13:43,785 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:14:13,789 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:14:43,796 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:15:03,911 DEBUG [http-bio-7070-exec-1] controller.StreamingController:255 : Saving StreamingConfig {"uuid":"8613b0e1-40ac-438c-bdf5-72be4d91c230","last_modified":1507705685859,"version":"2.1.0","name":"DEFAULT.STREAMING_SALES_TABLE","type":"kafka"}
2017-10-11 02:15:03,913 DEBUG [http-bio-7070-exec-1] controller.StreamingController:273 : Saving KafkaConfig {"uuid":"87dc6ab5-5141-4bd8-8e00-c16ec86dce41","last_modified":1507705685916,"version":"2.1.0","name":"DEFAULT.STREAMING_SALES_TABLE","clusters":[{"brokers":[{"id":"1","host":"sandbox","port":"9092"}]}],"topic":"kylin_demo","timeout":60000,"parserName":"org.apache.kylin.source.kafka.TimedJsonStreamParser","parserTimeStampField":null,"margin":0,"parserProperties":"tsColName=order_time"}
2017-10-11 02:15:03,963 DEBUG [pool-7-thread-1] cachesync.Broadcaster:132 : Servers in the cluster: [localhost:7070]
2017-10-11 02:15:04,000 DEBUG [pool-7-thread-1] cachesync.Broadcaster:139 : Announcing new broadcast event: BroadcastEvent{entity=streaming, event=update, cacheKey=DEFAULT.STREAMING_SALES_TABLE}
2017-10-11 02:15:04,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:132 : Servers in the cluster: [localhost:7070]
2017-10-11 02:15:04,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:139 : Announcing new broadcast event: BroadcastEvent{entity=kafka, event=update, cacheKey=DEFAULT.STREAMING_SALES_TABLE}
2017-10-11 02:15:04,164 DEBUG [http-bio-7070-exec-9] cachesync.Broadcaster:236 : Done broadcasting metadata change: entity=streaming, event=UPDATE, cacheKey=DEFAULT.STREAMING_SALES_TABLE
2017-10-11 02:15:04,192 DEBUG [http-bio-7070-exec-10] cachesync.Broadcaster:236 : Done broadcasting metadata change: entity=kafka, event=UPDATE, cacheKey=DEFAULT.STREAMING_SALES_TABLE
2017-10-11 02:15:13,789 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:15:23,780 DEBUG [http-bio-7070-exec-7] kafka.KafkaSource:83 : Last segment doesn't exist, and didn't initiate the start offset, will seek from topic's earliest offset.
2017-10-11 20:50:42,558 INFO [http-bio-7070-exec-8] utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
2017-10-11 20:50:42,563 INFO [http-bio-7070-exec-8] utils.AppInfoParser:84 : Kafka commitId : unknown
2017-10-11 20:50:42,570 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:105 : Seek end offsets from topic
2017-10-11 20:50:42,570 INFO [http-bio-7070-exec-8] consumer.ConsumerConfig:196 : ConsumerConfig values:
auto.commit.interval.ms<http://auto.commit.interval.ms> = 5000
auto.offset.reset = latest
bootstrap.servers = [localhost:9092]
check.crcs = true
client.id<http://client.id> =
connections.max.idle.ms<http://connections.max.idle.ms> = 540000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms<http://fetch.max.wait.ms> = 500
fetch.min.bytes = 1
group.id<http://group.id> = streaming_cube
heartbeat.interval.ms<http://heartbeat.interval.ms> = 3000
interceptor.classes = null
internal.leave.group.on.close = true
key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms<http://max.poll.interval.ms> = 300000
max.poll.records = 500
metadata.max.age.ms<http://metadata.max.age.ms> = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms<http://metrics.sample.window.ms> = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.ms<http://reconnect.backoff.ms> = 50
request.timeout.ms<http://request.timeout.ms> = 305000
retry.backoff.ms<http://retry.backoff.ms> = 100
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name<http://sasl.kerberos.service.name> = null
sasl.kerberos.ticket.renew.jitter = 0.05
request.timeout.ms<http://request.timeout.ms> = 305000
retry.backoff.ms<http://retry.backoff.ms> = 100
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name<http://sasl.kerberos.service.name> = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
session.timeout.ms<http://session.timeout.ms> = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.al<http://ssl.endpoint.identification.al>gorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] utils.AppInfoParser:84 : Kafka commitId : unknown
2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:107 : The end offsets are {0=0}
2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8] controller.CubeController:305 : No new message comes, startOffset = endOffset:0
java.lang.IllegalArgumentException: No new message comes, startOffset = endOffset:0
at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:134)
at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:236)
Regards,
Manoj
From: Billy Liu [mailto:billyliu@apache.org<ma...@apache.org>]
Sent: Thursday, October 12, 2017 1:06 PM
To: user
Subject: Re: Kafka Streaming data - Error while building the Cube
Hi Kumar,
Could you paste more Kafka Consumer related log in kylin.log? And also check from the Kafka broker side, if the Kylin client has connected to Broker.
2017-10-12 14:29 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>>:
Building the Cube from Kylin UI - Although Messages are there in Kafka topic but Kylin is not able read the offset. Can someone help on this?
2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] utils.AppInfoParser:84 : Kafka commitId : unknown
2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:107 : The end offsets are {0=0}
2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8] controller.CubeController:305 : No new message comes, startOffset = endOffset:0
java.lang.IllegalArgumentException: No new message comes, startOffset = endOffset:0
at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:134)
at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:236)
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:208)
at org.apache.kylin.rest.service.JobService$$FastClassBySpringCGLIB$$83a44b2a.invoke(<generated>)
Regards,
Manoj
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
Re: Kafka Streaming data - Error while building the Cube
Posted by Billy Liu <bi...@apache.org>.
And in Kylin tutorial, the topic is kylindemo, in your sample, the topic is
kylin_demo. Please double check the topic name.
2017-10-13 14:27 GMT+08:00 Billy Liu <bi...@apache.org>:
> If you could package the source code, please try to add more debug log
> when retrieving partition info by Kafka consumer. Check which topic and how
> many partitions you got.
>
> 2017-10-12 23:19 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
>
>> I guess KafkaSource Class file where it has
>> enrichSourcePartitionBeforeBuild() to get partition values. There its
>> getting Error out. Do we know how can we test out to find why its coming as
>> 0 for start & end offset?
>>
>>
>>
>> Regards,
>>
>> Manoj
>>
>>
>>
>> *From:* Kumar, Manoj H
>> *Sent:* Thursday, October 12, 2017 3:35 PM
>> *To:* 'user@kylin.apache.org'
>> *Subject:* RE: Kafka Streaming data - Error while building the Cube
>>
>>
>>
>> Yes its there.. I could see the messages..
>>
>>
>>
>> Regards,
>>
>> Manoj
>>
>>
>>
>> *From:* Billy Liu [mailto:billyliu@apache.org <bi...@apache.org>]
>> *Sent:* Thursday, October 12, 2017 3:11 PM
>>
>> *To:* user
>> *Subject:* Re: Kafka Streaming data - Error while building the Cube
>>
>>
>>
>> STREAMING_SALES_TABLE table reads messages from Kafka topic kylin_demo，but
>> got 0 message.
>>
>>
>>
>> Could you check if the topic has incoming message: bin/kafka-console-consumer.sh
>> --zookeeper localhost:2181 --bootstrap-server localhost:9092 --topic
>> kylin_demo
>>
>>
>>
>> 2017-10-12 17:19 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
>>
>> Pls. find below information about consumer config from Kylin log file.
>>
>>
>>
>> 2017-10-11 02:11:43,787 INFO [pool-8-thread-1]
>> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0
>> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0
>> discarded, 0 others
>>
>> 2017-10-11 02:12:13,783 INFO [pool-8-thread-1]
>> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0
>> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0
>> discarded, 0 others
>>
>> 2017-10-11 02:12:40,734 INFO [http-bio-7070-exec-3]
>> streaming.StreamingManager:222 : Reloading Streaming Metadata from folder
>> kylin_metadata(key='/streaming')@kylin_metadata@hbase
>>
>> 2017-10-11 02:12:40,760 DEBUG [http-bio-7070-exec-3]
>> streaming.StreamingManager:247 : Loaded 1 StreamingConfig(s)
>>
>> 2017-10-11 02:12:43,789 INFO [pool-8-thread-1]
>> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0
>> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0
>> discarded, 0 others
>>
>> 2017-10-11 02:13:13,788 INFO [pool-8-thread-1]
>> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0
>> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0
>> discarded, 0 others
>>
>> 2017-10-11 02:13:43,785 INFO [pool-8-thread-1]
>> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0
>> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0
>> discarded, 0 others
>>
>> 2017-10-11 02:14:13,789 INFO [pool-8-thread-1]
>> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0
>> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0
>> discarded, 0 others
>>
>> 2017-10-11 02:14:43,796 INFO [pool-8-thread-1]
>> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0
>> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0
>> discarded, 0 others
>>
>> 2017-10-11 02:15:03,911 DEBUG [http-bio-7070-exec-1]
>> controller.StreamingController:255 : Saving StreamingConfig
>> {"uuid":"8613b0e1-40ac-438c-bdf5-72be4d91c230","last_modifie
>> d":1507705685859,"version":"2.1.0","name":"DEFAULT.
>> STREAMING_SALES_TABLE","type":"kafka"}
>>
>> 2017-10-11 02:15:03,913 DEBUG [http-bio-7070-exec-1]
>> controller.StreamingController:273 : Saving KafkaConfig
>> {"uuid":"87dc6ab5-5141-4bd8-8e00-c16ec86dce41","last_modifie
>> d":1507705685916,"version":"2.1.0","name":"DEFAULT.
>> STREAMING_SALES_TABLE","clusters":[{"brokers":[{"id":"
>> 1","host":"sandbox","port":"9092"}]}],"topic":"kylin_demo","
>> timeout":60000,"parserName":"org.apache.kylin.source.kafka.
>> TimedJsonStreamParser","parserTimeStampField":null,"margin":
>> 0,"parserProperties":"tsColName=order_time"}
>>
>> 2017-10-11 02:15:03,963 DEBUG [pool-7-thread-1] cachesync.Broadcaster:132
>> : Servers in the cluster: [localhost:7070]
>>
>> 2017-10-11 02:15:04,000 DEBUG [pool-7-thread-1] cachesync.Broadcaster:139
>> : Announcing new broadcast event: BroadcastEvent{entity=streaming,
>> event=update, cacheKey=DEFAULT.STREAMING_SALES_TABLE}
>>
>> 2017-10-11 02:15:04,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:132
>> : Servers in the cluster: [localhost:7070]
>>
>> 2017-10-11 02:15:04,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:139
>> : Announcing new broadcast event: BroadcastEvent{entity=kafka,
>> event=update, cacheKey=DEFAULT.STREAMING_SALES_TABLE}
>>
>> 2017-10-11 02:15:04,164 DEBUG [http-bio-7070-exec-9]
>> cachesync.Broadcaster:236 : Done broadcasting metadata change:
>> entity=streaming, event=UPDATE, cacheKey=DEFAULT.STREAMING_SALES_TABLE
>>
>> 2017-10-11 02:15:04,192 DEBUG [http-bio-7070-exec-10]
>> cachesync.Broadcaster:236 : Done broadcasting metadata change:
>> entity=kafka, event=UPDATE, cacheKey=DEFAULT.STREAMING_SALES_TABLE
>>
>> 2017-10-11 02:15:13,789 INFO [pool-8-thread-1]
>> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0
>> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0
>> discarded, 0 others
>>
>> 2017-10-11 02:15:23,780 DEBUG [http-bio-7070-exec-7] kafka.KafkaSource:83
>> : Last segment doesn't exist, and didn't initiate the start offset, will
>> seek from topic's earliest offset.
>>
>>
>>
>> 2017-10-11 20:50:42,558 INFO [http-bio-7070-exec-8]
>> utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
>>
>> 2017-10-11 20:50:42,563 INFO [http-bio-7070-exec-8]
>> utils.AppInfoParser:84 : Kafka commitId : unknown
>>
>> 2017-10-11 20:50:42,570 DEBUG [http-bio-7070-exec-8]
>> kafka.KafkaSource:105 : Seek end offsets from topic
>>
>> 2017-10-11 20:50:42,570 INFO [http-bio-7070-exec-8]
>> consumer.ConsumerConfig:196 : ConsumerConfig values:
>>
>> auto.commit.interval.ms = 5000
>>
>> auto.offset.reset = latest
>>
>> bootstrap.servers = [localhost:9092]
>>
>> check.crcs = true
>>
>> client.id =
>>
>> connections.max.idle.ms = 540000
>>
>> enable.auto.commit = false
>>
>> exclude.internal.topics = true
>>
>> fetch.max.bytes = 52428800
>>
>> fetch.max.wait.ms = 500
>>
>> fetch.min.bytes = 1
>>
>> group.id = streaming_cube
>>
>> heartbeat.interval.ms = 3000
>>
>> interceptor.classes = null
>>
>> internal.leave.group.on.close = true
>>
>> key.deserializer = class org.apache.kafka.common.serial
>> ization.StringDeserializer
>>
>> max.partition.fetch.bytes = 1048576
>>
>> max.poll.interval.ms = 300000
>>
>> max.poll.records = 500
>>
>> metadata.max.age.ms = 300000
>>
>> metric.reporters = []
>>
>> metrics.num.samples = 2
>>
>> metrics.recording.level = INFO
>>
>> metrics.sample.window.ms = 30000
>>
>> partition.assignment.strategy = [class
>> org.apache.kafka.clients.consumer.RangeAssignor]
>>
>> receive.buffer.bytes = 65536
>>
>> reconnect.backoff.ms = 50
>>
>> request.timeout.ms = 305000
>>
>> retry.backoff.ms = 100
>>
>> sasl.jaas.config = null
>>
>> sasl.kerberos.kinit.cmd = /usr/bin/kinit
>>
>> sasl.kerberos.min.time.before.relogin = 60000
>>
>> sasl.kerberos.service.name = null
>>
>> sasl.kerberos.ticket.renew.jitter = 0.05
>>
>> request.timeout.ms = 305000
>>
>> retry.backoff.ms = 100
>>
>> sasl.jaas.config = null
>>
>> sasl.kerberos.kinit.cmd = /usr/bin/kinit
>>
>> sasl.kerberos.min.time.before.relogin = 60000
>>
>> sasl.kerberos.service.name = null
>>
>> sasl.kerberos.ticket.renew.jitter = 0.05
>>
>> sasl.kerberos.ticket.renew.window.factor = 0.8
>>
>> sasl.mechanism = GSSAPI
>>
>> security.protocol = PLAINTEXT
>>
>> send.buffer.bytes = 131072
>>
>> session.timeout.ms = 10000
>>
>> ssl.cipher.suites = null
>>
>> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
>>
>> ssl.endpoint.identification.algorithm = null
>>
>> ssl.key.password = null
>>
>> ssl.keymanager.algorithm = SunX509
>>
>> ssl.keystore.location = null
>>
>> ssl.keystore.password = null
>>
>> ssl.keystore.type = JKS
>>
>> ssl.protocol = TLS
>>
>> ssl.provider = null
>>
>> ssl.secure.random.implementation = null
>>
>> ssl.trustmanager.algorithm = PKIX
>>
>> ssl.truststore.location = null
>>
>> ssl.truststore.password = null
>>
>> ssl.truststore.type = JKS
>>
>> value.deserializer = class org.apache.kafka.common.serial
>> ization.StringDeserializer
>>
>>
>>
>> 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8]
>> utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
>>
>> 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8]
>> utils.AppInfoParser:84 : Kafka commitId : unknown
>>
>> 2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8]
>> kafka.KafkaSource:107 : The end offsets are {0=0}
>>
>> 2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8]
>> controller.CubeController:305 : No new message comes, startOffset =
>> endOffset:0
>>
>> java.lang.IllegalArgumentException: No new message comes, startOffset =
>> endOffset:0
>>
>> at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartit
>> ionBeforeBuild(KafkaSource.java:134)
>>
>> at org.apache.kylin.rest.service.JobService.submitJobInternal(J
>> obService.java:236)
>>
>> Regards,
>>
>> Manoj
>>
>>
>>
>> *From:* Billy Liu [mailto:billyliu@apache.org]
>> *Sent:* Thursday, October 12, 2017 1:06 PM
>> *To:* user
>> *Subject:* Re: Kafka Streaming data - Error while building the Cube
>>
>>
>>
>> Hi Kumar,
>>
>>
>>
>> Could you paste more Kafka Consumer related log in kylin.log? And also
>> check from the Kafka broker side, if the Kylin client has connected to
>> Broker.
>>
>>
>>
>> 2017-10-12 14:29 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
>>
>> Building the Cube from Kylin UI - Although Messages are there in Kafka
>> topic but Kylin is not able read the offset. Can someone help on this?
>>
>> 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8]
>> utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
>> 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8]
>> utils.AppInfoParser:84 : Kafka commitId : unknown
>> 2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8]
>> kafka.KafkaSource:107 : The end offsets are {0=0}
>> 2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8]
>> controller.CubeController:305 : No new message comes, startOffset =
>> endOffset:0
>> java.lang.IllegalArgumentException: No new message comes, startOffset =
>> endOffset:0
>> at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartit
>> ionBeforeBuild(KafkaSource.java:134)
>> at org.apache.kylin.rest.service.JobService.submitJobInternal(J
>> obService.java:236)
>> at org.apache.kylin.rest.service.JobService.submitJob(JobServic
>> e.java:208)
>> at org.apache.kylin.rest.service.JobService$$FastClassBySpringC
>> GLIB$$83a44b2a.invoke(<generated>)
>>
>> Regards,
>> Manoj
>>
>>
>> This message is confidential and subject to terms at:
>> http://www.jpmorgan.com/emaildisclaimer including on confidentiality,
>> legal privilege, viruses and monitoring of electronic messages. If you are
>> not the intended recipient, please delete this message and notify the
>> sender immediately. Any unauthorized use is strictly prohibited.
>>
>>
>>
>> This message is confidential and subject to terms at: http://
>> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
>> privilege, viruses and monitoring of electronic messages. If you are not
>> the intended recipient, please delete this message and notify the sender
>> immediately. Any unauthorized use is strictly prohibited.
>>
>>
>>
>> This message is confidential and subject to terms at: http://
>> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
>> privilege, viruses and monitoring of electronic messages. If you are not
>> the intended recipient, please delete this message and notify the sender
>> immediately. Any unauthorized use is strictly prohibited.
>>
>
>
Re: Kafka Streaming data - Error while building the Cube
Posted by Billy Liu <bi...@apache.org>.
If you could package the source code, please try to add more debug log when
retrieving partition info by Kafka consumer. Check which topic and how many
partitions you got.
2017-10-12 23:19 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
> I guess KafkaSource Class file where it has enrichSourcePartitionBeforeBuild()
> to get partition values. There its getting Error out. Do we know how can we
> test out to find why its coming as 0 for start & end offset?
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* Kumar, Manoj H
> *Sent:* Thursday, October 12, 2017 3:35 PM
> *To:* 'user@kylin.apache.org'
> *Subject:* RE: Kafka Streaming data - Error while building the Cube
>
>
>
> Yes its there.. I could see the messages..
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* Billy Liu [mailto:billyliu@apache.org <bi...@apache.org>]
> *Sent:* Thursday, October 12, 2017 3:11 PM
>
> *To:* user
> *Subject:* Re: Kafka Streaming data - Error while building the Cube
>
>
>
> STREAMING_SALES_TABLE table reads messages from Kafka topic kylin_demo，but
> got 0 message.
>
>
>
> Could you check if the topic has incoming message: bin/kafka-console-consumer.sh
> --zookeeper localhost:2181 --bootstrap-server localhost:9092 --topic
> kylin_demo
>
>
>
> 2017-10-12 17:19 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
>
> Pls. find below information about consumer config from Kylin log file.
>
>
>
> 2017-10-11 02:11:43,787 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0
> others
>
> 2017-10-11 02:12:13,783 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0
> others
>
> 2017-10-11 02:12:40,734 INFO [http-bio-7070-exec-3]
> streaming.StreamingManager:222 : Reloading Streaming Metadata from folder
> kylin_metadata(key='/streaming')@kylin_metadata@hbase
>
> 2017-10-11 02:12:40,760 DEBUG [http-bio-7070-exec-3]
> streaming.StreamingManager:247 : Loaded 1 StreamingConfig(s)
>
> 2017-10-11 02:12:43,789 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0
> others
>
> 2017-10-11 02:13:13,788 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0
> others
>
> 2017-10-11 02:13:43,785 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0
> others
>
> 2017-10-11 02:14:13,789 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0
> others
>
> 2017-10-11 02:14:43,796 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0
> others
>
> 2017-10-11 02:15:03,911 DEBUG [http-bio-7070-exec-1] controller.StreamingController:255
> : Saving StreamingConfig {"uuid":"8613b0e1-40ac-438c-
> bdf5-72be4d91c230","last_modified":1507705685859,"
> version":"2.1.0","name":"DEFAULT.STREAMING_SALES_TABLE","type":"kafka"}
>
> 2017-10-11 02:15:03,913 DEBUG [http-bio-7070-exec-1] controller.StreamingController:273
> : Saving KafkaConfig {"uuid":"87dc6ab5-5141-4bd8-8e00-c16ec86dce41","last_
> modified":1507705685916,"version":"2.1.0","name":"
> DEFAULT.STREAMING_SALES_TABLE","clusters":[{"brokers":[{"id"
> :"1","host":"sandbox","port":"9092"}]}],"topic":"kylin_demo"
> ,"timeout":60000,"parserName":"org.apache.kylin.source.
> kafka.TimedJsonStreamParser","parserTimeStampField":null,"
> margin":0,"parserProperties":"tsColName=order_time"}
>
> 2017-10-11 02:15:03,963 DEBUG [pool-7-thread-1] cachesync.Broadcaster:132
> : Servers in the cluster: [localhost:7070]
>
> 2017-10-11 02:15:04,000 DEBUG [pool-7-thread-1] cachesync.Broadcaster:139
> : Announcing new broadcast event: BroadcastEvent{entity=streaming,
> event=update, cacheKey=DEFAULT.STREAMING_SALES_TABLE}
>
> 2017-10-11 02:15:04,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:132
> : Servers in the cluster: [localhost:7070]
>
> 2017-10-11 02:15:04,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:139
> : Announcing new broadcast event: BroadcastEvent{entity=kafka,
> event=update, cacheKey=DEFAULT.STREAMING_SALES_TABLE}
>
> 2017-10-11 02:15:04,164 DEBUG [http-bio-7070-exec-9]
> cachesync.Broadcaster:236 : Done broadcasting metadata change:
> entity=streaming, event=UPDATE, cacheKey=DEFAULT.STREAMING_SALES_TABLE
>
> 2017-10-11 02:15:04,192 DEBUG [http-bio-7070-exec-10]
> cachesync.Broadcaster:236 : Done broadcasting metadata change:
> entity=kafka, event=UPDATE, cacheKey=DEFAULT.STREAMING_SALES_TABLE
>
> 2017-10-11 02:15:13,789 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0
> others
>
> 2017-10-11 02:15:23,780 DEBUG [http-bio-7070-exec-7] kafka.KafkaSource:83
> : Last segment doesn't exist, and didn't initiate the start offset, will
> seek from topic's earliest offset.
>
>
>
> 2017-10-11 20:50:42,558 INFO [http-bio-7070-exec-8]
> utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
>
> 2017-10-11 20:50:42,563 INFO [http-bio-7070-exec-8]
> utils.AppInfoParser:84 : Kafka commitId : unknown
>
> 2017-10-11 20:50:42,570 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:105
> : Seek end offsets from topic
>
> 2017-10-11 20:50:42,570 INFO [http-bio-7070-exec-8]
> consumer.ConsumerConfig:196 : ConsumerConfig values:
>
> auto.commit.interval.ms = 5000
>
> auto.offset.reset = latest
>
> bootstrap.servers = [localhost:9092]
>
> check.crcs = true
>
> client.id =
>
> connections.max.idle.ms = 540000
>
> enable.auto.commit = false
>
> exclude.internal.topics = true
>
> fetch.max.bytes = 52428800
>
> fetch.max.wait.ms = 500
>
> fetch.min.bytes = 1
>
> group.id = streaming_cube
>
> heartbeat.interval.ms = 3000
>
> interceptor.classes = null
>
> internal.leave.group.on.close = true
>
> key.deserializer = class org.apache.kafka.common.serialization.
> StringDeserializer
>
> max.partition.fetch.bytes = 1048576
>
> max.poll.interval.ms = 300000
>
> max.poll.records = 500
>
> metadata.max.age.ms = 300000
>
> metric.reporters = []
>
> metrics.num.samples = 2
>
> metrics.recording.level = INFO
>
> metrics.sample.window.ms = 30000
>
> partition.assignment.strategy = [class org.apache.kafka.clients.
> consumer.RangeAssignor]
>
> receive.buffer.bytes = 65536
>
> reconnect.backoff.ms = 50
>
> request.timeout.ms = 305000
>
> retry.backoff.ms = 100
>
> sasl.jaas.config = null
>
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
>
> sasl.kerberos.min.time.before.relogin = 60000
>
> sasl.kerberos.service.name = null
>
> sasl.kerberos.ticket.renew.jitter = 0.05
>
> request.timeout.ms = 305000
>
> retry.backoff.ms = 100
>
> sasl.jaas.config = null
>
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
>
> sasl.kerberos.min.time.before.relogin = 60000
>
> sasl.kerberos.service.name = null
>
> sasl.kerberos.ticket.renew.jitter = 0.05
>
> sasl.kerberos.ticket.renew.window.factor = 0.8
>
> sasl.mechanism = GSSAPI
>
> security.protocol = PLAINTEXT
>
> send.buffer.bytes = 131072
>
> session.timeout.ms = 10000
>
> ssl.cipher.suites = null
>
> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
>
> ssl.endpoint.identification.algorithm = null
>
> ssl.key.password = null
>
> ssl.keymanager.algorithm = SunX509
>
> ssl.keystore.location = null
>
> ssl.keystore.password = null
>
> ssl.keystore.type = JKS
>
> ssl.protocol = TLS
>
> ssl.provider = null
>
> ssl.secure.random.implementation = null
>
> ssl.trustmanager.algorithm = PKIX
>
> ssl.truststore.location = null
>
> ssl.truststore.password = null
>
> ssl.truststore.type = JKS
>
> value.deserializer = class org.apache.kafka.common.serialization.
> StringDeserializer
>
>
>
> 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8]
> utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
>
> 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8]
> utils.AppInfoParser:84 : Kafka commitId : unknown
>
> 2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:107
> : The end offsets are {0=0}
>
> 2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8]
> controller.CubeController:305 : No new message comes, startOffset =
> endOffset:0
>
> java.lang.IllegalArgumentException: No new message comes, startOffset =
> endOffset:0
>
> at org.apache.kylin.source.kafka.KafkaSource.
> enrichSourcePartitionBeforeBuild(KafkaSource.java:134)
>
> at org.apache.kylin.rest.service.JobService.submitJobInternal(
> JobService.java:236)
>
> Regards,
>
> Manoj
>
>
>
> *From:* Billy Liu [mailto:billyliu@apache.org]
> *Sent:* Thursday, October 12, 2017 1:06 PM
> *To:* user
> *Subject:* Re: Kafka Streaming data - Error while building the Cube
>
>
>
> Hi Kumar,
>
>
>
> Could you paste more Kafka Consumer related log in kylin.log? And also
> check from the Kafka broker side, if the Kylin client has connected to
> Broker.
>
>
>
> 2017-10-12 14:29 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
>
> Building the Cube from Kylin UI - Although Messages are there in Kafka
> topic but Kylin is not able read the offset. Can someone help on this?
>
> 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8]
> utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
> 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8]
> utils.AppInfoParser:84 : Kafka commitId : unknown
> 2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:107
> : The end offsets are {0=0}
> 2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8]
> controller.CubeController:305 : No new message comes, startOffset =
> endOffset:0
> java.lang.IllegalArgumentException: No new message comes, startOffset =
> endOffset:0
> at org.apache.kylin.source.kafka.KafkaSource.
> enrichSourcePartitionBeforeBuild(KafkaSource.java:134)
> at org.apache.kylin.rest.service.JobService.submitJobInternal(
> JobService.java:236)
> at org.apache.kylin.rest.service.JobService.submitJob(
> JobService.java:208)
> at org.apache.kylin.rest.service.JobService$$
> FastClassBySpringCGLIB$$83a44b2a.invoke(<generated>)
>
> Regards,
> Manoj
>
>
> This message is confidential and subject to terms at:
> http://www.jpmorgan.com/emaildisclaimer including on confidentiality,
> legal privilege, viruses and monitoring of electronic messages. If you are
> not the intended recipient, please delete this message and notify the
> sender immediately. Any unauthorized use is strictly prohibited.
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
RE: Kafka Streaming data - Error while building the Cube
Posted by "Kumar, Manoj H" <ma...@jpmorgan.com>.
I guess KafkaSource Class file where it has enrichSourcePartitionBeforeBuild() to get partition values. There its getting Error out. Do we know how can we test out to find why its coming as 0 for start & end offset?
Regards,
Manoj
From: Kumar, Manoj H
Sent: Thursday, October 12, 2017 3:35 PM
To: 'user@kylin.apache.org'
Subject: RE: Kafka Streaming data - Error while building the Cube
Yes its there.. I could see the messages..
Regards,
Manoj
From: Billy Liu [mailto:billyliu@apache.org]
Sent: Thursday, October 12, 2017 3:11 PM
To: user
Subject: Re: Kafka Streaming data - Error while building the Cube
STREAMING_SALES_TABLE table reads messages from Kafka topic kylin_demo，but got 0 message.
Could you check if the topic has incoming message: bin/kafka-console-consumer.sh --zookeeper localhost:2181 --bootstrap-server localhost:9092 --topic kylin_demo
2017-10-12 17:19 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>>:
Pls. find below information about consumer config from Kylin log file.
2017-10-11 02:11:43,787 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:12:13,783 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:12:40,734 INFO [http-bio-7070-exec-3] streaming.StreamingManager:222 : Reloading Streaming Metadata from folder kylin_metadata(key='/streaming')@kylin_metadata@hbase
2017-10-11 02:12:40,760 DEBUG [http-bio-7070-exec-3] streaming.StreamingManager:247 : Loaded 1 StreamingConfig(s)
2017-10-11 02:12:43,789 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:13:13,788 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:13:43,785 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:14:13,789 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:14:43,796 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:15:03,911 DEBUG [http-bio-7070-exec-1] controller.StreamingController:255 : Saving StreamingConfig {"uuid":"8613b0e1-40ac-438c-bdf5-72be4d91c230","last_modified":1507705685859,"version":"2.1.0","name":"DEFAULT.STREAMING_SALES_TABLE","type":"kafka"}
2017-10-11 02:15:03,913 DEBUG [http-bio-7070-exec-1] controller.StreamingController:273 : Saving KafkaConfig {"uuid":"87dc6ab5-5141-4bd8-8e00-c16ec86dce41","last_modified":1507705685916,"version":"2.1.0","name":"DEFAULT.STREAMING_SALES_TABLE","clusters":[{"brokers":[{"id":"1","host":"sandbox","port":"9092"}]}],"topic":"kylin_demo","timeout":60000,"parserName":"org.apache.kylin.source.kafka.TimedJsonStreamParser","parserTimeStampField":null,"margin":0,"parserProperties":"tsColName=order_time"}
2017-10-11 02:15:03,963 DEBUG [pool-7-thread-1] cachesync.Broadcaster:132 : Servers in the cluster: [localhost:7070]
2017-10-11 02:15:04,000 DEBUG [pool-7-thread-1] cachesync.Broadcaster:139 : Announcing new broadcast event: BroadcastEvent{entity=streaming, event=update, cacheKey=DEFAULT.STREAMING_SALES_TABLE}
2017-10-11 02:15:04,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:132 : Servers in the cluster: [localhost:7070]
2017-10-11 02:15:04,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:139 : Announcing new broadcast event: BroadcastEvent{entity=kafka, event=update, cacheKey=DEFAULT.STREAMING_SALES_TABLE}
2017-10-11 02:15:04,164 DEBUG [http-bio-7070-exec-9] cachesync.Broadcaster:236 : Done broadcasting metadata change: entity=streaming, event=UPDATE, cacheKey=DEFAULT.STREAMING_SALES_TABLE
2017-10-11 02:15:04,192 DEBUG [http-bio-7070-exec-10] cachesync.Broadcaster:236 : Done broadcasting metadata change: entity=kafka, event=UPDATE, cacheKey=DEFAULT.STREAMING_SALES_TABLE
2017-10-11 02:15:13,789 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:15:23,780 DEBUG [http-bio-7070-exec-7] kafka.KafkaSource:83 : Last segment doesn't exist, and didn't initiate the start offset, will seek from topic's earliest offset.
2017-10-11 20:50:42,558 INFO [http-bio-7070-exec-8] utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
2017-10-11 20:50:42,563 INFO [http-bio-7070-exec-8] utils.AppInfoParser:84 : Kafka commitId : unknown
2017-10-11 20:50:42,570 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:105 : Seek end offsets from topic
2017-10-11 20:50:42,570 INFO [http-bio-7070-exec-8] consumer.ConsumerConfig:196 : ConsumerConfig values:
auto.commit.interval.ms<http://auto.commit.interval.ms> = 5000
auto.offset.reset = latest
bootstrap.servers = [localhost:9092]
check.crcs = true
client.id<http://client.id> =
connections.max.idle.ms<http://connections.max.idle.ms> = 540000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms<http://fetch.max.wait.ms> = 500
fetch.min.bytes = 1
group.id<http://group.id> = streaming_cube
heartbeat.interval.ms<http://heartbeat.interval.ms> = 3000
interceptor.classes = null
internal.leave.group.on.close = true
key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms<http://max.poll.interval.ms> = 300000
max.poll.records = 500
metadata.max.age.ms<http://metadata.max.age.ms> = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms<http://metrics.sample.window.ms> = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.ms<http://reconnect.backoff.ms> = 50
request.timeout.ms<http://request.timeout.ms> = 305000
retry.backoff.ms<http://retry.backoff.ms> = 100
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name<http://sasl.kerberos.service.name> = null
sasl.kerberos.ticket.renew.jitter = 0.05
request.timeout.ms<http://request.timeout.ms> = 305000
retry.backoff.ms<http://retry.backoff.ms> = 100
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name<http://sasl.kerberos.service.name> = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
session.timeout.ms<http://session.timeout.ms> = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] utils.AppInfoParser:84 : Kafka commitId : unknown
2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:107 : The end offsets are {0=0}
2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8] controller.CubeController:305 : No new message comes, startOffset = endOffset:0
java.lang.IllegalArgumentException: No new message comes, startOffset = endOffset:0
at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:134)
at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:236)
Regards,
Manoj
From: Billy Liu [mailto:billyliu@apache.org<ma...@apache.org>]
Sent: Thursday, October 12, 2017 1:06 PM
To: user
Subject: Re: Kafka Streaming data - Error while building the Cube
Hi Kumar,
Could you paste more Kafka Consumer related log in kylin.log? And also check from the Kafka broker side, if the Kylin client has connected to Broker.
2017-10-12 14:29 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>>:
Building the Cube from Kylin UI - Although Messages are there in Kafka topic but Kylin is not able read the offset. Can someone help on this?
2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] utils.AppInfoParser:84 : Kafka commitId : unknown
2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:107 : The end offsets are {0=0}
2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8] controller.CubeController:305 : No new message comes, startOffset = endOffset:0
java.lang.IllegalArgumentException: No new message comes, startOffset = endOffset:0
at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:134)
at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:236)
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:208)
at org.apache.kylin.rest.service.JobService$$FastClassBySpringCGLIB$$83a44b2a.invoke(<generated>)
Regards,
Manoj
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
RE: Kafka Streaming data - Error while building the Cube
Posted by "Kumar, Manoj H" <ma...@jpmorgan.com>.
Yes its there.. I could see the messages..
Regards,
Manoj
From: Billy Liu [mailto:billyliu@apache.org]
Sent: Thursday, October 12, 2017 3:11 PM
To: user
Subject: Re: Kafka Streaming data - Error while building the Cube
STREAMING_SALES_TABLE table reads messages from Kafka topic kylin_demo，but got 0 message.
Could you check if the topic has incoming message: bin/kafka-console-consumer.sh --zookeeper localhost:2181 --bootstrap-server localhost:9092 --topic kylin_demo
2017-10-12 17:19 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>>:
Pls. find below information about consumer config from Kylin log file.
2017-10-11 02:11:43,787 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:12:13,783 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:12:40,734 INFO [http-bio-7070-exec-3] streaming.StreamingManager:222 : Reloading Streaming Metadata from folder kylin_metadata(key='/streaming')@kylin_metadata@hbase
2017-10-11 02:12:40,760 DEBUG [http-bio-7070-exec-3] streaming.StreamingManager:247 : Loaded 1 StreamingConfig(s)
2017-10-11 02:12:43,789 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:13:13,788 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:13:43,785 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:14:13,789 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:14:43,796 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:15:03,911 DEBUG [http-bio-7070-exec-1] controller.StreamingController:255 : Saving StreamingConfig {"uuid":"8613b0e1-40ac-438c-bdf5-72be4d91c230","last_modified":1507705685859,"version":"2.1.0","name":"DEFAULT.STREAMING_SALES_TABLE","type":"kafka"}
2017-10-11 02:15:03,913 DEBUG [http-bio-7070-exec-1] controller.StreamingController:273 : Saving KafkaConfig {"uuid":"87dc6ab5-5141-4bd8-8e00-c16ec86dce41","last_modified":1507705685916,"version":"2.1.0","name":"DEFAULT.STREAMING_SALES_TABLE","clusters":[{"brokers":[{"id":"1","host":"sandbox","port":"9092"}]}],"topic":"kylin_demo","timeout":60000,"parserName":"org.apache.kylin.source.kafka.TimedJsonStreamParser","parserTimeStampField":null,"margin":0,"parserProperties":"tsColName=order_time"}
2017-10-11 02:15:03,963 DEBUG [pool-7-thread-1] cachesync.Broadcaster:132 : Servers in the cluster: [localhost:7070]
2017-10-11 02:15:04,000 DEBUG [pool-7-thread-1] cachesync.Broadcaster:139 : Announcing new broadcast event: BroadcastEvent{entity=streaming, event=update, cacheKey=DEFAULT.STREAMING_SALES_TABLE}
2017-10-11 02:15:04,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:132 : Servers in the cluster: [localhost:7070]
2017-10-11 02:15:04,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:139 : Announcing new broadcast event: BroadcastEvent{entity=kafka, event=update, cacheKey=DEFAULT.STREAMING_SALES_TABLE}
2017-10-11 02:15:04,164 DEBUG [http-bio-7070-exec-9] cachesync.Broadcaster:236 : Done broadcasting metadata change: entity=streaming, event=UPDATE, cacheKey=DEFAULT.STREAMING_SALES_TABLE
2017-10-11 02:15:04,192 DEBUG [http-bio-7070-exec-10] cachesync.Broadcaster:236 : Done broadcasting metadata change: entity=kafka, event=UPDATE, cacheKey=DEFAULT.STREAMING_SALES_TABLE
2017-10-11 02:15:13,789 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:15:23,780 DEBUG [http-bio-7070-exec-7] kafka.KafkaSource:83 : Last segment doesn't exist, and didn't initiate the start offset, will seek from topic's earliest offset.
2017-10-11 20:50:42,558 INFO [http-bio-7070-exec-8] utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
2017-10-11 20:50:42,563 INFO [http-bio-7070-exec-8] utils.AppInfoParser:84 : Kafka commitId : unknown
2017-10-11 20:50:42,570 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:105 : Seek end offsets from topic
2017-10-11 20:50:42,570 INFO [http-bio-7070-exec-8] consumer.ConsumerConfig:196 : ConsumerConfig values:
auto.commit.interval.ms<http://auto.commit.interval.ms> = 5000
auto.offset.reset = latest
bootstrap.servers = [localhost:9092]
check.crcs = true
client.id<http://client.id> =
connections.max.idle.ms<http://connections.max.idle.ms> = 540000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms<http://fetch.max.wait.ms> = 500
fetch.min.bytes = 1
group.id<http://group.id> = streaming_cube
heartbeat.interval.ms<http://heartbeat.interval.ms> = 3000
interceptor.classes = null
internal.leave.group.on.close = true
key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms<http://max.poll.interval.ms> = 300000
max.poll.records = 500
metadata.max.age.ms<http://metadata.max.age.ms> = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms<http://metrics.sample.window.ms> = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.ms<http://reconnect.backoff.ms> = 50
request.timeout.ms<http://request.timeout.ms> = 305000
retry.backoff.ms<http://retry.backoff.ms> = 100
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name<http://sasl.kerberos.service.name> = null
sasl.kerberos.ticket.renew.jitter = 0.05
request.timeout.ms<http://request.timeout.ms> = 305000
retry.backoff.ms<http://retry.backoff.ms> = 100
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name<http://sasl.kerberos.service.name> = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
session.timeout.ms<http://session.timeout.ms> = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] utils.AppInfoParser:84 : Kafka commitId : unknown
2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:107 : The end offsets are {0=0}
2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8] controller.CubeController:305 : No new message comes, startOffset = endOffset:0
java.lang.IllegalArgumentException: No new message comes, startOffset = endOffset:0
at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:134)
at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:236)
Regards,
Manoj
From: Billy Liu [mailto:billyliu@apache.org<ma...@apache.org>]
Sent: Thursday, October 12, 2017 1:06 PM
To: user
Subject: Re: Kafka Streaming data - Error while building the Cube
Hi Kumar,
Could you paste more Kafka Consumer related log in kylin.log? And also check from the Kafka broker side, if the Kylin client has connected to Broker.
2017-10-12 14:29 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>>:
Building the Cube from Kylin UI - Although Messages are there in Kafka topic but Kylin is not able read the offset. Can someone help on this?
2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] utils.AppInfoParser:84 : Kafka commitId : unknown
2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:107 : The end offsets are {0=0}
2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8] controller.CubeController:305 : No new message comes, startOffset = endOffset:0
java.lang.IllegalArgumentException: No new message comes, startOffset = endOffset:0
at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:134)
at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:236)
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:208)
at org.apache.kylin.rest.service.JobService$$FastClassBySpringCGLIB$$83a44b2a.invoke(<generated>)
Regards,
Manoj
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
Re: Kafka Streaming data - Error while building the Cube
Posted by Billy Liu <bi...@apache.org>.
STREAMING_SALES_TABLE table reads messages from Kafka topic kylin_demo，but
got 0 message.
Could you check if the topic has incoming message:
bin/kafka-console-consumer.sh
--zookeeper localhost:2181 --bootstrap-server localhost:9092 --topic
kylin_demo
2017-10-12 17:19 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
> Pls. find below information about consumer config from Kylin log file.
>
>
>
> 2017-10-11 02:11:43,787 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0
> others
>
> 2017-10-11 02:12:13,783 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0
> others
>
> 2017-10-11 02:12:40,734 INFO [http-bio-7070-exec-3]
> streaming.StreamingManager:222 : Reloading Streaming Metadata from folder
> kylin_metadata(key='/streaming')@kylin_metadata@hbase
>
> 2017-10-11 02:12:40,760 DEBUG [http-bio-7070-exec-3]
> streaming.StreamingManager:247 : Loaded 1 StreamingConfig(s)
>
> 2017-10-11 02:12:43,789 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0
> others
>
> 2017-10-11 02:13:13,788 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0
> others
>
> 2017-10-11 02:13:43,785 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0
> others
>
> 2017-10-11 02:14:13,789 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0
> others
>
> 2017-10-11 02:14:43,796 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0
> others
>
> 2017-10-11 02:15:03,911 DEBUG [http-bio-7070-exec-1] controller.StreamingController:255
> : Saving StreamingConfig {"uuid":"8613b0e1-40ac-438c-
> bdf5-72be4d91c230","last_modified":1507705685859,"
> version":"2.1.0","name":"DEFAULT.STREAMING_SALES_TABLE","type":"kafka"}
>
> 2017-10-11 02:15:03,913 DEBUG [http-bio-7070-exec-1] controller.StreamingController:273
> : Saving KafkaConfig {"uuid":"87dc6ab5-5141-4bd8-8e00-c16ec86dce41","last_
> modified":1507705685916,"version":"2.1.0","name":"
> DEFAULT.STREAMING_SALES_TABLE","clusters":[{"brokers":[{"id"
> :"1","host":"sandbox","port":"9092"}]}],"topic":"kylin_demo"
> ,"timeout":60000,"parserName":"org.apache.kylin.source.
> kafka.TimedJsonStreamParser","parserTimeStampField":null,"
> margin":0,"parserProperties":"tsColName=order_time"}
>
> 2017-10-11 02:15:03,963 DEBUG [pool-7-thread-1] cachesync.Broadcaster:132
> : Servers in the cluster: [localhost:7070]
>
> 2017-10-11 02:15:04,000 DEBUG [pool-7-thread-1] cachesync.Broadcaster:139
> : Announcing new broadcast event: BroadcastEvent{entity=streaming,
> event=update, cacheKey=DEFAULT.STREAMING_SALES_TABLE}
>
> 2017-10-11 02:15:04,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:132
> : Servers in the cluster: [localhost:7070]
>
> 2017-10-11 02:15:04,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:139
> : Announcing new broadcast event: BroadcastEvent{entity=kafka,
> event=update, cacheKey=DEFAULT.STREAMING_SALES_TABLE}
>
> 2017-10-11 02:15:04,164 DEBUG [http-bio-7070-exec-9]
> cachesync.Broadcaster:236 : Done broadcasting metadata change:
> entity=streaming, event=UPDATE, cacheKey=DEFAULT.STREAMING_SALES_TABLE
>
> 2017-10-11 02:15:04,192 DEBUG [http-bio-7070-exec-10]
> cachesync.Broadcaster:236 : Done broadcasting metadata change:
> entity=kafka, event=UPDATE, cacheKey=DEFAULT.STREAMING_SALES_TABLE
>
> 2017-10-11 02:15:13,789 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0
> others
>
> 2017-10-11 02:15:23,780 DEBUG [http-bio-7070-exec-7] kafka.KafkaSource:83
> : Last segment doesn't exist, and didn't initiate the start offset, will
> seek from topic's earliest offset.
>
>
>
> 2017-10-11 20:50:42,558 INFO [http-bio-7070-exec-8]
> utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
>
> 2017-10-11 20:50:42,563 INFO [http-bio-7070-exec-8]
> utils.AppInfoParser:84 : Kafka commitId : unknown
>
> 2017-10-11 20:50:42,570 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:105
> : Seek end offsets from topic
>
> 2017-10-11 20:50:42,570 INFO [http-bio-7070-exec-8]
> consumer.ConsumerConfig:196 : ConsumerConfig values:
>
> auto.commit.interval.ms = 5000
>
> auto.offset.reset = latest
>
> bootstrap.servers = [localhost:9092]
>
> check.crcs = true
>
> client.id =
>
> connections.max.idle.ms = 540000
>
> enable.auto.commit = false
>
> exclude.internal.topics = true
>
> fetch.max.bytes = 52428800
>
> fetch.max.wait.ms = 500
>
> fetch.min.bytes = 1
>
> group.id = streaming_cube
>
> heartbeat.interval.ms = 3000
>
> interceptor.classes = null
>
> internal.leave.group.on.close = true
>
> key.deserializer = class org.apache.kafka.common.serialization.
> StringDeserializer
>
> max.partition.fetch.bytes = 1048576
>
> max.poll.interval.ms = 300000
>
> max.poll.records = 500
>
> metadata.max.age.ms = 300000
>
> metric.reporters = []
>
> metrics.num.samples = 2
>
> metrics.recording.level = INFO
>
> metrics.sample.window.ms = 30000
>
> partition.assignment.strategy = [class org.apache.kafka.clients.
> consumer.RangeAssignor]
>
> receive.buffer.bytes = 65536
>
> reconnect.backoff.ms = 50
>
> request.timeout.ms = 305000
>
> retry.backoff.ms = 100
>
> sasl.jaas.config = null
>
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
>
> sasl.kerberos.min.time.before.relogin = 60000
>
> sasl.kerberos.service.name = null
>
> sasl.kerberos.ticket.renew.jitter = 0.05
>
> request.timeout.ms = 305000
>
> retry.backoff.ms = 100
>
> sasl.jaas.config = null
>
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
>
> sasl.kerberos.min.time.before.relogin = 60000
>
> sasl.kerberos.service.name = null
>
> sasl.kerberos.ticket.renew.jitter = 0.05
>
> sasl.kerberos.ticket.renew.window.factor = 0.8
>
> sasl.mechanism = GSSAPI
>
> security.protocol = PLAINTEXT
>
> send.buffer.bytes = 131072
>
> session.timeout.ms = 10000
>
> ssl.cipher.suites = null
>
> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
>
> ssl.endpoint.identification.algorithm = null
>
> ssl.key.password = null
>
> ssl.keymanager.algorithm = SunX509
>
> ssl.keystore.location = null
>
> ssl.keystore.password = null
>
> ssl.keystore.type = JKS
>
> ssl.protocol = TLS
>
> ssl.provider = null
>
> ssl.secure.random.implementation = null
>
> ssl.trustmanager.algorithm = PKIX
>
> ssl.truststore.location = null
>
> ssl.truststore.password = null
>
> ssl.truststore.type = JKS
>
> value.deserializer = class org.apache.kafka.common.serialization.
> StringDeserializer
>
>
>
> 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8]
> utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
>
> 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8]
> utils.AppInfoParser:84 : Kafka commitId : unknown
>
> 2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:107
> : The end offsets are {0=0}
>
> 2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8]
> controller.CubeController:305 : No new message comes, startOffset =
> endOffset:0
>
> java.lang.IllegalArgumentException: No new message comes, startOffset =
> endOffset:0
>
> at org.apache.kylin.source.kafka.KafkaSource.
> enrichSourcePartitionBeforeBuild(KafkaSource.java:134)
>
> at org.apache.kylin.rest.service.JobService.submitJobInternal(
> JobService.java:236)
>
> Regards,
>
> Manoj
>
>
>
> *From:* Billy Liu [mailto:billyliu@apache.org]
> *Sent:* Thursday, October 12, 2017 1:06 PM
> *To:* user
> *Subject:* Re: Kafka Streaming data - Error while building the Cube
>
>
>
> Hi Kumar,
>
>
>
> Could you paste more Kafka Consumer related log in kylin.log? And also
> check from the Kafka broker side, if the Kylin client has connected to
> Broker.
>
>
>
> 2017-10-12 14:29 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
>
> Building the Cube from Kylin UI - Although Messages are there in Kafka
> topic but Kylin is not able read the offset. Can someone help on this?
>
> 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8]
> utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
> 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8]
> utils.AppInfoParser:84 : Kafka commitId : unknown
> 2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:107
> : The end offsets are {0=0}
> 2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8]
> controller.CubeController:305 : No new message comes, startOffset =
> endOffset:0
> java.lang.IllegalArgumentException: No new message comes, startOffset =
> endOffset:0
> at org.apache.kylin.source.kafka.KafkaSource.
> enrichSourcePartitionBeforeBuild(KafkaSource.java:134)
> at org.apache.kylin.rest.service.JobService.submitJobInternal(
> JobService.java:236)
> at org.apache.kylin.rest.service.JobService.submitJob(
> JobService.java:208)
> at org.apache.kylin.rest.service.JobService$$
> FastClassBySpringCGLIB$$83a44b2a.invoke(<generated>)
>
> Regards,
> Manoj
>
>
> This message is confidential and subject to terms at:
> http://www.jpmorgan.com/emaildisclaimer including on confidentiality,
> legal privilege, viruses and monitoring of electronic messages. If you are
> not the intended recipient, please delete this message and notify the
> sender immediately. Any unauthorized use is strictly prohibited.
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
RE: Kafka Streaming data - Error while building the Cube
Posted by "Kumar, Manoj H" <ma...@jpmorgan.com>.
Pls. find below information about consumer config from Kylin log file.
2017-10-11 02:11:43,787 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:12:13,783 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:12:40,734 INFO [http-bio-7070-exec-3] streaming.StreamingManager:222 : Reloading Streaming Metadata from folder kylin_metadata(key='/streaming')@kylin_metadata@hbase
2017-10-11 02:12:40,760 DEBUG [http-bio-7070-exec-3] streaming.StreamingManager:247 : Loaded 1 StreamingConfig(s)
2017-10-11 02:12:43,789 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:13:13,788 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:13:43,785 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:14:13,789 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:14:43,796 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:15:03,911 DEBUG [http-bio-7070-exec-1] controller.StreamingController:255 : Saving StreamingConfig {"uuid":"8613b0e1-40ac-438c-bdf5-72be4d91c230","last_modified":1507705685859,"version":"2.1.0","name":"DEFAULT.STREAMING_SALES_TABLE","type":"kafka"}
2017-10-11 02:15:03,913 DEBUG [http-bio-7070-exec-1] controller.StreamingController:273 : Saving KafkaConfig {"uuid":"87dc6ab5-5141-4bd8-8e00-c16ec86dce41","last_modified":1507705685916,"version":"2.1.0","name":"DEFAULT.STREAMING_SALES_TABLE","clusters":[{"brokers":[{"id":"1","host":"sandbox","port":"9092"}]}],"topic":"kylin_demo","timeout":60000,"parserName":"org.apache.kylin.source.kafka.TimedJsonStreamParser","parserTimeStampField":null,"margin":0,"parserProperties":"tsColName=order_time"}
2017-10-11 02:15:03,963 DEBUG [pool-7-thread-1] cachesync.Broadcaster:132 : Servers in the cluster: [localhost:7070]
2017-10-11 02:15:04,000 DEBUG [pool-7-thread-1] cachesync.Broadcaster:139 : Announcing new broadcast event: BroadcastEvent{entity=streaming, event=update, cacheKey=DEFAULT.STREAMING_SALES_TABLE}
2017-10-11 02:15:04,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:132 : Servers in the cluster: [localhost:7070]
2017-10-11 02:15:04,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:139 : Announcing new broadcast event: BroadcastEvent{entity=kafka, event=update, cacheKey=DEFAULT.STREAMING_SALES_TABLE}
2017-10-11 02:15:04,164 DEBUG [http-bio-7070-exec-9] cachesync.Broadcaster:236 : Done broadcasting metadata change: entity=streaming, event=UPDATE, cacheKey=DEFAULT.STREAMING_SALES_TABLE
2017-10-11 02:15:04,192 DEBUG [http-bio-7070-exec-10] cachesync.Broadcaster:236 : Done broadcasting metadata change: entity=kafka, event=UPDATE, cacheKey=DEFAULT.STREAMING_SALES_TABLE
2017-10-11 02:15:13,789 INFO [pool-8-thread-1] threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 others
2017-10-11 02:15:23,780 DEBUG [http-bio-7070-exec-7] kafka.KafkaSource:83 : Last segment doesn't exist, and didn't initiate the start offset, will seek from topic's earliest offset.
2017-10-11 20:50:42,558 INFO [http-bio-7070-exec-8] utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
2017-10-11 20:50:42,563 INFO [http-bio-7070-exec-8] utils.AppInfoParser:84 : Kafka commitId : unknown
2017-10-11 20:50:42,570 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:105 : Seek end offsets from topic
2017-10-11 20:50:42,570 INFO [http-bio-7070-exec-8] consumer.ConsumerConfig:196 : ConsumerConfig values:
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [localhost:9092]
check.crcs = true
client.id =
connections.max.idle.ms = 540000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = streaming_cube
heartbeat.interval.ms = 3000
interceptor.classes = null
internal.leave.group.on.close = true
key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.ms = 50
request.timeout.ms = 305000
retry.backoff.ms = 100
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
request.timeout.ms = 305000
retry.backoff.ms = 100
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
session.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] utils.AppInfoParser:84 : Kafka commitId : unknown
2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:107 : The end offsets are {0=0}
2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8] controller.CubeController:305 : No new message comes, startOffset = endOffset:0
java.lang.IllegalArgumentException: No new message comes, startOffset = endOffset:0
at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:134)
at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:236)
Regards,
Manoj
From: Billy Liu [mailto:billyliu@apache.org]
Sent: Thursday, October 12, 2017 1:06 PM
To: user
Subject: Re: Kafka Streaming data - Error while building the Cube
Hi Kumar,
Could you paste more Kafka Consumer related log in kylin.log? And also check from the Kafka broker side, if the Kylin client has connected to Broker.
2017-10-12 14:29 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>>:
Building the Cube from Kylin UI - Although Messages are there in Kafka topic but Kylin is not able read the offset. Can someone help on this?
2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] utils.AppInfoParser:84 : Kafka commitId : unknown
2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:107 : The end offsets are {0=0}
2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8] controller.CubeController:305 : No new message comes, startOffset = endOffset:0
java.lang.IllegalArgumentException: No new message comes, startOffset = endOffset:0
at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:134)
at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:236)
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:208)
at org.apache.kylin.rest.service.JobService$$FastClassBySpringCGLIB$$83a44b2a.invoke(<generated>)
Regards,
Manoj
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
Re: Kafka Streaming data - Error while building the Cube
Posted by Billy Liu <bi...@apache.org>.
Hi Kumar,
Could you paste more Kafka Consumer related log in kylin.log? And also
check from the Kafka broker side, if the Kylin client has connected to
Broker.
2017-10-12 14:29 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
> Building the Cube from Kylin UI - Although Messages are there in Kafka
> topic but Kylin is not able read the offset. Can someone help on this?
>
> 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8]
> utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
> 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8]
> utils.AppInfoParser:84 : Kafka commitId : unknown
> 2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:107
> : The end offsets are {0=0}
> 2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8]
> controller.CubeController:305 : No new message comes, startOffset =
> endOffset:0
> java.lang.IllegalArgumentException: No new message comes, startOffset =
> endOffset:0
> at org.apache.kylin.source.kafka.KafkaSource.
> enrichSourcePartitionBeforeBuild(KafkaSource.java:134)
> at org.apache.kylin.rest.service.JobService.submitJobInternal(
> JobService.java:236)
> at org.apache.kylin.rest.service.JobService.submitJob(
> JobService.java:208)
> at org.apache.kylin.rest.service.JobService$$
> FastClassBySpringCGLIB$$83a44b2a.invoke(<generated>)
>
> Regards,
> Manoj
>
>
> This message is confidential and subject to terms at:
> http://www.jpmorgan.com/emaildisclaimer including on confidentiality,
> legal privilege, viruses and monitoring of electronic messages. If you are
> not the intended recipient, please delete this message and notify the
> sender immediately. Any unauthorized use is strictly prohibited.
>
>