You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by "liyang (JIRA)" <ji...@apache.org> on 2016/12/15 05:06:59 UTC
[jira] [Created] (KYLIN-2283) Have a general purpose data
generation tool
liyang created KYLIN-2283:
-----------------------------
Summary: Have a general purpose data generation tool
Key: KYLIN-2283
URL: https://issues.apache.org/jira/browse/KYLIN-2283
Project: Kylin
Issue Type: Improvement
Reporter: liyang
To generate some random data according to a data model. Help troubleshooting and testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: When building streaming cube，i find two issues
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi lei,
for question 1, the API returns the start offset for each partition; In
your case, Kylin will start from from 172400 on parittion 0, instead of
from the very begining;
for question 2, I'm afraid changing partition number before a build be
completed isn't supported in 1.6.0; please discard the current job and
resubmit a new one. You raised a good point that this may not be avoided,
especially when there are a couple of streaming cubes being built
frequently; Could you please open a JIRA to Kylin?
Thanks for your feedback;
2016-12-15 15:10 GMT+08:00 张磊 <12...@qq.com>:
> I follow the tutorial build a streaming cube, the partitions of the topic
> is 1
> 1、I test the blow resuful api, the result is {"result":"success","offsets":"{0=172400}"},
> but how can i use this result?curl -X PUT --user ADMIN:KYLIN -H
> "Content-Type: application/json;charset=utf-8" -d '{ "sourceOffsetStart":
> 0, "sourceOffsetEnd": 9223372036854775807, "buildType": "BUILD"}'
> http://localhost:7070/kylin/api/cubes/{your_cube_name}/init_start_offsets
>
>
> 2、in the middle of build cube，i alter the topic partitions to 2，the cube
> build error。Can not i alter the partitions of the topic？
> The error message is：
> java.lang.IllegalArgumentException: partition number mismatch with server
> side at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
> at org.apache.kylin.source.kafka.hadoop.KafkaInputFormat.
> getSplits(KafkaInputFormat.java:71) at org.apache.hadoop.mapreduce.
> JobSubmitter.writeNewSplits(JobSubmitter.java:301) at
> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:318)
> at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) at
> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1657) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
> at org.apache.kylin.engine.mr.common.AbstractHadoopJob.
> waitForCompletion(AbstractHadoopJob.java:149) at
> org.apache.kylin.source.kafka.hadoop.KafkaFlatTableJob.run(KafkaFlatTableJob.java:122)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:92) at
> org.apache.kylin.engine.mr.common.MapReduceExecutable.
> doWork(MapReduceExecutable.java:120) at org.apache.kylin.job.
> execution.AbstractExecutable.execute(AbstractExecutable.java:113)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:57) at org.apache.kylin.job.
> execution.AbstractExecutable.execute(AbstractExecutable.java:113)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745) result code:2
--
Best regards,
Shaofeng Shi 史少锋
When building streaming cube，i find two issues
Posted by 张磊 <12...@qq.com>.
I follow the tutorial build a streaming cube, the partitions of the topic is 1
1、I test the blow resuful api, the result is {"result":"success","offsets":"{0=172400}"}, but how can i use this result?curl -X PUT --user ADMIN:KYLIN -H "Content-Type: application/json;charset=utf-8" -d '{ "sourceOffsetStart": 0, "sourceOffsetEnd": 9223372036854775807, "buildType": "BUILD"}' http://localhost:7070/kylin/api/cubes/{your_cube_name}/init_start_offsets
2、in the middle of build cube，i alter the topic partitions to 2，the cube build error。Can not i alter the partitions of the topic？
The error message is：
java.lang.IllegalArgumentException: partition number mismatch with server side at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92) at org.apache.kylin.source.kafka.hadoop.KafkaInputFormat.getSplits(KafkaInputFormat.java:71) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:301) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:318) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) at org.apache.kylin.engine.mr.common.AbstractHadoopJob.waitForCompletion(AbstractHadoopJob.java:149) at org.apache.kylin.source.kafka.hadoop.KafkaFlatTableJob.run(KafkaFlatTableJob.java:122) at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:92) at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:120) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) result code:2