You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by 梅秋莹 <32...@qq.com> on 2020/04/21 03:22:45 UTC
When build streaming cube in kylin, I got an error tip.
Hello，dear kyliners:
Lately, We tried to create a streaming model in kylin ,which refer to doc "Cubing from Kafka(Near RT Streaming)" in kylin official website,
and we created the model and cube successfully, but when build the cube, it appeared an error,like this:
"
ERROR [http-nio-7070-exec-9] controller.CubeController:398 : org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;"
our environment: kylin 2.6.3; kafka 2.0.0.
If you know where is the problem or you know how to solve it, please contact me, thank you very much!
回复： erros on building cube in kylin
Posted by 我是天才 <50...@qq.com>.
Thanks !I tried you way,but i didn't work.
------------------ 原始邮件 ------------------
发件人: "Fatemeh Ahmadi"<fahmadi281994@gmail.com>;
发送时间: 2020年5月6日(星期三) 晚上8:39
收件人: "user"<user@kylin.apache.org>;"我是天才"<50308969@qq.com>;
主题: Re: erros on building cube in kylin
Hi,According to this link, “Too high cardinality is not suitable for dictionary”. Have you tried other encoding methods such as “fixed_length” for the UHC column? I am not sure maybe this is not the best solution but it works for me.
You can also check "Deal with Ultra-High-Cardinality Columns" section in this page.
On Wed, May 6, 2020 at 4:15 PM 我是天才 <50308969@qq.com> wrote:
Hello:
I have a table and it has three columns,it has 5736604 rows,the three columns distinct value counts is 1843696 5736485 1742356.I build cube and got a error on 3rd step,it said:"Error: Java heap space",the step name is " Extract Fact Table Distinct Columns".
Can you help me? thanks!
Re: erros on building cube in kylin
Posted by Fatemeh Ahmadi <fa...@gmail.com>.
Hi,
According to this link
<http://kylin.apache.org/docs/howto/howto_optimize_build.html>, “Too high
cardinality is not suitable for dictionary”. Have you tried other encoding
methods such as “fixed_length” for the UHC column? I am not sure maybe
this is not the best solution but it works for me.
You can also check "Deal with Ultra-High-Cardinality Columns" section in this
page <http://kylin.apache.org/docs/install/configuration.html>.
On Wed, May 6, 2020 at 4:15 PM 我是天才 <50...@qq.com> wrote:
> Hello:
> I have a table and it has three columns,it has 5736604 rows,the three
> columns distinct value counts is 1843696 5736485 1742356.I build cube and
> got a error on 3rd step,it said:"Error: Java heap space",the step name is
> " Extract Fact Table Distinct Columns".
> Can you help me? thanks!
>
回复：erros on building cube in kylin
Posted by 我是天才 <50...@qq.com>.
if i want to add "kylin.engine.mr.config-override" in the front of them.
MY config below as:
kylin_job_conf.xml
<property>
<name>mapreduce.map.java.opts.max.heap</name>
<value>204800</value>
</property>
<property>
<name>mapreduce.reduce.java.opts.max.heap</name>
<value>204800</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx8000m -XX:OnOutOfMemoryError='kill -9 %p'</value>
<description></description>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx8000m -XX:OnOutOfMemoryError='kill -9 %p'</value>
<description></description>
</property>
<property>
<name>mapred.child.java.opt</name>
<value>-Xmx8000m -XX:OnOutOfMemoryError='kill -9 %p'</value>
<description></description>
kylin_job_conf_inmem.xml
<property>
<name>mapreduce.map.memory.mb</name>
<value>8192</value>
<description></description>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx12g -XX:OnOutOfMemoryError='kill -9 %p'</value>
<description></description>
</property>
<property>
<name>mapreduce.task.io.sort.mb</name>
<value>20</value>
<description></description>
</property>
------------------ 原始邮件 ------------------
发件人: "Xiaoxiang Yu"<xxyu@apache.org>;
发送时间: 2020年5月6日(星期三) 晚上7:57
收件人: "user"<user@kylin.apache.org>;"我是天才"<50308969@qq.com>;
主题: Re:erros on building cube in kylin
Hi,
Have you ever check this doc : http://kylin.apache.org/docs/install/configuration.html#mr-config-override and increase the heap of Mapper or Reducer?
--
Best wishes to you !
From ：Xiaoxiang Yu
At 2020-05-06 19:45:43, "我是天才" <50308969@qq.com> wrote:
Hello:
I have a table and it has three columns,it has 5736604 rows,the three columns distinct value counts is 1843696 5736485 1742356.I build cube and got a error on 3rd step,it said:"Error: Java heap space",the step name is " Extract Fact Table Distinct Columns".
Can you help me? thanks!
Re:erros on building cube in kylin
Posted by Xiaoxiang Yu <xx...@apache.org>.
Hi,
Have you ever check this doc : http://kylin.apache.org/docs/install/configuration.html#mr-config-override and increase the heap of Mapper or Reducer?
--
Best wishes to you !
From ：Xiaoxiang Yu
At 2020-05-06 19:45:43, "我是天才" <50...@qq.com> wrote:
Hello:
I have a table and it has three columns,it has 5736604 rows,the three columns distinct value counts is 184369657364851742356.I build cube and got a error on 3rd step,it said:"Error: Java heap space",the step name is " Extract Fact Table Distinct Columns".
Can you help me? thanks!
erros on building cube in kylin
Posted by 我是天才 <50...@qq.com>.
Hello:
I have a table and it has three columns,it has 5736604 rows,the three columns distinct value counts is 1843696 5736485 1742356.I build cube and got a error on 3rd step,it said:"Error: Java heap space",the step name is " Extract Fact Table Distinct Columns".
Can you help me? thanks!
回复：When build streaming cube in kylin, I got an error tip.
Posted by 梅秋莹 <32...@qq.com>.
Hello,
My Hadoop version is HDP 3.1.1. Yesterday, I have solved the problem with the method as follows:
In the follow path， we have repalced the "kafka-client-{version}.jar " from a lower version to a higher. Then we can build the streaming cube successfully.
------------------ 原始邮件 ------------------
发件人: "wangrupeng"<wangrupeng@live.cn>;
发送时间: 2020年4月22日(星期三) 下午3:52
收件人: "user@kylin.apache.org"<user@kylin.apache.org>;
主题: Re: 回复： When build streaming cube in kylin, I got an error tip.
发件人: Rupeng Wang <wangrupeng@live.cn>
日期: 2020年4月22日 星期三 15:45
收件人: 梅秋莹 <3281438086@qq.com>
主题: Re: 回复： When build streaming cube in kylin, I got an error tip.
Well, I test it in Hadoop cluster with Kylin2.6.3 and Kafka2.0.0. But it run normally. My Hadoop version is cdh5.7. What’s your Hadoop version?
-------------------
Best wishes,
Rupeng Wang
发件人: 梅秋莹 <3281438086@qq.com>
日期: 2020年4月21日 星期二 16:46
收件人: user <user@kylin.apache.org>
抄送: wangrupeng <wangrupeng@live.cn>, Yaqian_Zhang <Yaqian_Zhang@126.com>
主题: 回复： When build streaming cube in kylin, I got an error tip.
Thank you for your suggestions!
When I tried with those plans you provided, I can find the kafka-client-{version}.jar and it also can be loaded by kylin. But when we restart the kylin, and recreate the model and cube，it also goes wrong with the same
error message.
------------------ 原始邮件 ------------------
发件人: "Rupeng Wang"<wangrupeng@live.cn>;
发送时间: 2020年4月21日(星期二) 下午3:14
收件人: "user"<user@kylin.apache.org>;
主题: Re: When build streaming cube in kylin, I got an error tip.
Yes, you can try this command to determine whether kafka-client-{version}.jar can be found.
find -L $KAFKA_HOME -name 'kafka-clients-[a-z0-9A-Z\.-]*.jar' ! -name '*doc*' ! -name '*test*' ! -name '*sources*' ''-printf '%p:' | sed 's/:$//'
And also check check if kafka-client-{version}.jar is loaded by kylin.
cat ${KYLIN_HOME}/pid #get kylin proess id
ps -ef | grep ${pid}
Then take a look if the kafka-client.jar is included in the jars which are loaded by kylin when kylin start.
-------------------
Best wishes,
Rupeng Wang
发件人: Yaqian Zhang <Yaqian_Zhang@126.com>
答复: <user@kylin.apache.org>
日期: 2020年4月21日 星期二 14:51
收件人: <user@kylin.apache.org>
主题: Re: When build streaming cube in kylin, I got an error tip.
Hi：
Is Kafka home set before kylin is started？If not, you can set KAFKA_HOME and restart kylin.
If you still encounter this problem after setting KAFKA_HOME, and your hadoop environment is hdp3.0, you can take a look at this issue https://issues.apache.org/jira/browse/KYLIN-3970. I'm not sure if it will help you.
在 2020年4月21日，11:22，梅秋莹 <3281438086@qq.com> 写道：
Hello，dear kyliners:
Lately, We tried to create a streaming model in kylin ,which refer to doc "Cubing from Kafka(Near RT Streaming)" in kylin official website,
and we created the model and cube successfully, but when build the cube, it appeared an error,like this:
"
ERROR [http-nio-7070-exec-9] controller.CubeController:398 : org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;"
<403E365D@D00A1012.05679E5E.png.jpg>
our environment: kylin 2.6.3; kafka 2.0.0.
If you know where is the problem or you know how to solve it, please contact me, thank you very much!
Re: 回复： When build streaming cube in kylin, I got an error tip.
Posted by Rupeng Wang <wa...@live.cn>.
发件人: Rupeng Wang <wa...@live.cn>
日期: 2020年4月22日 星期三 15:45
收件人: 梅秋莹 <32...@qq.com>
主题: Re: 回复： When build streaming cube in kylin, I got an error tip.
Well, I test it in Hadoop cluster with Kylin2.6.3 and Kafka2.0.0. But it run normally. My Hadoop version is cdh5.7. What’s your Hadoop version?
-------------------
Best wishes,
Rupeng Wang
发件人: 梅秋莹 <32...@qq.com>
日期: 2020年4月21日 星期二 16:46
收件人: user <us...@kylin.apache.org>
抄送: wangrupeng <wa...@live.cn>, Yaqian_Zhang <Ya...@126.com>
主题: 回复： When build streaming cube in kylin, I got an error tip.
Thank you for your suggestions!
When I tried with those plans you provided, I can find the kafka-client-{version}.jar and it also can be loaded by kylin. But when we restart the kylin, and recreate the model and cube，it also goes wrong with the same
error message.
------------------ 原始邮件 ------------------
发件人: "Rupeng Wang"<wa...@live.cn>;
发送时间: 2020年4月21日(星期二) 下午3:14
收件人: "user"<us...@kylin.apache.org>;
主题: Re: When build streaming cube in kylin, I got an error tip.
Yes, you can try this command to determine whether kafka-client-{version}.jar can be found.
find -L $KAFKA_HOME -name 'kafka-clients-[a-z0-9A-Z\.-]*.jar' ! -name '*doc*' ! -name '*test*' ! -name '*sources*' ''-printf '%p:' | sed 's/:$//'
And also check check if kafka-client-{version}.jar is loaded by kylin.
cat ${KYLIN_HOME}/pid #get kylin proess id
ps -ef | grep ${pid}
Then take a look if the kafka-client.jar is included in the jars which are loaded by kylin when kylin start.
-------------------
Best wishes,
Rupeng Wang
发件人: Yaqian Zhang <Ya...@126.com>
答复: <us...@kylin.apache.org>
日期: 2020年4月21日 星期二 14:51
收件人: <us...@kylin.apache.org>
主题: Re: When build streaming cube in kylin, I got an error tip.
Hi：
Is Kafka home set before kylin is started？If not, you can set KAFKA_HOME and restart kylin.
If you still encounter this problem after setting KAFKA_HOME, and your hadoop environment is hdp3.0, you can take a look at this issue https://issues.apache.org/jira/browse/KYLIN-3970. I'm not sure if it will help you.
在 2020年4月21日，11:22，梅秋莹 <32...@qq.com> 写道：
Hello，dear kyliners:
Lately, We tried to create a streaming model in kylin ,which refer to doc "Cubing from Kafka(Near RT Streaming)" in kylin official website,
and we created the model and cube successfully, but when build the cube, it appeared an error,like this:
"
ERROR [http-nio-7070-exec-9] controller.CubeController:398 : org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;"
<40...@D00A1012.05679E5E.png.jpg>
our environment: kylin 2.6.3; kafka 2.0.0.
If you know where is the problem or you know how to solve it, please contact me, thank you very much!
回复： When build streaming cube in kylin, I got an error tip.
Posted by 梅秋莹 <32...@qq.com>.
Thank you for your suggestions!
When I tried with those plans you provided, I can find the kafka-client-{version}.jar and it also can be loaded by kylin. But when we restart the kylin, and recreate the model and cube，it also goes wrong with the same
error message.
------------------ 原始邮件 ------------------
发件人: "Rupeng Wang"<wangrupeng@live.cn>;
发送时间: 2020年4月21日(星期二) 下午3:14
收件人: "user"<user@kylin.apache.org>;
主题: Re: When build streaming cube in kylin, I got an error tip.
Yes, you can try this command to determine whether kafka-client-{version}.jar can be found.
find -L $KAFKA_HOME -name 'kafka-clients-[a-z0-9A-Z\.-]*.jar' ! -name '*doc*' ! -name '*test*' ! -name '*sources*' ''-printf '%p:' | sed 's/:$//'
And also check check if kafka-client-{version}.jar is loaded by kylin.
cat ${KYLIN_HOME}/pid #get kylin proess id
ps -ef | grep ${pid}
Then take a look if the kafka-client.jar is included in the jars which are loaded by kylin when kylin start.
-------------------
Best wishes,
Rupeng Wang
发件人: Yaqian Zhang <Yaqian_Zhang@126.com>
答复: <user@kylin.apache.org>
日期: 2020年4月21日 星期二 14:51
收件人: <user@kylin.apache.org>
主题: Re: When build streaming cube in kylin, I got an error tip.
Hi：
Is Kafka home set before kylin is started？If not, you can set KAFKA_HOME and restart kylin.
If you still encounter this problem after setting KAFKA_HOME, and your hadoop environment is hdp3.0, you can take a look at this issue https://issues.apache.org/jira/browse/KYLIN-3970. I'm not sure if it will help you.
在 2020年4月21日，11:22，梅秋莹 <3281438086@qq.com> 写道：
Hello，dear kyliners:
Lately, We tried to create a streaming model in kylin ,which refer to doc "Cubing from Kafka(Near RT Streaming)" in kylin official website,
and we created the model and cube successfully, but when build the cube, it appeared an error,like this:
"
ERROR [http-nio-7070-exec-9] controller.CubeController:398 : org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;"
<403E365D@D00A1012.05679E5E.png.jpg>
our environment: kylin 2.6.3; kafka 2.0.0.
If you know where is the problem or you know how to solve it, please contact me, thank you very much!
Re: When build streaming cube in kylin, I got an error tip.
Posted by Rupeng Wang <wa...@live.cn>.
Yes, you can try this command to determine whether kafka-client-{version}.jar can be found.
find -L $KAFKA_HOME -name 'kafka-clients-[a-z0-9A-Z\.-]*.jar' ! -name '*doc*' ! -name '*test*' ! -name '*sources*' ''-printf '%p:' | sed 's/:$//'
And also check check if kafka-client-{version}.jar is loaded by kylin.
cat ${KYLIN_HOME}/pid #get kylin proess id
ps -ef | grep ${pid}
Then take a look if the kafka-client.jar is included in the jars which are loaded by kylin when kylin start.
-------------------
Best wishes,
Rupeng Wang
发件人: Yaqian Zhang <Ya...@126.com>
答复: <us...@kylin.apache.org>
日期: 2020年4月21日 星期二 14:51
收件人: <us...@kylin.apache.org>
主题: Re: When build streaming cube in kylin, I got an error tip.
Hi：
Is Kafka home set before kylin is started？If not, you can set KAFKA_HOME and restart kylin.
If you still encounter this problem after setting KAFKA_HOME, and your hadoop environment is hdp3.0, you can take a look at this issue https://issues.apache.org/jira/browse/KYLIN-3970. I'm not sure if it will help you.
在 2020年4月21日，11:22，梅秋莹 <32...@qq.com> 写道：
Hello，dear kyliners:
Lately, We tried to create a streaming model in kylin ,which refer to doc "Cubing from Kafka(Near RT Streaming)" in kylin official website,
and we created the model and cube successfully, but when build the cube, it appeared an error,like this:
"
ERROR [http-nio-7070-exec-9] controller.CubeController:398 : org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;"
<40...@D00A1012.05679E5E.png.jpg>
our environment: kylin 2.6.3; kafka 2.0.0.
If you know where is the problem or you know how to solve it, please contact me, thank you very much!
Re: When build streaming cube in kylin, I got an error tip.
Posted by Yaqian Zhang <Ya...@126.com>.
Hi：
Is Kafka home set before kylin is started？If not, you can set KAFKA_HOME and restart kylin.
If you still encounter this problem after setting KAFKA_HOME, and your hadoop environment is hdp3.0, you can take a look at this issue https://issues.apache.org/jira/browse/KYLIN-3970 <https://issues.apache.org/jira/browse/KYLIN-3970>. I'm not sure if it will help you.
> 在 2020年4月21日，11:22，梅秋莹 <32...@qq.com> 写道：
>
> Hello，dear kyliners:
>
> Lately, We tried to create a streaming model in kylin ,which refer to doc "Cubing from Kafka(Near RT Streaming)" in kylin official website,
> and we created the model and cube successfully, but when build the cube, it appeared an error,like this:
>
> "
> ERROR [http-nio-7070-exec-9] controller.CubeController:398 : org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
> java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;"
>
> <40...@D00A1012.05679E5E.png.jpg>
>
> our environment: kylin 2.6.3; kafka 2.0.0.
>
> If you know where is the problem or you know how to solve it, please contact me, thank you very much!