You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by Xiaoxiang Yu <xi...@kyligence.io> on 2019/12/09 10:55:57 UTC
Re: 构建cube的第一步到了如下错误, 版本 apache-kylin-2.6.4-bin-cdh60.tar.gz
Hi friend,
I saw log contains "hadoop.fs.FileStatus.isErasureCoded", which indicated that you are using a Hadoop3 env, is it CDH6?
I guess the error maybe caused by some mistake in HDFS configuration.
I also saw other information which seems be helpful, did you set the correct value for dfs.encryption.key.provider.uri?
I think you should copy the SQL generated by Kylin and pasted them into Hive CLI, and change your configuration with Hive and HDFS to ensure sql can be executed successfully.
----------------
Best wishes,
Xiaoxiang Yu
发件人: 欧阳苗 <a9...@qq.com>
答复: "user@kylin.apache.org" <us...@kylin.apache.org>
日期: 2019年12月9日 星期一 18:15
收件人: user <us...@kylin.apache.org>
主题: 构建cube的第一步到了如下错误, 版本 apache-kylin-2.6.4-bin-cdh60.tar.gz
java.io.IOException: OS command error exit with return code: 155, error message: SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/modules/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hive-common-2.1.1-cdh6.2.0.jar!/hive-log4j2.properties Async: false
OK
Time taken: 2.513 seconds
OK
Time taken: 0.385 seconds
OK
Time taken: 0.529 seconds
OK
Time taken: 0.521 seconds
Query ID = root_20191208010521_cb1bca53-9910-4ab1-ac6e-a4de6dcdff39
Total jobs = 1
Execution completed successfully
MapredLocal task succeeded
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
19/12/08 01:05:42 INFO Configuration.deprecation: mapred.submit.replication is deprecated. Instead, use mapreduce.client.submit.file.replication
19/12/08 01:05:44 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm164
19/12/08 01:05:44 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
19/12/08 01:05:49 INFO input.FileInputFormat: Total input paths to process : 1
19/12/08 01:05:49 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 3, size left: 0
19/12/08 01:05:49 INFO mapreduce.JobSubmitter: number of splits:1
19/12/08 01:05:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1575269010295_0006
19/12/08 01:05:50 INFO impl.YarnClientImpl: Submitted application application_1575269010295_0006
19/12/08 01:05:50 INFO mapreduce.Job: The url to track the job: http://miao2.com:8088/proxy/application_1575269010295_0006/
Starting Job = job_1575269010295_0006, Tracking URL = http://miao2.com:8088/proxy/application_1575269010295_0006/
Kill Command = /opt/modules/hadoop-2.7.7/bin/hadoop job -kill job_1575269010295_0006
Hadoop job information for Stage-4: number of mappers: 1; number of reducers: 0
19/12/08 01:06:05 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2019-12-08 01:06:05,920 Stage-4 map = 0%, reduce = 0%
19/12/08 01:06:19 WARN counters.FileSystemCounterGroup: HDFS_BYTES_READ_EC is not a recognized counter.
19/12/08 01:06:19 WARN counters.FrameworkCounterGroup: MAP_PHYSICAL_MEMORY_BYTES_MAX is not a recognized counter.
19/12/08 01:06:19 WARN counters.FrameworkCounterGroup: MAP_VIRTUAL_MEMORY_BYTES_MAX is not a recognized counter.
2019-12-08 01:06:19,383 Stage-4 map = 100%, reduce = 0%, Cumulative CPU 2.52 sec
19/12/08 01:06:20 WARN counters.FileSystemCounterGroup: HDFS_BYTES_READ_EC is not a recognized counter.
19/12/08 01:06:20 WARN counters.FrameworkCounterGroup: MAP_PHYSICAL_MEMORY_BYTES_MAX is not a recognized counter.
19/12/08 01:06:20 WARN counters.FrameworkCounterGroup: MAP_VIRTUAL_MEMORY_BYTES_MAX is not a recognized counter.
19/12/08 01:06:20 WARN counters.FileSystemCounterGroup: HDFS_BYTES_READ_EC is not a recognized counter.
19/12/08 01:06:20 WARN counters.FrameworkCounterGroup: MAP_PHYSICAL_MEMORY_BYTES_MAX is not a recognized counter.
19/12/08 01:06:20 WARN counters.FrameworkCounterGroup: MAP_VIRTUAL_MEMORY_BYTES_MAX is not a recognized counter.
MapReduce Total cumulative CPU time: 2 seconds 520 msec
Ended Job = job_1575269010295_0006
Loading data to table default.kylin_intermediate_emp_cube_da04d657_54f0_d436_1e6f_9b5c1d48d135
19/12/08 01:06:20 ERROR hdfs.KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.StatsTask. org.apache.hadoop.fs.FileStatus.isErasureCoded()Z
MapReduce Jobs Launched:
Stage-Stage-4: Map: 1 Cumulative CPU: 2.52 sec HDFS Read: 8969 HDFS Write: 551 SUCCESS
Total MapReduce CPU Time Spent: 2 seconds 520 msec
The command is:
hive -e "USE default;
DROP TABLE IF EXISTS kylin_intermediate_emp_cube_da04d657_54f0_d436_1e6f_9b5c1d48d135;
CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_emp_cube_da04d657_54f0_d436_1e6f_9b5c1d48d135
(
\`EMP_JOB\` string
,\`EMP_HIREDATE\` string
,\`EMP_DEPTNO\` int
,\`DEPT_DNAME\` string
,\`DEPT_LOC\` int
,\`EMP_SAL\` double
)
STORED AS SEQUENCEFILE
LOCATION 'hdfs://nameservice1/kylin/kylin_metadata/kylin-99121fe3-9dce-cdf0-f684-d97cb998c629/kylin_intermediate_emp_cube_da04d657_54f0_d436_1e6f_9b5c1d48d135';
ALTER TABLE kylin_intermediate_emp_cube_da04d657_54f0_d436_1e6f_9b5c1d48d135 SET TBLPROPERTIES('auto.purge'='true');
INSERT OVERWRITE TABLE \`kylin_intermediate_emp_cube_da04d657_54f0_d436_1e6f_9b5c1d48d135\` SELECT
\`EMP\`.\`JOB\` as \`EMP_JOB\`
,\`EMP\`.\`HIREDATE\` as \`EMP_HIREDATE\`
,\`EMP\`.\`DEPTNO\` as \`EMP_DEPTNO\`
,\`DEPT\`.\`DNAME\` as \`DEPT_DNAME\`
,\`DEPT\`.\`LOC\` as \`DEPT_LOC\`
,\`EMP\`.\`SAL\` as \`EMP_SAL\`
FROM \`DEFAULT\`.\`EMP\` as \`EMP\`
INNER JOIN \`DEFAULT\`.\`DEPT\` as \`DEPT\`
ON \`EMP\`.\`DEPTNO\` = \`DEPT\`.\`DEPTNO\`
WHERE 1=1;
" --hiveconf hive.merge.mapredfiles=false --hiveconf hive.auto.convert.join=true --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true --hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false --hiveconf hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf hive.stats.autogather=true
at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:96)
at org.apache.kylin.source.hive.CreateFlatHiveTableStep.createFlatHiveTable(CreateFlatHiveTableStep.java:62)
at org.apache.kylin.source.hive.CreateFlatHiveTableStep.doWork(CreateFlatHiveTableStep.java:99)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
回复： 构建cube的第一步到了如下错误, 版本 apache-kylin-2.6.4-bin-cdh60.tar.gz
Posted by 欧阳苗 <a9...@qq.com>.
当我执行kylin的sample.sh表跟数据都插入了进去，但是在webUI上，看不到learn_kylin这个项目，而且报了如下错误
Loading data to table default.kylin_sales
19/12/09 19:55:52 ERROR hdfs.KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
chgrp: changing ownership of 'hdfs://nameservice1/user/hive/warehouse/kylin_sales/DEFAULT.KYLIN_SALES.csv': User root does not belong to hive
FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.StatsTask. org.apache.hadoop.fs.FileStatus.isErasureCoded()Z
我的版本是cdh6.2.0，我的hive是可以正常插入数据的，所以我想知道kylin的哪一个版本是支持cdh6.2.0的
------------------ 原始邮件 ------------------
发件人: "Xiaoxiang Yu"<xiaoxiang.yu@kyligence.io>;
发送时间: 2019年12月9日(星期一) 晚上6:55
收件人: "user@kylin.apache.org"<user@kylin.apache.org>;"欧阳苗"<a952064610@qq.com>;
主题: Re: 构建cube的第一步到了如下错误, 版本 apache-kylin-2.6.4-bin-cdh60.tar.gz
Hi friend,
I saw log contains "hadoop.fs.FileStatus.isErasureCoded", which indicated that you are using a Hadoop3 env, is it CDH6?
I guess the error maybe caused by some mistake in HDFS configuration.
I also saw other information which seems be helpful, did you set the correct value for dfs.encryption.key.provider.uri?
I think you should copy the SQL generated by Kylin and pasted them into Hive CLI, and change your configuration with Hive and HDFS to ensure sql can be executed successfully.
----------------
Best wishes,
Xiaoxiang Yu
发件人: 欧阳苗 <a952064610@qq.com>
答复: "user@kylin.apache.org" <user@kylin.apache.org>
日期: 2019年12月9日 星期一 18:15
收件人: user <user@kylin.apache.org>
主题: 构建cube的第一步到了如下错误, 版本 apache-kylin-2.6.4-bin-cdh60.tar.gz
java.io.IOException: OS command error exit with return code: 155, error message: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/modules/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hive-common-2.1.1-cdh6.2.0.jar!/hive-log4j2.properties Async: false OK Time taken: 2.513 seconds OK Time taken: 0.385 seconds OK Time taken: 0.529 seconds OK Time taken: 0.521 seconds Query ID = root_20191208010521_cb1bca53-9910-4ab1-ac6e-a4de6dcdff39 Total jobs = 1 Execution completed successfully MapredLocal task succeeded Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator 19/12/08 01:05:42 INFO Configuration.deprecation: mapred.submit.replication is deprecated. Instead, use mapreduce.client.submit.file.replication 19/12/08 01:05:44 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm164 19/12/08 01:05:44 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 19/12/08 01:05:49 INFO input.FileInputFormat: Total input paths to process : 1 19/12/08 01:05:49 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 3, size left: 0 19/12/08 01:05:49 INFO mapreduce.JobSubmitter: number of splits:1 19/12/08 01:05:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1575269010295_0006 19/12/08 01:05:50 INFO impl.YarnClientImpl: Submitted application application_1575269010295_0006 19/12/08 01:05:50 INFO mapreduce.Job: The url to track the job: http://miao2.com:8088/proxy/application_1575269010295_0006/ Starting Job = job_1575269010295_0006, Tracking URL = http://miao2.com:8088/proxy/application_1575269010295_0006/ Kill Command = /opt/modules/hadoop-2.7.7/bin/hadoop job -kill job_1575269010295_0006 Hadoop job information for Stage-4: number of mappers: 1; number of reducers: 0 19/12/08 01:06:05 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 2019-12-08 01:06:05,920 Stage-4 map = 0%, reduce = 0% 19/12/08 01:06:19 WARN counters.FileSystemCounterGroup: HDFS_BYTES_READ_EC is not a recognized counter. 19/12/08 01:06:19 WARN counters.FrameworkCounterGroup: MAP_PHYSICAL_MEMORY_BYTES_MAX is not a recognized counter. 19/12/08 01:06:19 WARN counters.FrameworkCounterGroup: MAP_VIRTUAL_MEMORY_BYTES_MAX is not a recognized counter. 2019-12-08 01:06:19,383 Stage-4 map = 100%, reduce = 0%, Cumulative CPU 2.52 sec 19/12/08 01:06:20 WARN counters.FileSystemCounterGroup: HDFS_BYTES_READ_EC is not a recognized counter. 19/12/08 01:06:20 WARN counters.FrameworkCounterGroup: MAP_PHYSICAL_MEMORY_BYTES_MAX is not a recognized counter. 19/12/08 01:06:20 WARN counters.FrameworkCounterGroup: MAP_VIRTUAL_MEMORY_BYTES_MAX is not a recognized counter. 19/12/08 01:06:20 WARN counters.FileSystemCounterGroup: HDFS_BYTES_READ_EC is not a recognized counter. 19/12/08 01:06:20 WARN counters.FrameworkCounterGroup: MAP_PHYSICAL_MEMORY_BYTES_MAX is not a recognized counter. 19/12/08 01:06:20 WARN counters.FrameworkCounterGroup: MAP_VIRTUAL_MEMORY_BYTES_MAX is not a recognized counter. MapReduce Total cumulative CPU time: 2 seconds 520 msec Ended Job = job_1575269010295_0006 Loading data to table default.kylin_intermediate_emp_cube_da04d657_54f0_d436_1e6f_9b5c1d48d135 19/12/08 01:06:20 ERROR hdfs.KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !! FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.StatsTask. org.apache.hadoop.fs.FileStatus.isErasureCoded()Z MapReduce Jobs Launched: Stage-Stage-4: Map: 1 Cumulative CPU: 2.52 sec HDFS Read: 8969 HDFS Write: 551 SUCCESS Total MapReduce CPU Time Spent: 2 seconds 520 msec The command is: hive -e "USE default; DROP TABLE IF EXISTS kylin_intermediate_emp_cube_da04d657_54f0_d436_1e6f_9b5c1d48d135; CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_emp_cube_da04d657_54f0_d436_1e6f_9b5c1d48d135 ( \`EMP_JOB\` string ,\`EMP_HIREDATE\` string ,\`EMP_DEPTNO\` int ,\`DEPT_DNAME\` string ,\`DEPT_LOC\` int ,\`EMP_SAL\` double ) STORED AS SEQUENCEFILE LOCATION 'hdfs://nameservice1/kylin/kylin_metadata/kylin-99121fe3-9dce-cdf0-f684-d97cb998c629/kylin_intermediate_emp_cube_da04d657_54f0_d436_1e6f_9b5c1d48d135'; ALTER TABLE kylin_intermediate_emp_cube_da04d657_54f0_d436_1e6f_9b5c1d48d135 SET TBLPROPERTIES('auto.purge'='true'); INSERT OVERWRITE TABLE \`kylin_intermediate_emp_cube_da04d657_54f0_d436_1e6f_9b5c1d48d135\` SELECT \`EMP\`.\`JOB\` as \`EMP_JOB\` ,\`EMP\`.\`HIREDATE\` as \`EMP_HIREDATE\` ,\`EMP\`.\`DEPTNO\` as \`EMP_DEPTNO\` ,\`DEPT\`.\`DNAME\` as \`DEPT_DNAME\` ,\`DEPT\`.\`LOC\` as \`DEPT_LOC\` ,\`EMP\`.\`SAL\` as \`EMP_SAL\` FROM \`DEFAULT\`.\`EMP\` as \`EMP\` INNER JOIN \`DEFAULT\`.\`DEPT\` as \`DEPT\` ON \`EMP\`.\`DEPTNO\` = \`DEPT\`.\`DEPTNO\` WHERE 1=1; " --hiveconf hive.merge.mapredfiles=false --hiveconf hive.auto.convert.join=true --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true --hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false --hiveconf hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf hive.stats.autogather=true at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:96) at org.apache.kylin.source.hive.CreateFlatHiveTableStep.createFlatHiveTable(CreateFlatHiveTableStep.java:62) at org.apache.kylin.source.hive.CreateFlatHiveTableStep.doWork(CreateFlatHiveTableStep.java:99) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)