You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by Gao Michael <mi...@hotmail.com> on 2022/05/19 11:41:20 UTC
回复: File does not exist ....... fairscheduler.xml
Hi all,
Who can Help?
发件人: Gao Michael <mi...@hotmail.com>
发送时间: 2022年5月18日 19:23
收件人: user@kylin.apache.org
主题: File does not exist ....... fairscheduler.xml
Hi all
I got follow error when I try to add table to my kylin project via web interface:
2022-05-18 18:48:37,904 INFO [Thread-7] ui.SparkUI:57 : Bound SparkUI to 0.0.0.0, and started at http://hadoop-cluster-001:4040
2022-05-18 18:48:38,681 ERROR [Thread-7] scheduler.FairSchedulableBuilder:94 : Error while building the fair scheduler pools
java.io.FileNotFoundException: File does not exist: /home/michael/opt/modules/kylin/apache-kylin-4.0.1-bin-spark3/conf/fairscheduler.xml
and I’m sure file exist!
michael@hadoop-cluster-001:~/opt/modules/kylin/apache-kylin-4.0.1-bin-spark3$<mailto:michael@hadoop-cluster-001:~/opt/modules/kylin/apache-kylin-4.0.1-bin-spark3$> ls conf/
fairscheduler.xml kylin_job_conf_inmem.xml kylin.properties setenv.sh spark-executor-log4j.properties
kylin_hive_conf.xml kylin_job_conf.xml kylin-server-log4j.properties setenv-tool.sh
kylin_job_conf_cube_merge.xml kylin-kafka-consumer.xml kylin-tools-log4j.properties spark-driver-log4j.properties
What’s the problem and how to solve? Thanks!
Re:回复: File does not exist ....... fairscheduler.xml
Posted by Mukvin <bo...@163.com>.
Hi Gao Michael,
Spark 3.2.1 has differences from Spark 3.1.1. If you want to run on Spark 3.2.1 maybe after Kylin adapted it.
You can run Spark 3.2.1 and Spark 3.1.1 together, but two sparks will separate the resource of the cluster.
And current the package of "kylin-4.0.1-bin-spark3.tar.gz" can only run on Spark 3.1.1.
--
Best regards.
Tengting Xu
在 2022-05-20 23:24:13,"Gao Michael" <mi...@hotmail.com> 写道:
Hi Tengting Xu
Thanks for your help, it works!
I download spark3.2.1 and modify SPARK_HOME=`path in .bashrc to spark 3.1.1`, now I can import hive tables via kylin web interface.
So is there any way to keep spark3.2.1 run on cluster and just keep kylin run on spark3.1.1?
Thanks!
发件人: Mukvin <bo...@163.com>
发送时间: 2022年5月20日 9:22
收件人: michael.gaocl@hotmail.com; user@kylin.apache.org
主题: Re:回复: Re:回复: Re:回复: File does not exist ....... fairscheduler.xml
Hi Gao Michael,
I mean you can export SPARK_HOME=`path to spark 3.1.1` for KYLIN, do not need to downgrade 3.2.1.
You can keep them both.
--
Best regards.
Tengting Xu
At 2022-05-19 22:47:14, "Gao Michael" <mi...@hotmail.com> wrote:
Hi Tengting Xu:
I already deployed spark3.2.1 on my cluster, you mean I should degrade it to 3.1.1?
发件人: Mukvin <bo...@163.com>
发送时间: 2022年5月19日 22:36
收件人:michael.gaocl@hotmail.com; user@kylin.apache.org
主题: Re:回复: Re:回复: File does not exist ....... fairscheduler.xml
Hi Gao Michael,
As I checked the log file, I found two errors in it.
"controller.TableController:131 : Failed to load Hive Table java.lang.StackOverflowError" and "File not found exception".
They are related to spark.
As you have mentioned your env is with spark 3.2.1, but currently, kylin-4.0.1-bin-spark3.tar.gz only supports spark 3.1.1 as Kylin's official site mentioned, so you may need to download spark 3.1.1 and export the path to be `spark home`. Then you can try to restart Kylin again and check.
--
Best regards.
Tengting Xu
At 2022-05-19 22:15:15, "Gao Michael" <mi...@hotmail.com> wrote:
Hi Mukin
The log file too long and I only can only send pats of it as an attachment.
I import table via kylin web interface, and watch log file by command “tail -f”,
The log file watch terminal was rolling continuous in the process of import.
Attachment cover a complete loading process
Thanks
发件人: Mukvin <bo...@163.com>
发送时间: 2022年5月19日 21:28
收件人:michael.gaocl@hotmail.com; user@kylin.apache.org
主题: Re:回复: File does not exist ....... fairscheduler.xml
Hi Gao Michael,
Could you give a full log about kylin.log located in $KYLIN_HOME/logs/kylin.log.
Let me get the full trace of this error.
--
Best regards.
Tengting Xu
At 2022-05-19 19:41:20, "Gao Michael" <mi...@hotmail.com> wrote:
Hi all,
Who can Help?
发件人: Gao Michael <mi...@hotmail.com>
发送时间: 2022年5月18日 19:23
收件人:user@kylin.apache.org
主题: File does not exist ....... fairscheduler.xml
Hi all
I got follow error when I try to add table to my kylin project via web interface:
2022-05-1818:48:37,904INFO [Thread-7] ui.SparkUI:57 : Bound SparkUI to 0.0.0.0, and started at http://hadoop-cluster-001:4040
2022-05-1818:48:38,681ERROR [Thread-7] scheduler.FairSchedulableBuilder:94 : Error while building the fair scheduler pools
java.io.FileNotFoundException: File does not exist: /home/michael/opt/modules/kylin/apache-kylin-4.0.1-bin-spark3/conf/fairscheduler.xml
and I’m sure file exist!
michael@hadoop-cluster-001:~/opt/modules/kylin/apache-kylin-4.0.1-bin-spark3$ ls conf/
fairscheduler.xml kylin_job_conf_inmem.xml kylin.properties setenv.sh spark-executor-log4j.properties
kylin_hive_conf.xml kylin_job_conf.xml kylin-server-log4j.properties setenv-tool.sh
kylin_job_conf_cube_merge.xml kylin-kafka-consumer.xml kylin-tools-log4j.properties spark-driver-log4j.properties
What’s the problem and how to solve? Thanks!
回复: Re:回复: Re:回复: Re:回复: File does not exist ....... fairscheduler.xml
Posted by Gao Michael <mi...@hotmail.com>.
Hi Tengting Xu
Thanks for your help, it works!
I download spark3.2.1 and modify SPARK_HOME=`path in .bashrc to spark 3.1.1`, now I can import hive tables via kylin web interface.
So is there any way to keep spark3.2.1 run on cluster and just keep kylin run on spark3.1.1?
Thanks!
发件人: Mukvin <bo...@163.com>
发送时间: 2022年5月20日 9:22
收件人: michael.gaocl@hotmail.com; user@kylin.apache.org
主题: Re:回复: Re:回复: Re:回复: File does not exist ....... fairscheduler.xml
Hi Gao Michael,
I mean you can export SPARK_HOME=`path to spark 3.1.1` for KYLIN, do not need to downgrade 3.2.1.
You can keep them both.
--
Best regards.
Tengting Xu
At 2022-05-19 22:47:14, "Gao Michael" <mi...@hotmail.com>> wrote:
Hi Tengting Xu:
I already deployed spark3.2.1 on my cluster, you mean I should degrade it to 3.1.1?
发件人: Mukvin <bo...@163.com>>
发送时间: 2022年5月19日 22:36
收件人: michael.gaocl@hotmail.com<ma...@hotmail.com>; user@kylin.apache.org<ma...@kylin.apache.org>
主题: Re:回复: Re:回复: File does not exist ....... fairscheduler.xml
Hi Gao Michael,
As I checked the log file, I found two errors in it.
"controller.TableController:131 : Failed to load Hive Table java.lang.StackOverflowError" and "File not found exception".
They are related to spark.
As you have mentioned your env is with spark 3.2.1, but currently, kylin-4.0.1-bin-spark3.tar.gz only supports spark 3.1.1 as Kylin's official site mentioned, so you may need to download spark 3.1.1 and export the path to be `spark home`. Then you can try to restart Kylin again and check.
--
Best regards.
Tengting Xu
At 2022-05-19 22:15:15, "Gao Michael" <mi...@hotmail.com>> wrote:
Hi Mukin
The log file too long and I only can only send pats of it as an attachment.
I import table via kylin web interface, and watch log file by command “tail -f”,
The log file watch terminal was rolling continuous in the process of import.
Attachment cover a complete loading process
Thanks
发件人: Mukvin <bo...@163.com>>
发送时间: 2022年5月19日 21:28
收件人: michael.gaocl@hotmail.com<ma...@hotmail.com>; user@kylin.apache.org<ma...@kylin.apache.org>
主题: Re:回复: File does not exist ....... fairscheduler.xml
Hi Gao Michael,
Could you give a full log about kylin.log located in $KYLIN_HOME/logs/kylin.log.
Let me get the full trace of this error.
--
Best regards.
Tengting Xu
At 2022-05-19 19:41:20, "Gao Michael" <mi...@hotmail.com>> wrote:
Hi all,
Who can Help?
发件人: Gao Michael <mi...@hotmail.com>>
发送时间: 2022年5月18日 19:23
收件人: user@kylin.apache.org<ma...@kylin.apache.org>
主题: File does not exist ....... fairscheduler.xml
Hi all
I got follow error when I try to add table to my kylin project via web interface:
2022-05-18 18:48:37,904 INFO [Thread-7] ui.SparkUI:57 : Bound SparkUI to 0.0.0.0, and started at http://hadoop-cluster-001:4040
2022-05-18 18:48:38,681 ERROR [Thread-7] scheduler.FairSchedulableBuilder:94 : Error while building the fair scheduler pools
java.io.FileNotFoundException: File does not exist: /home/michael/opt/modules/kylin/apache-kylin-4.0.1-bin-spark3/conf/fairscheduler.xml
and I’m sure file exist!
michael@hadoop-cluster-001:~/opt/modules/kylin/apache-kylin-4.0.1-bin-spark3$<mailto:michael@hadoop-cluster-001:~/opt/modules/kylin/apache-kylin-4.0.1-bin-spark3$> ls conf/
fairscheduler.xml kylin_job_conf_inmem.xml kylin.properties setenv.sh spark-executor-log4j.properties
kylin_hive_conf.xml kylin_job_conf.xml kylin-server-log4j.properties setenv-tool.sh
kylin_job_conf_cube_merge.xml kylin-kafka-consumer.xml kylin-tools-log4j.properties spark-driver-log4j.properties
What’s the problem and how to solve? Thanks!
Re:回复: Re:回复: Re:回复: File does not exist ....... fairscheduler.xml
Posted by Mukvin <bo...@163.com>.
Hi Gao Michael,
I mean you can export SPARK_HOME=`path to spark 3.1.1` for KYLIN, do not need to downgrade 3.2.1.
You can keep them both.
--
Best regards.
Tengting Xu
At 2022-05-19 22:47:14, "Gao Michael" <mi...@hotmail.com> wrote:
Hi Tengting Xu:
I already deployed spark3.2.1 on my cluster, you mean I should degrade it to 3.1.1?
发件人: Mukvin <bo...@163.com>
发送时间: 2022年5月19日 22:36
收件人: michael.gaocl@hotmail.com; user@kylin.apache.org
主题: Re:回复: Re:回复: File does not exist ....... fairscheduler.xml
Hi Gao Michael,
As I checked the log file, I found two errors in it.
"controller.TableController:131 : Failed to load Hive Table java.lang.StackOverflowError" and "File not found exception".
They are related to spark.
As you have mentioned your env is with spark 3.2.1, but currently, kylin-4.0.1-bin-spark3.tar.gz only supports spark 3.1.1 as Kylin's official site mentioned, so you may need to download spark 3.1.1 and export the path to be `spark home`. Then you can try to restart Kylin again and check.
--
Best regards.
Tengting Xu
At 2022-05-19 22:15:15, "Gao Michael" <mi...@hotmail.com> wrote:
Hi Mukin
The log file too long and I only can only send pats of it as an attachment.
I import table via kylin web interface, and watch log file by command “tail -f”,
The log file watch terminal was rolling continuous in the process of import.
Attachment cover a complete loading process
Thanks
发件人: Mukvin <bo...@163.com>
发送时间: 2022年5月19日 21:28
收件人: michael.gaocl@hotmail.com; user@kylin.apache.org
主题: Re:回复: File does not exist ....... fairscheduler.xml
Hi Gao Michael,
Could you give a full log about kylin.log located in $KYLIN_HOME/logs/kylin.log.
Let me get the full trace of this error.
--
Best regards.
Tengting Xu
At 2022-05-19 19:41:20, "Gao Michael" <mi...@hotmail.com> wrote:
Hi all,
Who can Help?
发件人: Gao Michael <mi...@hotmail.com>
发送时间: 2022年5月18日 19:23
收件人:user@kylin.apache.org
主题: File does not exist ....... fairscheduler.xml
Hi all
I got follow error when I try to add table to my kylin project via web interface:
2022-05-1818:48:37,904INFO [Thread-7] ui.SparkUI:57 : Bound SparkUI to 0.0.0.0, and started at http://hadoop-cluster-001:4040
2022-05-1818:48:38,681ERROR [Thread-7] scheduler.FairSchedulableBuilder:94 : Error while building the fair scheduler pools
java.io.FileNotFoundException: File does not exist: /home/michael/opt/modules/kylin/apache-kylin-4.0.1-bin-spark3/conf/fairscheduler.xml
and I’m sure file exist!
michael@hadoop-cluster-001:~/opt/modules/kylin/apache-kylin-4.0.1-bin-spark3$ ls conf/
fairscheduler.xml kylin_job_conf_inmem.xml kylin.properties setenv.sh spark-executor-log4j.properties
kylin_hive_conf.xml kylin_job_conf.xml kylin-server-log4j.properties setenv-tool.sh
kylin_job_conf_cube_merge.xml kylin-kafka-consumer.xml kylin-tools-log4j.properties spark-driver-log4j.properties
What’s the problem and how to solve? Thanks!
回复: Re:回复: Re:回复: File does not exist ....... fairscheduler.xml
Posted by Gao Michael <mi...@hotmail.com>.
Hi Tengting Xu:
I already deployed spark3.2.1 on my cluster, you mean I should degrade it to 3.1.1?
发件人: Mukvin <bo...@163.com>
发送时间: 2022年5月19日 22:36
收件人: michael.gaocl@hotmail.com; user@kylin.apache.org
主题: Re:回复: Re:回复: File does not exist ....... fairscheduler.xml
Hi Gao Michael,
As I checked the log file, I found two errors in it.
"controller.TableController:131 : Failed to load Hive Table java.lang.StackOverflowError" and "File not found exception".
They are related to spark.
As you have mentioned your env is with spark 3.2.1, but currently, kylin-4.0.1-bin-spark3.tar.gz only supports spark 3.1.1 as Kylin's official site mentioned, so you may need to download spark 3.1.1 and export the path to be `spark home`. Then you can try to restart Kylin again and check.
--
Best regards.
Tengting Xu
At 2022-05-19 22:15:15, "Gao Michael" <mi...@hotmail.com>> wrote:
Hi Mukin
The log file too long and I only can only send pats of it as an attachment.
I import table via kylin web interface, and watch log file by command “tail -f”,
The log file watch terminal was rolling continuous in the process of import.
Attachment cover a complete loading process
Thanks
发件人: Mukvin <bo...@163.com>>
发送时间: 2022年5月19日 21:28
收件人: michael.gaocl@hotmail.com<ma...@hotmail.com>; user@kylin.apache.org<ma...@kylin.apache.org>
主题: Re:回复: File does not exist ....... fairscheduler.xml
Hi Gao Michael,
Could you give a full log about kylin.log located in $KYLIN_HOME/logs/kylin.log.
Let me get the full trace of this error.
--
Best regards.
Tengting Xu
At 2022-05-19 19:41:20, "Gao Michael" <mi...@hotmail.com>> wrote:
Hi all,
Who can Help?
发件人: Gao Michael <mi...@hotmail.com>>
发送时间: 2022年5月18日 19:23
收件人: user@kylin.apache.org<ma...@kylin.apache.org>
主题: File does not exist ....... fairscheduler.xml
Hi all
I got follow error when I try to add table to my kylin project via web interface:
2022-05-18 18:48:37,904 INFO [Thread-7] ui.SparkUI:57 : Bound SparkUI to 0.0.0.0, and started at http://hadoop-cluster-001:4040
2022-05-18 18:48:38,681 ERROR [Thread-7] scheduler.FairSchedulableBuilder:94 : Error while building the fair scheduler pools
java.io.FileNotFoundException: File does not exist: /home/michael/opt/modules/kylin/apache-kylin-4.0.1-bin-spark3/conf/fairscheduler.xml
and I’m sure file exist!
michael@hadoop-cluster-001:~/opt/modules/kylin/apache-kylin-4.0.1-bin-spark3$<mailto:michael@hadoop-cluster-001:~/opt/modules/kylin/apache-kylin-4.0.1-bin-spark3$> ls conf/
fairscheduler.xml kylin_job_conf_inmem.xml kylin.properties setenv.sh spark-executor-log4j.properties
kylin_hive_conf.xml kylin_job_conf.xml kylin-server-log4j.properties setenv-tool.sh
kylin_job_conf_cube_merge.xml kylin-kafka-consumer.xml kylin-tools-log4j.properties spark-driver-log4j.properties
What’s the problem and how to solve? Thanks!
Re:回复: Re:回复: File does not exist ....... fairscheduler.xml
Posted by Mukvin <bo...@163.com>.
Hi Gao Michael,
As I checked the log file, I found two errors in it.
"controller.TableController:131 : Failed to load Hive Table java.lang.StackOverflowError" and "File not found exception".
They are related to spark.
As you have mentioned your env is with spark 3.2.1, but currently, kylin-4.0.1-bin-spark3.tar.gz only supports spark 3.1.1 as Kylin's official site mentioned, so you may need to download spark 3.1.1 and export the path to be `spark home`. Then you can try to restart Kylin again and check.
--
Best regards.
Tengting Xu
At 2022-05-19 22:15:15, "Gao Michael" <mi...@hotmail.com> wrote:
Hi Mukin
The log file too long and I only can only send pats of it as an attachment.
I import table via kylin web interface, and watch log file by command “tail -f”,
The log file watch terminal was rolling continuous in the process of import.
Attachment cover a complete loading process
Thanks
发件人: Mukvin <bo...@163.com>
发送时间: 2022年5月19日 21:28
收件人: michael.gaocl@hotmail.com; user@kylin.apache.org
主题: Re:回复: File does not exist ....... fairscheduler.xml
Hi Gao Michael,
Could you give a full log about kylin.log located in $KYLIN_HOME/logs/kylin.log.
Let me get the full trace of this error.
--
Best regards.
Tengting Xu
At 2022-05-19 19:41:20, "Gao Michael" <mi...@hotmail.com> wrote:
Hi all,
Who can Help?
发件人: Gao Michael <mi...@hotmail.com>
发送时间: 2022年5月18日 19:23
收件人:user@kylin.apache.org
主题: File does not exist ....... fairscheduler.xml
Hi all
I got follow error when I try to add table to my kylin project via web interface:
2022-05-1818:48:37,904INFO [Thread-7] ui.SparkUI:57 : Bound SparkUI to 0.0.0.0, and started at http://hadoop-cluster-001:4040
2022-05-1818:48:38,681ERROR [Thread-7] scheduler.FairSchedulableBuilder:94 : Error while building the fair scheduler pools
java.io.FileNotFoundException: File does not exist: /home/michael/opt/modules/kylin/apache-kylin-4.0.1-bin-spark3/conf/fairscheduler.xml
and I’m sure file exist!
michael@hadoop-cluster-001:~/opt/modules/kylin/apache-kylin-4.0.1-bin-spark3$ ls conf/
fairscheduler.xml kylin_job_conf_inmem.xml kylin.properties setenv.sh spark-executor-log4j.properties
kylin_hive_conf.xml kylin_job_conf.xml kylin-server-log4j.properties setenv-tool.sh
kylin_job_conf_cube_merge.xml kylin-kafka-consumer.xml kylin-tools-log4j.properties spark-driver-log4j.properties
What’s the problem and how to solve? Thanks!
回复: Re:回复: File does not exist ....... fairscheduler.xml
Posted by Gao Michael <mi...@hotmail.com>.
Hi Mukin
The log file too long and I only can only send pats of it as an attachment.
I import table via kylin web interface, and watch log file by command “tail -f”,
The log file watch terminal was rolling continuous in the process of import.
Attachment cover a complete loading process
Thanks
发件人: Mukvin <bo...@163.com>
发送时间: 2022年5月19日 21:28
收件人: michael.gaocl@hotmail.com; user@kylin.apache.org
主题: Re:回复: File does not exist ....... fairscheduler.xml
Hi Gao Michael,
Could you give a full log about kylin.log located in $KYLIN_HOME/logs/kylin.log.
Let me get the full trace of this error.
--
Best regards.
Tengting Xu
At 2022-05-19 19:41:20, "Gao Michael" <mi...@hotmail.com>> wrote:
Hi all,
Who can Help?
发件人: Gao Michael <mi...@hotmail.com>>
发送时间: 2022年5月18日 19:23
收件人: user@kylin.apache.org<ma...@kylin.apache.org>
主题: File does not exist ....... fairscheduler.xml
Hi all
I got follow error when I try to add table to my kylin project via web interface:
2022-05-18 18:48:37,904 INFO [Thread-7] ui.SparkUI:57 : Bound SparkUI to 0.0.0.0, and started at http://hadoop-cluster-001:4040
2022-05-18 18:48:38,681 ERROR [Thread-7] scheduler.FairSchedulableBuilder:94 : Error while building the fair scheduler pools
java.io.FileNotFoundException: File does not exist: /home/michael/opt/modules/kylin/apache-kylin-4.0.1-bin-spark3/conf/fairscheduler.xml
and I’m sure file exist!
michael@hadoop-cluster-001:~/opt/modules/kylin/apache-kylin-4.0.1-bin-spark3$<mailto:michael@hadoop-cluster-001:~/opt/modules/kylin/apache-kylin-4.0.1-bin-spark3$> ls conf/
fairscheduler.xml kylin_job_conf_inmem.xml kylin.properties setenv.sh spark-executor-log4j.properties
kylin_hive_conf.xml kylin_job_conf.xml kylin-server-log4j.properties setenv-tool.sh
kylin_job_conf_cube_merge.xml kylin-kafka-consumer.xml kylin-tools-log4j.properties spark-driver-log4j.properties
What’s the problem and how to solve? Thanks!
Re:回复: File does not exist ....... fairscheduler.xml
Posted by Mukvin <bo...@163.com>.
Hi Gao Michael,
Could you give a full log about kylin.log located in $KYLIN_HOME/logs/kylin.log.
Let me get the full trace of this error.
--
Best regards.
Tengting Xu
At 2022-05-19 19:41:20, "Gao Michael" <mi...@hotmail.com> wrote:
Hi all,
Who can Help?
发件人: Gao Michael <mi...@hotmail.com>
发送时间: 2022年5月18日 19:23
收件人: user@kylin.apache.org
主题: File does not exist ....... fairscheduler.xml
Hi all
I got follow error when I try to add table to my kylin project via web interface:
2022-05-1818:48:37,904INFO [Thread-7] ui.SparkUI:57 : Bound SparkUI to 0.0.0.0, and started at http://hadoop-cluster-001:4040
2022-05-1818:48:38,681ERROR [Thread-7] scheduler.FairSchedulableBuilder:94 : Error while building the fair scheduler pools
java.io.FileNotFoundException: File does not exist: /home/michael/opt/modules/kylin/apache-kylin-4.0.1-bin-spark3/conf/fairscheduler.xml
and I’m sure file exist!
michael@hadoop-cluster-001:~/opt/modules/kylin/apache-kylin-4.0.1-bin-spark3$ ls conf/
fairscheduler.xml kylin_job_conf_inmem.xml kylin.properties setenv.sh spark-executor-log4j.properties
kylin_hive_conf.xml kylin_job_conf.xml kylin-server-log4j.properties setenv-tool.sh
kylin_job_conf_cube_merge.xml kylin-kafka-consumer.xml kylin-tools-log4j.properties spark-driver-log4j.properties
What’s the problem and how to solve? Thanks!