You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by 梅秋莹 <32...@qq.com> on 2020/07/08 03:16:49 UTC
执行cube在第4步报错，提示“打开的文件过多” 是怎么回事？
------------------ 原始邮件 ------------------
发件人: "梅秋莹"<3281438086@qq.com>;
发送时间: 2020年7月3日(星期五) 下午4:55
收件人: "user"<user@kylin.apache.org>;
主题: When build cube in kylin, it failed at the 4th step "Build Dimension Dictionary"
Hello everyone:
Recently, we have encountered a new problem. That is, when build cube in kylin, it failed at the 4th step "Build Dimension Dictionary".
and the error message is like below. Before tha， we have never met such quetion, and our model and cube have functioned properly for at least one month. After reading some articles and blogs, someone think the errors could be related to the cube which has been set some measures like, "TOP-N" and "COUNT_DISTINCT". Indeed，I have set the measures like "TOP-N" and "COUNT_DISTINCT" when create cube. What should we do to solve the proble? Please tell me if you know the reason. Thank you very much.
our kylin is 2.6.3.
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.FileNotFoundException: /etc/hadoop/3.1.0.0-78/0/core-site.xml (打开的文件过多) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3000) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2926) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2806) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1200) at org.apache.kylin.common.util.HadoopUtil.healSickConfig(HadoopUtil.java:69) at org.apache.kylin.common.util.HadoopUtil.getCurrentConfiguration(HadoopUtil.java:59) at org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:103) at org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:95) at org.apache.kylin.common.util.HadoopUtil.getWorkingFileSystem(HadoopUtil.java:82) at org.apache.kylin.engine.mr.steps.CreateDictionaryJob$2.getDictionary(CreateDictionaryJob.java:92) at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:95) at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:69) at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73) at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93) at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.FileNotFoundException: /etc/hadoop/3.1.0.0-78/0/core-site.xml (打开的文件过多) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at java.io.FileInputStream.<init>(FileInputStream.java:93) at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90) at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188) at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2901) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2953) ... 21 more result code:2 at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:73) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
回复： 执行cube在第4步报错，提示“打开的文件过多” 是怎么回事？
Posted by 梅秋莹 <32...@qq.com>.
We have tried to restart kylin and rebuild the job many times，but it also returned the same errors. If you konw how to solve it ,please contact me . . Thank you very much!
------------------ 原始邮件 ------------------
发件人: "Yaqian Zhang"<Yaqian_Zhang@126.com>;
发送时间: 2020年7月9日(星期四) 上午10:39
收件人: "user"<user@kylin.apache.org>;
主题: Re: 执行cube在第4步报错，提示“打开的文件过多” 是怎么回事？
Hi:
If it's useful to restart kylin or rebuild the job?
在 2020年7月8日，11:16，梅秋莹 <3281438086@qq.com> 写道：
------------------ 原始邮件 ------------------
发件人: "梅秋莹"<3281438086@qq.com>;
发送时间: 2020年7月3日(星期五) 下午4:55
收件人: "user"<user@kylin.apache.org>;
主题: When build cube in kylin, it failed at the 4th step "Build Dimension Dictionary"
Hello everyone:
Recently, we have encountered a new problem. That is, when build cube in kylin, it failed at the 4th step "Build Dimension Dictionary".
<EB7E8108@66E50C04.A13A055F.jpg>
and the error message is like below. Before tha， we have never met such quetion, and our model and cube have functioned properly for at least one month. After reading some articles and blogs, someone think the errors could be related to the cube which has been set some measures like, "TOP-N" and "COUNT_DISTINCT". Indeed，I have set the measures like "TOP-N" and "COUNT_DISTINCT" when create cube. What should we do to solve the proble? Please tell me if you know the reason. Thank you very much.
our kylin is 2.6.3.
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.FileNotFoundException: /etc/hadoop/3.1.0.0-78/0/core-site.xml (打开的文件过多) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3000) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2926) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2806) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1200) at org.apache.kylin.common.util.HadoopUtil.healSickConfig(HadoopUtil.java:69) at org.apache.kylin.common.util.HadoopUtil.getCurrentConfiguration(HadoopUtil.java:59) at org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:103) at org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:95) at org.apache.kylin.common.util.HadoopUtil.getWorkingFileSystem(HadoopUtil.java:82) at org.apache.kylin.engine.mr.steps.CreateDictionaryJob$2.getDictionary(CreateDictionaryJob.java:92) at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:95) at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:69) at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73) at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93) at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.FileNotFoundException: /etc/hadoop/3.1.0.0-78/0/core-site.xml (打开的文件过多) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at java.io.FileInputStream.<init>(FileInputStream.java:93) at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90) at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188) at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2901) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2953) ... 21 more result code:2 at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:73) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)<E4D43A7C@7AE6845F.A13A055F.jpg>
<E4D43A7C@7AE6845F.A13A055F.jpg><EB7E8108@66E50C04.A13A055F.jpg><kylin(1).log>
Re: Re: 执行cube在第4步报错，提示“打开的文件过多” 是怎么回事？
Posted by ShaoFeng Shi <sh...@apache.org>.
https://askubuntu.com/questions/181215/too-many-open-files-how-to-find-the-culprit
You can do some investigation to see which process open so many files,
please refer to above link. I did encounter such a problem several years
ago but didn't see it for some time.
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org
恩爸 <44...@qq.com> 于2020年7月9日周四 下午10:31写道：
> Hi, you can use command 'ulimit -a' to see the limitation of opening
> files, and google how to modify this limitation.
>
> ------------------------------
>
>
> Best regards,
> Zhichao Zhang
>
>
>
>
> ------------------ 原始邮件 ------------------
> *发件人:* "user" <32...@qq.com>;
> *发送时间:* 2020年7月9日(星期四) 下午5:43
> *收件人:* "user"<us...@kylin.apache.org>;
> *抄送:* "crowgns"<cr...@163.com>;
> *主题:* 回复：Re: 执行cube在第4步报错，提示“打开的文件过多” 是怎么回事？
>
> 不好意思，对于kylin我了解还不是很深刻，也不知道句柄是啥意思。请问要改哪个的句柄数？具体操作是怎么样的呢？
>
>
> ------------------ 原始邮件 ------------------
> *发件人:* "初晓"<cr...@163.com>;
> *发送时间:* 2020年7月9日(星期四) 下午3:53
> *收件人:* "user"<us...@kylin.apache.org>;
> *主题:* Re:Re: 执行cube在第4步报错，提示“打开的文件过多” 是怎么回事？
>
> 句柄用光了吧，改一下句柄数
>
>
>
>
>
>
> At 2020-07-09 10:39:53, "Yaqian Zhang" <Ya...@126.com> wrote:
>
> Hi:
>
> If it's useful to restart kylin or rebuild the job?
>
> 在 2020年7月8日，11:16，梅秋莹 <32...@qq.com> 写道：
>
>
>
>
> ------------------ 原始邮件 ------------------
> *发件人:* "梅秋莹"<32...@qq.com>;
> *发送时间:* 2020年7月3日(星期五) 下午4:55
> *收件人:* "user"<us...@kylin.apache.org>;
> *主题:* When build cube in kylin, it failed at the 4th step "Build
> Dimension Dictionary"
>
> Hello everyone:
>
> Recently, we have encountered a new problem. That is, when build
> cube in kylin, it failed at the 4th step "Build Dimension Dictionary".
>
> <EB...@66E50C04.A13A055F.jpg>
> and the error message is like below. Before tha， we have never met such
> quetion, and our model and cube have functioned properly for at least one
> month. After reading some articles and blogs, someone think the errors
> could be related to the cube which has been set some measures like, "TOP-N"
> and "COUNT_DISTINCT". Indeed，I have set the measures like "TOP-N" and
> "COUNT_DISTINCT" when create cube. What should we do to solve the proble?
> Please tell me if you know the reason. Thank you very much.
>
> our kylin is 2.6.3.
>
> org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.FileNotFoundException: /etc/hadoop/3.1.0.0-78/0/core-site.xml (打开的文件过多)
> at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3000)
> at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2926)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2806)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1200)
> at org.apache.kylin.common.util.HadoopUtil.healSickConfig(HadoopUtil.java:69)
> at org.apache.kylin.common.util.HadoopUtil.getCurrentConfiguration(HadoopUtil.java:59)
> at org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:103)
> at org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:95)
> at org.apache.kylin.common.util.HadoopUtil.getWorkingFileSystem(HadoopUtil.java:82)
> at org.apache.kylin.engine.mr.steps.CreateDictionaryJob$2.getDictionary(CreateDictionaryJob.java:92)
> at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:95)
> at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:69)
> at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
> at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.FileNotFoundException: /etc/hadoop/3.1.0.0-78/0/core-site.xml (打开的文件过多)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.<init>(FileInputStream.java:138)
> at java.io.FileInputStream.<init>(FileInputStream.java:93)
> at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
> at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2901)
> at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2953)
> ... 21 more
>
> result code:2
> at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:73)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
> <E4...@7AE6845F.A13A055F.jpg>
>
> <E4...@66E50C04.A13A055F.jpg>
> <kylin(1).log>
>
>
>
>
>
>
回复：Re: 执行cube在第4步报错，提示“打开的文件过多” 是怎么回事？
Posted by 恩爸 <44...@qq.com>.
Hi, you can use command 'ulimit -a' to see the limitation of opening files, and google how to modify this limitation.
Best regards,
Zhichao Zhang
------------------ 原始邮件 ------------------
发件人: "user" <3281438086@qq.com>;
发送时间: 2020年7月9日(星期四) 下午5:43
收件人: "user"<user@kylin.apache.org>;
抄送: "crowgns"<crowgns@163.com>;
主题: 回复：Re: 执行cube在第4步报错，提示“打开的文件过多” 是怎么回事？
不好意思，对于kylin我了解还不是很深刻，也不知道句柄是啥意思。请问要改哪个的句柄数？具体操作是怎么样的呢？
------------------ 原始邮件 ------------------
发件人: "初晓"<crowgns@163.com>;
发送时间: 2020年7月9日(星期四) 下午3:53
收件人: "user"<user@kylin.apache.org>;
主题: Re:Re: 执行cube在第4步报错，提示“打开的文件过多” 是怎么回事？
句柄用光了吧，改一下句柄数
At 2020-07-09 10:39:53, "Yaqian Zhang" <Yaqian_Zhang@126.com> wrote:
Hi:
If it's useful to restart kylin or rebuild the job?
在 2020年7月8日，11:16，梅秋莹 <3281438086@qq.com> 写道：
------------------ 原始邮件 ------------------
发件人: "梅秋莹"<3281438086@qq.com>;
发送时间: 2020年7月3日(星期五) 下午4:55
收件人: "user"<user@kylin.apache.org>;
主题: When build cube in kylin, it failed at the 4th step "Build Dimension Dictionary"
Hello everyone:
Recently, we have encountered a new problem. That is, when build cube in kylin, it failed at the 4th step "Build Dimension Dictionary".
<EB7E8108@66E50C04.A13A055F.jpg>
and the error message is like below. Before tha， we have never met such quetion, and our model and cube have functioned properly for at least one month. After reading some articles and blogs, someone think the errors could be related to the cube which has been set some measures like, "TOP-N" and "COUNT_DISTINCT". Indeed，I have set the measures like "TOP-N" and "COUNT_DISTINCT" when create cube. What should we do to solve the proble? Please tell me if you know the reason. Thank you very much.
our kylin is 2.6.3.
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.FileNotFoundException: /etc/hadoop/3.1.0.0-78/0/core-site.xml (打开的文件过多) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3000) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2926) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2806) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1200) at org.apache.kylin.common.util.HadoopUtil.healSickConfig(HadoopUtil.java:69) at org.apache.kylin.common.util.HadoopUtil.getCurrentConfiguration(HadoopUtil.java:59) at org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:103) at org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:95) at org.apache.kylin.common.util.HadoopUtil.getWorkingFileSystem(HadoopUtil.java:82) at org.apache.kylin.engine.mr.steps.CreateDictionaryJob$2.getDictionary(CreateDictionaryJob.java:92) at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:95) at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:69) at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73) at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93) at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.FileNotFoundException: /etc/hadoop/3.1.0.0-78/0/core-site.xml (打开的文件过多) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at java.io.FileInputStream.<init>(FileInputStream.java:93) at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90) at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188) at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2901) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2953) ... 21 more result code:2 at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:73) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)<E4D43A7C@7AE6845F.A13A055F.jpg>
<E4D43A7C@7AE6845F.A13A055F.jpg><EB7E8108@66E50C04.A13A055F.jpg><kylin(1).log>
回复：Re: 执行cube在第4步报错，提示“打开的文件过多” 是怎么回事？
Posted by 梅秋莹 <32...@qq.com>.
不好意思，对于kylin我了解还不是很深刻，也不知道句柄是啥意思。请问要改哪个的句柄数？具体操作是怎么样的呢？
------------------ 原始邮件 ------------------
发件人: "初晓"<crowgns@163.com>;
发送时间: 2020年7月9日(星期四) 下午3:53
收件人: "user"<user@kylin.apache.org>;
主题: Re:Re: 执行cube在第4步报错，提示“打开的文件过多” 是怎么回事？
句柄用光了吧，改一下句柄数
At 2020-07-09 10:39:53, "Yaqian Zhang" <Yaqian_Zhang@126.com> wrote:
Hi:
If it's useful to restart kylin or rebuild the job?
在 2020年7月8日，11:16，梅秋莹 <3281438086@qq.com> 写道：
------------------ 原始邮件 ------------------
发件人: "梅秋莹"<3281438086@qq.com>;
发送时间: 2020年7月3日(星期五) 下午4:55
收件人: "user"<user@kylin.apache.org>;
主题: When build cube in kylin, it failed at the 4th step "Build Dimension Dictionary"
Hello everyone:
Recently, we have encountered a new problem. That is, when build cube in kylin, it failed at the 4th step "Build Dimension Dictionary".
<EB7E8108@66E50C04.A13A055F.jpg>
and the error message is like below. Before tha， we have never met such quetion, and our model and cube have functioned properly for at least one month. After reading some articles and blogs, someone think the errors could be related to the cube which has been set some measures like, "TOP-N" and "COUNT_DISTINCT". Indeed，I have set the measures like "TOP-N" and "COUNT_DISTINCT" when create cube. What should we do to solve the proble? Please tell me if you know the reason. Thank you very much.
our kylin is 2.6.3.
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.FileNotFoundException: /etc/hadoop/3.1.0.0-78/0/core-site.xml (打开的文件过多) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3000) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2926) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2806) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1200) at org.apache.kylin.common.util.HadoopUtil.healSickConfig(HadoopUtil.java:69) at org.apache.kylin.common.util.HadoopUtil.getCurrentConfiguration(HadoopUtil.java:59) at org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:103) at org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:95) at org.apache.kylin.common.util.HadoopUtil.getWorkingFileSystem(HadoopUtil.java:82) at org.apache.kylin.engine.mr.steps.CreateDictionaryJob$2.getDictionary(CreateDictionaryJob.java:92) at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:95) at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:69) at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73) at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93) at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.FileNotFoundException: /etc/hadoop/3.1.0.0-78/0/core-site.xml (打开的文件过多) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at java.io.FileInputStream.<init>(FileInputStream.java:93) at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90) at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188) at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2901) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2953) ... 21 more result code:2 at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:73) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)<E4D43A7C@7AE6845F.A13A055F.jpg>
<E4D43A7C@7AE6845F.A13A055F.jpg><EB7E8108@66E50C04.A13A055F.jpg><kylin(1).log>
Re:Re: 执行cube在第4步报错，提示“打开的文件过多” 是怎么回事？
Posted by 初晓 <cr...@163.com>.
句柄用光了吧，改一下句柄数
At 2020-07-09 10:39:53, "Yaqian Zhang" <Ya...@126.com> wrote:
Hi:
If it's useful to restart kylin or rebuild the job?
在 2020年7月8日，11:16，梅秋莹 <32...@qq.com> 写道：
------------------ 原始邮件 ------------------
发件人: "梅秋莹"<32...@qq.com>;
发送时间: 2020年7月3日(星期五) 下午4:55
收件人: "user"<us...@kylin.apache.org>;
主题: When build cube in kylin, it failed at the 4th step "Build Dimension Dictionary"
Hello everyone:
Recently, we have encountered a new problem. That is, when build cube in kylin, it failed at the 4th step "Build Dimension Dictionary".
<EB...@66E50C04.A13A055F.jpg>
and the error message is like below. Before tha， we have never met such quetion, and our model and cube have functioned properly for at least one month. After reading some articles and blogs, someone think the errors could be related to the cube which has been set some measures like, "TOP-N" and "COUNT_DISTINCT". Indeed，I have set the measures like "TOP-N" and "COUNT_DISTINCT" when create cube. What should we do to solve the proble? Please tell me if you know the reason. Thank you very much.
our kylin is 2.6.3.
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.FileNotFoundException: /etc/hadoop/3.1.0.0-78/0/core-site.xml (打开的文件过多)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3000)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2926)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2806)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1200)
at org.apache.kylin.common.util.HadoopUtil.healSickConfig(HadoopUtil.java:69)
at org.apache.kylin.common.util.HadoopUtil.getCurrentConfiguration(HadoopUtil.java:59)
at org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:103)
at org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:95)
at org.apache.kylin.common.util.HadoopUtil.getWorkingFileSystem(HadoopUtil.java:82)
at org.apache.kylin.engine.mr.steps.CreateDictionaryJob$2.getDictionary(CreateDictionaryJob.java:92)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:95)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:69)
at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: /etc/hadoop/3.1.0.0-78/0/core-site.xml (打开的文件过多)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2901)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2953)
... 21 more
result code:2
at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:73)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
<E4...@7AE6845F.A13A055F.jpg>
<E4...@66E50C04.A13A055F.jpg><kylin(1).log>
Re:Re: 执行cube在第4步报错，提示“打开的文件过多” 是怎么回事？
Posted by 初晓 <cr...@163.com>.
https://blog.csdn.net/kuyuyingzi/article/details/80049973
不过你的异常为什么是中文，还是第一次见
At 2020-07-09 10:39:53, "Yaqian Zhang" <Ya...@126.com> wrote:
Hi:
If it's useful to restart kylin or rebuild the job?
在 2020年7月8日，11:16，梅秋莹 <32...@qq.com> 写道：
------------------ 原始邮件 ------------------
发件人: "梅秋莹"<32...@qq.com>;
发送时间: 2020年7月3日(星期五) 下午4:55
收件人: "user"<us...@kylin.apache.org>;
主题: When build cube in kylin, it failed at the 4th step "Build Dimension Dictionary"
Hello everyone:
Recently, we have encountered a new problem. That is, when build cube in kylin, it failed at the 4th step "Build Dimension Dictionary".
<EB...@66E50C04.A13A055F.jpg>
and the error message is like below. Before tha， we have never met such quetion, and our model and cube have functioned properly for at least one month. After reading some articles and blogs, someone think the errors could be related to the cube which has been set some measures like, "TOP-N" and "COUNT_DISTINCT". Indeed，I have set the measures like "TOP-N" and "COUNT_DISTINCT" when create cube. What should we do to solve the proble? Please tell me if you know the reason. Thank you very much.
our kylin is 2.6.3.
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.FileNotFoundException: /etc/hadoop/3.1.0.0-78/0/core-site.xml (打开的文件过多)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3000)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2926)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2806)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1200)
at org.apache.kylin.common.util.HadoopUtil.healSickConfig(HadoopUtil.java:69)
at org.apache.kylin.common.util.HadoopUtil.getCurrentConfiguration(HadoopUtil.java:59)
at org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:103)
at org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:95)
at org.apache.kylin.common.util.HadoopUtil.getWorkingFileSystem(HadoopUtil.java:82)
at org.apache.kylin.engine.mr.steps.CreateDictionaryJob$2.getDictionary(CreateDictionaryJob.java:92)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:95)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:69)
at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: /etc/hadoop/3.1.0.0-78/0/core-site.xml (打开的文件过多)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2901)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2953)
... 21 more
result code:2
at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:73)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
<E4...@7AE6845F.A13A055F.jpg>
<E4...@66E50C04.A13A055F.jpg><kylin(1).log>
Re: 执行cube在第4步报错，提示“打开的文件过多” 是怎么回事？
Posted by Yaqian Zhang <Ya...@126.com>.
Hi:
If it's useful to restart kylin or rebuild the job?
> 在 2020年7月8日，11:16，梅秋莹 <32...@qq.com> 写道：
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "梅秋莹"<32...@qq.com>;
> 发送时间: 2020年7月3日(星期五) 下午4:55
> 收件人: "user"<us...@kylin.apache.org>;
> 主题: When build cube in kylin, it failed at the 4th step "Build Dimension Dictionary"
>
> Hello everyone:
>
> Recently, we have encountered a new problem. That is, when build cube in kylin, it failed at the 4th step "Build Dimension Dictionary".
>
> <EB...@66E50C04.A13A055F.jpg>
> and the error message is like below. Before tha， we have never met such quetion, and our model and cube have functioned properly for at least one month. After reading some articles and blogs, someone think the errors could be related to the cube which has been set some measures like, "TOP-N" and "COUNT_DISTINCT". Indeed，I have set the measures like "TOP-N" and "COUNT_DISTINCT" when create cube. What should we do to solve the proble? Please tell me if you know the reason. Thank you very much.
>
> our kylin is 2.6.3.
>
> org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.FileNotFoundException: /etc/hadoop/3.1.0.0-78/0/core-site.xml (打开的文件过多)
> at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3000)
> at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2926)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2806)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1200)
> at org.apache.kylin.common.util.HadoopUtil.healSickConfig(HadoopUtil.java:69)
> at org.apache.kylin.common.util.HadoopUtil.getCurrentConfiguration(HadoopUtil.java:59)
> at org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:103)
> at org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:95)
> at org.apache.kylin.common.util.HadoopUtil.getWorkingFileSystem(HadoopUtil.java:82)
> at org.apache.kylin.engine.mr.steps.CreateDictionaryJob$2.getDictionary(CreateDictionaryJob.java:92)
> at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:95)
> at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:69)
> at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
> at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.FileNotFoundException: /etc/hadoop/3.1.0.0-78/0/core-site.xml (打开的文件过多)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.<init>(FileInputStream.java:138)
> at java.io.FileInputStream.<init>(FileInputStream.java:93)
> at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
> at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2901)
> at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2953)
> ... 21 more
>
> result code:2
> at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:73)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> <E4...@7AE6845F.A13A055F.jpg>
>
> <E4...@66E50C04.A13A055F.jpg><kylin(1).log>