You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by li_cong521 <li...@126.com> on 2021/01/06 10:47:30 UTC
来自li_cong521的邮件
hello
kylin创建cube的时候，在第四步出现了错误：
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.IllegalStateException: Table snapshot should be no greater than 300 MB, but TableDesc{name='TIME_TABLE', columns=[ColumnDesc{id='1', name='VIN_TIME', datatype='varchar(256)', comment='null'}, ColumnDesc{id='2', name='GPSTIME', datatype='timestamp', comment='null'}, ColumnDesc{id='3', name='PRINT', datatype='date', comment='null'}], sourceType=0, tableType='MANAGED_TABLE', database=DatabaseDesc [name=YIQI], identity='YIQI.TIME_TABLE'} size is 2161852416
at org.apache.kylin.dict.lookup.SnapshotManager.getReusableSnapShot(SnapshotManager.java:168)
然后将kylin的配置文件中将
kylin.table.snapshot.max_mb设置为3000M
日志中又会出现kylin.log：
System free space 不足100M，不知道这里的空间是指什么 ，服务器的内存是256G，应该不是指服务器，请问有人知道如何解决吗？谢谢~
Re:答复: Re:答复: 来自li_cong521的邮件
Posted by li_cong521 <li...@126.com>.
thanks.
But the available memory is very large.
so there is no other reason i can think . May be have a parameter can control the system distribute how many space for kylin?
At 2021-01-08 18:14:31, "Wang rupeng" <wa...@live.cn> wrote:
“System free memory less than 100 MB ” means that “System available memory Is less than 100MB” Available memory is “Maximum heap size minus the current amount used”
So It’s not about disk space but memory space.
You can use the command “free -m” to check the available memory.
发件人: li_cong521 <li...@126.com>
日期:星期五, 2021年1月8日 11:22
收件人: user@kylin.apache.org <us...@kylin.apache.org>
主题: Re:答复: 来自li_cong521的邮件
The error full log is:
2021-01-08 11:18:02,901 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2021-01-08 11:18:02,901 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:18:31,611 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:19:00,110 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:19:13,300 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2021-01-08 11:19:29,908 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:20:11,884 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:20:13,301 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2021-01-08 11:20:13,302 INFO [BadQueryDetector] service.BadQueryDetector:165 : System free memory less than 100 MB. 0 queries running.
the disk free space
At 2021-01-08 10:39:57, "Wang rupeng" <wa...@live.cn> wrote:
Hi,
Isn’t this mean the out of space of your system disk or HDFS? The log “System free space”may not be printed by Kylin.
Or you can show the detailed log so that we can find out the root cause.
-------------------
Best wishes,
Rupeng Wang
发件人: li_cong521 <li...@126.com>
日期:星期三, 2021年1月6日 18:48
收件人: kylin solve error <us...@kylin.apache.org>
主题:来自li_cong521的邮件
hello
kylin创建cube的时候，在第四步出现了错误：
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.IllegalStateException: Table snapshot should be no greater than 300 MB, but TableDesc{name='TIME_TABLE', columns=[ColumnDesc{id='1', name='VIN_TIME', datatype='varchar(256)', comment='null'}, ColumnDesc{id='2', name='GPSTIME', datatype='timestamp', comment='null'}, ColumnDesc{id='3', name='PRINT', datatype='date', comment='null'}], sourceType=0, tableType='MANAGED_TABLE', database=DatabaseDesc [name=YIQI], identity='YIQI.TIME_TABLE'} size is 2161852416
at org.apache.kylin.dict.lookup.SnapshotManager.getReusableSnapShot(SnapshotManager.java:168)
然后将kylin的配置文件中将
kylin.table.snapshot.max_mb设置为3000M
日志中又会出现kylin.log：
System free space 不足100M，不知道这里的空间是指什么，服务器的内存是256G，应该不是指服务器，请问有人知道如何解决吗？谢谢~
答复: Re:答复: 来自li_cong521的邮件
Posted by Wang rupeng <wa...@live.cn>.
“System free memory less than 100 MB ” means that “System available memory Is less than 100MB” Available memory is “Maximum heap size minus the current amount used”
So It’s not about disk space but memory space.
You can use the command “free -m” to check the available memory.
发件人: li_cong521 <li...@126.com>
日期: 星期五, 2021年1月8日 11:22
收件人: user@kylin.apache.org <us...@kylin.apache.org>
主题: Re:答复: 来自li_cong521的邮件
The error full log is:
2021-01-08 11:18:02,901 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2021-01-08 11:18:02,901 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:18:31,611 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:19:00,110 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:19:13,300 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2021-01-08 11:19:29,908 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:20:11,884 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:20:13,301 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2021-01-08 11:20:13,302 INFO [BadQueryDetector] service.BadQueryDetector:165 : System free memory less than 100 MB. 0 queries running.
the disk free space
[cid:image001.png@01D6E5EA.1956B870]
At 2021-01-08 10:39:57, "Wang rupeng" <wa...@live.cn> wrote:
Hi,
Isn’t this mean the out of space of your system disk or HDFS? The log “System free space”may not be printed by Kylin.
Or you can show the detailed log so that we can find out the root cause.
-------------------
Best wishes,
Rupeng Wang
发件人: li_cong521 <li...@126.com>
日期: 星期三, 2021年1月6日 18:48
收件人: kylin solve error <us...@kylin.apache.org>
主题: 来自li_cong521的邮件
hello
kylin创建cube的时候，在第四步出现了错误：
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.IllegalStateException: Table snapshot should be no greater than 300 MB, but TableDesc{name='TIME_TABLE', columns=[ColumnDesc{id='1', name='VIN_TIME', datatype='varchar(256)', comment='null'}, ColumnDesc{id='2', name='GPSTIME', datatype='timestamp', comment='null'}, ColumnDesc{id='3', name='PRINT', datatype='date', comment='null'}], sourceType=0, tableType='MANAGED_TABLE', database=DatabaseDesc [name=YIQI], identity='YIQI.TIME_TABLE'} size is 2161852416
at org.apache.kylin.dict.lookup.SnapshotManager.getReusableSnapShot(SnapshotManager.java:168)
然后将kylin的配置文件中将
kylin.table.snapshot.max_mb设置为3000M
日志中又会出现kylin.log：
System free space 不足100M，不知道这里的空间是指什么 ，服务器的内存是256G，应该不是指服务器，请问有人知道如何解决吗？谢谢~
Re:Re: 来自li_cong521的邮件
Posted by li_cong521 <li...@126.com>.
Thanks for your replay.
At 2021-01-08 15:29:37, "Xiaoxiang Yu" <xx...@apache.org> wrote:
In short, if the size of dimension table is greater than 300MB, please choose to use normal dimension(s) other than derived dimension(s) unless you have a good reason. Too huge snapshot will cause query server's performance decline .
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2021-01-08 11:22:07，"li_cong521" <li...@126.com> 写道：
The error full log is:
2021-01-08 11:18:02,901 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2021-01-08 11:18:02,901 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:18:31,611 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:19:00,110 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:19:13,300 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2021-01-08 11:19:29,908 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:20:11,884 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:20:13,301 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2021-01-08 11:20:13,302 INFO [BadQueryDetector] service.BadQueryDetector:165 : System free memory less than 100 MB. 0 queries running.
the disk free space
At 2021-01-08 10:39:57, "Wang rupeng" <wa...@live.cn> wrote:
Hi,
Isn’t this mean the out of space of your system disk or HDFS? The log “System free space”may not be printed by Kylin.
Or you can show the detailed log so that we can find out the root cause.
-------------------
Best wishes,
Rupeng Wang
发件人: li_cong521 <li...@126.com>
日期:星期三, 2021年1月6日 18:48
收件人: kylin solve error <us...@kylin.apache.org>
主题:来自li_cong521的邮件
hello
kylin创建cube的时候，在第四步出现了错误：
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.IllegalStateException: Table snapshot should be no greater than 300 MB, but TableDesc{name='TIME_TABLE', columns=[ColumnDesc{id='1', name='VIN_TIME', datatype='varchar(256)', comment='null'}, ColumnDesc{id='2', name='GPSTIME', datatype='timestamp', comment='null'}, ColumnDesc{id='3', name='PRINT', datatype='date', comment='null'}], sourceType=0, tableType='MANAGED_TABLE', database=DatabaseDesc [name=YIQI], identity='YIQI.TIME_TABLE'} size is 2161852416
at org.apache.kylin.dict.lookup.SnapshotManager.getReusableSnapShot(SnapshotManager.java:168)
然后将kylin的配置文件中将
kylin.table.snapshot.max_mb设置为3000M
日志中又会出现kylin.log：
System free space 不足100M，不知道这里的空间是指什么，服务器的内存是256G，应该不是指服务器，请问有人知道如何解决吗？谢谢~
Re: 来自li_cong521的邮件
Posted by Xiaoxiang Yu <xx...@apache.org>.
In short, if the size of dimension table is greater than 300MB, please choose to use normal dimension(s) other than derived dimension(s) unless you have a good reason. Too huge snapshot will cause query server's performance decline .
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2021-01-08 11:22:07，"li_cong521" <li...@126.com> 写道：
The error full log is:
2021-01-08 11:18:02,901 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2021-01-08 11:18:02,901 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:18:31,611 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:19:00,110 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:19:13,300 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2021-01-08 11:19:29,908 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:20:11,884 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:20:13,301 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2021-01-08 11:20:13,302 INFO [BadQueryDetector] service.BadQueryDetector:165 : System free memory less than 100 MB. 0 queries running.
the disk free space
At 2021-01-08 10:39:57, "Wang rupeng" <wa...@live.cn> wrote:
Hi,
Isn’t this mean the out of space of your system disk or HDFS? The log “System free space”may not be printed by Kylin.
Or you can show the detailed log so that we can find out the root cause.
-------------------
Best wishes,
Rupeng Wang
发件人: li_cong521 <li...@126.com>
日期:星期三, 2021年1月6日 18:48
收件人: kylin solve error <us...@kylin.apache.org>
主题:来自li_cong521的邮件
hello
kylin创建cube的时候，在第四步出现了错误：
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.IllegalStateException: Table snapshot should be no greater than 300 MB, but TableDesc{name='TIME_TABLE', columns=[ColumnDesc{id='1', name='VIN_TIME', datatype='varchar(256)', comment='null'}, ColumnDesc{id='2', name='GPSTIME', datatype='timestamp', comment='null'}, ColumnDesc{id='3', name='PRINT', datatype='date', comment='null'}], sourceType=0, tableType='MANAGED_TABLE', database=DatabaseDesc [name=YIQI], identity='YIQI.TIME_TABLE'} size is 2161852416
at org.apache.kylin.dict.lookup.SnapshotManager.getReusableSnapShot(SnapshotManager.java:168)
然后将kylin的配置文件中将
kylin.table.snapshot.max_mb设置为3000M
日志中又会出现kylin.log：
System free space 不足100M，不知道这里的空间是指什么，服务器的内存是256G，应该不是指服务器，请问有人知道如何解决吗？谢谢~
Re:答复: 来自li_cong521的邮件
Posted by li_cong521 <li...@126.com>.
The error full log is:
2021-01-08 11:18:02,901 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2021-01-08 11:18:02,901 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:18:31,611 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:19:00,110 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:19:13,300 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2021-01-08 11:19:29,908 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:20:11,884 INFO [FetcherRunner 2065467594-56] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 8 already succeed, 0 error, 0 discarded, 0 others
2021-01-08 11:20:13,301 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2021-01-08 11:20:13,302 INFO [BadQueryDetector] service.BadQueryDetector:165 : System free memory less than 100 MB. 0 queries running.
the disk free space
At 2021-01-08 10:39:57, "Wang rupeng" <wa...@live.cn> wrote:
Hi,
Isn’t this mean the out of space of your system disk or HDFS? The log “System free space”may not be printed by Kylin.
Or you can show the detailed log so that we can find out the root cause.
-------------------
Best wishes,
Rupeng Wang
发件人: li_cong521 <li...@126.com>
日期:星期三, 2021年1月6日 18:48
收件人: kylin solve error <us...@kylin.apache.org>
主题:来自li_cong521的邮件
hello
kylin创建cube的时候，在第四步出现了错误：
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.IllegalStateException: Table snapshot should be no greater than 300 MB, but TableDesc{name='TIME_TABLE', columns=[ColumnDesc{id='1', name='VIN_TIME', datatype='varchar(256)', comment='null'}, ColumnDesc{id='2', name='GPSTIME', datatype='timestamp', comment='null'}, ColumnDesc{id='3', name='PRINT', datatype='date', comment='null'}], sourceType=0, tableType='MANAGED_TABLE', database=DatabaseDesc [name=YIQI], identity='YIQI.TIME_TABLE'} size is 2161852416
at org.apache.kylin.dict.lookup.SnapshotManager.getReusableSnapShot(SnapshotManager.java:168)
然后将kylin的配置文件中将
kylin.table.snapshot.max_mb设置为3000M
日志中又会出现kylin.log：
System free space 不足100M，不知道这里的空间是指什么，服务器的内存是256G，应该不是指服务器，请问有人知道如何解决吗？谢谢~
Re:答复: 来自li_cong521的邮件
Posted by li_cong521 <li...@126.com>.
Thanks for your replay.
The HDFS space is 1.7T*10, may be is the parameter set in hadoop or kylin error? such as this ?
the full error log is
At 2021-01-08 10:39:57, "Wang rupeng" <wa...@live.cn> wrote:
Hi,
Isn’t this mean the out of space of your system disk or HDFS? The log “System free space”may not be printed by Kylin.
Or you can show the detailed log so that we can find out the root cause.
-------------------
Best wishes,
Rupeng Wang
发件人: li_cong521 <li...@126.com>
日期:星期三, 2021年1月6日 18:48
收件人: kylin solve error <us...@kylin.apache.org>
主题:来自li_cong521的邮件
hello
kylin创建cube的时候，在第四步出现了错误：
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.IllegalStateException: Table snapshot should be no greater than 300 MB, but TableDesc{name='TIME_TABLE', columns=[ColumnDesc{id='1', name='VIN_TIME', datatype='varchar(256)', comment='null'}, ColumnDesc{id='2', name='GPSTIME', datatype='timestamp', comment='null'}, ColumnDesc{id='3', name='PRINT', datatype='date', comment='null'}], sourceType=0, tableType='MANAGED_TABLE', database=DatabaseDesc [name=YIQI], identity='YIQI.TIME_TABLE'} size is 2161852416
at org.apache.kylin.dict.lookup.SnapshotManager.getReusableSnapShot(SnapshotManager.java:168)
然后将kylin的配置文件中将
kylin.table.snapshot.max_mb设置为3000M
日志中又会出现kylin.log：
System free space 不足100M，不知道这里的空间是指什么，服务器的内存是256G，应该不是指服务器，请问有人知道如何解决吗？谢谢~
答复: 来自li_cong521的邮件
Posted by Wang rupeng <wa...@live.cn>.
Hi,
Isn’t this mean the out of space of your system disk or HDFS? The log “System free space”may not be printed by Kylin.
Or you can show the detailed log so that we can find out the root cause.
-------------------
Best wishes,
Rupeng Wang
发件人: li_cong521 <li...@126.com>
日期: 星期三, 2021年1月6日 18:48
收件人: kylin solve error <us...@kylin.apache.org>
主题: 来自li_cong521的邮件
hello
kylin创建cube的时候，在第四步出现了错误：
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.IllegalStateException: Table snapshot should be no greater than 300 MB, but TableDesc{name='TIME_TABLE', columns=[ColumnDesc{id='1', name='VIN_TIME', datatype='varchar(256)', comment='null'}, ColumnDesc{id='2', name='GPSTIME', datatype='timestamp', comment='null'}, ColumnDesc{id='3', name='PRINT', datatype='date', comment='null'}], sourceType=0, tableType='MANAGED_TABLE', database=DatabaseDesc [name=YIQI], identity='YIQI.TIME_TABLE'} size is 2161852416
at org.apache.kylin.dict.lookup.SnapshotManager.getReusableSnapShot(SnapshotManager.java:168)
然后将kylin的配置文件中将
kylin.table.snapshot.max_mb设置为3000M
日志中又会出现kylin.log：
System free space 不足100M，不知道这里的空间是指什么 ，服务器的内存是256G，应该不是指服务器，请问有人知道如何解决吗？谢谢~