You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by ashishjainm <as...@gmail.com> on 2017/04/14 05:53:58 UTC
Kylin Build Cube failing with the following error
Hello Team,
I am new to Kylin and just created the sample cube. however, when i create a
model with my own data and this is a very small data set with 10 rows in the
fact table. The process is failing with the following error message. I am
using the CDH quickstart VM.
May be this is a Yarn related issue and i wanted to know an optimum
configuration that i need to set for me to be able to build a sample cube of
my own.
82252_0024_01_000003/meta/cube_desc
2017-04-13 22:42:34,902 INFO [main]
org.apache.kylin.metadata.project.ProjectManager: Initializing
ProjectManager with metadata url
/yarn/nm/usercache/root/appcache/application_1492137382252_0024/container_1492137382252_0024_01_000003/meta
2017-04-13 22:42:35,008 INFO [main]
org.apache.kylin.measure.MeasureTypeFactory: Checking custom measure types
from kylin config
2017-04-13 22:42:35,011 INFO [main]
org.apache.kylin.measure.MeasureTypeFactory: registering
COUNT_DISTINCT(hllc), class
org.apache.kylin.measure.hllc.HLLCMeasureType$Factory
2017-04-13 22:42:35,028 INFO [main]
org.apache.kylin.measure.MeasureTypeFactory: registering
COUNT_DISTINCT(bitmap), class
org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory
2017-04-13 22:42:35,038 INFO [main]
org.apache.kylin.measure.MeasureTypeFactory: registering TOP_N(topn), class
org.apache.kylin.measure.topn.TopNMeasureType$Factory
2017-04-13 22:42:35,042 INFO [main]
org.apache.kylin.measure.MeasureTypeFactory: registering RAW(raw), class
org.apache.kylin.measure.raw.RawMeasureType$Factory
2017-04-13 22:42:35,046 INFO [main]
org.apache.kylin.measure.MeasureTypeFactory: registering
EXTENDED_COLUMN(extendedcolumn), class
org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory
2017-04-13 22:42:35,049 INFO [main]
org.apache.kylin.measure.MeasureTypeFactory: registering
PERCENTILE(percentile), class
org.apache.kylin.measure.percentile.PercentileMeasureType$Factory
2017-04-13 22:42:35,155 INFO [main] org.apache.kylin.cube.CubeDescManager:
Loaded 1 Cube(s)
2017-04-13 22:42:35,159 INFO [main] org.apache.kylin.cube.CubeManager:
Reloaded cube spend_cube being CUBE[name=spend_cube] having 1 segments
2017-04-13 22:42:35,159 INFO [main] org.apache.kylin.cube.CubeManager:
Loaded 1 cubes, fail on 0 cubes
2017-04-13 22:42:38,384 INFO [main] org.apache.hadoop.mapred.MapTask:
Starting flush of map output
2017-04-13 22:42:41,220 FATAL [main] org.apache.hadoop.mapred.YarnChild:
Error running child : java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300)
at java.lang.StringCoding.encode(StringCoding.java:344)
at java.lang.String.getBytes(String.java:916)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:216)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
at
org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:295)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:281)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:901)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:863)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1843)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1508)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
2017-04-13 22:42:41,332 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics
system...
2017-04-13 22:42:41,333 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
stopped.
2017-04-13 22:42:41,333 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
shutdown complete.
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/Kylin-Build-Cube-failing-with-the-following-error-tp7646.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: 答复: 答复: 答复: Kylin Build Cube failing with the following error
Posted by ashishjainm <as...@gmail.com>.
Please find attached the log. Request you to provide additional information
adn guidance.
On Fri, Apr 14, 2017 at 2:58 AM, Roger Shi [via Apache Kylin] <
ml-node+s74782n7659h86@n6.nabble.com> wrote:
> Please follow the link to generate diagnose package. It helps to identify
> the root cause.
>
>
> http://kylin.apache.org/blog/2016/06/10/diagnosis-tool/
>
> Apache Kylin | Diagnosis Tool Introduction<http://kylin.
> apache.org/blog/2016/06/10/diagnosis-tool/>
> kylin.apache.org
> Diagnosis Tool Introduction. Jun 10, 2016 • Dong Li. Introduction. Since
> Apache Kylin 1.5.2, there’s a diagnosis tool on Web UI, which aims to help
> Kylin admins ...
>
>
>
>
> ________________________________
> 发件人: ashishjainm <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=7659&i=0>>
> 发送时间: 2017年4月14日 17:39:04
> 收件人: [hidden email] <http:///user/SendEmail.jtp?type=node&node=7659&i=1>
> 主题: Re: 答复: 答复: Kylin Build Cube failing with the following error
>
> any suggestions on the changes of the properties and corresponding values.
> also, the sample cube with Kylin builds without any problems. then why
> this
> small cube build fails again and again, beats me.
>
> On Fri, Apr 14, 2017 at 2:21 AM, Roger Shi [via Apache Kylin] <
> [hidden email] <http:///user/SendEmail.jtp?type=node&node=7659&i=2>>
> wrote:
>
> > Yarn resource allocation is not that enough in CDH quickstart version
> 5.8.
> > Here "enough" doesn't only refer to total resource, but also map and
> reduce
> > memory configuration. You could have a try to configure them properly.
> > ________________________________
> > 发件人: ashishjainm <[hidden email]
> > <http:///user/SendEmail.jtp?type=node&node=7656&i=0>>
> > 发送时间: 2017年4月14日 16:44:36
> > 收件人: [hidden email] <http:///user/SendEmail.jtp?type=node&node=7656&i=1>
>
> > 主题: Re: 答复: Kylin Build Cube failing with the following error
> >
> > Thanks Roger.
> > Below step is failing:
> > #3 Step Name: Extract Fact Table Distinct Columns
> > Duration: 1.62 mins Waiting: 23 seconds
> >
> > I am running on CDH distribution quickstart vm version 5.8
> >
> > On Thu, Apr 13, 2017 at 11:57 PM, Roger Shi [via Apache Kylin] <
> > [hidden email] <http:///user/SendEmail.jtp?type=node&node=7656&i=2>>
> > wrote:
> >
> > > Which step throws this error? From the log, it seems hadoop is opening
> > > local file. Please check the default FileSystem of HDFS and also check
> > the
> > > hadoop job's memory configuration. There might be some hint.
> > > ________________________________
> > > 发件人: ashishjainm <[hidden email]
> > > <http:///user/SendEmail.jtp?type=node&node=7647&i=0>>
> > > 发送时间: 2017年4月14日 13:53:58
> > > 收件人: [hidden email] <http:///user/SendEmail.jtp?
> type=node&node=7647&i=1>
> >
> > > 主题: Kylin Build Cube failing with the following error
> > >
> > > Hello Team,
> > >
> > > I am new to Kylin and just created the sample cube. however, when i
> > create
> > > a
> > > model with my own data and this is a very small data set with 10 rows
> in
> > > the
> > > fact table. The process is failing with the following error message. I
> > am
> > > using the CDH quickstart VM.
> > >
> > > May be this is a Yarn related issue and i wanted to know an optimum
> > > configuration that i need to set for me to be able to build a sample
> > cube
> > > of
> > > my own.
> > >
> > > 82252_0024_01_000003/meta/cube_desc
> > > 2017-04-13 22:42:34,902 INFO [main]
> > > org.apache.kylin.metadata.project.ProjectManager: Initializing
> > > ProjectManager with metadata url
> > > /yarn/nm/usercache/root/appcache/application_1492137382252_0024/container_
>
> >
> > > 1492137382252_0024_01_000003/meta
> > > 2017-04-13 22:42:35,008 INFO [main]
> > > org.apache.kylin.measure.MeasureTypeFactory: Checking custom measure
> > > types
> > > from kylin config
> > > 2017-04-13 22:42:35,011 INFO [main]
> > > org.apache.kylin.measure.MeasureTypeFactory: registering
> > > COUNT_DISTINCT(hllc), class
> > > org.apache.kylin.measure.hllc.HLLCMeasureType$Factory
> > > 2017-04-13 22:42:35,028 INFO [main]
> > > org.apache.kylin.measure.MeasureTypeFactory: registering
> > > COUNT_DISTINCT(bitmap), class
> > > org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory
> > > 2017-04-13 22:42:35,038 INFO [main]
> > > org.apache.kylin.measure.MeasureTypeFactory: registering TOP_N(topn),
> > > class
> > > org.apache.kylin.measure.topn.TopNMeasureType$Factory
> > > 2017-04-13 22:42:35,042 INFO [main]
> > > org.apache.kylin.measure.MeasureTypeFactory: registering RAW(raw),
> > class
> > > org.apache.kylin.measure.raw.RawMeasureType$Factory
> > > 2017-04-13 22:42:35,046 INFO [main]
> > > org.apache.kylin.measure.MeasureTypeFactory: registering
> > > EXTENDED_COLUMN(extendedcolumn), class
> > > org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory
>
> >
> > > 2017-04-13 22:42:35,049 INFO [main]
> > > org.apache.kylin.measure.MeasureTypeFactory: registering
> > > PERCENTILE(percentile), class
> > > org.apache.kylin.measure.percentile.PercentileMeasureType$Factory
> > > 2017-04-13 22:42:35,155 INFO [main] org.apache.kylin.cube.CubeDescManager:
>
> >
> > >
> > > Loaded 1 Cube(s)
> > > 2017-04-13 22:42:35,159 INFO [main] org.apache.kylin.cube.CubeManager:
>
> > > Reloaded cube spend_cube being CUBE[name=spend_cube] having 1 segments
> > > 2017-04-13 22:42:35,159 INFO [main] org.apache.kylin.cube.CubeManager:
>
> > > Loaded 1 cubes, fail on 0 cubes
> > > 2017-04-13 22:42:38,384 INFO [main] org.apache.hadoop.mapred.MapTask:
> > > Starting flush of map output
> > > 2017-04-13 22:42:41,220 FATAL [main] org.apache.hadoop.mapred.YarnChild:
>
> >
> > > Error running child : java.lang.OutOfMemoryError: GC overhead limit
> > > exceeded
> > > at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300)
>
> >
> > >
> > > at java.lang.StringCoding.encode(StringCoding.java:344)
> > > at java.lang.String.getBytes(String.java:916)
> > > at java.io.FileOutputStream.open(Native Method)
> > > at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
> > > at
> > > org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:216)
>
> >
> > >
> > > at
> > > org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
>
> >
> > >
> > > at
> > > org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
>
> >
> > >
> > > at
> > > org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:295)
>
> >
> > >
> > > at
> > > org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:281)
>
> >
> > >
> > > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:901)
>
> > > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:863)
>
> > > at
> > > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1843)
>
> >
> > >
> > > at
> > > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1508)
>
> >
> > > at
> > > org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
>
> >
> > >
> > > at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
>
> >
> > >
> > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
>
> >
> > >
> > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> > > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at javax.security.auth.Subject.doAs(Subject.java:415)
> > > at
> > > org.apache.hadoop.security.UserGroupInformation.doAs(
> > > UserGroupInformation.java:1693)
> > > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>
> > >
> > > 2017-04-13 22:42:41,332 INFO [main]
> > > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask
> > > metrics
> > > system...
> > > 2017-04-13 22:42:41,333 INFO [main]
> > > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics
> > system
> > > stopped.
> > > 2017-04-13 22:42:41,333 INFO [main]
> > > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics
> > system
> > > shutdown complete.
> > >
> > > --
> > > View this message in context: http://apache-kylin.74782.x6.
> > > nabble.com/Kylin-Build-Cube-failing-with-the-following-
> error-tp7646.html
> > > Sent from the Apache Kylin mailing list archive at Nabble.com.
> > >
> > >
> > > ------------------------------
> > > If you reply to this email, your message will be added to the
> discussion
> > > below:
> > > http://apache-kylin.74782.x6.nabble.com/Kylin-Build-Cube-
> > > failing-with-the-following-error-tp7646p7647.html
> > > To unsubscribe from Kylin Build Cube failing with the following error,
> > click
> > > here
> > > <
> > > .
> > > NAML
> > > <http://apache-kylin.74782.x6.nabble.com/template/
> > NamlServlet.jtp?macro=macro_viewer&id=instant_html%
> > 21nabble%3Aemail.naml&base=nabble.naml.namespaces.
> > BasicNamespace-nabble.view.web.template.NabbleNamespace-
> > nabble.view.web.template.NodeNamespace&breadcrumbs=
> > notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> > 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> > >
> >
> >
> >
> > --
> >
> >
> >
> > [image: --]
> >
> > Ashish Jain
> > [image: http://]about.me/ashishjainm
> > <http://about.me/ashishjainm?promo=email_sig>
> >
> >
> > --
> > View this message in context: http://apache-kylin.74782.x6.
> > nabble.com/Kylin-Build-Cube-failing-with-the-following-
> > error-tp7646p7653.html
> > Sent from the Apache Kylin mailing list archive at Nabble.com.
> >
> >
> > ------------------------------
> > If you reply to this email, your message will be added to the discussion
> > below:
> > http://apache-kylin.74782.x6.nabble.com/Kylin-Build-Cube-
> > failing-with-the-following-error-tp7646p7656.html
> > To unsubscribe from Kylin Build Cube failing with the following error,
> click
> > here
> > <
> > .
> > NAML
> > <http://apache-kylin.74782.x6.nabble.com/template/
> NamlServlet.jtp?macro=macro_viewer&id=instant_html%
> 21nabble%3Aemail.naml&base=nabble.naml.namespaces.
> BasicNamespace-nabble.view.web.template.NabbleNamespace-
> nabble.view.web.template.NodeNamespace&breadcrumbs=
> notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> >
>
>
>
> --
>
>
>
> [image: --]
>
> Ashish Jain
> [image: http://]about.me/ashishjainm
> <http://about.me/ashishjainm?promo=email_sig>
>
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/Kylin-Build-Cube-failing-with-the-following-
> error-tp7646p7658.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-kylin.74782.x6.nabble.com/Kylin-Build-Cube-
> failing-with-the-following-error-tp7646p7659.html
> To unsubscribe from Kylin Build Cube failing with the following error, click
> here
> <http://apache-kylin.74782.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7646&code=YXNoaXNoamFpbm1AZ21haWwuY29tfDc2NDZ8MTk3MTk2MjI2>
> .
> NAML
> <http://apache-kylin.74782.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
[image: --]
Ashish Jain
[image: http://]about.me/ashishjainm
<http://about.me/ashishjainm?promo=email_sig>
job_2017_04_14_03_10_14.zip (768K) <http://apache-kylin.74782.x6.nabble.com/attachment/7660/0/job_2017_04_14_03_10_14.zip>
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/Kylin-Build-Cube-failing-with-the-following-error-tp7646p7660.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
答复: 答复: 答复: Kylin Build Cube failing with the following error
Posted by roger shi <ro...@hotmail.com>.
Please follow the link to generate diagnose package. It helps to identify the root cause.
http://kylin.apache.org/blog/2016/06/10/diagnosis-tool/
Apache Kylin | Diagnosis Tool Introduction<http://kylin.apache.org/blog/2016/06/10/diagnosis-tool/>
kylin.apache.org
Diagnosis Tool Introduction. Jun 10, 2016 • Dong Li. Introduction. Since Apache Kylin 1.5.2, there’s a diagnosis tool on Web UI, which aims to help Kylin admins ...
________________________________
发件人: ashishjainm <as...@gmail.com>
发送时间: 2017年4月14日 17:39:04
收件人: dev@kylin.apache.org
主题: Re: 答复: 答复: Kylin Build Cube failing with the following error
any suggestions on the changes of the properties and corresponding values.
also, the sample cube with Kylin builds without any problems. then why this
small cube build fails again and again, beats me.
On Fri, Apr 14, 2017 at 2:21 AM, Roger Shi [via Apache Kylin] <
ml-node+s74782n7656h93@n6.nabble.com> wrote:
> Yarn resource allocation is not that enough in CDH quickstart version 5.8.
> Here "enough" doesn't only refer to total resource, but also map and reduce
> memory configuration. You could have a try to configure them properly.
> ________________________________
> 发件人: ashishjainm <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=7656&i=0>>
> 发送时间: 2017年4月14日 16:44:36
> 收件人: [hidden email] <http:///user/SendEmail.jtp?type=node&node=7656&i=1>
> 主题: Re: 答复: Kylin Build Cube failing with the following error
>
> Thanks Roger.
> Below step is failing:
> #3 Step Name: Extract Fact Table Distinct Columns
> Duration: 1.62 mins Waiting: 23 seconds
>
> I am running on CDH distribution quickstart vm version 5.8
>
> On Thu, Apr 13, 2017 at 11:57 PM, Roger Shi [via Apache Kylin] <
> [hidden email] <http:///user/SendEmail.jtp?type=node&node=7656&i=2>>
> wrote:
>
> > Which step throws this error? From the log, it seems hadoop is opening
> > local file. Please check the default FileSystem of HDFS and also check
> the
> > hadoop job's memory configuration. There might be some hint.
> > ________________________________
> > 发件人: ashishjainm <[hidden email]
> > <http:///user/SendEmail.jtp?type=node&node=7647&i=0>>
> > 发送时间: 2017年4月14日 13:53:58
> > 收件人: [hidden email] <http:///user/SendEmail.jtp?type=node&node=7647&i=1>
>
> > 主题: Kylin Build Cube failing with the following error
> >
> > Hello Team,
> >
> > I am new to Kylin and just created the sample cube. however, when i
> create
> > a
> > model with my own data and this is a very small data set with 10 rows in
> > the
> > fact table. The process is failing with the following error message. I
> am
> > using the CDH quickstart VM.
> >
> > May be this is a Yarn related issue and i wanted to know an optimum
> > configuration that i need to set for me to be able to build a sample
> cube
> > of
> > my own.
> >
> > 82252_0024_01_000003/meta/cube_desc
> > 2017-04-13 22:42:34,902 INFO [main]
> > org.apache.kylin.metadata.project.ProjectManager: Initializing
> > ProjectManager with metadata url
> > /yarn/nm/usercache/root/appcache/application_1492137382252_0024/container_
>
> > 1492137382252_0024_01_000003/meta
> > 2017-04-13 22:42:35,008 INFO [main]
> > org.apache.kylin.measure.MeasureTypeFactory: Checking custom measure
> > types
> > from kylin config
> > 2017-04-13 22:42:35,011 INFO [main]
> > org.apache.kylin.measure.MeasureTypeFactory: registering
> > COUNT_DISTINCT(hllc), class
> > org.apache.kylin.measure.hllc.HLLCMeasureType$Factory
> > 2017-04-13 22:42:35,028 INFO [main]
> > org.apache.kylin.measure.MeasureTypeFactory: registering
> > COUNT_DISTINCT(bitmap), class
> > org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory
> > 2017-04-13 22:42:35,038 INFO [main]
> > org.apache.kylin.measure.MeasureTypeFactory: registering TOP_N(topn),
> > class
> > org.apache.kylin.measure.topn.TopNMeasureType$Factory
> > 2017-04-13 22:42:35,042 INFO [main]
> > org.apache.kylin.measure.MeasureTypeFactory: registering RAW(raw),
> class
> > org.apache.kylin.measure.raw.RawMeasureType$Factory
> > 2017-04-13 22:42:35,046 INFO [main]
> > org.apache.kylin.measure.MeasureTypeFactory: registering
> > EXTENDED_COLUMN(extendedcolumn), class
> > org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory
>
> > 2017-04-13 22:42:35,049 INFO [main]
> > org.apache.kylin.measure.MeasureTypeFactory: registering
> > PERCENTILE(percentile), class
> > org.apache.kylin.measure.percentile.PercentileMeasureType$Factory
> > 2017-04-13 22:42:35,155 INFO [main] org.apache.kylin.cube.CubeDescManager:
>
> >
> > Loaded 1 Cube(s)
> > 2017-04-13 22:42:35,159 INFO [main] org.apache.kylin.cube.CubeManager:
> > Reloaded cube spend_cube being CUBE[name=spend_cube] having 1 segments
> > 2017-04-13 22:42:35,159 INFO [main] org.apache.kylin.cube.CubeManager:
> > Loaded 1 cubes, fail on 0 cubes
> > 2017-04-13 22:42:38,384 INFO [main] org.apache.hadoop.mapred.MapTask:
> > Starting flush of map output
> > 2017-04-13 22:42:41,220 FATAL [main] org.apache.hadoop.mapred.YarnChild:
>
> > Error running child : java.lang.OutOfMemoryError: GC overhead limit
> > exceeded
> > at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300)
>
> >
> > at java.lang.StringCoding.encode(StringCoding.java:344)
> > at java.lang.String.getBytes(String.java:916)
> > at java.io.FileOutputStream.open(Native Method)
> > at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
> > at
> > org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:216)
>
> >
> > at
> > org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
>
> >
> > at
> > org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
>
> >
> > at
> > org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:295)
>
> >
> > at
> > org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:281)
>
> >
> > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:901)
> > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:863)
> > at
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1843)
>
> >
> > at
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1508)
>
> > at
> > org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
>
> >
> > at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
>
> >
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
>
> >
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
> > org.apache.hadoop.security.UserGroupInformation.doAs(
> > UserGroupInformation.java:1693)
> > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> >
> > 2017-04-13 22:42:41,332 INFO [main]
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask
> > metrics
> > system...
> > 2017-04-13 22:42:41,333 INFO [main]
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics
> system
> > stopped.
> > 2017-04-13 22:42:41,333 INFO [main]
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics
> system
> > shutdown complete.
> >
> > --
> > View this message in context: http://apache-kylin.74782.x6.
> > nabble.com/Kylin-Build-Cube-failing-with-the-following-error-tp7646.html
> > Sent from the Apache Kylin mailing list archive at Nabble.com.
> >
> >
> > ------------------------------
> > If you reply to this email, your message will be added to the discussion
> > below:
> > http://apache-kylin.74782.x6.nabble.com/Kylin-Build-Cube-
> > failing-with-the-following-error-tp7646p7647.html
> > To unsubscribe from Kylin Build Cube failing with the following error,
> click
> > here
> > <
> > .
> > NAML
> > <http://apache-kylin.74782.x6.nabble.com/template/
> NamlServlet.jtp?macro=macro_viewer&id=instant_html%
> 21nabble%3Aemail.naml&base=nabble.naml.namespaces.
> BasicNamespace-nabble.view.web.template.NabbleNamespace-
> nabble.view.web.template.NodeNamespace&breadcrumbs=
> notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> >
>
>
>
> --
>
>
>
> [image: --]
>
> Ashish Jain
> [image: http://]about.me/ashishjainm
> <http://about.me/ashishjainm?promo=email_sig>
>
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/Kylin-Build-Cube-failing-with-the-following-
> error-tp7646p7653.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-kylin.74782.x6.nabble.com/Kylin-Build-Cube-
> failing-with-the-following-error-tp7646p7656.html
> To unsubscribe from Kylin Build Cube failing with the following error, click
> here
> <http://apache-kylin.74782.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7646&code=YXNoaXNoamFpbm1AZ21haWwuY29tfDc2NDZ8MTk3MTk2MjI2>
> .
> NAML
> <http://apache-kylin.74782.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
[image: --]
Ashish Jain
[image: http://]about.me/ashishjainm
<http://about.me/ashishjainm?promo=email_sig>
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/Kylin-Build-Cube-failing-with-the-following-error-tp7646p7658.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: 答复: 答复: Kylin Build Cube failing with the following error
Posted by ashishjainm <as...@gmail.com>.
any suggestions on the changes of the properties and corresponding values.
also, the sample cube with Kylin builds without any problems. then why this
small cube build fails again and again, beats me.
On Fri, Apr 14, 2017 at 2:21 AM, Roger Shi [via Apache Kylin] <
ml-node+s74782n7656h93@n6.nabble.com> wrote:
> Yarn resource allocation is not that enough in CDH quickstart version 5.8.
> Here "enough" doesn't only refer to total resource, but also map and reduce
> memory configuration. You could have a try to configure them properly.
> ________________________________
> 发件人: ashishjainm <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=7656&i=0>>
> 发送时间: 2017年4月14日 16:44:36
> 收件人: [hidden email] <http:///user/SendEmail.jtp?type=node&node=7656&i=1>
> 主题: Re: 答复: Kylin Build Cube failing with the following error
>
> Thanks Roger.
> Below step is failing:
> #3 Step Name: Extract Fact Table Distinct Columns
> Duration: 1.62 mins Waiting: 23 seconds
>
> I am running on CDH distribution quickstart vm version 5.8
>
> On Thu, Apr 13, 2017 at 11:57 PM, Roger Shi [via Apache Kylin] <
> [hidden email] <http:///user/SendEmail.jtp?type=node&node=7656&i=2>>
> wrote:
>
> > Which step throws this error? From the log, it seems hadoop is opening
> > local file. Please check the default FileSystem of HDFS and also check
> the
> > hadoop job's memory configuration. There might be some hint.
> > ________________________________
> > 发件人: ashishjainm <[hidden email]
> > <http:///user/SendEmail.jtp?type=node&node=7647&i=0>>
> > 发送时间: 2017年4月14日 13:53:58
> > 收件人: [hidden email] <http:///user/SendEmail.jtp?type=node&node=7647&i=1>
>
> > 主题: Kylin Build Cube failing with the following error
> >
> > Hello Team,
> >
> > I am new to Kylin and just created the sample cube. however, when i
> create
> > a
> > model with my own data and this is a very small data set with 10 rows in
> > the
> > fact table. The process is failing with the following error message. I
> am
> > using the CDH quickstart VM.
> >
> > May be this is a Yarn related issue and i wanted to know an optimum
> > configuration that i need to set for me to be able to build a sample
> cube
> > of
> > my own.
> >
> > 82252_0024_01_000003/meta/cube_desc
> > 2017-04-13 22:42:34,902 INFO [main]
> > org.apache.kylin.metadata.project.ProjectManager: Initializing
> > ProjectManager with metadata url
> > /yarn/nm/usercache/root/appcache/application_1492137382252_0024/container_
>
> > 1492137382252_0024_01_000003/meta
> > 2017-04-13 22:42:35,008 INFO [main]
> > org.apache.kylin.measure.MeasureTypeFactory: Checking custom measure
> > types
> > from kylin config
> > 2017-04-13 22:42:35,011 INFO [main]
> > org.apache.kylin.measure.MeasureTypeFactory: registering
> > COUNT_DISTINCT(hllc), class
> > org.apache.kylin.measure.hllc.HLLCMeasureType$Factory
> > 2017-04-13 22:42:35,028 INFO [main]
> > org.apache.kylin.measure.MeasureTypeFactory: registering
> > COUNT_DISTINCT(bitmap), class
> > org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory
> > 2017-04-13 22:42:35,038 INFO [main]
> > org.apache.kylin.measure.MeasureTypeFactory: registering TOP_N(topn),
> > class
> > org.apache.kylin.measure.topn.TopNMeasureType$Factory
> > 2017-04-13 22:42:35,042 INFO [main]
> > org.apache.kylin.measure.MeasureTypeFactory: registering RAW(raw),
> class
> > org.apache.kylin.measure.raw.RawMeasureType$Factory
> > 2017-04-13 22:42:35,046 INFO [main]
> > org.apache.kylin.measure.MeasureTypeFactory: registering
> > EXTENDED_COLUMN(extendedcolumn), class
> > org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory
>
> > 2017-04-13 22:42:35,049 INFO [main]
> > org.apache.kylin.measure.MeasureTypeFactory: registering
> > PERCENTILE(percentile), class
> > org.apache.kylin.measure.percentile.PercentileMeasureType$Factory
> > 2017-04-13 22:42:35,155 INFO [main] org.apache.kylin.cube.CubeDescManager:
>
> >
> > Loaded 1 Cube(s)
> > 2017-04-13 22:42:35,159 INFO [main] org.apache.kylin.cube.CubeManager:
> > Reloaded cube spend_cube being CUBE[name=spend_cube] having 1 segments
> > 2017-04-13 22:42:35,159 INFO [main] org.apache.kylin.cube.CubeManager:
> > Loaded 1 cubes, fail on 0 cubes
> > 2017-04-13 22:42:38,384 INFO [main] org.apache.hadoop.mapred.MapTask:
> > Starting flush of map output
> > 2017-04-13 22:42:41,220 FATAL [main] org.apache.hadoop.mapred.YarnChild:
>
> > Error running child : java.lang.OutOfMemoryError: GC overhead limit
> > exceeded
> > at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300)
>
> >
> > at java.lang.StringCoding.encode(StringCoding.java:344)
> > at java.lang.String.getBytes(String.java:916)
> > at java.io.FileOutputStream.open(Native Method)
> > at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
> > at
> > org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:216)
>
> >
> > at
> > org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
>
> >
> > at
> > org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
>
> >
> > at
> > org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:295)
>
> >
> > at
> > org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:281)
>
> >
> > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:901)
> > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:863)
> > at
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1843)
>
> >
> > at
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1508)
>
> > at
> > org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
>
> >
> > at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
>
> >
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
>
> >
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
> > org.apache.hadoop.security.UserGroupInformation.doAs(
> > UserGroupInformation.java:1693)
> > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> >
> > 2017-04-13 22:42:41,332 INFO [main]
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask
> > metrics
> > system...
> > 2017-04-13 22:42:41,333 INFO [main]
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics
> system
> > stopped.
> > 2017-04-13 22:42:41,333 INFO [main]
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics
> system
> > shutdown complete.
> >
> > --
> > View this message in context: http://apache-kylin.74782.x6.
> > nabble.com/Kylin-Build-Cube-failing-with-the-following-error-tp7646.html
> > Sent from the Apache Kylin mailing list archive at Nabble.com.
> >
> >
> > ------------------------------
> > If you reply to this email, your message will be added to the discussion
> > below:
> > http://apache-kylin.74782.x6.nabble.com/Kylin-Build-Cube-
> > failing-with-the-following-error-tp7646p7647.html
> > To unsubscribe from Kylin Build Cube failing with the following error,
> click
> > here
> > <
> > .
> > NAML
> > <http://apache-kylin.74782.x6.nabble.com/template/
> NamlServlet.jtp?macro=macro_viewer&id=instant_html%
> 21nabble%3Aemail.naml&base=nabble.naml.namespaces.
> BasicNamespace-nabble.view.web.template.NabbleNamespace-
> nabble.view.web.template.NodeNamespace&breadcrumbs=
> notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> >
>
>
>
> --
>
>
>
> [image: --]
>
> Ashish Jain
> [image: http://]about.me/ashishjainm
> <http://about.me/ashishjainm?promo=email_sig>
>
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/Kylin-Build-Cube-failing-with-the-following-
> error-tp7646p7653.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-kylin.74782.x6.nabble.com/Kylin-Build-Cube-
> failing-with-the-following-error-tp7646p7656.html
> To unsubscribe from Kylin Build Cube failing with the following error, click
> here
> <http://apache-kylin.74782.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7646&code=YXNoaXNoamFpbm1AZ21haWwuY29tfDc2NDZ8MTk3MTk2MjI2>
> .
> NAML
> <http://apache-kylin.74782.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
[image: --]
Ashish Jain
[image: http://]about.me/ashishjainm
<http://about.me/ashishjainm?promo=email_sig>
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/Kylin-Build-Cube-failing-with-the-following-error-tp7646p7658.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
答复: 答复: Kylin Build Cube failing with the following error
Posted by roger shi <ro...@hotmail.com>.
Yarn resource allocation is not that enough in CDH quickstart version 5.8. Here "enough" doesn't only refer to total resource, but also map and reduce memory configuration. You could have a try to configure them properly.
________________________________
发件人: ashishjainm <as...@gmail.com>
发送时间: 2017年4月14日 16:44:36
收件人: dev@kylin.apache.org
主题: Re: 答复: Kylin Build Cube failing with the following error
Thanks Roger.
Below step is failing:
#3 Step Name: Extract Fact Table Distinct Columns
Duration: 1.62 mins Waiting: 23 seconds
I am running on CDH distribution quickstart vm version 5.8
On Thu, Apr 13, 2017 at 11:57 PM, Roger Shi [via Apache Kylin] <
ml-node+s74782n7647h14@n6.nabble.com> wrote:
> Which step throws this error? From the log, it seems hadoop is opening
> local file. Please check the default FileSystem of HDFS and also check the
> hadoop job's memory configuration. There might be some hint.
> ________________________________
> 发件人: ashishjainm <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=7647&i=0>>
> 发送时间: 2017年4月14日 13:53:58
> 收件人: [hidden email] <http:///user/SendEmail.jtp?type=node&node=7647&i=1>
> 主题: Kylin Build Cube failing with the following error
>
> Hello Team,
>
> I am new to Kylin and just created the sample cube. however, when i create
> a
> model with my own data and this is a very small data set with 10 rows in
> the
> fact table. The process is failing with the following error message. I am
> using the CDH quickstart VM.
>
> May be this is a Yarn related issue and i wanted to know an optimum
> configuration that i need to set for me to be able to build a sample cube
> of
> my own.
>
> 82252_0024_01_000003/meta/cube_desc
> 2017-04-13 22:42:34,902 INFO [main]
> org.apache.kylin.metadata.project.ProjectManager: Initializing
> ProjectManager with metadata url
> /yarn/nm/usercache/root/appcache/application_1492137382252_0024/container_
> 1492137382252_0024_01_000003/meta
> 2017-04-13 22:42:35,008 INFO [main]
> org.apache.kylin.measure.MeasureTypeFactory: Checking custom measure
> types
> from kylin config
> 2017-04-13 22:42:35,011 INFO [main]
> org.apache.kylin.measure.MeasureTypeFactory: registering
> COUNT_DISTINCT(hllc), class
> org.apache.kylin.measure.hllc.HLLCMeasureType$Factory
> 2017-04-13 22:42:35,028 INFO [main]
> org.apache.kylin.measure.MeasureTypeFactory: registering
> COUNT_DISTINCT(bitmap), class
> org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory
> 2017-04-13 22:42:35,038 INFO [main]
> org.apache.kylin.measure.MeasureTypeFactory: registering TOP_N(topn),
> class
> org.apache.kylin.measure.topn.TopNMeasureType$Factory
> 2017-04-13 22:42:35,042 INFO [main]
> org.apache.kylin.measure.MeasureTypeFactory: registering RAW(raw), class
> org.apache.kylin.measure.raw.RawMeasureType$Factory
> 2017-04-13 22:42:35,046 INFO [main]
> org.apache.kylin.measure.MeasureTypeFactory: registering
> EXTENDED_COLUMN(extendedcolumn), class
> org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory
> 2017-04-13 22:42:35,049 INFO [main]
> org.apache.kylin.measure.MeasureTypeFactory: registering
> PERCENTILE(percentile), class
> org.apache.kylin.measure.percentile.PercentileMeasureType$Factory
> 2017-04-13 22:42:35,155 INFO [main] org.apache.kylin.cube.CubeDescManager:
>
> Loaded 1 Cube(s)
> 2017-04-13 22:42:35,159 INFO [main] org.apache.kylin.cube.CubeManager:
> Reloaded cube spend_cube being CUBE[name=spend_cube] having 1 segments
> 2017-04-13 22:42:35,159 INFO [main] org.apache.kylin.cube.CubeManager:
> Loaded 1 cubes, fail on 0 cubes
> 2017-04-13 22:42:38,384 INFO [main] org.apache.hadoop.mapred.MapTask:
> Starting flush of map output
> 2017-04-13 22:42:41,220 FATAL [main] org.apache.hadoop.mapred.YarnChild:
> Error running child : java.lang.OutOfMemoryError: GC overhead limit
> exceeded
> at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300)
>
> at java.lang.StringCoding.encode(StringCoding.java:344)
> at java.lang.String.getBytes(String.java:916)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:216)
>
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
>
> at
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
>
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:295)
>
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:281)
>
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:901)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:863)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1843)
>
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1508)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
>
> at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
>
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
>
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1693)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>
> 2017-04-13 22:42:41,332 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask
> metrics
> system...
> 2017-04-13 22:42:41,333 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
> stopped.
> 2017-04-13 22:42:41,333 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
> shutdown complete.
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/Kylin-Build-Cube-failing-with-the-following-error-tp7646.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-kylin.74782.x6.nabble.com/Kylin-Build-Cube-
> failing-with-the-following-error-tp7646p7647.html
> To unsubscribe from Kylin Build Cube failing with the following error, click
> here
> <http://apache-kylin.74782.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7646&code=YXNoaXNoamFpbm1AZ21haWwuY29tfDc2NDZ8MTk3MTk2MjI2>
> .
> NAML
> <http://apache-kylin.74782.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
[image: --]
Ashish Jain
[image: http://]about.me/ashishjainm
<http://about.me/ashishjainm?promo=email_sig>
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/Kylin-Build-Cube-failing-with-the-following-error-tp7646p7653.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: 答复: Kylin Build Cube failing with the following error
Posted by ashishjainm <as...@gmail.com>.
Thanks Roger.
Below step is failing:
#3 Step Name: Extract Fact Table Distinct Columns
Duration: 1.62 mins Waiting: 23 seconds
I am running on CDH distribution quickstart vm version 5.8
On Thu, Apr 13, 2017 at 11:57 PM, Roger Shi [via Apache Kylin] <
ml-node+s74782n7647h14@n6.nabble.com> wrote:
> Which step throws this error? From the log, it seems hadoop is opening
> local file. Please check the default FileSystem of HDFS and also check the
> hadoop job's memory configuration. There might be some hint.
> ________________________________
> 发件人: ashishjainm <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=7647&i=0>>
> 发送时间: 2017年4月14日 13:53:58
> 收件人: [hidden email] <http:///user/SendEmail.jtp?type=node&node=7647&i=1>
> 主题: Kylin Build Cube failing with the following error
>
> Hello Team,
>
> I am new to Kylin and just created the sample cube. however, when i create
> a
> model with my own data and this is a very small data set with 10 rows in
> the
> fact table. The process is failing with the following error message. I am
> using the CDH quickstart VM.
>
> May be this is a Yarn related issue and i wanted to know an optimum
> configuration that i need to set for me to be able to build a sample cube
> of
> my own.
>
> 82252_0024_01_000003/meta/cube_desc
> 2017-04-13 22:42:34,902 INFO [main]
> org.apache.kylin.metadata.project.ProjectManager: Initializing
> ProjectManager with metadata url
> /yarn/nm/usercache/root/appcache/application_1492137382252_0024/container_
> 1492137382252_0024_01_000003/meta
> 2017-04-13 22:42:35,008 INFO [main]
> org.apache.kylin.measure.MeasureTypeFactory: Checking custom measure
> types
> from kylin config
> 2017-04-13 22:42:35,011 INFO [main]
> org.apache.kylin.measure.MeasureTypeFactory: registering
> COUNT_DISTINCT(hllc), class
> org.apache.kylin.measure.hllc.HLLCMeasureType$Factory
> 2017-04-13 22:42:35,028 INFO [main]
> org.apache.kylin.measure.MeasureTypeFactory: registering
> COUNT_DISTINCT(bitmap), class
> org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory
> 2017-04-13 22:42:35,038 INFO [main]
> org.apache.kylin.measure.MeasureTypeFactory: registering TOP_N(topn),
> class
> org.apache.kylin.measure.topn.TopNMeasureType$Factory
> 2017-04-13 22:42:35,042 INFO [main]
> org.apache.kylin.measure.MeasureTypeFactory: registering RAW(raw), class
> org.apache.kylin.measure.raw.RawMeasureType$Factory
> 2017-04-13 22:42:35,046 INFO [main]
> org.apache.kylin.measure.MeasureTypeFactory: registering
> EXTENDED_COLUMN(extendedcolumn), class
> org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory
> 2017-04-13 22:42:35,049 INFO [main]
> org.apache.kylin.measure.MeasureTypeFactory: registering
> PERCENTILE(percentile), class
> org.apache.kylin.measure.percentile.PercentileMeasureType$Factory
> 2017-04-13 22:42:35,155 INFO [main] org.apache.kylin.cube.CubeDescManager:
>
> Loaded 1 Cube(s)
> 2017-04-13 22:42:35,159 INFO [main] org.apache.kylin.cube.CubeManager:
> Reloaded cube spend_cube being CUBE[name=spend_cube] having 1 segments
> 2017-04-13 22:42:35,159 INFO [main] org.apache.kylin.cube.CubeManager:
> Loaded 1 cubes, fail on 0 cubes
> 2017-04-13 22:42:38,384 INFO [main] org.apache.hadoop.mapred.MapTask:
> Starting flush of map output
> 2017-04-13 22:42:41,220 FATAL [main] org.apache.hadoop.mapred.YarnChild:
> Error running child : java.lang.OutOfMemoryError: GC overhead limit
> exceeded
> at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300)
>
> at java.lang.StringCoding.encode(StringCoding.java:344)
> at java.lang.String.getBytes(String.java:916)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:216)
>
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
>
> at
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
>
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:295)
>
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:281)
>
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:901)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:863)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1843)
>
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1508)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
>
> at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
>
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
>
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1693)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>
> 2017-04-13 22:42:41,332 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask
> metrics
> system...
> 2017-04-13 22:42:41,333 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
> stopped.
> 2017-04-13 22:42:41,333 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
> shutdown complete.
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/Kylin-Build-Cube-failing-with-the-following-error-tp7646.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-kylin.74782.x6.nabble.com/Kylin-Build-Cube-
> failing-with-the-following-error-tp7646p7647.html
> To unsubscribe from Kylin Build Cube failing with the following error, click
> here
> <http://apache-kylin.74782.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7646&code=YXNoaXNoamFpbm1AZ21haWwuY29tfDc2NDZ8MTk3MTk2MjI2>
> .
> NAML
> <http://apache-kylin.74782.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
[image: --]
Ashish Jain
[image: http://]about.me/ashishjainm
<http://about.me/ashishjainm?promo=email_sig>
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/Kylin-Build-Cube-failing-with-the-following-error-tp7646p7653.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
答复: Kylin Build Cube failing with the following error
Posted by roger shi <ro...@hotmail.com>.
Which step throws this error? From the log, it seems hadoop is opening local file. Please check the default FileSystem of HDFS and also check the hadoop job's memory configuration. There might be some hint.
________________________________
发件人: ashishjainm <as...@gmail.com>
发送时间: 2017年4月14日 13:53:58
收件人: dev@kylin.apache.org
主题: Kylin Build Cube failing with the following error
Hello Team,
I am new to Kylin and just created the sample cube. however, when i create a
model with my own data and this is a very small data set with 10 rows in the
fact table. The process is failing with the following error message. I am
using the CDH quickstart VM.
May be this is a Yarn related issue and i wanted to know an optimum
configuration that i need to set for me to be able to build a sample cube of
my own.
82252_0024_01_000003/meta/cube_desc
2017-04-13 22:42:34,902 INFO [main]
org.apache.kylin.metadata.project.ProjectManager: Initializing
ProjectManager with metadata url
/yarn/nm/usercache/root/appcache/application_1492137382252_0024/container_1492137382252_0024_01_000003/meta
2017-04-13 22:42:35,008 INFO [main]
org.apache.kylin.measure.MeasureTypeFactory: Checking custom measure types
from kylin config
2017-04-13 22:42:35,011 INFO [main]
org.apache.kylin.measure.MeasureTypeFactory: registering
COUNT_DISTINCT(hllc), class
org.apache.kylin.measure.hllc.HLLCMeasureType$Factory
2017-04-13 22:42:35,028 INFO [main]
org.apache.kylin.measure.MeasureTypeFactory: registering
COUNT_DISTINCT(bitmap), class
org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory
2017-04-13 22:42:35,038 INFO [main]
org.apache.kylin.measure.MeasureTypeFactory: registering TOP_N(topn), class
org.apache.kylin.measure.topn.TopNMeasureType$Factory
2017-04-13 22:42:35,042 INFO [main]
org.apache.kylin.measure.MeasureTypeFactory: registering RAW(raw), class
org.apache.kylin.measure.raw.RawMeasureType$Factory
2017-04-13 22:42:35,046 INFO [main]
org.apache.kylin.measure.MeasureTypeFactory: registering
EXTENDED_COLUMN(extendedcolumn), class
org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory
2017-04-13 22:42:35,049 INFO [main]
org.apache.kylin.measure.MeasureTypeFactory: registering
PERCENTILE(percentile), class
org.apache.kylin.measure.percentile.PercentileMeasureType$Factory
2017-04-13 22:42:35,155 INFO [main] org.apache.kylin.cube.CubeDescManager:
Loaded 1 Cube(s)
2017-04-13 22:42:35,159 INFO [main] org.apache.kylin.cube.CubeManager:
Reloaded cube spend_cube being CUBE[name=spend_cube] having 1 segments
2017-04-13 22:42:35,159 INFO [main] org.apache.kylin.cube.CubeManager:
Loaded 1 cubes, fail on 0 cubes
2017-04-13 22:42:38,384 INFO [main] org.apache.hadoop.mapred.MapTask:
Starting flush of map output
2017-04-13 22:42:41,220 FATAL [main] org.apache.hadoop.mapred.YarnChild:
Error running child : java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300)
at java.lang.StringCoding.encode(StringCoding.java:344)
at java.lang.String.getBytes(String.java:916)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:216)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
at
org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:295)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:281)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:901)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:863)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1843)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1508)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
2017-04-13 22:42:41,332 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics
system...
2017-04-13 22:42:41,333 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
stopped.
2017-04-13 22:42:41,333 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
shutdown complete.
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/Kylin-Build-Cube-failing-with-the-following-error-tp7646.html
Sent from the Apache Kylin mailing list archive at Nabble.com.