You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by skyyws <sk...@163.com> on 2018/05/24 12:58:42 UTC
NegativeArraySizeException on "Extract Fact Table Distinct Columns" step
Hello guys,
I used kylin 2.0.0-hbase 1.x to build a cube, and I met the error like this on step “Extract Fact Table Distinct Columns", it occured at one of the reduce task:
----------------------------------------------------------------------------------------------------------------------------------------------------------
Error: java.lang.NegativeArraySizeException at org.apache.commons.io.output.ByteArrayOutputStream.needNewBuffer(ByteArrayOutputStream.java:122)
at org.apache.commons.io.output.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
at org.apache.kylin.dict.TrieDictionary.write(TrieDictionary.java:315)
at org.apache.kylin.dict.TrieDictionaryForest.writeBody(TrieDictionaryForest.java:201)
at org.apache.kylin.dict.TrieDictionaryForest.write(TrieDictionaryForest.java:169)
at org.apache.kylin.engine.mr.steps.FactDistinctColumnsReducer.outputDict(FactDistinctColumnsReducer.java:256)
at org.apache.kylin.engine.mr.steps.FactDistinctColumnsReducer.doCleanup(FactDistinctColumnsReducer.java:232)
at org.apache.kylin.engine.mr.KylinReducer.cleanup(KylinReducer.java:71)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:179)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143
----------------------------------------------------------------------------------------------------------------------------------------------------------
I'm not sure, it there any probelm with my raw data or my cube model? Anyone who can give me some advice, thanks a lot!
2018-05-24
Wang
Re: NegativeArraySizeException on "Extract Fact Table Distinct
Columns" step
Posted by Li Yang <li...@apache.org>.
This is a known issue https://issues.apache.org/jira/browse/KYLIN-2662 .
Will fix it very soon.
However the fix won't help too much as it will just throw a clearer error
message and will not let dictionary build succeed.
The root cause is Kylin does not allow any dictionary go beyond 2GB. So for
a high cardinality column which does not fit in
a dictionary, you must use a different encoding like "integer" or
"fixed_length".
Cheers
Yang
On Fri, May 25, 2018 at 6:30 AM, Ruslan Dautkhanov <da...@gmail.com>
wrote:
> 143 can also simply mean yarn container was preempted where another user's
> job started so your fair share has dropped
> to give more resources to another resource pool.
>
>
>
> --
> Ruslan Dautkhanov
>
> On Thu, May 24, 2018 at 4:24 PM, Alberto Ramón <a....@gmail.com>
> wrote:
>
>> The YANR error 143 is very common, you are using more resources than you
>> configured in YARN
>> You will find a lot of doc on Internet
>>
>> On 24 May 2018 at 13:58, skyyws <sk...@163.com> wrote:
>>
>>> Hello guys,
>>> I used kylin 2.0.0-hbase 1.x to build a cube, and I met the error like
>>> this on step “*Extract Fact Table Distinct Columns*", it occured at one
>>> of the reduce task:
>>> ------------------------------------------------------------
>>> ------------------------------------------------------------
>>> ----------------------------------
>>> Error: java.lang.NegativeArraySizeException at
>>> org.apache.commons.io.output.ByteArrayOutputStream.needNewBu
>>> ffer(ByteArrayOutputStream.java:122)
>>> at org.apache.commons.io.output.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
>>>
>>> at java.io.DataOutputStream.write(DataOutputStream.java:107)
>>> at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
>>> at org.apache.kylin.dict.TrieDictionary.write(TrieDictionary.java:315)
>>> at org.apache.kylin.dict.TrieDictionaryForest.writeBody(TrieDic
>>> tionaryForest.java:201)
>>> at org.apache.kylin.dict.TrieDictionaryForest.write(TrieDictionaryForest.java:169)
>>>
>>> at org.apache.kylin.engine.mr.steps.FactDistinctColumnsReducer.
>>> outputDict(FactDistinctColumnsReducer.java:256)
>>> at org.apache.kylin.engine.mr.steps.FactDistinctColumnsReducer.
>>> doCleanup(FactDistinctColumnsReducer.java:232)
>>> at org.apache.kylin.engine.mr.KylinReducer.cleanup(KylinReducer.java:71)
>>>
>>> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:179)
>>> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
>>>
>>> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>>>
>>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
>>> Container killed by the ApplicationMaster. Container killed on request.
>>> Exit code is 143 Container exited with a non-zero exit code 143
>>> ------------------------------------------------------------
>>> ------------------------------------------------------------
>>> ----------------------------------
>>> I'm not sure, it there any probelm with my raw data or my cube model?
>>> Anyone who can give me some advice, thanks a lot!
>>> 2018-05-24
>>> ------------------------------
>>> Wang
>>>
>>
>>
>
Re: NegativeArraySizeException on "Extract Fact Table Distinct
Columns" step
Posted by Ruslan Dautkhanov <da...@gmail.com>.
143 can also simply mean yarn container was preempted where another user's
job started so your fair share has dropped
to give more resources to another resource pool.
--
Ruslan Dautkhanov
On Thu, May 24, 2018 at 4:24 PM, Alberto Ramón <a....@gmail.com>
wrote:
> The YANR error 143 is very common, you are using more resources than you
> configured in YARN
> You will find a lot of doc on Internet
>
> On 24 May 2018 at 13:58, skyyws <sk...@163.com> wrote:
>
>> Hello guys,
>> I used kylin 2.0.0-hbase 1.x to build a cube, and I met the error like
>> this on step “*Extract Fact Table Distinct Columns*", it occured at one
>> of the reduce task:
>> ------------------------------------------------------------
>> ------------------------------------------------------------
>> ----------------------------------
>> Error: java.lang.NegativeArraySizeException at
>> org.apache.commons.io.output.ByteArrayOutputStream.needNewBu
>> ffer(ByteArrayOutputStream.java:122)
>> at org.apache.commons.io.output.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
>>
>> at java.io.DataOutputStream.write(DataOutputStream.java:107)
>> at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
>> at org.apache.kylin.dict.TrieDictionary.write(TrieDictionary.java:315)
>> at org.apache.kylin.dict.TrieDictionaryForest.writeBody(TrieDic
>> tionaryForest.java:201)
>> at org.apache.kylin.dict.TrieDictionaryForest.write(TrieDictionaryForest.java:169)
>>
>> at org.apache.kylin.engine.mr.steps.FactDistinctColumnsReducer.
>> outputDict(FactDistinctColumnsReducer.java:256)
>> at org.apache.kylin.engine.mr.steps.FactDistinctColumnsReducer.
>> doCleanup(FactDistinctColumnsReducer.java:232)
>> at org.apache.kylin.engine.mr.KylinReducer.cleanup(KylinReducer.java:71)
>> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:179)
>> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
>>
>> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>>
>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
>> Container killed by the ApplicationMaster. Container killed on request.
>> Exit code is 143 Container exited with a non-zero exit code 143
>> ------------------------------------------------------------
>> ------------------------------------------------------------
>> ----------------------------------
>> I'm not sure, it there any probelm with my raw data or my cube model?
>> Anyone who can give me some advice, thanks a lot!
>> 2018-05-24
>> ------------------------------
>> Wang
>>
>
>
Re: NegativeArraySizeException on "Extract Fact Table Distinct
Columns" step
Posted by Alberto Ramón <a....@gmail.com>.
The YANR error 143 is very common, you are using more resources than you
configured in YARN
You will find a lot of doc on Internet
On 24 May 2018 at 13:58, skyyws <sk...@163.com> wrote:
> Hello guys,
> I used kylin 2.0.0-hbase 1.x to build a cube, and I met the error like
> this on step “*Extract Fact Table Distinct Columns*", it occured at one
> of the reduce task:
> ------------------------------------------------------------
> ------------------------------------------------------------
> ----------------------------------
> Error: java.lang.NegativeArraySizeException at
> org.apache.commons.io.output.ByteArrayOutputStream.needNewBuffer(
> ByteArrayOutputStream.java:122)
> at org.apache.commons.io.output.ByteArrayOutputStream.write(
> ByteArrayOutputStream.java:153)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
> at org.apache.kylin.dict.TrieDictionary.write(TrieDictionary.java:315)
> at org.apache.kylin.dict.TrieDictionaryForest.writeBody(
> TrieDictionaryForest.java:201)
> at org.apache.kylin.dict.TrieDictionaryForest.write(TrieDictionaryForest.java:169)
>
> at org.apache.kylin.engine.mr.steps.FactDistinctColumnsReducer.outputDict(
> FactDistinctColumnsReducer.java:256)
> at org.apache.kylin.engine.mr.steps.FactDistinctColumnsReducer.doCleanup(
> FactDistinctColumnsReducer.java:232)
> at org.apache.kylin.engine.mr.KylinReducer.cleanup(KylinReducer.java:71)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:179)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1698)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Container killed by the ApplicationMaster. Container killed on request.
> Exit code is 143 Container exited with a non-zero exit code 143
> ------------------------------------------------------------
> ------------------------------------------------------------
> ----------------------------------
> I'm not sure, it there any probelm with my raw data or my cube model?
> Anyone who can give me some advice, thanks a lot!
> 2018-05-24
> ------------------------------
> Wang
>