You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by "Jason Wang (WDG)" <ja...@microsoft.com> on 2016/03/09 01:27:03 UTC

java.lang.IllegalArgumentException: Too many digits for NumberDictionary

I hit into an error during cube build. The "Build Dimension Dictionary" step failed after 0.15 minutes.
Any idea?

java.lang.IllegalArgumentException: Too many digits for NumberDictionary: 635696813322678783. Expect 16 digits before decimal point at max.
         at org.apache.kylin.dict.NumberDictionary$NumberBytesCodec.encodeNumber(NumberDictionary.java:78)
         at org.apache.kylin.dict.NumberDictionaryBuilder.addValue(NumberDictionaryBuilder.java:37)
         at org.apache.kylin.dict.TrieDictionaryBuilder.addValue(TrieDictionaryBuilder.java:83)
         at org.apache.kylin.dict.DictionaryGenerator.buildNumberDict(DictionaryGenerator.java:170)
         at org.apache.kylin.dict.DictionaryGenerator.buildDictionaryFromValueEnumerator(DictionaryGenerator.java:67)
         at org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:101)
         at org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.java:211)
         at org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:166)
         at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:52)
         at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:41)
         at org.apache.kylin.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:52)
         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
         at org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
         at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
         at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
         at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
         at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
         at java.lang.Thread.run(Thread.java:745)

result code:2

Re: java.lang.IllegalArgumentException: Too many digits for NumberDictionary

Posted by yu feng <ol...@gmail.com>.
I has encountered this problem before, Now you can modify the column to
string type(by create a view and cast this column) to skip this problem, My
suggestion is try to use TrieDictionary if there is some error while
building NumberDictionary or  DateStrDictionary, This will more robust.

2016-03-09 9:00 GMT+08:00 Jason Wang (WDG) <ja...@microsoft.com>:

> So I traced into the code a little bit and queried my HIVE table.
>
> Turns out one of my dimension table contains a bigint column, and the max
> value of that column is 9223372036854775807, which is the upper bound of
> bigint type. I guess they must have used this value to indicate a special
> case. But regardless, the NumberDictionary class should be able to handle
> bigint right?
>
>
>
>
>
> *From:* Jason Wang (WDG) [mailto:jasowang@microsoft.com]
> *Sent:* Tuesday, March 8, 2016 4:27 PM
> *To:* user@kylin.apache.org
> *Subject:* java.lang.IllegalArgumentException: Too many digits for
> NumberDictionary
>
>
>
> I hit into an error during cube build. The “Build Dimension Dictionary”
> step failed after 0.15 minutes.
>
> Any idea?
>
>
>
> java.lang.IllegalArgumentException: Too many digits for NumberDictionary:
> 635696813322678783. Expect 16 digits before decimal point at max.
>
>          at
> org.apache.kylin.dict.NumberDictionary$NumberBytesCodec.encodeNumber(NumberDictionary.java:78)
>
>          at
> org.apache.kylin.dict.NumberDictionaryBuilder.addValue(NumberDictionaryBuilder.java:37)
>
>          at
> org.apache.kylin.dict.TrieDictionaryBuilder.addValue(TrieDictionaryBuilder.java:83)
>
>          at
> org.apache.kylin.dict.DictionaryGenerator.buildNumberDict(DictionaryGenerator.java:170)
>
>          at
> org.apache.kylin.dict.DictionaryGenerator.buildDictionaryFromValueEnumerator(DictionaryGenerator.java:67)
>
>          at
> org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:101)
>
>          at
> org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.java:211)
>
>          at
> org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:166)
>
>          at
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:52)
>
>          at
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:41)
>
>          at
> org.apache.kylin.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:52)
>
>          at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>
>          at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
>
>          at
> org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
>
>          at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
>
>          at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
>
>          at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
>
>          at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
>
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
>          at java.lang.Thread.run(Thread.java:745)
>
>
>
> result code:2
>
>
>

Re: java.lang.IllegalArgumentException: Too many digits for NumberDictionary

Posted by "Jason Wang (WDG)" <ja...@microsoft.com>.
https://issues.apache.org/jira/browse/KYLIN-1486


Sent from Outlook<http://aka.ms/weboutlook>

________________________________
From: Li Yang <li...@apache.org>
Sent: Tuesday, March 8, 2016 10:39:04 PM
To: user@kylin.apache.org
Subject: Re: java.lang.IllegalArgumentException: Too many digits for NumberDictionary

Agree. NumberDictionary should handle this case. Could you open a JIRA to track?

On Wed, Mar 9, 2016 at 9:00 AM, Jason Wang (WDG) <ja...@microsoft.com>> wrote:
So I traced into the code a little bit and queried my HIVE table.
Turns out one of my dimension table contains a bigint column, and the max value of that column is 9223372036854775807, which is the upper bound of bigint type. I guess they must have used this value to indicate a special case. But regardless, the NumberDictionary class should be able to handle bigint right?


From: Jason Wang (WDG) [mailto:jasowang@microsoft.com<ma...@microsoft.com>]
Sent: Tuesday, March 8, 2016 4:27 PM
To: user@kylin.apache.org<ma...@kylin.apache.org>
Subject: java.lang.IllegalArgumentException: Too many digits for NumberDictionary

I hit into an error during cube build. The "Build Dimension Dictionary" step failed after 0.15 minutes.
Any idea?

java.lang.IllegalArgumentException: Too many digits for NumberDictionary: 635696813322678783. Expect 16 digits before decimal point at max.
         at org.apache.kylin.dict.NumberDictionary$NumberBytesCodec.encodeNumber(NumberDictionary.java:78)
         at org.apache.kylin.dict.NumberDictionaryBuilder.addValue(NumberDictionaryBuilder.java:37)
         at org.apache.kylin.dict.TrieDictionaryBuilder.addValue(TrieDictionaryBuilder.java:83)
         at org.apache.kylin.dict.DictionaryGenerator.buildNumberDict(DictionaryGenerator.java:170)
         at org.apache.kylin.dict.DictionaryGenerator.buildDictionaryFromValueEnumerator(DictionaryGenerator.java:67)
         at org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:101)
         at org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.java:211)
         at org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:166)
         at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:52)
         at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:41)
         at org.apache.kylin.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:52)
         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
         at org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
         at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
         at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
         at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
         at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
         at java.lang.Thread.run(Thread.java:745)

result code:2

Re: java.lang.IllegalArgumentException: Too many digits for NumberDictionary

Posted by Li Yang <li...@apache.org>.
Agree. NumberDictionary should handle this case. Could you open a JIRA to
track?

On Wed, Mar 9, 2016 at 9:00 AM, Jason Wang (WDG) <ja...@microsoft.com>
wrote:

> So I traced into the code a little bit and queried my HIVE table.
>
> Turns out one of my dimension table contains a bigint column, and the max
> value of that column is 9223372036854775807, which is the upper bound of
> bigint type. I guess they must have used this value to indicate a special
> case. But regardless, the NumberDictionary class should be able to handle
> bigint right?
>
>
>
>
>
> *From:* Jason Wang (WDG) [mailto:jasowang@microsoft.com]
> *Sent:* Tuesday, March 8, 2016 4:27 PM
> *To:* user@kylin.apache.org
> *Subject:* java.lang.IllegalArgumentException: Too many digits for
> NumberDictionary
>
>
>
> I hit into an error during cube build. The “Build Dimension Dictionary”
> step failed after 0.15 minutes.
>
> Any idea?
>
>
>
> java.lang.IllegalArgumentException: Too many digits for NumberDictionary:
> 635696813322678783. Expect 16 digits before decimal point at max.
>
>          at
> org.apache.kylin.dict.NumberDictionary$NumberBytesCodec.encodeNumber(NumberDictionary.java:78)
>
>          at
> org.apache.kylin.dict.NumberDictionaryBuilder.addValue(NumberDictionaryBuilder.java:37)
>
>          at
> org.apache.kylin.dict.TrieDictionaryBuilder.addValue(TrieDictionaryBuilder.java:83)
>
>          at
> org.apache.kylin.dict.DictionaryGenerator.buildNumberDict(DictionaryGenerator.java:170)
>
>          at
> org.apache.kylin.dict.DictionaryGenerator.buildDictionaryFromValueEnumerator(DictionaryGenerator.java:67)
>
>          at
> org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:101)
>
>          at
> org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.java:211)
>
>          at
> org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:166)
>
>          at
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:52)
>
>          at
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:41)
>
>          at
> org.apache.kylin.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:52)
>
>          at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>
>          at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
>
>          at
> org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
>
>          at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
>
>          at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
>
>          at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
>
>          at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
>
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
>          at java.lang.Thread.run(Thread.java:745)
>
>
>
> result code:2
>
>
>

RE: java.lang.IllegalArgumentException: Too many digits for NumberDictionary

Posted by "Jason Wang (WDG)" <ja...@microsoft.com>.
So I traced into the code a little bit and queried my HIVE table.
Turns out one of my dimension table contains a bigint column, and the max value of that column is 9223372036854775807, which is the upper bound of bigint type. I guess they must have used this value to indicate a special case. But regardless, the NumberDictionary class should be able to handle bigint right?


From: Jason Wang (WDG) [mailto:jasowang@microsoft.com]
Sent: Tuesday, March 8, 2016 4:27 PM
To: user@kylin.apache.org
Subject: java.lang.IllegalArgumentException: Too many digits for NumberDictionary

I hit into an error during cube build. The "Build Dimension Dictionary" step failed after 0.15 minutes.
Any idea?

java.lang.IllegalArgumentException: Too many digits for NumberDictionary: 635696813322678783. Expect 16 digits before decimal point at max.
         at org.apache.kylin.dict.NumberDictionary$NumberBytesCodec.encodeNumber(NumberDictionary.java:78)
         at org.apache.kylin.dict.NumberDictionaryBuilder.addValue(NumberDictionaryBuilder.java:37)
         at org.apache.kylin.dict.TrieDictionaryBuilder.addValue(TrieDictionaryBuilder.java:83)
         at org.apache.kylin.dict.DictionaryGenerator.buildNumberDict(DictionaryGenerator.java:170)
         at org.apache.kylin.dict.DictionaryGenerator.buildDictionaryFromValueEnumerator(DictionaryGenerator.java:67)
         at org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:101)
         at org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.java:211)
         at org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:166)
         at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:52)
         at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:41)
         at org.apache.kylin.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:52)
         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
         at org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
         at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
         at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
         at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
         at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
         at java.lang.Thread.run(Thread.java:745)

result code:2