You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by Benjamin Leonhardi <bl...@hortonworks.com> on 2015/10/01 07:30:13 UTC
commons-codec problem with HDP 2.3 and HBase1.x branch
Hello All,
I try to get Kylin to run with HDP 2.3 sandbox image. Using the HBase1.x branch.
https://github.com/apache/incubator-kylin/tree/1.x-HBase1.x
It compiles fine and the tests run through until they reach the job creation. I followed the steps below.
http://kylin.incubator.apache.org/development/dev_env.html
When I run the tests he reaches the job part and then fails because he misses a method in commons-codec.jar. The method is encodeBase64(byte[], isChunk boolean, isUrlSafe boolean) and it is in commons-codec1.4+ but not in older versions. I checked but my environment only has codec jars that are newer.
However when I build kylin he for some reason compiled an older codec into the kylin-job-1.1-incubating-SNAPSHOT-job.jar. If I unzip this I see a Base64 class that only has the method encodeBase64(byte[], isChunk boolean ) but not the urlsafe method. ( so I suppose codec 1.3 or older ) I tried to check the pom.xml because I thought I might find an older codec dependency inside but did not find anything.
I also have to say I am not the best Maven person so I might be overlooking something easy. One question I would have is why he compiles the code into the -job.jar at all.
Any help welcome
Best regards,
Benjamin
Starting: Kylin_Fact_Distinct_Columns_test_kylin_cube_without_slr_empty_Step
L4J [2015-10-01 03:51:24,166][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - append job jar: /root/kylin_build/incubator-kylin/job/../job/target/kylin-job-1.1-incu
bating-SNAPSHOT-job.jar
L4J [2015-10-01 03:51:24,169][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - append kylin.hive.dependency: null and kylin.hbase.dependency: null to mapreduce.appli
cation.classpath
L4J [2015-10-01 03:51:24,167][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - append job jar: /root/kylin_build/incubator-kylin/job/../job/target/kylin-job-1.1-incu
bating-SNAPSHOT-job.jar
L4J [2015-10-01 03:51:24,170][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - append kylin.hive.dependency: null and kylin.hbase.dependency: null to mapreduce.appli
cation.classpath
L4J [2015-10-01 03:51:24,170][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - Hadoop job classpath is: /tmp/kylin/*,$HADOOP_CONF_DIR,/usr/hdp/2.3.0.0-2530/hbase/lib/hbase-common.jar,/usr/hdp/current/hive-client/conf/,/usr/hdp/2.3.0.0-2530/hive/lib/hive-metastore.jar,/usr/hdp/2.3.0.0-2530/hive/lib/hive-exec.jar,/usr/hdp/2.3.0.0-2530/hive-hcatalog/share/hcatalog/*,$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/2.3.0.0-2530/hadoop/lib/hadoop-lzo-0.6.0.2.3.0.0-2530.jar:/etc/hadoop/conf/secure
L4J [2015-10-01 03:51:24,169][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - Hadoop job classpath is: /tmp/kylin/*,$HADOOP_CONF_DIR,/usr/hdp/2.3.0.0-2530/hbase/lib/hbase-common.jar,/usr/hdp/current/hive-client/conf/,/usr/hdp/2.3.0.0-2530/hive/lib/hive-metastore.jar,/usr/hdp/2.3.0.0-2530/hive/lib/hive-exec.jar,/usr/hdp/2.3.0.0-2530/hive-hcatalog/share/hcatalog/*,$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/2.3.0.0-2530/hadoop/lib/hadoop-lzo-0.6.0.2.3.0.0-2530.jar:/etc/hadoop/conf/secure
L4J [2015-10-01 03:51:24,229][WARN][org.apache.hadoop.hive.conf.HiveConf] - HiveConf of name hive.heapsize does not exist
L4J [2015-10-01 03:51:24,230][WARN][org.apache.hadoop.hive.conf.HiveConf] - HiveConf of name hive.server2.enable.impersonation does not exist
L4J [2015-10-01 03:51:24,236][WARN][org.apache.hadoop.hive.conf.HiveConf] - HiveConf of name hive.heapsize does not exist
L4J [2015-10-01 03:51:24,237][WARN][org.apache.hadoop.hive.conf.HiveConf] - HiveConf of name hive.server2.enable.impersonation does not exist
L4J [2015-10-01 03:51:24,265][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - tempMetaFileString is : null
L4J [2015-10-01 03:51:24,266][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - tempMetaFileString is : null
L4J [2015-10-01 03:51:24,267][ERROR][org.apache.kylin.job.execution.AbstractExecutable] - error running Executable
java.lang.NoSuchMethodError: org.apache.commons.codec.binary.Base64.encodeBase64([BZZ)[B
at org.apache.hive.hcatalog.common.HCatUtil.encodeBytes(HCatUtil.java:125)
at org.apache.hive.hcatalog.common.HCatUtil.serialize(HCatUtil.java:104)
at org.apache.hive.hcatalog.common.HCatUtil.getHiveConf(HCatUtil.java:585)
at org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:100)
at org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
at org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:101)
at org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:77)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
L4J [2015-10-01 03:51:24,267][ERROR][org.apache.kylin.job.execution.AbstractExecutable] - error running Executable
java.lang.NoSuchMethodError: org.apache.commons.codec.binary.Base64.encodeBase64([BZZ)[B
at org.apache.hive.hcatalog.common.HCatUtil.encodeBytes(HCatUtil.java:125)
at org.apache.hive.hcatalog.common.HCatUtil.serialize(HCatUtil.java:104)
at org.apache.hive.hcatalog.common.HCatUtil.getHiveConf(HCatUtil.java:585)
at org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:100)
at org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
at org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:101)
at org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:77)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Re: commons-codec problem with HDP 2.3 and HBase1.x branch
Posted by Benjamin Leonhardi <bl...@hortonworks.com>.
Hello Everybody,
We figured out the 1.2 coded comes from the first dependency. Will try and update.
Thanks for your help.
Ben
On 10/6/15, 12:47 AM, "周千昊" <qh...@apache.org> wrote:
>Hi, Benjamin,
> It is a bug, a jira ticket is created as follow:
>https://issues.apache.org/jira/browse/KYLIN-1059
> and it should have been fixed. since I don't have the hdp 2.3 env
>currently, it would be appreciated if you can help to verify with the
>latest code(branch 1.x-HBase1.x)
>
>Luke Han <lu...@gmail.com>于2015年10月5日周一 下午8:35写道：
>
>> Hi Benjamin,
>> Did you generate binary package by yourself with below instruction? or
>> just compile war/jar from maven?
>>
>> http://kylin.incubator.apache.org/development/howto_package.html
>> <http://kylin.incubator.apache.org/development/howto_package.html>
>>
>> Thanks.
>>
>>
>> Best Regards!
>> ---------------------
>>
>> Luke Han
>>
>> On Thu, Oct 1, 2015 at 1:30 PM, Benjamin Leonhardi <
>> bleonhardi@hortonworks.com> wrote:
>>
>> > Hello All,
>> >
>> > I try to get Kylin to run with HDP 2.3 sandbox image. Using the HBase1.x
>> > branch.
>> >
>> > https://github.com/apache/incubator-kylin/tree/1.x-HBase1.x
>> >
>> > It compiles fine and the tests run through until they reach the job
>> > creation. I followed the steps below.
>> >
>> > http://kylin.incubator.apache.org/development/dev_env.html
>> >
>> > When I run the tests he reaches the job part and then fails because he
>> > misses a method in commons-codec.jar. The method is encodeBase64(byte[],
>> > isChunk boolean, isUrlSafe boolean) and it is in commons-codec1.4+ but
>> not
>> > in older versions. I checked but my environment only has codec jars that
>> > are newer.
>> >
>> > However when I build kylin he for some reason compiled an older codec
>> into
>> > the kylin-job-1.1-incubating-SNAPSHOT-job.jar. If I unzip this I see a
>> > Base64 class that only has the method encodeBase64(byte[], isChunk
>> boolean
>> > ) but not the urlsafe method. ( so I suppose codec 1.3 or older ) I
>> tried
>> > to check the pom.xml because I thought I might find an older codec
>> > dependency inside but did not find anything.
>> >
>> > I also have to say I am not the best Maven person so I might be
>> > overlooking something easy. One question I would have is why he compiles
>> > the code into the -job.jar at all.
>> >
>> >
>> > Any help welcome
>> >
>> > Best regards,
>> >
>> > Benjamin
>> >
>> >
>> > Starting:
>> > Kylin_Fact_Distinct_Columns_test_kylin_cube_without_slr_empty_Step
>> > L4J [2015-10-01
>> > 03:51:24,166][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
>> append
>> > job jar:
>> > /root/kylin_build/incubator-kylin/job/../job/target/kylin-job-1.1-incu
>> > bating-SNAPSHOT-job.jar
>> > L4J [2015-10-01
>> > 03:51:24,169][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
>> append
>> > kylin.hive.dependency: null and kylin.hbase.dependency: null to
>> > mapreduce.appli
>> > cation.classpath
>> > L4J [2015-10-01
>> > 03:51:24,167][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
>> append
>> > job jar:
>> > /root/kylin_build/incubator-kylin/job/../job/target/kylin-job-1.1-incu
>> > bating-SNAPSHOT-job.jar
>> > L4J [2015-10-01
>> > 03:51:24,170][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
>> append
>> > kylin.hive.dependency: null and kylin.hbase.dependency: null to
>> > mapreduce.appli
>> > cation.classpath
>> > L4J [2015-10-01
>> > 03:51:24,170][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
>> Hadoop
>> > job classpath is:
>> >
>> /tmp/kylin/*,$HADOOP_CONF_DIR,/usr/hdp/2.3.0.0-2530/hbase/lib/hbase-common.jar,/usr/hdp/current/hive-client/conf/,/usr/hdp/2.3.0.0-2530/hive/lib/hive-metastore.jar,/usr/hdp/2.3.0.0-2530/hive/lib/hive-exec.jar,/usr/hdp/2.3.0.0-2530/hive-hcatalog/share/hcatalog/*,$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/2.3.0.0-2530/hadoop/lib/hadoop-lzo-0.6.0.2.3.0.0-2530.jar:/etc/hadoop/conf/secure
>> > L4J [2015-10-01
>> > 03:51:24,169][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
>> Hadoop
>> > job classpath is:
>> >
>> /tmp/kylin/*,$HADOOP_CONF_DIR,/usr/hdp/2.3.0.0-2530/hbase/lib/hbase-common.jar,/usr/hdp/current/hive-client/conf/,/usr/hdp/2.3.0.0-2530/hive/lib/hive-metastore.jar,/usr/hdp/2.3.0.0-2530/hive/lib/hive-exec.jar,/usr/hdp/2.3.0.0-2530/hive-hcatalog/share/hcatalog/*,$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/2.3.0.0-2530/hadoop/lib/hadoop-lzo-0.6.0.2.3.0.0-2530.jar:/etc/hadoop/conf/secure
>> > L4J [2015-10-01 03:51:24,229][WARN][org.apache.hadoop.hive.conf.HiveConf]
>> > - HiveConf of name hive.heapsize does not exist
>> > L4J [2015-10-01 03:51:24,230][WARN][org.apache.hadoop.hive.conf.HiveConf]
>> > - HiveConf of name hive.server2.enable.impersonation does not exist
>> > L4J [2015-10-01 03:51:24,236][WARN][org.apache.hadoop.hive.conf.HiveConf]
>> > - HiveConf of name hive.heapsize does not exist
>> > L4J [2015-10-01 03:51:24,237][WARN][org.apache.hadoop.hive.conf.HiveConf]
>> > - HiveConf of name hive.server2.enable.impersonation does not exist
>> > L4J [2015-10-01
>> > 03:51:24,265][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
>> > tempMetaFileString is : null
>> > L4J [2015-10-01
>> > 03:51:24,266][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
>> > tempMetaFileString is : null
>> > L4J [2015-10-01
>> > 03:51:24,267][ERROR][org.apache.kylin.job.execution.AbstractExecutable] -
>> > error running Executable
>> > java.lang.NoSuchMethodError:
>> > org.apache.commons.codec.binary.Base64.encodeBase64([BZZ)[B
>> > at
>> > org.apache.hive.hcatalog.common.HCatUtil.encodeBytes(HCatUtil.java:125)
>> > at
>> > org.apache.hive.hcatalog.common.HCatUtil.serialize(HCatUtil.java:104)
>> > at
>> > org.apache.hive.hcatalog.common.HCatUtil.getHiveConf(HCatUtil.java:585)
>> > at
>> >
>> org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:100)
>> > at
>> >
>> org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
>> > at
>> >
>> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
>> > at
>> >
>> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
>> > at
>> >
>> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:101)
>> > at
>> >
>> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:77)
>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>> > at
>> >
>> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113)
>> > at
>> >
>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
>> > at
>> >
>> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
>> > at
>> >
>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
>> > at
>> >
>> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> > at java.lang.Thread.run(Thread.java:745)
>> > L4J [2015-10-01
>> > 03:51:24,267][ERROR][org.apache.kylin.job.execution.AbstractExecutable] -
>> > error running Executable
>> > java.lang.NoSuchMethodError:
>> > org.apache.commons.codec.binary.Base64.encodeBase64([BZZ)[B
>> > at
>> > org.apache.hive.hcatalog.common.HCatUtil.encodeBytes(HCatUtil.java:125)
>> > at
>> > org.apache.hive.hcatalog.common.HCatUtil.serialize(HCatUtil.java:104)
>> > at
>> > org.apache.hive.hcatalog.common.HCatUtil.getHiveConf(HCatUtil.java:585)
>> > at
>> >
>> org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:100)
>> > at
>> >
>> org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
>> > at
>> >
>> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
>> > at
>> >
>> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
>> > at
>> >
>> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:101)
>> > at
>> >
>> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:77)
>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>> > at
>> >
>> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113)
>> > at
>> >
>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
>> > at
>> >
>> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
>> > at
>> >
>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
>> > at
>> >
>> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>>
Re: commons-codec problem with HDP 2.3 and HBase1.x branch
Posted by 周千昊 <qh...@apache.org>.
it seems yang has already ported changes to the 1.x-staging branch, thanks
yang.
周千昊 <qh...@apache.org>于2015年10月6日周二 下午7:26写道：
> OK, I will re-port the changes to 1.x-staging
>
> Li Yang <li...@apache.org>于2015年10月6日周二 下午3:28写道：
>
>> Qianhao, pls don't patch 1.x-HBase1.x directly. All changes should go into
>> 1.x-staging first, then apply an HBase patch to become 1.x-HBase1.x
>>
>> Ported the fix to 1.x-staging and promoted 1.x-HBase1.x branch again. The
>> result should be same. The effective commons-codec is 1.9 as shown in
>> eclipse.
>>
>> On Mon, Oct 5, 2015 at 11:47 PM, 周千昊 <qh...@apache.org> wrote:
>>
>> > Hi, Benjamin,
>> > It is a bug, a jira ticket is created as follow:
>> > https://issues.apache.org/jira/browse/KYLIN-1059
>> > and it should have been fixed. since I don't have the hdp 2.3 env
>> > currently, it would be appreciated if you can help to verify with the
>> > latest code(branch 1.x-HBase1.x)
>> >
>> > Luke Han <lu...@gmail.com>于2015年10月5日周一 下午8:35写道：
>> >
>> > > Hi Benjamin,
>> > > Did you generate binary package by yourself with below
>> instruction?
>> > or
>> > > just compile war/jar from maven?
>> > >
>> > > http://kylin.incubator.apache.org/development/howto_package.html
>> > > <http://kylin.incubator.apache.org/development/howto_package.html>
>> > >
>> > > Thanks.
>> > >
>> > >
>> > > Best Regards!
>> > > ---------------------
>> > >
>> > > Luke Han
>> > >
>> > > On Thu, Oct 1, 2015 at 1:30 PM, Benjamin Leonhardi <
>> > > bleonhardi@hortonworks.com> wrote:
>> > >
>> > > > Hello All,
>> > > >
>> > > > I try to get Kylin to run with HDP 2.3 sandbox image. Using the
>> > HBase1.x
>> > > > branch.
>> > > >
>> > > > https://github.com/apache/incubator-kylin/tree/1.x-HBase1.x
>> > > >
>> > > > It compiles fine and the tests run through until they reach the job
>> > > > creation. I followed the steps below.
>> > > >
>> > > > http://kylin.incubator.apache.org/development/dev_env.html
>> > > >
>> > > > When I run the tests he reaches the job part and then fails because
>> he
>> > > > misses a method in commons-codec.jar. The method is
>> > encodeBase64(byte[],
>> > > > isChunk boolean, isUrlSafe boolean) and it is in commons-codec1.4+
>> but
>> > > not
>> > > > in older versions. I checked but my environment only has codec jars
>> > that
>> > > > are newer.
>> > > >
>> > > > However when I build kylin he for some reason compiled an older
>> codec
>> > > into
>> > > > the kylin-job-1.1-incubating-SNAPSHOT-job.jar. If I unzip this I
>> see a
>> > > > Base64 class that only has the method encodeBase64(byte[], isChunk
>> > > boolean
>> > > > ) but not the urlsafe method. ( so I suppose codec 1.3 or older ) I
>> > > tried
>> > > > to check the pom.xml because I thought I might find an older codec
>> > > > dependency inside but did not find anything.
>> > > >
>> > > > I also have to say I am not the best Maven person so I might be
>> > > > overlooking something easy. One question I would have is why he
>> > compiles
>> > > > the code into the -job.jar at all.
>> > > >
>> > > >
>> > > > Any help welcome
>> > > >
>> > > > Best regards,
>> > > >
>> > > > Benjamin
>> > > >
>> > > >
>> > > > Starting:
>> > > > Kylin_Fact_Distinct_Columns_test_kylin_cube_without_slr_empty_Step
>> > > > L4J [2015-10-01
>> > > > 03:51:24,166][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
>> > > append
>> > > > job jar:
>> > > >
>> /root/kylin_build/incubator-kylin/job/../job/target/kylin-job-1.1-incu
>> > > > bating-SNAPSHOT-job.jar
>> > > > L4J [2015-10-01
>> > > > 03:51:24,169][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
>> > > append
>> > > > kylin.hive.dependency: null and kylin.hbase.dependency: null to
>> > > > mapreduce.appli
>> > > > cation.classpath
>> > > > L4J [2015-10-01
>> > > > 03:51:24,167][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
>> > > append
>> > > > job jar:
>> > > >
>> /root/kylin_build/incubator-kylin/job/../job/target/kylin-job-1.1-incu
>> > > > bating-SNAPSHOT-job.jar
>> > > > L4J [2015-10-01
>> > > > 03:51:24,170][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
>> > > append
>> > > > kylin.hive.dependency: null and kylin.hbase.dependency: null to
>> > > > mapreduce.appli
>> > > > cation.classpath
>> > > > L4J [2015-10-01
>> > > > 03:51:24,170][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
>> > > Hadoop
>> > > > job classpath is:
>> > > >
>> > >
>> >
>> /tmp/kylin/*,$HADOOP_CONF_DIR,/usr/hdp/2.3.0.0-2530/hbase/lib/hbase-common.jar,/usr/hdp/current/hive-client/conf/,/usr/hdp/2.3.0.0-2530/hive/lib/hive-metastore.jar,/usr/hdp/2.3.0.0-2530/hive/lib/hive-exec.jar,/usr/hdp/2.3.0.0-2530/hive-hcatalog/share/hcatalog/*,$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/2.3.0.0-2530/hadoop/lib/hadoop-lzo-0.6.0.2.3.0.0-2530.jar:/etc/hadoop/conf/secure
>> > > > L4J [2015-10-01
>> > > > 03:51:24,169][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
>> > > Hadoop
>> > > > job classpath is:
>> > > >
>> > >
>> >
>> /tmp/kylin/*,$HADOOP_CONF_DIR,/usr/hdp/2.3.0.0-2530/hbase/lib/hbase-common.jar,/usr/hdp/current/hive-client/conf/,/usr/hdp/2.3.0.0-2530/hive/lib/hive-metastore.jar,/usr/hdp/2.3.0.0-2530/hive/lib/hive-exec.jar,/usr/hdp/2.3.0.0-2530/hive-hcatalog/share/hcatalog/*,$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/2.3.0.0-2530/hadoop/lib/hadoop-lzo-0.6.0.2.3.0.0-2530.jar:/etc/hadoop/conf/secure
>> > > > L4J [2015-10-01
>> > 03:51:24,229][WARN][org.apache.hadoop.hive.conf.HiveConf]
>> > > > - HiveConf of name hive.heapsize does not exist
>> > > > L4J [2015-10-01
>> > 03:51:24,230][WARN][org.apache.hadoop.hive.conf.HiveConf]
>> > > > - HiveConf of name hive.server2.enable.impersonation does not exist
>> > > > L4J [2015-10-01
>> > 03:51:24,236][WARN][org.apache.hadoop.hive.conf.HiveConf]
>> > > > - HiveConf of name hive.heapsize does not exist
>> > > > L4J [2015-10-01
>> > 03:51:24,237][WARN][org.apache.hadoop.hive.conf.HiveConf]
>> > > > - HiveConf of name hive.server2.enable.impersonation does not exist
>> > > > L4J [2015-10-01
>> > > > 03:51:24,265][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
>> > > > tempMetaFileString is : null
>> > > > L4J [2015-10-01
>> > > > 03:51:24,266][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
>> > > > tempMetaFileString is : null
>> > > > L4J [2015-10-01
>> > > >
>> > 03:51:24,267][ERROR][org.apache.kylin.job.execution.AbstractExecutable]
>> -
>> > > > error running Executable
>> > > > java.lang.NoSuchMethodError:
>> > > > org.apache.commons.codec.binary.Base64.encodeBase64([BZZ)[B
>> > > > at
>> > > >
>> org.apache.hive.hcatalog.common.HCatUtil.encodeBytes(HCatUtil.java:125)
>> > > > at
>> > > >
>> org.apache.hive.hcatalog.common.HCatUtil.serialize(HCatUtil.java:104)
>> > > > at
>> > > >
>> org.apache.hive.hcatalog.common.HCatUtil.getHiveConf(HCatUtil.java:585)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:100)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:101)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:77)
>> > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
>> > > > at
>> > > >
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> > > > at
>> > > >
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> > > > at java.lang.Thread.run(Thread.java:745)
>> > > > L4J [2015-10-01
>> > > >
>> > 03:51:24,267][ERROR][org.apache.kylin.job.execution.AbstractExecutable]
>> -
>> > > > error running Executable
>> > > > java.lang.NoSuchMethodError:
>> > > > org.apache.commons.codec.binary.Base64.encodeBase64([BZZ)[B
>> > > > at
>> > > >
>> org.apache.hive.hcatalog.common.HCatUtil.encodeBytes(HCatUtil.java:125)
>> > > > at
>> > > >
>> org.apache.hive.hcatalog.common.HCatUtil.serialize(HCatUtil.java:104)
>> > > > at
>> > > >
>> org.apache.hive.hcatalog.common.HCatUtil.getHiveConf(HCatUtil.java:585)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:100)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:101)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:77)
>> > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
>> > > > at
>> > > >
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> > > > at
>> > > >
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> > > >
>> > > > at java.lang.Thread.run(Thread.java:745)
>> > > >
>> > >
>> >
>>
>
Re: commons-codec problem with HDP 2.3 and HBase1.x branch
Posted by 周千昊 <qh...@apache.org>.
OK, I will re-port the changes to 1.x-staging
Li Yang <li...@apache.org>于2015年10月6日周二 下午3:28写道：
> Qianhao, pls don't patch 1.x-HBase1.x directly. All changes should go into
> 1.x-staging first, then apply an HBase patch to become 1.x-HBase1.x
>
> Ported the fix to 1.x-staging and promoted 1.x-HBase1.x branch again. The
> result should be same. The effective commons-codec is 1.9 as shown in
> eclipse.
>
> On Mon, Oct 5, 2015 at 11:47 PM, 周千昊 <qh...@apache.org> wrote:
>
> > Hi, Benjamin,
> > It is a bug, a jira ticket is created as follow:
> > https://issues.apache.org/jira/browse/KYLIN-1059
> > and it should have been fixed. since I don't have the hdp 2.3 env
> > currently, it would be appreciated if you can help to verify with the
> > latest code(branch 1.x-HBase1.x)
> >
> > Luke Han <lu...@gmail.com>于2015年10月5日周一 下午8:35写道：
> >
> > > Hi Benjamin,
> > > Did you generate binary package by yourself with below instruction?
> > or
> > > just compile war/jar from maven?
> > >
> > > http://kylin.incubator.apache.org/development/howto_package.html
> > > <http://kylin.incubator.apache.org/development/howto_package.html>
> > >
> > > Thanks.
> > >
> > >
> > > Best Regards!
> > > ---------------------
> > >
> > > Luke Han
> > >
> > > On Thu, Oct 1, 2015 at 1:30 PM, Benjamin Leonhardi <
> > > bleonhardi@hortonworks.com> wrote:
> > >
> > > > Hello All,
> > > >
> > > > I try to get Kylin to run with HDP 2.3 sandbox image. Using the
> > HBase1.x
> > > > branch.
> > > >
> > > > https://github.com/apache/incubator-kylin/tree/1.x-HBase1.x
> > > >
> > > > It compiles fine and the tests run through until they reach the job
> > > > creation. I followed the steps below.
> > > >
> > > > http://kylin.incubator.apache.org/development/dev_env.html
> > > >
> > > > When I run the tests he reaches the job part and then fails because
> he
> > > > misses a method in commons-codec.jar. The method is
> > encodeBase64(byte[],
> > > > isChunk boolean, isUrlSafe boolean) and it is in commons-codec1.4+
> but
> > > not
> > > > in older versions. I checked but my environment only has codec jars
> > that
> > > > are newer.
> > > >
> > > > However when I build kylin he for some reason compiled an older codec
> > > into
> > > > the kylin-job-1.1-incubating-SNAPSHOT-job.jar. If I unzip this I see
> a
> > > > Base64 class that only has the method encodeBase64(byte[], isChunk
> > > boolean
> > > > ) but not the urlsafe method. ( so I suppose codec 1.3 or older ) I
> > > tried
> > > > to check the pom.xml because I thought I might find an older codec
> > > > dependency inside but did not find anything.
> > > >
> > > > I also have to say I am not the best Maven person so I might be
> > > > overlooking something easy. One question I would have is why he
> > compiles
> > > > the code into the -job.jar at all.
> > > >
> > > >
> > > > Any help welcome
> > > >
> > > > Best regards,
> > > >
> > > > Benjamin
> > > >
> > > >
> > > > Starting:
> > > > Kylin_Fact_Distinct_Columns_test_kylin_cube_without_slr_empty_Step
> > > > L4J [2015-10-01
> > > > 03:51:24,166][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> > > append
> > > > job jar:
> > > >
> /root/kylin_build/incubator-kylin/job/../job/target/kylin-job-1.1-incu
> > > > bating-SNAPSHOT-job.jar
> > > > L4J [2015-10-01
> > > > 03:51:24,169][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> > > append
> > > > kylin.hive.dependency: null and kylin.hbase.dependency: null to
> > > > mapreduce.appli
> > > > cation.classpath
> > > > L4J [2015-10-01
> > > > 03:51:24,167][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> > > append
> > > > job jar:
> > > >
> /root/kylin_build/incubator-kylin/job/../job/target/kylin-job-1.1-incu
> > > > bating-SNAPSHOT-job.jar
> > > > L4J [2015-10-01
> > > > 03:51:24,170][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> > > append
> > > > kylin.hive.dependency: null and kylin.hbase.dependency: null to
> > > > mapreduce.appli
> > > > cation.classpath
> > > > L4J [2015-10-01
> > > > 03:51:24,170][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> > > Hadoop
> > > > job classpath is:
> > > >
> > >
> >
> /tmp/kylin/*,$HADOOP_CONF_DIR,/usr/hdp/2.3.0.0-2530/hbase/lib/hbase-common.jar,/usr/hdp/current/hive-client/conf/,/usr/hdp/2.3.0.0-2530/hive/lib/hive-metastore.jar,/usr/hdp/2.3.0.0-2530/hive/lib/hive-exec.jar,/usr/hdp/2.3.0.0-2530/hive-hcatalog/share/hcatalog/*,$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/2.3.0.0-2530/hadoop/lib/hadoop-lzo-0.6.0.2.3.0.0-2530.jar:/etc/hadoop/conf/secure
> > > > L4J [2015-10-01
> > > > 03:51:24,169][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> > > Hadoop
> > > > job classpath is:
> > > >
> > >
> >
> /tmp/kylin/*,$HADOOP_CONF_DIR,/usr/hdp/2.3.0.0-2530/hbase/lib/hbase-common.jar,/usr/hdp/current/hive-client/conf/,/usr/hdp/2.3.0.0-2530/hive/lib/hive-metastore.jar,/usr/hdp/2.3.0.0-2530/hive/lib/hive-exec.jar,/usr/hdp/2.3.0.0-2530/hive-hcatalog/share/hcatalog/*,$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/2.3.0.0-2530/hadoop/lib/hadoop-lzo-0.6.0.2.3.0.0-2530.jar:/etc/hadoop/conf/secure
> > > > L4J [2015-10-01
> > 03:51:24,229][WARN][org.apache.hadoop.hive.conf.HiveConf]
> > > > - HiveConf of name hive.heapsize does not exist
> > > > L4J [2015-10-01
> > 03:51:24,230][WARN][org.apache.hadoop.hive.conf.HiveConf]
> > > > - HiveConf of name hive.server2.enable.impersonation does not exist
> > > > L4J [2015-10-01
> > 03:51:24,236][WARN][org.apache.hadoop.hive.conf.HiveConf]
> > > > - HiveConf of name hive.heapsize does not exist
> > > > L4J [2015-10-01
> > 03:51:24,237][WARN][org.apache.hadoop.hive.conf.HiveConf]
> > > > - HiveConf of name hive.server2.enable.impersonation does not exist
> > > > L4J [2015-10-01
> > > > 03:51:24,265][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> > > > tempMetaFileString is : null
> > > > L4J [2015-10-01
> > > > 03:51:24,266][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> > > > tempMetaFileString is : null
> > > > L4J [2015-10-01
> > > >
> > 03:51:24,267][ERROR][org.apache.kylin.job.execution.AbstractExecutable] -
> > > > error running Executable
> > > > java.lang.NoSuchMethodError:
> > > > org.apache.commons.codec.binary.Base64.encodeBase64([BZZ)[B
> > > > at
> > > >
> org.apache.hive.hcatalog.common.HCatUtil.encodeBytes(HCatUtil.java:125)
> > > > at
> > > > org.apache.hive.hcatalog.common.HCatUtil.serialize(HCatUtil.java:104)
> > > > at
> > > >
> org.apache.hive.hcatalog.common.HCatUtil.getHiveConf(HCatUtil.java:585)
> > > > at
> > > >
> > >
> >
> org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:100)
> > > > at
> > > >
> > >
> >
> org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
> > > > at
> > > >
> > >
> >
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
> > > > at
> > > >
> > >
> >
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
> > > > at
> > > >
> > >
> >
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:101)
> > > > at
> > > >
> > >
> >
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:77)
> > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> > > > at
> > > >
> > >
> >
> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113)
> > > > at
> > > >
> > >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> > > > at
> > > >
> > >
> >
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
> > > > at
> > > >
> > >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> > > > at
> > > >
> > >
> >
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
> > > > at
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > > at
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > > at java.lang.Thread.run(Thread.java:745)
> > > > L4J [2015-10-01
> > > >
> > 03:51:24,267][ERROR][org.apache.kylin.job.execution.AbstractExecutable] -
> > > > error running Executable
> > > > java.lang.NoSuchMethodError:
> > > > org.apache.commons.codec.binary.Base64.encodeBase64([BZZ)[B
> > > > at
> > > >
> org.apache.hive.hcatalog.common.HCatUtil.encodeBytes(HCatUtil.java:125)
> > > > at
> > > > org.apache.hive.hcatalog.common.HCatUtil.serialize(HCatUtil.java:104)
> > > > at
> > > >
> org.apache.hive.hcatalog.common.HCatUtil.getHiveConf(HCatUtil.java:585)
> > > > at
> > > >
> > >
> >
> org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:100)
> > > > at
> > > >
> > >
> >
> org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
> > > > at
> > > >
> > >
> >
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
> > > > at
> > > >
> > >
> >
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
> > > > at
> > > >
> > >
> >
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:101)
> > > > at
> > > >
> > >
> >
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:77)
> > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> > > > at
> > > >
> > >
> >
> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113)
> > > > at
> > > >
> > >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> > > > at
> > > >
> > >
> >
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
> > > > at
> > > >
> > >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> > > > at
> > > >
> > >
> >
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
> > > > at
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > > at
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > >
> > > > at java.lang.Thread.run(Thread.java:745)
> > > >
> > >
> >
>
Re: commons-codec problem with HDP 2.3 and HBase1.x branch
Posted by Li Yang <li...@apache.org>.
Qianhao, pls don't patch 1.x-HBase1.x directly. All changes should go into
1.x-staging first, then apply an HBase patch to become 1.x-HBase1.x
Ported the fix to 1.x-staging and promoted 1.x-HBase1.x branch again. The
result should be same. The effective commons-codec is 1.9 as shown in
eclipse.
On Mon, Oct 5, 2015 at 11:47 PM, 周千昊 <qh...@apache.org> wrote:
> Hi, Benjamin,
> It is a bug, a jira ticket is created as follow:
> https://issues.apache.org/jira/browse/KYLIN-1059
> and it should have been fixed. since I don't have the hdp 2.3 env
> currently, it would be appreciated if you can help to verify with the
> latest code(branch 1.x-HBase1.x)
>
> Luke Han <lu...@gmail.com>于2015年10月5日周一 下午8:35写道：
>
> > Hi Benjamin,
> > Did you generate binary package by yourself with below instruction?
> or
> > just compile war/jar from maven?
> >
> > http://kylin.incubator.apache.org/development/howto_package.html
> > <http://kylin.incubator.apache.org/development/howto_package.html>
> >
> > Thanks.
> >
> >
> > Best Regards!
> > ---------------------
> >
> > Luke Han
> >
> > On Thu, Oct 1, 2015 at 1:30 PM, Benjamin Leonhardi <
> > bleonhardi@hortonworks.com> wrote:
> >
> > > Hello All,
> > >
> > > I try to get Kylin to run with HDP 2.3 sandbox image. Using the
> HBase1.x
> > > branch.
> > >
> > > https://github.com/apache/incubator-kylin/tree/1.x-HBase1.x
> > >
> > > It compiles fine and the tests run through until they reach the job
> > > creation. I followed the steps below.
> > >
> > > http://kylin.incubator.apache.org/development/dev_env.html
> > >
> > > When I run the tests he reaches the job part and then fails because he
> > > misses a method in commons-codec.jar. The method is
> encodeBase64(byte[],
> > > isChunk boolean, isUrlSafe boolean) and it is in commons-codec1.4+ but
> > not
> > > in older versions. I checked but my environment only has codec jars
> that
> > > are newer.
> > >
> > > However when I build kylin he for some reason compiled an older codec
> > into
> > > the kylin-job-1.1-incubating-SNAPSHOT-job.jar. If I unzip this I see a
> > > Base64 class that only has the method encodeBase64(byte[], isChunk
> > boolean
> > > ) but not the urlsafe method. ( so I suppose codec 1.3 or older ) I
> > tried
> > > to check the pom.xml because I thought I might find an older codec
> > > dependency inside but did not find anything.
> > >
> > > I also have to say I am not the best Maven person so I might be
> > > overlooking something easy. One question I would have is why he
> compiles
> > > the code into the -job.jar at all.
> > >
> > >
> > > Any help welcome
> > >
> > > Best regards,
> > >
> > > Benjamin
> > >
> > >
> > > Starting:
> > > Kylin_Fact_Distinct_Columns_test_kylin_cube_without_slr_empty_Step
> > > L4J [2015-10-01
> > > 03:51:24,166][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> > append
> > > job jar:
> > > /root/kylin_build/incubator-kylin/job/../job/target/kylin-job-1.1-incu
> > > bating-SNAPSHOT-job.jar
> > > L4J [2015-10-01
> > > 03:51:24,169][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> > append
> > > kylin.hive.dependency: null and kylin.hbase.dependency: null to
> > > mapreduce.appli
> > > cation.classpath
> > > L4J [2015-10-01
> > > 03:51:24,167][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> > append
> > > job jar:
> > > /root/kylin_build/incubator-kylin/job/../job/target/kylin-job-1.1-incu
> > > bating-SNAPSHOT-job.jar
> > > L4J [2015-10-01
> > > 03:51:24,170][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> > append
> > > kylin.hive.dependency: null and kylin.hbase.dependency: null to
> > > mapreduce.appli
> > > cation.classpath
> > > L4J [2015-10-01
> > > 03:51:24,170][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> > Hadoop
> > > job classpath is:
> > >
> >
> /tmp/kylin/*,$HADOOP_CONF_DIR,/usr/hdp/2.3.0.0-2530/hbase/lib/hbase-common.jar,/usr/hdp/current/hive-client/conf/,/usr/hdp/2.3.0.0-2530/hive/lib/hive-metastore.jar,/usr/hdp/2.3.0.0-2530/hive/lib/hive-exec.jar,/usr/hdp/2.3.0.0-2530/hive-hcatalog/share/hcatalog/*,$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/2.3.0.0-2530/hadoop/lib/hadoop-lzo-0.6.0.2.3.0.0-2530.jar:/etc/hadoop/conf/secure
> > > L4J [2015-10-01
> > > 03:51:24,169][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> > Hadoop
> > > job classpath is:
> > >
> >
> /tmp/kylin/*,$HADOOP_CONF_DIR,/usr/hdp/2.3.0.0-2530/hbase/lib/hbase-common.jar,/usr/hdp/current/hive-client/conf/,/usr/hdp/2.3.0.0-2530/hive/lib/hive-metastore.jar,/usr/hdp/2.3.0.0-2530/hive/lib/hive-exec.jar,/usr/hdp/2.3.0.0-2530/hive-hcatalog/share/hcatalog/*,$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/2.3.0.0-2530/hadoop/lib/hadoop-lzo-0.6.0.2.3.0.0-2530.jar:/etc/hadoop/conf/secure
> > > L4J [2015-10-01
> 03:51:24,229][WARN][org.apache.hadoop.hive.conf.HiveConf]
> > > - HiveConf of name hive.heapsize does not exist
> > > L4J [2015-10-01
> 03:51:24,230][WARN][org.apache.hadoop.hive.conf.HiveConf]
> > > - HiveConf of name hive.server2.enable.impersonation does not exist
> > > L4J [2015-10-01
> 03:51:24,236][WARN][org.apache.hadoop.hive.conf.HiveConf]
> > > - HiveConf of name hive.heapsize does not exist
> > > L4J [2015-10-01
> 03:51:24,237][WARN][org.apache.hadoop.hive.conf.HiveConf]
> > > - HiveConf of name hive.server2.enable.impersonation does not exist
> > > L4J [2015-10-01
> > > 03:51:24,265][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> > > tempMetaFileString is : null
> > > L4J [2015-10-01
> > > 03:51:24,266][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> > > tempMetaFileString is : null
> > > L4J [2015-10-01
> > >
> 03:51:24,267][ERROR][org.apache.kylin.job.execution.AbstractExecutable] -
> > > error running Executable
> > > java.lang.NoSuchMethodError:
> > > org.apache.commons.codec.binary.Base64.encodeBase64([BZZ)[B
> > > at
> > > org.apache.hive.hcatalog.common.HCatUtil.encodeBytes(HCatUtil.java:125)
> > > at
> > > org.apache.hive.hcatalog.common.HCatUtil.serialize(HCatUtil.java:104)
> > > at
> > > org.apache.hive.hcatalog.common.HCatUtil.getHiveConf(HCatUtil.java:585)
> > > at
> > >
> >
> org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:100)
> > > at
> > >
> >
> org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
> > > at
> > >
> >
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
> > > at
> > >
> >
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
> > > at
> > >
> >
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:101)
> > > at
> > >
> >
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:77)
> > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> > > at
> > >
> >
> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113)
> > > at
> > >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> > > at
> > >
> >
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
> > > at
> > >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> > > at
> > >
> >
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > at java.lang.Thread.run(Thread.java:745)
> > > L4J [2015-10-01
> > >
> 03:51:24,267][ERROR][org.apache.kylin.job.execution.AbstractExecutable] -
> > > error running Executable
> > > java.lang.NoSuchMethodError:
> > > org.apache.commons.codec.binary.Base64.encodeBase64([BZZ)[B
> > > at
> > > org.apache.hive.hcatalog.common.HCatUtil.encodeBytes(HCatUtil.java:125)
> > > at
> > > org.apache.hive.hcatalog.common.HCatUtil.serialize(HCatUtil.java:104)
> > > at
> > > org.apache.hive.hcatalog.common.HCatUtil.getHiveConf(HCatUtil.java:585)
> > > at
> > >
> >
> org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:100)
> > > at
> > >
> >
> org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
> > > at
> > >
> >
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
> > > at
> > >
> >
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
> > > at
> > >
> >
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:101)
> > > at
> > >
> >
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:77)
> > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> > > at
> > >
> >
> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113)
> > > at
> > >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> > > at
> > >
> >
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
> > > at
> > >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> > > at
> > >
> >
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > >
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> >
>
Re: commons-codec problem with HDP 2.3 and HBase1.x branch
Posted by 周千昊 <qh...@apache.org>.
Hi, Benjamin,
It is a bug, a jira ticket is created as follow:
https://issues.apache.org/jira/browse/KYLIN-1059
and it should have been fixed. since I don't have the hdp 2.3 env
currently, it would be appreciated if you can help to verify with the
latest code(branch 1.x-HBase1.x)
Luke Han <lu...@gmail.com>于2015年10月5日周一 下午8:35写道：
> Hi Benjamin,
> Did you generate binary package by yourself with below instruction? or
> just compile war/jar from maven?
>
> http://kylin.incubator.apache.org/development/howto_package.html
> <http://kylin.incubator.apache.org/development/howto_package.html>
>
> Thanks.
>
>
> Best Regards!
> ---------------------
>
> Luke Han
>
> On Thu, Oct 1, 2015 at 1:30 PM, Benjamin Leonhardi <
> bleonhardi@hortonworks.com> wrote:
>
> > Hello All,
> >
> > I try to get Kylin to run with HDP 2.3 sandbox image. Using the HBase1.x
> > branch.
> >
> > https://github.com/apache/incubator-kylin/tree/1.x-HBase1.x
> >
> > It compiles fine and the tests run through until they reach the job
> > creation. I followed the steps below.
> >
> > http://kylin.incubator.apache.org/development/dev_env.html
> >
> > When I run the tests he reaches the job part and then fails because he
> > misses a method in commons-codec.jar. The method is encodeBase64(byte[],
> > isChunk boolean, isUrlSafe boolean) and it is in commons-codec1.4+ but
> not
> > in older versions. I checked but my environment only has codec jars that
> > are newer.
> >
> > However when I build kylin he for some reason compiled an older codec
> into
> > the kylin-job-1.1-incubating-SNAPSHOT-job.jar. If I unzip this I see a
> > Base64 class that only has the method encodeBase64(byte[], isChunk
> boolean
> > ) but not the urlsafe method. ( so I suppose codec 1.3 or older ) I
> tried
> > to check the pom.xml because I thought I might find an older codec
> > dependency inside but did not find anything.
> >
> > I also have to say I am not the best Maven person so I might be
> > overlooking something easy. One question I would have is why he compiles
> > the code into the -job.jar at all.
> >
> >
> > Any help welcome
> >
> > Best regards,
> >
> > Benjamin
> >
> >
> > Starting:
> > Kylin_Fact_Distinct_Columns_test_kylin_cube_without_slr_empty_Step
> > L4J [2015-10-01
> > 03:51:24,166][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> append
> > job jar:
> > /root/kylin_build/incubator-kylin/job/../job/target/kylin-job-1.1-incu
> > bating-SNAPSHOT-job.jar
> > L4J [2015-10-01
> > 03:51:24,169][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> append
> > kylin.hive.dependency: null and kylin.hbase.dependency: null to
> > mapreduce.appli
> > cation.classpath
> > L4J [2015-10-01
> > 03:51:24,167][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> append
> > job jar:
> > /root/kylin_build/incubator-kylin/job/../job/target/kylin-job-1.1-incu
> > bating-SNAPSHOT-job.jar
> > L4J [2015-10-01
> > 03:51:24,170][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> append
> > kylin.hive.dependency: null and kylin.hbase.dependency: null to
> > mapreduce.appli
> > cation.classpath
> > L4J [2015-10-01
> > 03:51:24,170][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> Hadoop
> > job classpath is:
> >
> /tmp/kylin/*,$HADOOP_CONF_DIR,/usr/hdp/2.3.0.0-2530/hbase/lib/hbase-common.jar,/usr/hdp/current/hive-client/conf/,/usr/hdp/2.3.0.0-2530/hive/lib/hive-metastore.jar,/usr/hdp/2.3.0.0-2530/hive/lib/hive-exec.jar,/usr/hdp/2.3.0.0-2530/hive-hcatalog/share/hcatalog/*,$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/2.3.0.0-2530/hadoop/lib/hadoop-lzo-0.6.0.2.3.0.0-2530.jar:/etc/hadoop/conf/secure
> > L4J [2015-10-01
> > 03:51:24,169][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> Hadoop
> > job classpath is:
> >
> /tmp/kylin/*,$HADOOP_CONF_DIR,/usr/hdp/2.3.0.0-2530/hbase/lib/hbase-common.jar,/usr/hdp/current/hive-client/conf/,/usr/hdp/2.3.0.0-2530/hive/lib/hive-metastore.jar,/usr/hdp/2.3.0.0-2530/hive/lib/hive-exec.jar,/usr/hdp/2.3.0.0-2530/hive-hcatalog/share/hcatalog/*,$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/2.3.0.0-2530/hadoop/lib/hadoop-lzo-0.6.0.2.3.0.0-2530.jar:/etc/hadoop/conf/secure
> > L4J [2015-10-01 03:51:24,229][WARN][org.apache.hadoop.hive.conf.HiveConf]
> > - HiveConf of name hive.heapsize does not exist
> > L4J [2015-10-01 03:51:24,230][WARN][org.apache.hadoop.hive.conf.HiveConf]
> > - HiveConf of name hive.server2.enable.impersonation does not exist
> > L4J [2015-10-01 03:51:24,236][WARN][org.apache.hadoop.hive.conf.HiveConf]
> > - HiveConf of name hive.heapsize does not exist
> > L4J [2015-10-01 03:51:24,237][WARN][org.apache.hadoop.hive.conf.HiveConf]
> > - HiveConf of name hive.server2.enable.impersonation does not exist
> > L4J [2015-10-01
> > 03:51:24,265][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> > tempMetaFileString is : null
> > L4J [2015-10-01
> > 03:51:24,266][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> > tempMetaFileString is : null
> > L4J [2015-10-01
> > 03:51:24,267][ERROR][org.apache.kylin.job.execution.AbstractExecutable] -
> > error running Executable
> > java.lang.NoSuchMethodError:
> > org.apache.commons.codec.binary.Base64.encodeBase64([BZZ)[B
> > at
> > org.apache.hive.hcatalog.common.HCatUtil.encodeBytes(HCatUtil.java:125)
> > at
> > org.apache.hive.hcatalog.common.HCatUtil.serialize(HCatUtil.java:104)
> > at
> > org.apache.hive.hcatalog.common.HCatUtil.getHiveConf(HCatUtil.java:585)
> > at
> >
> org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:100)
> > at
> >
> org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
> > at
> >
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
> > at
> >
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
> > at
> >
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:101)
> > at
> >
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:77)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> > at
> >
> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113)
> > at
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> > at
> >
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
> > at
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> > at
> >
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> > L4J [2015-10-01
> > 03:51:24,267][ERROR][org.apache.kylin.job.execution.AbstractExecutable] -
> > error running Executable
> > java.lang.NoSuchMethodError:
> > org.apache.commons.codec.binary.Base64.encodeBase64([BZZ)[B
> > at
> > org.apache.hive.hcatalog.common.HCatUtil.encodeBytes(HCatUtil.java:125)
> > at
> > org.apache.hive.hcatalog.common.HCatUtil.serialize(HCatUtil.java:104)
> > at
> > org.apache.hive.hcatalog.common.HCatUtil.getHiveConf(HCatUtil.java:585)
> > at
> >
> org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:100)
> > at
> >
> org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
> > at
> >
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
> > at
> >
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
> > at
> >
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:101)
> > at
> >
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:77)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> > at
> >
> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113)
> > at
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> > at
> >
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
> > at
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> > at
> >
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >
> > at java.lang.Thread.run(Thread.java:745)
> >
>
Re: commons-codec problem with HDP 2.3 and HBase1.x branch
Posted by Luke Han <lu...@gmail.com>.
Hi Benjamin,
Did you generate binary package by yourself with below instruction? or
just compile war/jar from maven?
http://kylin.incubator.apache.org/development/howto_package.html
<http://kylin.incubator.apache.org/development/howto_package.html>
Thanks.
Best Regards!
---------------------
Luke Han
On Thu, Oct 1, 2015 at 1:30 PM, Benjamin Leonhardi <
bleonhardi@hortonworks.com> wrote:
> Hello All,
>
> I try to get Kylin to run with HDP 2.3 sandbox image. Using the HBase1.x
> branch.
>
> https://github.com/apache/incubator-kylin/tree/1.x-HBase1.x
>
> It compiles fine and the tests run through until they reach the job
> creation. I followed the steps below.
>
> http://kylin.incubator.apache.org/development/dev_env.html
>
> When I run the tests he reaches the job part and then fails because he
> misses a method in commons-codec.jar. The method is encodeBase64(byte[],
> isChunk boolean, isUrlSafe boolean) and it is in commons-codec1.4+ but not
> in older versions. I checked but my environment only has codec jars that
> are newer.
>
> However when I build kylin he for some reason compiled an older codec into
> the kylin-job-1.1-incubating-SNAPSHOT-job.jar. If I unzip this I see a
> Base64 class that only has the method encodeBase64(byte[], isChunk boolean
> ) but not the urlsafe method. ( so I suppose codec 1.3 or older ) I tried
> to check the pom.xml because I thought I might find an older codec
> dependency inside but did not find anything.
>
> I also have to say I am not the best Maven person so I might be
> overlooking something easy. One question I would have is why he compiles
> the code into the -job.jar at all.
>
>
> Any help welcome
>
> Best regards,
>
> Benjamin
>
>
> Starting:
> Kylin_Fact_Distinct_Columns_test_kylin_cube_without_slr_empty_Step
> L4J [2015-10-01
> 03:51:24,166][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - append
> job jar:
> /root/kylin_build/incubator-kylin/job/../job/target/kylin-job-1.1-incu
> bating-SNAPSHOT-job.jar
> L4J [2015-10-01
> 03:51:24,169][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - append
> kylin.hive.dependency: null and kylin.hbase.dependency: null to
> mapreduce.appli
> cation.classpath
> L4J [2015-10-01
> 03:51:24,167][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - append
> job jar:
> /root/kylin_build/incubator-kylin/job/../job/target/kylin-job-1.1-incu
> bating-SNAPSHOT-job.jar
> L4J [2015-10-01
> 03:51:24,170][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - append
> kylin.hive.dependency: null and kylin.hbase.dependency: null to
> mapreduce.appli
> cation.classpath
> L4J [2015-10-01
> 03:51:24,170][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - Hadoop
> job classpath is:
> /tmp/kylin/*,$HADOOP_CONF_DIR,/usr/hdp/2.3.0.0-2530/hbase/lib/hbase-common.jar,/usr/hdp/current/hive-client/conf/,/usr/hdp/2.3.0.0-2530/hive/lib/hive-metastore.jar,/usr/hdp/2.3.0.0-2530/hive/lib/hive-exec.jar,/usr/hdp/2.3.0.0-2530/hive-hcatalog/share/hcatalog/*,$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/2.3.0.0-2530/hadoop/lib/hadoop-lzo-0.6.0.2.3.0.0-2530.jar:/etc/hadoop/conf/secure
> L4J [2015-10-01
> 03:51:24,169][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - Hadoop
> job classpath is:
> /tmp/kylin/*,$HADOOP_CONF_DIR,/usr/hdp/2.3.0.0-2530/hbase/lib/hbase-common.jar,/usr/hdp/current/hive-client/conf/,/usr/hdp/2.3.0.0-2530/hive/lib/hive-metastore.jar,/usr/hdp/2.3.0.0-2530/hive/lib/hive-exec.jar,/usr/hdp/2.3.0.0-2530/hive-hcatalog/share/hcatalog/*,$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/2.3.0.0-2530/hadoop/lib/hadoop-lzo-0.6.0.2.3.0.0-2530.jar:/etc/hadoop/conf/secure
> L4J [2015-10-01 03:51:24,229][WARN][org.apache.hadoop.hive.conf.HiveConf]
> - HiveConf of name hive.heapsize does not exist
> L4J [2015-10-01 03:51:24,230][WARN][org.apache.hadoop.hive.conf.HiveConf]
> - HiveConf of name hive.server2.enable.impersonation does not exist
> L4J [2015-10-01 03:51:24,236][WARN][org.apache.hadoop.hive.conf.HiveConf]
> - HiveConf of name hive.heapsize does not exist
> L4J [2015-10-01 03:51:24,237][WARN][org.apache.hadoop.hive.conf.HiveConf]
> - HiveConf of name hive.server2.enable.impersonation does not exist
> L4J [2015-10-01
> 03:51:24,265][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> tempMetaFileString is : null
> L4J [2015-10-01
> 03:51:24,266][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] -
> tempMetaFileString is : null
> L4J [2015-10-01
> 03:51:24,267][ERROR][org.apache.kylin.job.execution.AbstractExecutable] -
> error running Executable
> java.lang.NoSuchMethodError:
> org.apache.commons.codec.binary.Base64.encodeBase64([BZZ)[B
> at
> org.apache.hive.hcatalog.common.HCatUtil.encodeBytes(HCatUtil.java:125)
> at
> org.apache.hive.hcatalog.common.HCatUtil.serialize(HCatUtil.java:104)
> at
> org.apache.hive.hcatalog.common.HCatUtil.getHiveConf(HCatUtil.java:585)
> at
> org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:100)
> at
> org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
> at
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
> at
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
> at
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:101)
> at
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:77)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> L4J [2015-10-01
> 03:51:24,267][ERROR][org.apache.kylin.job.execution.AbstractExecutable] -
> error running Executable
> java.lang.NoSuchMethodError:
> org.apache.commons.codec.binary.Base64.encodeBase64([BZZ)[B
> at
> org.apache.hive.hcatalog.common.HCatUtil.encodeBytes(HCatUtil.java:125)
> at
> org.apache.hive.hcatalog.common.HCatUtil.serialize(HCatUtil.java:104)
> at
> org.apache.hive.hcatalog.common.HCatUtil.getHiveConf(HCatUtil.java:585)
> at
> org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:100)
> at
> org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
> at
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
> at
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
> at
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:101)
> at
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:77)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:745)
>