You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by hongbin ma <ma...@apache.org> on 2015/01/14 13:20:28 UTC

upgrading kylin's hadoop depedencies version

hi, yang

I tried our deploy script on hdp 2.2 (with current kylin dependencies in
pom untouched), it turns out that MR job submission will fail. the log is
attached at the end.
I found that "mapreduce.application.framework.path" is something new in hdp
2.2's mapred-site.xml. I guess the obsolete MR client might not be able to
handle it properly.

I suggest we upgrade the hadoop dependencies versions in our pom, and see
if it will work out in both hdp 2.1 and 2.2.
I remember you tried upgrading the versions recently, Did you checked them
in?

hongbin.

=================================

[QuartzScheduler_Worker-1]:[2015-01-14
09:24:45,832][ERROR][com.kylinolap.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:83)]
- Unable to parse
'/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a
URI, check the setting for mapreduce.application.framework.path

java.lang.IllegalArgumentException: Unable to parse
'/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a
URI, check the setting for mapreduce.application.framework.path

at
org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(JobSubmitter.java:739)

at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:435)

at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)

at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)

at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)

at
com.kylinolap.job.hadoop.AbstractHadoopJob.waitForCompletion(AbstractHadoopJob.java:114)

at
com.kylinolap.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:79)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)

at com.kylinolap.job.cmd.JavaHadoopCmd.execute(JavaHadoopCmd.java:53)

at com.kylinolap.job.flow.AsyncJobFlowNode.execute(AsyncJobFlowNode.java:70)

at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

Caused by: java.net.URISyntaxException: Illegal character in path at index
11: /hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework

at java.net.URI$Parser.fail(URI.java:2829)

at java.net.URI$Parser.checkChars(URI.java:3002)

at java.net.URI$Parser.parseHierarchical(URI.java:3086)

at java.net.URI$Parser.parse(URI.java:3044)

at java.net.URI.<init>(URI.java:595)

at
org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(JobSubmitter.java:737)

... 15 more

Re: upgrading kylin's hadoop depedencies version

Posted by Li Yang <li...@apache.org>.
(only dev)

Let's aim to make Kylin compatible with all Hadoop 2.x distributions. We
shall be pretty close since we have been sticking to the stable Hadoop
APIs. I don't remember played any tricks around Hadoop.

Cheers
Yang


On Wed, Jan 14, 2015 at 9:58 PM, Luke Han <lu...@apache.org> wrote:

> Hi Hongbin, please send mail direct to dev@ list, not cc :-)
>
> Thanks.
>
> Luke
>
> 2015-01-14 20:20 GMT+08:00 hongbin ma <ma...@apache.org>:
>
>> hi, yang
>>
>> I tried our deploy script on hdp 2.2 (with current kylin dependencies in
>> pom untouched), it turns out that MR job submission will fail. the log is
>> attached at the end.
>> I found that "mapreduce.application.framework.path" is something new in
>> hdp
>> 2.2's mapred-site.xml. I guess the obsolete MR client might not be able to
>> handle it properly.
>>
>> I suggest we upgrade the hadoop dependencies versions in our pom, and see
>> if it will work out in both hdp 2.1 and 2.2.
>> I remember you tried upgrading the versions recently, Did you checked them
>> in?
>>
>> hongbin.
>>
>> =================================
>>
>> [QuartzScheduler_Worker-1]:[2015-01-14
>>
>> 09:24:45,832][ERROR][com.kylinolap.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:83)]
>> - Unable to parse
>> '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a
>> URI, check the setting for mapreduce.application.framework.path
>>
>> java.lang.IllegalArgumentException: Unable to parse
>> '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a
>> URI, check the setting for mapreduce.application.framework.path
>>
>> at
>>
>> org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(JobSubmitter.java:739)
>>
>> at
>>
>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:435)
>>
>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
>>
>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
>>
>> at java.security.AccessController.doPrivileged(Native Method)
>>
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>
>> at
>>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>>
>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
>>
>> at
>>
>> com.kylinolap.job.hadoop.AbstractHadoopJob.waitForCompletion(AbstractHadoopJob.java:114)
>>
>> at
>>
>> com.kylinolap.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:79)
>>
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>
>> at com.kylinolap.job.cmd.JavaHadoopCmd.execute(JavaHadoopCmd.java:53)
>>
>> at
>> com.kylinolap.job.flow.AsyncJobFlowNode.execute(AsyncJobFlowNode.java:70)
>>
>> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
>>
>> at
>>
>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
>>
>> Caused by: java.net.URISyntaxException: Illegal character in path at index
>> 11: /hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework
>>
>> at java.net.URI$Parser.fail(URI.java:2829)
>>
>> at java.net.URI$Parser.checkChars(URI.java:3002)
>>
>> at java.net.URI$Parser.parseHierarchical(URI.java:3086)
>>
>> at java.net.URI$Parser.parse(URI.java:3044)
>>
>> at java.net.URI.<init>(URI.java:595)
>>
>> at
>>
>> org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(JobSubmitter.java:737)
>>
>> ... 15 more
>>
>
>

Re: upgrading kylin's hadoop depedencies version

Posted by Luke Han <lu...@apache.org>.
Hi Hongbin, please send mail direct to dev@ list, not cc :-)

Thanks.

Luke

2015-01-14 20:20 GMT+08:00 hongbin ma <ma...@apache.org>:

> hi, yang
>
> I tried our deploy script on hdp 2.2 (with current kylin dependencies in
> pom untouched), it turns out that MR job submission will fail. the log is
> attached at the end.
> I found that "mapreduce.application.framework.path" is something new in hdp
> 2.2's mapred-site.xml. I guess the obsolete MR client might not be able to
> handle it properly.
>
> I suggest we upgrade the hadoop dependencies versions in our pom, and see
> if it will work out in both hdp 2.1 and 2.2.
> I remember you tried upgrading the versions recently, Did you checked them
> in?
>
> hongbin.
>
> =================================
>
> [QuartzScheduler_Worker-1]:[2015-01-14
>
> 09:24:45,832][ERROR][com.kylinolap.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:83)]
> - Unable to parse
> '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a
> URI, check the setting for mapreduce.application.framework.path
>
> java.lang.IllegalArgumentException: Unable to parse
> '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a
> URI, check the setting for mapreduce.application.framework.path
>
> at
>
> org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(JobSubmitter.java:739)
>
> at
>
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:435)
>
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
>
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
>
> at
>
> com.kylinolap.job.hadoop.AbstractHadoopJob.waitForCompletion(AbstractHadoopJob.java:114)
>
> at
>
> com.kylinolap.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:79)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>
> at com.kylinolap.job.cmd.JavaHadoopCmd.execute(JavaHadoopCmd.java:53)
>
> at
> com.kylinolap.job.flow.AsyncJobFlowNode.execute(AsyncJobFlowNode.java:70)
>
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
>
> at
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
>
> Caused by: java.net.URISyntaxException: Illegal character in path at index
> 11: /hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework
>
> at java.net.URI$Parser.fail(URI.java:2829)
>
> at java.net.URI$Parser.checkChars(URI.java:3002)
>
> at java.net.URI$Parser.parseHierarchical(URI.java:3086)
>
> at java.net.URI$Parser.parse(URI.java:3044)
>
> at java.net.URI.<init>(URI.java:595)
>
> at
>
> org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(JobSubmitter.java:737)
>
> ... 15 more
>

Re: upgrading kylin's hadoop depedencies version

Posted by "Zhou, Qianhao" <qi...@ebay.com>.
There are also some users who has reported the same issue.
However the list of supported hdp version is kind of out-of-date.

Best Regard
Zhou QianHao





On 1/14/15, 8:20 PM, "hongbin ma" <ma...@apache.org> wrote:

>hi, yang
>
>I tried our deploy script on hdp 2.2 (with current kylin dependencies in
>pom untouched), it turns out that MR job submission will fail. the log is
>attached at the end.
>I found that "mapreduce.application.framework.path" is something new in
>hdp
>2.2's mapred-site.xml. I guess the obsolete MR client might not be able to
>handle it properly.
>
>I suggest we upgrade the hadoop dependencies versions in our pom, and see
>if it will work out in both hdp 2.1 and 2.2.
>I remember you tried upgrading the versions recently, Did you checked them
>in?
>
>hongbin.
>
>=================================
>
>[QuartzScheduler_Worker-1]:[2015-01-14
>09:24:45,832][ERROR][com.kylinolap.job.hadoop.cube.FactDistinctColumnsJob.
>run(FactDistinctColumnsJob.java:83)]
>- Unable to parse
>'/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a
>URI, check the setting for mapreduce.application.framework.path
>
>java.lang.IllegalArgumentException: Unable to parse
>'/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a
>URI, check the setting for mapreduce.application.framework.path
>
>at
>org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(
>JobSubmitter.java:739)
>
>at
>org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.ja
>va:435)
>
>at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
>
>at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
>
>at java.security.AccessController.doPrivileged(Native Method)
>
>at javax.security.auth.Subject.doAs(Subject.java:415)
>
>at
>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>java:1628)
>
>at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
>
>at
>com.kylinolap.job.hadoop.AbstractHadoopJob.waitForCompletion(AbstractHadoo
>pJob.java:114)
>
>at
>com.kylinolap.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColum
>nsJob.java:79)
>
>at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
>at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>
>at com.kylinolap.job.cmd.JavaHadoopCmd.execute(JavaHadoopCmd.java:53)
>
>at 
>com.kylinolap.job.flow.AsyncJobFlowNode.execute(AsyncJobFlowNode.java:70)
>
>at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
>
>at
>org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:5
>73)
>
>Caused by: java.net.URISyntaxException: Illegal character in path at index
>11: /hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework
>
>at java.net.URI$Parser.fail(URI.java:2829)
>
>at java.net.URI$Parser.checkChars(URI.java:3002)
>
>at java.net.URI$Parser.parseHierarchical(URI.java:3086)
>
>at java.net.URI$Parser.parse(URI.java:3044)
>
>at java.net.URI.<init>(URI.java:595)
>
>at
>org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(
>JobSubmitter.java:737)
>
>... 15 more