You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by "Billy(Yiming) Liu" <li...@gmail.com> on 2016/11/10 04:01:53 UTC
Re: hadoop2.2 imcompatible proble
Based on my limited experience, here are some source code changes needed
because some YARN APIs changed since Hadoop 2.2.
The code is in JobTaskCounterExtractor and HadoopStatusChecker
2016-11-10 11:23 GMT+08:00 胡志华(万里通科技及数据中心商务智能团队数据分析组) <
HUZHIHUA160@pingan.com.cn>:
> Hi all
>
> As the request of department, I need to deploy another kylin
> cluster. The problem is hadoop cluster is too old , hadoop2.2.0, cube
> building will stop at step 3 “Extract Fact Table Distinct Columns”, the
> error code from mapreduce is as below:
>
> I don’t know whether kylin support hadoop 2.2.0, by the way, I have
> run “wordcount” on kylin’s hadoop client ,it worked.
>
> Hadoop cluster won’t be upgraded easily, So I can’t doploy kylin
> with my present environment ?
>
>
> “Application application_1468678294220_132943 failed 2 times due to AM
> Container for appattempt_1468678294220_132943_000002 exited with
> exitCode: 1 due to: Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(
> Shell.java:589)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.
> launchContainer(DefaultContainerExecutor.java:195)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.
> launcher.ContainerLaunch.call(ContainerLaunch.java:283)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.
> launcher.ContainerLaunch.call(ContainerLaunch.java:79)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> .Failing this attempt.. Failing the application.”
>
>
> ************************************************************
> ********************************************************************
> The information in this email is confidential and may be legally
> privileged. If you have received this email in error or are not the
> intended recipient, please immediately notify the sender and delete this
> message from your computer. Any use, distribution, or copying of this email
> other than by the intended recipient is strictly prohibited. All messages
> sent to and from us may be monitored to ensure compliance with internal
> policies and to protect our business.
> Emails are not secure and cannot be guaranteed to be error free as they
> can be intercepted, amended, lost or destroyed, or contain viruses. Anyone
> who communicates with us by email is taken to accept these risks.
>
> 收发邮件者请注意：
> 本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。
> 进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。
> ************************************************************
> ********************************************************************
>
--
With Warm regards
Yiming Liu (刘一鸣)
答复: 答复: hadoop2.2 imcompatible proble
Posted by "胡志华 (万里通科技及数据中心商务智能团队数据分析组)" <HU...@pingan.com.cn>.
I will never do that either, thank you all.
-----邮件原件-----
发件人: ShaoFeng Shi [mailto:shaofengshi@apache.org]
发送时间: 2016年11月10日 18:40
收件人: dev
主题: Re: 答复: hadoop2.2 imcompatible proble
You can try to compile Kylin source code with Hadoop 2.2; we never did that; There are some Hadoop API change (as Billy mentioned) so you need re-write some codes. Good luck!
2016-11-10 16:42 GMT+08:00 胡志华(万里通科技及数据中心商务智能团队数据分析组) <
HUZHIHUA160@pingan.com.cn>:
> You mean , it's impossible to deploy kylin on hadoop 2.2?
>
> Give up or upgrade hadoop version , another way?
>
> -----邮件原件-----
> 发件人: Billy(Yiming) Liu [mailto:liuyiming.vip@gmail.com]
> 发送时间: 2016年11月10日 12:02
> 收件人: dev
> 主题: Re: hadoop2.2 imcompatible proble
>
> Based on my limited experience, here are some source code changes
> needed because some YARN APIs changed since Hadoop 2.2.
>
> The code is in JobTaskCounterExtractor and HadoopStatusChecker
>
> 2016-11-10 11:23 GMT+08:00 胡志华(万里通科技及数据中心商务智能团队数据分析组) <
> HUZHIHUA160@pingan.com.cn>:
>
> > Hi all
> >
> > As the request of department, I need to deploy another kylin
> > cluster. The problem is hadoop cluster is too old , hadoop2.2.0,
> > cube building will stop at step 3 “Extract Fact Table Distinct
> > Columns”, the error code from mapreduce is as below:
> >
> > I don’t know whether kylin support hadoop 2.2.0, by the way,
> > I have run “wordcount” on kylin’s hadoop client ,it worked.
> >
> > Hadoop cluster won’t be upgraded easily, So I can’t doploy
> > kylin with my present environment ?
> >
> >
> > “Application application_1468678294220_132943 failed 2 times due to
> > AM Container for appattempt_1468678294220_132943_000002 exited with
> > exitCode: 1 due to: Exception from container-launch:
> > org.apache.hadoop.util.Shell$ExitCodeException:
> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> > at org.apache.hadoop.util.Shell.run(Shell.java:379)
> > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(
> > Shell.java:589)
> > at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.
> > launchContainer(DefaultContainerExecutor.java:195)
> > at org.apache.hadoop.yarn.server.nodemanager.containermanager.
> > launcher.ContainerLaunch.call(ContainerLaunch.java:283)
> > at org.apache.hadoop.yarn.server.nodemanager.containermanager.
> > launcher.ContainerLaunch.call(ContainerLaunch.java:79)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1145)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> > .Failing this attempt.. Failing the application.”
> >
> >
> > ************************************************************
> > ********************************************************************
> > The information in this email is confidential and may be legally
> > privileged. If you have received this email in error or are not the
> > intended recipient, please immediately notify the sender and delete
> > this message from your computer. Any use, distribution, or copying
> > of this email other than by the intended recipient is strictly
> > prohibited. All messages sent to and from us may be monitored to
> > ensure compliance with internal policies and to protect our business.
> > Emails are not secure and cannot be guaranteed to be error free as
> > they can be intercepted, amended, lost or destroyed, or contain
> > viruses. Anyone who communicates with us by email is taken to accept
> these risks.
> >
> > 收发邮件者请注意：
> > 本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。
> > 进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。
> > ************************************************************
> > ********************************************************************
> >
>
>
>
> --
> With Warm regards
>
> Yiming Liu (刘一鸣)
>
> ************************************************************
> ********************************************************************
> The information in this email is confidential and may be legally
> privileged. If you have received this email in error or are not the
> intended recipient, please immediately notify the sender and delete
> this message from your computer. Any use, distribution, or copying of
> this email other than by the intended recipient is strictly
> prohibited. All messages sent to and from us may be monitored to
> ensure compliance with internal policies and to protect our business.
> Emails are not secure and cannot be guaranteed to be error free as
> they can be intercepted, amended, lost or destroyed, or contain
> viruses. Anyone who communicates with us by email is taken to accept these risks.
>
> 收发邮件者请注意：
> 本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。
> 进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。
> ************************************************************
> ********************************************************************
>
--
Best regards,
Shaofeng Shi 史少锋
********************************************************************************************************************************
The information in this email is confidential and may be legally privileged. If you have received this email in error or are not the intended recipient, please immediately notify the sender and delete this message from your computer. Any use, distribution, or copying of this email other than by the intended recipient is strictly prohibited. All messages sent to and from us may be monitored to ensure compliance with internal policies and to protect our business.
Emails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, lost or destroyed, or contain viruses. Anyone who communicates with us by email is taken to accept these risks.
收发邮件者请注意：
本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。
进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。
********************************************************************************************************************************
Re: 答复: hadoop2.2 imcompatible proble
Posted by ShaoFeng Shi <sh...@apache.org>.
You can try to compile Kylin source code with Hadoop 2.2; we never did
that; There are some Hadoop API change (as Billy mentioned) so you need
re-write some codes. Good luck!
2016-11-10 16:42 GMT+08:00 胡志华(万里通科技及数据中心商务智能团队数据分析组) <
HUZHIHUA160@pingan.com.cn>:
> You mean , it's impossible to deploy kylin on hadoop 2.2?
>
> Give up or upgrade hadoop version , another way?
>
> -----邮件原件-----
> 发件人: Billy(Yiming) Liu [mailto:liuyiming.vip@gmail.com]
> 发送时间: 2016年11月10日 12:02
> 收件人: dev
> 主题: Re: hadoop2.2 imcompatible proble
>
> Based on my limited experience, here are some source code changes needed
> because some YARN APIs changed since Hadoop 2.2.
>
> The code is in JobTaskCounterExtractor and HadoopStatusChecker
>
> 2016-11-10 11:23 GMT+08:00 胡志华(万里通科技及数据中心商务智能团队数据分析组) <
> HUZHIHUA160@pingan.com.cn>:
>
> > Hi all
> >
> > As the request of department, I need to deploy another kylin
> > cluster. The problem is hadoop cluster is too old , hadoop2.2.0, cube
> > building will stop at step 3 “Extract Fact Table Distinct Columns”,
> > the error code from mapreduce is as below:
> >
> > I don’t know whether kylin support hadoop 2.2.0, by the way, I
> > have run “wordcount” on kylin’s hadoop client ,it worked.
> >
> > Hadoop cluster won’t be upgraded easily, So I can’t doploy
> > kylin with my present environment ?
> >
> >
> > “Application application_1468678294220_132943 failed 2 times due to AM
> > Container for appattempt_1468678294220_132943_000002 exited with
> > exitCode: 1 due to: Exception from container-launch:
> > org.apache.hadoop.util.Shell$ExitCodeException:
> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> > at org.apache.hadoop.util.Shell.run(Shell.java:379)
> > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(
> > Shell.java:589)
> > at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.
> > launchContainer(DefaultContainerExecutor.java:195)
> > at org.apache.hadoop.yarn.server.nodemanager.containermanager.
> > launcher.ContainerLaunch.call(ContainerLaunch.java:283)
> > at org.apache.hadoop.yarn.server.nodemanager.containermanager.
> > launcher.ContainerLaunch.call(ContainerLaunch.java:79)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1145)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> > .Failing this attempt.. Failing the application.”
> >
> >
> > ************************************************************
> > ********************************************************************
> > The information in this email is confidential and may be legally
> > privileged. If you have received this email in error or are not the
> > intended recipient, please immediately notify the sender and delete
> > this message from your computer. Any use, distribution, or copying of
> > this email other than by the intended recipient is strictly
> > prohibited. All messages sent to and from us may be monitored to
> > ensure compliance with internal policies and to protect our business.
> > Emails are not secure and cannot be guaranteed to be error free as
> > they can be intercepted, amended, lost or destroyed, or contain
> > viruses. Anyone who communicates with us by email is taken to accept
> these risks.
> >
> > 收发邮件者请注意：
> > 本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。
> > 进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。
> > ************************************************************
> > ********************************************************************
> >
>
>
>
> --
> With Warm regards
>
> Yiming Liu (刘一鸣)
>
> ************************************************************
> ********************************************************************
> The information in this email is confidential and may be legally
> privileged. If you have received this email in error or are not the
> intended recipient, please immediately notify the sender and delete this
> message from your computer. Any use, distribution, or copying of this email
> other than by the intended recipient is strictly prohibited. All messages
> sent to and from us may be monitored to ensure compliance with internal
> policies and to protect our business.
> Emails are not secure and cannot be guaranteed to be error free as they
> can be intercepted, amended, lost or destroyed, or contain viruses. Anyone
> who communicates with us by email is taken to accept these risks.
>
> 收发邮件者请注意：
> 本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。
> 进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。
> ************************************************************
> ********************************************************************
>
--
Best regards,
Shaofeng Shi 史少锋
答复: hadoop2.2 imcompatible proble
Posted by "胡志华 (万里通科技及数据中心商务智能团队数据分析组)" <HU...@pingan.com.cn>.
You mean , it's impossible to deploy kylin on hadoop 2.2?
Give up or upgrade hadoop version , another way?
-----邮件原件-----
发件人: Billy(Yiming) Liu [mailto:liuyiming.vip@gmail.com]
发送时间: 2016年11月10日 12:02
收件人: dev
主题: Re: hadoop2.2 imcompatible proble
Based on my limited experience, here are some source code changes needed because some YARN APIs changed since Hadoop 2.2.
The code is in JobTaskCounterExtractor and HadoopStatusChecker
2016-11-10 11:23 GMT+08:00 胡志华(万里通科技及数据中心商务智能团队数据分析组) <
HUZHIHUA160@pingan.com.cn>:
> Hi all
>
> As the request of department, I need to deploy another kylin
> cluster. The problem is hadoop cluster is too old , hadoop2.2.0, cube
> building will stop at step 3 “Extract Fact Table Distinct Columns”,
> the error code from mapreduce is as below:
>
> I don’t know whether kylin support hadoop 2.2.0, by the way, I
> have run “wordcount” on kylin’s hadoop client ,it worked.
>
> Hadoop cluster won’t be upgraded easily, So I can’t doploy
> kylin with my present environment ?
>
>
> “Application application_1468678294220_132943 failed 2 times due to AM
> Container for appattempt_1468678294220_132943_000002 exited with
> exitCode: 1 due to: Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(
> Shell.java:589)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.
> launchContainer(DefaultContainerExecutor.java:195)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.
> launcher.ContainerLaunch.call(ContainerLaunch.java:283)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.
> launcher.ContainerLaunch.call(ContainerLaunch.java:79)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> .Failing this attempt.. Failing the application.”
>
>
> ************************************************************
> ********************************************************************
> The information in this email is confidential and may be legally
> privileged. If you have received this email in error or are not the
> intended recipient, please immediately notify the sender and delete
> this message from your computer. Any use, distribution, or copying of
> this email other than by the intended recipient is strictly
> prohibited. All messages sent to and from us may be monitored to
> ensure compliance with internal policies and to protect our business.
> Emails are not secure and cannot be guaranteed to be error free as
> they can be intercepted, amended, lost or destroyed, or contain
> viruses. Anyone who communicates with us by email is taken to accept these risks.
>
> 收发邮件者请注意：
> 本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。
> 进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。
> ************************************************************
> ********************************************************************
>
--
With Warm regards
Yiming Liu (刘一鸣)
********************************************************************************************************************************
The information in this email is confidential and may be legally privileged. If you have received this email in error or are not the intended recipient, please immediately notify the sender and delete this message from your computer. Any use, distribution, or copying of this email other than by the intended recipient is strictly prohibited. All messages sent to and from us may be monitored to ensure compliance with internal policies and to protect our business.
Emails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, lost or destroyed, or contain viruses. Anyone who communicates with us by email is taken to accept these risks.
收发邮件者请注意：
本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。
进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。
********************************************************************************************************************************