You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by "250635732@qq.com" <25...@qq.com> on 2015/11/26 03:53:25 UTC
NPE throws doing status check
Hi ,
When running job of step 2 , the following exception can be seen. Using the 1.1-incubating release version, with
hadoop 2.7.1, hbase 0.98.10-hadoop2, hive 1.2
[pool-5-thread-2]:[2015-11-25 19:31:38,805][ERROR][org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:91)] - error check status
java.lang.NullPointerException
at org.apache.kylin.job.tools.HadoopStatusGetter.get(HadoopStatusGetter.java:56)
at org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:56)
at org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:137)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
What could be the root cause ?
Best,
Sun.
250635732@qq.com
Re: Re: NPE throws doing status check
Posted by hongbin ma <ma...@apache.org>.
if you're just beginning, it is highly recommended that you start from a
typical sandbox env to exclude all the installation factors.
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
Re: Re: NPE throws doing status check
Posted by "250635732@qq.com" <25...@qq.com>.
Ah, I see. The installations are all manuelly configured and indeed problems
exist with config side. I shall debug more to locate the cause and shall let u know
with any progress.
Best,
Sun.
250635732@qq.com
From: hongbin ma
Date: 2015-11-27 12:17
To: dev
Subject: Re: Re: NPE throws doing status check
Local mode might be a issue, however i have no idea with your env. but if
you're using a standard sandbox env (like hdp sandbox), you don't have to
do any modification as long as you follow our tutorial
On Fri, Nov 27, 2015 at 12:13 PM, 250635732@qq.com <25...@qq.com> wrote:
> Hi,
> The instances are all runing with local mode in only one virtual machine
> host. Could that be
> the issue ? We modified the kylin.properties with restserver as
> "tts2-VirtualBox:7070" , anywhere
> else do we need to modify the RM related hostname ?
>
> Thanks a lot for following the issue.
> Best,
> Sun.
>
>
>
> 250635732@qq.com
>
> From: hongbin ma
> Date: 2015-11-27 12:08
> To: dev
> Subject: Re: Re: NPE throws doing status check
> it seems RM is expecting a number format when you provide
> "local1591747746",
> are you sure "local1591747746" is your job id? why there's a "local"
> prefix?
>
> On Fri, Nov 27, 2015 at 12:00 PM, 250635732@qq.com <25...@qq.com>
> wrote:
>
> > Hi , hongbin
> >
> > I am trying with the following url :
> >
> >
> http://tts2-VirtualBox:8088/ws/v1/cluster/apps/application_local1591747746_0001?anonymous=true
> >
> > still get no luck and the response is just the seem :
> >
> > This XML file does not appear to have any style information associated
> > with it. The document tree is shown below.
> > <RemoteException>
> > <exception>NumberFormatException</exception>
> > <message>For input string: "local1591747746"</message>
> > <javaClassName>java.lang.NumberFormatException</javaClassName>
> > </RemoteException>
> >
> >
> >
> > 250635732@qq.com
> >
> > From: hongbin ma
> > Date: 2015-11-27 10:51
> > To: dev
> > Subject: Re: Re: NPE throws doing status check
> > according to the code logic you should try
> >
> > http://tts2-VirtualBox:8088/ws/v1/cluster/apps/
> > application_
> > <
> >
> http://tts2-virtualbox:8088/ws/v1/cluster/apps/$%7Bjob_id%7D?anonymous=true
> > >
> > local2013102403_0001?anonymous=true
> > <
> >
> http://tts2-virtualbox:8088/ws/v1/cluster/apps/$%7Bjob_id%7D?anonymous=true
> > >
> >
> >
> >
> >
> > --
> > Regards,
> >
> > *Bin Mahone | 马洪宾*
> > Apache Kylin: http://kylin.io
> > Github: https://github.com/binmahone
> >
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
Re: Re: NPE throws doing status check
Posted by hongbin ma <ma...@apache.org>.
Local mode might be a issue, however i have no idea with your env. but if
you're using a standard sandbox env (like hdp sandbox), you don't have to
do any modification as long as you follow our tutorial
On Fri, Nov 27, 2015 at 12:13 PM, 250635732@qq.com <25...@qq.com> wrote:
> Hi,
> The instances are all runing with local mode in only one virtual machine
> host. Could that be
> the issue ? We modified the kylin.properties with restserver as
> "tts2-VirtualBox:7070" , anywhere
> else do we need to modify the RM related hostname ?
>
> Thanks a lot for following the issue.
> Best,
> Sun.
>
>
>
> 250635732@qq.com
>
> From: hongbin ma
> Date: 2015-11-27 12:08
> To: dev
> Subject: Re: Re: NPE throws doing status check
> it seems RM is expecting a number format when you provide
> "local1591747746",
> are you sure "local1591747746" is your job id? why there's a "local"
> prefix?
>
> On Fri, Nov 27, 2015 at 12:00 PM, 250635732@qq.com <25...@qq.com>
> wrote:
>
> > Hi , hongbin
> >
> > I am trying with the following url :
> >
> >
> http://tts2-VirtualBox:8088/ws/v1/cluster/apps/application_local1591747746_0001?anonymous=true
> >
> > still get no luck and the response is just the seem :
> >
> > This XML file does not appear to have any style information associated
> > with it. The document tree is shown below.
> > <RemoteException>
> > <exception>NumberFormatException</exception>
> > <message>For input string: "local1591747746"</message>
> > <javaClassName>java.lang.NumberFormatException</javaClassName>
> > </RemoteException>
> >
> >
> >
> > 250635732@qq.com
> >
> > From: hongbin ma
> > Date: 2015-11-27 10:51
> > To: dev
> > Subject: Re: Re: NPE throws doing status check
> > according to the code logic you should try
> >
> > http://tts2-VirtualBox:8088/ws/v1/cluster/apps/
> > application_
> > <
> >
> http://tts2-virtualbox:8088/ws/v1/cluster/apps/$%7Bjob_id%7D?anonymous=true
> > >
> > local2013102403_0001?anonymous=true
> > <
> >
> http://tts2-virtualbox:8088/ws/v1/cluster/apps/$%7Bjob_id%7D?anonymous=true
> > >
> >
> >
> >
> >
> > --
> > Regards,
> >
> > *Bin Mahone | 马洪宾*
> > Apache Kylin: http://kylin.io
> > Github: https://github.com/binmahone
> >
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
Re: Re: NPE throws doing status check
Posted by "250635732@qq.com" <25...@qq.com>.
Hi,
The instances are all runing with local mode in only one virtual machine host. Could that be
the issue ? We modified the kylin.properties with restserver as "tts2-VirtualBox:7070" , anywhere
else do we need to modify the RM related hostname ?
Thanks a lot for following the issue.
Best,
Sun.
250635732@qq.com
From: hongbin ma
Date: 2015-11-27 12:08
To: dev
Subject: Re: Re: NPE throws doing status check
it seems RM is expecting a number format when you provide "local1591747746",
are you sure "local1591747746" is your job id? why there's a "local" prefix?
On Fri, Nov 27, 2015 at 12:00 PM, 250635732@qq.com <25...@qq.com> wrote:
> Hi , hongbin
>
> I am trying with the following url :
>
> http://tts2-VirtualBox:8088/ws/v1/cluster/apps/application_local1591747746_0001?anonymous=true
>
> still get no luck and the response is just the seem :
>
> This XML file does not appear to have any style information associated
> with it. The document tree is shown below.
> <RemoteException>
> <exception>NumberFormatException</exception>
> <message>For input string: "local1591747746"</message>
> <javaClassName>java.lang.NumberFormatException</javaClassName>
> </RemoteException>
>
>
>
> 250635732@qq.com
>
> From: hongbin ma
> Date: 2015-11-27 10:51
> To: dev
> Subject: Re: Re: NPE throws doing status check
> according to the code logic you should try
>
> http://tts2-VirtualBox:8088/ws/v1/cluster/apps/
> application_
> <
> http://tts2-virtualbox:8088/ws/v1/cluster/apps/$%7Bjob_id%7D?anonymous=true
> >
> local2013102403_0001?anonymous=true
> <
> http://tts2-virtualbox:8088/ws/v1/cluster/apps/$%7Bjob_id%7D?anonymous=true
> >
>
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
Re: Re: NPE throws doing status check
Posted by hongbin ma <ma...@apache.org>.
it seems RM is expecting a number format when you provide "local1591747746",
are you sure "local1591747746" is your job id? why there's a "local" prefix?
On Fri, Nov 27, 2015 at 12:00 PM, 250635732@qq.com <25...@qq.com> wrote:
> Hi , hongbin
>
> I am trying with the following url :
>
> http://tts2-VirtualBox:8088/ws/v1/cluster/apps/application_local1591747746_0001?anonymous=true
>
> still get no luck and the response is just the seem :
>
> This XML file does not appear to have any style information associated
> with it. The document tree is shown below.
> <RemoteException>
> <exception>NumberFormatException</exception>
> <message>For input string: "local1591747746"</message>
> <javaClassName>java.lang.NumberFormatException</javaClassName>
> </RemoteException>
>
>
>
> 250635732@qq.com
>
> From: hongbin ma
> Date: 2015-11-27 10:51
> To: dev
> Subject: Re: Re: NPE throws doing status check
> according to the code logic you should try
>
> http://tts2-VirtualBox:8088/ws/v1/cluster/apps/
> application_
> <
> http://tts2-virtualbox:8088/ws/v1/cluster/apps/$%7Bjob_id%7D?anonymous=true
> >
> local2013102403_0001?anonymous=true
> <
> http://tts2-virtualbox:8088/ws/v1/cluster/apps/$%7Bjob_id%7D?anonymous=true
> >
>
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
Re: Re: NPE throws doing status check
Posted by "250635732@qq.com" <25...@qq.com>.
Hi , hongbin
I am trying with the following url :
http://tts2-VirtualBox:8088/ws/v1/cluster/apps/application_local1591747746_0001?anonymous=true
still get no luck and the response is just the seem :
This XML file does not appear to have any style information associated with it. The document tree is shown below.
<RemoteException>
<exception>NumberFormatException</exception>
<message>For input string: "local1591747746"</message>
<javaClassName>java.lang.NumberFormatException</javaClassName>
</RemoteException>
250635732@qq.com
From: hongbin ma
Date: 2015-11-27 10:51
To: dev
Subject: Re: Re: NPE throws doing status check
according to the code logic you should try
http://tts2-VirtualBox:8088/ws/v1/cluster/apps/
application_
<http://tts2-virtualbox:8088/ws/v1/cluster/apps/$%7Bjob_id%7D?anonymous=true>
local2013102403_0001?anonymous=true
<http://tts2-virtualbox:8088/ws/v1/cluster/apps/$%7Bjob_id%7D?anonymous=true>
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
Re: Re: NPE throws doing status check
Posted by hongbin ma <ma...@apache.org>.
according to the code logic you should try
http://tts2-VirtualBox:8088/ws/v1/cluster/apps/
application_
<http://tts2-virtualbox:8088/ws/v1/cluster/apps/$%7Bjob_id%7D?anonymous=true>
local2013102403_0001?anonymous=true
<http://tts2-virtualbox:8088/ws/v1/cluster/apps/$%7Bjob_id%7D?anonymous=true>
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
Re: Re: NPE throws doing status check
Posted by "250635732@qq.com" <25...@qq.com>.
Sorry, maybe the pic did not show normally. I attached it.
250635732@qq.com
From: 250635732@qq.com
Date: 2015-11-27 10:48
To: dev
Subject: Re: Re: NPE throws doing status check
The full URL is as following :
http://tts2-VirtualBox:8088/ws/v1/cluster/apps/${job_id}?anonymous=true
I replace the job_id with job_local2013102403_0001 , The following pic show my debug screenshot
250635732@qq.com
From: hongbin ma
Date: 2015-11-27 10:44
To: dev
Subject: Re: Re: NPE throws doing status check
where did you get the "job_id", and can you show us the full URL?
in HadoopStatusGetter:53 it replaces "job" to "application", I don't know
if it suites your case
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
Re: Re: NPE throws doing status check
Posted by "250635732@qq.com" <25...@qq.com>.
The full URL is as following :
http://tts2-VirtualBox:8088/ws/v1/cluster/apps/${job_id}?anonymous=true
I replace the job_id with job_local2013102403_0001 , The following pic show my debug screenshot
250635732@qq.com
From: hongbin ma
Date: 2015-11-27 10:44
To: dev
Subject: Re: Re: NPE throws doing status check
where did you get the "job_id", and can you show us the full URL?
in HadoopStatusGetter:53 it replaces "job" to "application", I don't know
if it suites your case
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
Re: Re: NPE throws doing status check
Posted by hongbin ma <ma...@apache.org>.
where did you get the "job_id", and can you show us the full URL?
in HadoopStatusGetter:53 it replaces "job" to "application", I don't know
if it suites your case
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
Re: Re: NPE throws doing status check
Posted by "250635732@qq.com" <25...@qq.com>.
Hi, hongbin
Yes. After debugged into the code, I found the URL reponse is the following xml pattern :
This XML file does not appear to have any style information associated with it. The document tree is shown below.
<RemoteException>
<exception>NumberFormatException</exception>
<message>For input string: "local2013102403"</message>
<javaClassName>java.lang.NumberFormatException</javaClassName>
</RemoteException>
Our job_id is "job_local2013102403_0001"
I cannot see any clue for the numberformat exception. Please help on this.
Best,
Sun.
250635732@qq.com
From: hongbin ma
Date: 2015-11-27 10:34
To: dev
Subject: Re: NPE throws doing status check
the release correspond to branch: v1.1-release, you can checkout the code.
the HadoopStatusChecker visits a URL which contains job status in json, and
parse the text to get the values. The pattern of URL is:
http(s)://${yarn.resourcemanager.webapp.address}/ws/v1/cluster/apps/${job_id}?anonymous=true
(check MapReduceExecutable:177)
My guess is that the text returned from your cluster's resource manager is
not as expected, which causes NPE in HadoopStatusGetter.java:56
can you check what is returned from the URL? for the real URL you can
either check backend log(in tomcat/log/kylin.log) or the log icon for each
cubing step.
On Thu, Nov 26, 2015 at 10:53 AM, 250635732@qq.com <25...@qq.com> wrote:
> Hi ,
>
> When running job of step 2 , the following exception can be seen. Using
> the 1.1-incubating release version, with
> hadoop 2.7.1, hbase 0.98.10-hadoop2, hive 1.2
>
> [pool-5-thread-2]:[2015-11-25
> 19:31:38,805][ERROR][org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:91)]
> - error check status
> java.lang.NullPointerException
> at
> org.apache.kylin.job.tools.HadoopStatusGetter.get(HadoopStatusGetter.java:56)
> at
> org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:56)
> at
> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:137)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> What could be the root cause ?
>
> Best,
> Sun.
>
>
> 250635732@qq.com
>
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
Re: NPE throws doing status check
Posted by hongbin ma <ma...@apache.org>.
the release correspond to branch: v1.1-release, you can checkout the code.
the HadoopStatusChecker visits a URL which contains job status in json, and
parse the text to get the values. The pattern of URL is:
http(s)://${yarn.resourcemanager.webapp.address}/ws/v1/cluster/apps/${job_id}?anonymous=true
(check MapReduceExecutable:177)
My guess is that the text returned from your cluster's resource manager is
not as expected, which causes NPE in HadoopStatusGetter.java:56
can you check what is returned from the URL? for the real URL you can
either check backend log(in tomcat/log/kylin.log) or the log icon for each
cubing step.
On Thu, Nov 26, 2015 at 10:53 AM, 250635732@qq.com <25...@qq.com> wrote:
> Hi ,
>
> When running job of step 2 , the following exception can be seen. Using
> the 1.1-incubating release version, with
> hadoop 2.7.1, hbase 0.98.10-hadoop2, hive 1.2
>
> [pool-5-thread-2]:[2015-11-25
> 19:31:38,805][ERROR][org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:91)]
> - error check status
> java.lang.NullPointerException
> at
> org.apache.kylin.job.tools.HadoopStatusGetter.get(HadoopStatusGetter.java:56)
> at
> org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:56)
> at
> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:137)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> What could be the root cause ?
>
> Best,
> Sun.
>
>
> 250635732@qq.com
>
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone