You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "ChiaHung Lin (JIRA)" <ji...@apache.org> on 2011/04/18 08:16:06 UTC
[jira] [Created] (HAMA-378) Lost connection to BSP Master
[host/ip:port]. Retrying...
Lost connection to BSP Master [host/ip:port]. Retrying...
-----------------------------------------------------------
Key: HAMA-378
URL: https://issues.apache.org/jira/browse/HAMA-378
Project: Hama
Issue Type: Improvement
Components: bsp
Reporter: ChiaHung Lin
Priority: Minor
When submitting a job, at the server side the GroomServer reports message
Lost connection to BSP Master [host/ip:port]. Retrying... java.lang.NullPointerException
This seems because the script hama only includes jar files (such as hama-0.3.0-examples.jar) located under basedir, the same folder as build.xml. Also, a quick check shows that when submitting a job, the ClassLader used is sun.misc.Launcher$AppClassLoader, instead of java.net.URLClassLoader. This might not be the root cause, but a further investigation may be required.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-378) Lost connection to BSP Master
[host/ip:port]. Retrying...
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021045#comment-13021045 ]
Thomas Jungblut commented on HAMA-378:
--------------------------------------
Hey, I think I have found a related issue with the classloader:
Since message patch the user can implement his own message type. Running with this message tells me on the grooms:
{noformat}
2011-04-18 16:23:37,619 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 58648: readAndProcess threw exception java.lang.RuntimeException: readObject can't find class de.jungblut.hama.model.IntegerDoubleMessage. Count of bytes read: 0
java.lang.RuntimeException: readObject can't find class de.jungblut.hama.model.IntegerDoubleMessage
at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:233)
at org.apache.hadoop.ipc.RPC$Invocation.readFields(RPC.java:103)
at org.apache.hadoop.ipc.Server$Connection.processData(Server.java:908)
at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:847)
at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:421)
at org.apache.hadoop.ipc.Server$Listener.run(Server.java:330)
Caused by: java.lang.ClassNotFoundException: de.jungblut.hama.model.IntegerDoubleMessage
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:231)
... 5 more
{noformat}
Obviously the jar contains this class, the jar has been transfered to the grooms and got unpacked:
{noformat}
hadoop@raynor:/usr/local$ /srv/hama/bsp/local/groomServer/attempt_201104181622_0001_000001_0/
job.jar job.xml work/
hadoop@raynor:/usr/local$ /srv/hama/bsp/local/groomServer/attempt_201104181622_0001_000001_0/work/
de/ META-INF/
hadoop@raynor:/usr/local$ /srv/hama/bsp/local/groomServer/attempt_201104181622_0001_000001_0/work/de/jungblut/
graph/ hama/ mapreduce/
hadoop@raynor:/usr/local$ /srv/hama/bsp/local/groomServer/attempt_201104181622_0001_000001_0/work/de/jungblut/hama/
bsp/ graph/ mapreduce/ model/ partioning/
hadoop@raynor:/usr/local$ ls /srv/hama/bsp/local/groomServer/attempt_201104181622_0001_000001_0/work/de/jungblut/hama/model/
BooleanMessage.class IntegerDoubleMessage.class WeightedVertex.class
FullIntegerMessage.class IntegerMessage.class
hadoop@raynor:/usr/local$ ls /srv/hama/bsp/local/groomServer/attempt_201104181622_0001_000001_0/work/de/jungblut/hama/model/
BooleanMessage.class IntegerDoubleMessage.class WeightedVertex.class
FullIntegerMessage.class IntegerMessage.class
{noformat}
Looking at the stacktrace reveals the AppClassLoader too.
> Lost connection to BSP Master [host/ip:port]. Retrying...
> -----------------------------------------------------------
>
> Key: HAMA-378
> URL: https://issues.apache.org/jira/browse/HAMA-378
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Reporter: ChiaHung Lin
> Priority: Minor
>
> When submitting a job, at the server side the GroomServer reports message
> Lost connection to BSP Master [host/ip:port]. Retrying... java.lang.NullPointerException
> This seems because the script hama only includes jar files (such as hama-0.3.0-examples.jar) located under basedir, the same folder as build.xml. Also, a quick check shows that when submitting a job, the ClassLader used is sun.misc.Launcher$AppClassLoader, instead of java.net.URLClassLoader. This might not be the root cause, but a further investigation may be required.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-378) Lost connection to BSP Master
[host/ip:port]. Retrying...
Posted by "ChiaHung Lin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025568#comment-13025568 ]
ChiaHung Lin commented on HAMA-378:
-----------------------------------
The jar file exists on the groom server, but the file to be unpacked in the path is not a correct one.
>From what has been discovered, it looks like when executing findContainingJar() in BSPJob, my_class.getClassLoader() would be different from what has been proposed in RunJar, in which the context class loader is set to URLClassLoader. Also, the jar file in command e.g. `hama jar path/to/eaxmple.jar` is added to classpath (if what perceived is correct). But in the findContainingJar function, the context loader is not the one proposed i.e. URLClassLoader; therefore, url.getProtocol() only returns `file' instead of `jar', resulting the execution of findContaingJar function returns null. Thus, bsp.jar is not set in configuration.
This is just what is discovered, but it might be a red herring and a further investigation may be required.
> Lost connection to BSP Master [host/ip:port]. Retrying...
> -----------------------------------------------------------
>
> Key: HAMA-378
> URL: https://issues.apache.org/jira/browse/HAMA-378
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Reporter: ChiaHung Lin
> Priority: Minor
>
> When submitting a job, at the server side the GroomServer reports message
> Lost connection to BSP Master [host/ip:port]. Retrying... java.lang.NullPointerException
> This seems because the script hama only includes jar files (such as hama-0.3.0-examples.jar) located under basedir, the same folder as build.xml. Also, a quick check shows that when submitting a job, the ClassLader used is sun.misc.Launcher$AppClassLoader, instead of java.net.URLClassLoader. This might not be the root cause, but a further investigation may be required.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HAMA-378) Lost connection to BSP Master
[host/ip:port]. Retrying...
Posted by "Edward J. Yoon (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon resolved HAMA-378.
---------------------------------
Resolution: Invalid
Assignee: Edward J. Yoon
> Lost connection to BSP Master [host/ip:port]. Retrying...
> -----------------------------------------------------------
>
> Key: HAMA-378
> URL: https://issues.apache.org/jira/browse/HAMA-378
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Reporter: ChiaHung Lin
> Assignee: Edward J. Yoon
> Priority: Minor
>
> When submitting a job, at the server side the GroomServer reports message
> Lost connection to BSP Master [host/ip:port]. Retrying... java.lang.NullPointerException
> This seems because the script hama only includes jar files (such as hama-0.3.0-examples.jar) located under basedir, the same folder as build.xml. Also, a quick check shows that when submitting a job, the ClassLader used is sun.misc.Launcher$AppClassLoader, instead of java.net.URLClassLoader. This might not be the root cause, but a further investigation may be required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HAMA-378) Lost connection to BSP
Master [host/ip:port]. Retrying...
Posted by "ChiaHung Lin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025568#comment-13025568 ]
ChiaHung Lin edited comment on HAMA-378 at 4/27/11 2:38 AM:
------------------------------------------------------------
The groom server has path referred to the jar file copied it. And the jar file exists on the groom server, but the file to be unpacked in the path is not a correct one.
>From what has been discovered, it looks like when executing findContainingJar() in BSPJob, my_class.getClassLoader() would be different from what has been proposed in RunJar, in which the context class loader is set to URLClassLoader. Also, the jar file in command e.g. `hama jar path/to/eaxmple.jar` is added to classpath (if what perceived is correct). But in the findContainingJar function, the context loader is not the one proposed i.e. URLClassLoader; therefore, url.getProtocol() only returns `file' instead of `jar', resulting the execution of findContaingJar function returns null. Thus, bsp.jar is not set in configuration.
This is just what is discovered, but it might be a red herring and a further investigation may be required.
was (Author: chl501):
The jar file exists on the groom server, but the file to be unpacked in the path is not a correct one.
>From what has been discovered, it looks like when executing findContainingJar() in BSPJob, my_class.getClassLoader() would be different from what has been proposed in RunJar, in which the context class loader is set to URLClassLoader. Also, the jar file in command e.g. `hama jar path/to/eaxmple.jar` is added to classpath (if what perceived is correct). But in the findContainingJar function, the context loader is not the one proposed i.e. URLClassLoader; therefore, url.getProtocol() only returns `file' instead of `jar', resulting the execution of findContaingJar function returns null. Thus, bsp.jar is not set in configuration.
This is just what is discovered, but it might be a red herring and a further investigation may be required.
> Lost connection to BSP Master [host/ip:port]. Retrying...
> -----------------------------------------------------------
>
> Key: HAMA-378
> URL: https://issues.apache.org/jira/browse/HAMA-378
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Reporter: ChiaHung Lin
> Priority: Minor
>
> When submitting a job, at the server side the GroomServer reports message
> Lost connection to BSP Master [host/ip:port]. Retrying... java.lang.NullPointerException
> This seems because the script hama only includes jar files (such as hama-0.3.0-examples.jar) located under basedir, the same folder as build.xml. Also, a quick check shows that when submitting a job, the ClassLader used is sun.misc.Launcher$AppClassLoader, instead of java.net.URLClassLoader. This might not be the root cause, but a further investigation may be required.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-378) Lost connection to BSP Master
[host/ip:port]. Retrying...
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025632#comment-13025632 ]
Thomas Jungblut commented on HAMA-378:
--------------------------------------
Yea, this could be the problem... But this is basically the same in Hadoop's source code. The only difference is that this class searching is invoked in the context of a configuration.
> Lost connection to BSP Master [host/ip:port]. Retrying...
> -----------------------------------------------------------
>
> Key: HAMA-378
> URL: https://issues.apache.org/jira/browse/HAMA-378
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Reporter: ChiaHung Lin
> Priority: Minor
>
> When submitting a job, at the server side the GroomServer reports message
> Lost connection to BSP Master [host/ip:port]. Retrying... java.lang.NullPointerException
> This seems because the script hama only includes jar files (such as hama-0.3.0-examples.jar) located under basedir, the same folder as build.xml. Also, a quick check shows that when submitting a job, the ClassLader used is sun.misc.Launcher$AppClassLoader, instead of java.net.URLClassLoader. This might not be the root cause, but a further investigation may be required.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-378) Lost connection to BSP Master
[host/ip:port]. Retrying...
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025208#comment-13025208 ]
Thomas Jungblut commented on HAMA-378:
--------------------------------------
When deserializing the configuration on the grooms, does the classloader refer to the path where the jar has been unpacked?
> Lost connection to BSP Master [host/ip:port]. Retrying...
> -----------------------------------------------------------
>
> Key: HAMA-378
> URL: https://issues.apache.org/jira/browse/HAMA-378
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Reporter: ChiaHung Lin
> Priority: Minor
>
> When submitting a job, at the server side the GroomServer reports message
> Lost connection to BSP Master [host/ip:port]. Retrying... java.lang.NullPointerException
> This seems because the script hama only includes jar files (such as hama-0.3.0-examples.jar) located under basedir, the same folder as build.xml. Also, a quick check shows that when submitting a job, the ClassLader used is sun.misc.Launcher$AppClassLoader, instead of java.net.URLClassLoader. This might not be the root cause, but a further investigation may be required.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HAMA-378) Lost connection to BSP
Master [host/ip:port]. Retrying...
Posted by "ChiaHung Lin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025568#comment-13025568 ]
ChiaHung Lin edited comment on HAMA-378 at 4/27/11 2:39 AM:
------------------------------------------------------------
The groom server has path referred to the jar file copied to it. And the jar file exists on the groom server, but the file to be unpacked in the path is not a correct one.
>From what has been discovered, it looks like when executing findContainingJar() in BSPJob, my_class.getClassLoader() would be different from what has been proposed in RunJar, in which the context class loader is set to URLClassLoader. Also, the jar file in command e.g. `hama jar path/to/eaxmple.jar` is added to classpath (if what perceived is correct). But in the findContainingJar function, the context loader is not the one proposed i.e. URLClassLoader; therefore, url.getProtocol() only returns `file' instead of `jar', resulting the execution of findContaingJar function returns null. Thus, bsp.jar is not set in configuration.
This is just what is discovered, but it might be a red herring and a further investigation may be required.
was (Author: chl501):
The groom server has path referred to the jar file copied it. And the jar file exists on the groom server, but the file to be unpacked in the path is not a correct one.
>From what has been discovered, it looks like when executing findContainingJar() in BSPJob, my_class.getClassLoader() would be different from what has been proposed in RunJar, in which the context class loader is set to URLClassLoader. Also, the jar file in command e.g. `hama jar path/to/eaxmple.jar` is added to classpath (if what perceived is correct). But in the findContainingJar function, the context loader is not the one proposed i.e. URLClassLoader; therefore, url.getProtocol() only returns `file' instead of `jar', resulting the execution of findContaingJar function returns null. Thus, bsp.jar is not set in configuration.
This is just what is discovered, but it might be a red herring and a further investigation may be required.
> Lost connection to BSP Master [host/ip:port]. Retrying...
> -----------------------------------------------------------
>
> Key: HAMA-378
> URL: https://issues.apache.org/jira/browse/HAMA-378
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Reporter: ChiaHung Lin
> Priority: Minor
>
> When submitting a job, at the server side the GroomServer reports message
> Lost connection to BSP Master [host/ip:port]. Retrying... java.lang.NullPointerException
> This seems because the script hama only includes jar files (such as hama-0.3.0-examples.jar) located under basedir, the same folder as build.xml. Also, a quick check shows that when submitting a job, the ClassLader used is sun.misc.Launcher$AppClassLoader, instead of java.net.URLClassLoader. This might not be the root cause, but a further investigation may be required.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira