You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Thomas Jungblut (JIRA)" <ji...@apache.org> on 2012/05/27 21:45:22 UTC
[jira] [Created] (HAMA-583) Misc release tasks
Thomas Jungblut created HAMA-583:
------------------------------------
Summary: Misc release tasks
Key: HAMA-583
URL: https://issues.apache.org/jira/browse/HAMA-583
Project: Hama
Issue Type: Improvement
Components: build
Affects Versions: 0.5.0
Reporter: Thomas Jungblut
Priority: Blocker
Fix For: 0.5.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-583) Misc release tasks
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284230#comment-13284230 ]
Thomas Jungblut commented on HAMA-583:
--------------------------------------
This
{noformat}
2012-05-27 20:25:28,916 ERROR org.apache.hama.bsp.SimpleTaskScheduler: Error submitting job
java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.hama.bsp.SimpleTaskScheduler$JobProcessor.schedule(SimpleTaskScheduler.java:141)
at org.apache.hama.bsp.SimpleTaskScheduler$JobProcessor.run(SimpleTaskScheduler.java:116)
Caused by: java.lang.NullPointerException
at org.apache.hama.bsp.TaskInProgress.getTaskToRun(TaskInProgress.java:149)
at org.apache.hama.bsp.JobInProgress.obtainNewTask(JobInProgress.java:268)
at org.apache.hama.bsp.SimpleTaskScheduler$TaskWorker.call(SimpleTaskScheduler.java:191)
at org.apache.hama.bsp.SimpleTaskScheduler$TaskWorker.call(SimpleTaskScheduler.java:160)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
{noformat}
happens if the grooms are not correctly configured.
{{{
#TaskInProgress->line 145
String location = possibleLocations[i];
GroomServerStatus groom = grooms.get(location);
}}}
This checks the given DFS location, if it does not match the groom it will throw these NPEs.
We should throw a better exception here.
> Misc release tasks
> ------------------
>
> Key: HAMA-583
> URL: https://issues.apache.org/jira/browse/HAMA-583
> Project: Hama
> Issue Type: Improvement
> Components: build
> Affects Versions: 0.5.0
> Reporter: Thomas Jungblut
> Priority: Blocker
> Fix For: 0.5.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HAMA-583) Misc release tasks
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon resolved HAMA-583.
---------------------------------
Resolution: Fixed
> Misc release tasks
> ------------------
>
> Key: HAMA-583
> URL: https://issues.apache.org/jira/browse/HAMA-583
> Project: Hama
> Issue Type: Improvement
> Components: build
> Affects Versions: 0.5.0
> Reporter: Thomas Jungblut
> Assignee: Edward J. Yoon
> Priority: Blocker
> Fix For: 0.5.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-583) Misc release tasks
Posted by "Suraj Menon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398165#comment-13398165 ]
Suraj Menon commented on HAMA-583:
----------------------------------
The NPE is a bug. HAMA-595 fixes it. But getting groom as null is also possible when the GroomServer process is dead on the machine and datanode on the machine is up and has the interested block. This could also rise when the groomservers are not configured to run on all the machines where HDFS daemons are running. We are looking for a mechanism to validate location provided by HDFS. How about checking if the location is present in all current grooms and blacklisted ones? Even then we are not sure about throwing an error here by the aforesaid scenarios.
> Misc release tasks
> ------------------
>
> Key: HAMA-583
> URL: https://issues.apache.org/jira/browse/HAMA-583
> Project: Hama
> Issue Type: Improvement
> Components: build
> Affects Versions: 0.5.0
> Reporter: Thomas Jungblut
> Priority: Blocker
> Fix For: 0.5.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-583) Misc release tasks
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284240#comment-13284240 ]
Thomas Jungblut commented on HAMA-583:
--------------------------------------
Also pinging seems very aggressive, in Cloud where you have long response delay due to context switching and stuff it may be better to increase the interval, we should lift from 5s to 1 minute. If there is no connection to groom temporary, it should retry n-times.
> Misc release tasks
> ------------------
>
> Key: HAMA-583
> URL: https://issues.apache.org/jira/browse/HAMA-583
> Project: Hama
> Issue Type: Improvement
> Components: build
> Affects Versions: 0.5.0
> Reporter: Thomas Jungblut
> Priority: Blocker
> Fix For: 0.5.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HAMA-583) Misc release tasks
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon reassigned HAMA-583:
-----------------------------------
Assignee: Edward J. Yoon
> Misc release tasks
> ------------------
>
> Key: HAMA-583
> URL: https://issues.apache.org/jira/browse/HAMA-583
> Project: Hama
> Issue Type: Improvement
> Components: build
> Affects Versions: 0.5.0
> Reporter: Thomas Jungblut
> Assignee: Edward J. Yoon
> Priority: Blocker
> Fix For: 0.5.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HAMA-583) Misc release tasks
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284230#comment-13284230 ]
Thomas Jungblut edited comment on HAMA-583 at 5/28/12 6:02 AM:
---------------------------------------------------------------
This
{noformat}
2012-05-27 20:25:28,916 ERROR org.apache.hama.bsp.SimpleTaskScheduler: Error submitting job
java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.hama.bsp.SimpleTaskScheduler$JobProcessor.schedule(SimpleTaskScheduler.java:141)
at org.apache.hama.bsp.SimpleTaskScheduler$JobProcessor.run(SimpleTaskScheduler.java:116)
Caused by: java.lang.NullPointerException
at org.apache.hama.bsp.TaskInProgress.getTaskToRun(TaskInProgress.java:149)
at org.apache.hama.bsp.JobInProgress.obtainNewTask(JobInProgress.java:268)
at org.apache.hama.bsp.SimpleTaskScheduler$TaskWorker.call(SimpleTaskScheduler.java:191)
at org.apache.hama.bsp.SimpleTaskScheduler$TaskWorker.call(SimpleTaskScheduler.java:160)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
{noformat}
happens if the grooms are not correctly configured.
{noformat}
#TaskInProgress->line 145
String location = possibleLocations[i];
GroomServerStatus groom = grooms.get(location);
{noformat}
This checks the given DFS location, if it does not match the groom it will throw these NPEs.
We should throw a better exception here.
Example
{noformat}
2012-05-27 20:57:15,723 INFO org.apache.hama.bsp.TaskInProgress: GROOMS: {ip-10-xxx-73-25.eu-west-1.compute.internal=org.apache.hama.bsp.GroomServerStatus@74243402, ip-10-xxx-166-9.eu-west-1.compute.internal=org.apache.hama.bsp.GroomServerStatus@2883f70a, ip-10-xxx-97-203.eu-west-1.compute.internal=org.apache.hama.bsp.GroomServerStatus@775898e8, ip-10-xxx-129-154.eu-west-1.compute.internal=org.apache.hama.bsp.GroomServerStatus@a732038d}
2012-05-27 20:57:15,723 INFO org.apache.hama.bsp.TaskInProgress: LOCATION: 10.xxx.166.9
2012-05-27 20:57:15,723 INFO org.apache.hama.bsp.TaskInProgress: LOCATION: 10.xxx.73.25
2012-05-27 20:57:15,723 INFO org.apache.hama.bsp.TaskInProgress: LOCATION: 10.xxx.97.203
{noformat}
and it can't find the location by the splits in the groom map.
I believe this is because the datanode is not doing a reverse DNS anymore instead it takes just the hostname of the machine configured in /etc/hostname.
was (Author: thomas.jungblut):
This
{noformat}
2012-05-27 20:25:28,916 ERROR org.apache.hama.bsp.SimpleTaskScheduler: Error submitting job
java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.hama.bsp.SimpleTaskScheduler$JobProcessor.schedule(SimpleTaskScheduler.java:141)
at org.apache.hama.bsp.SimpleTaskScheduler$JobProcessor.run(SimpleTaskScheduler.java:116)
Caused by: java.lang.NullPointerException
at org.apache.hama.bsp.TaskInProgress.getTaskToRun(TaskInProgress.java:149)
at org.apache.hama.bsp.JobInProgress.obtainNewTask(JobInProgress.java:268)
at org.apache.hama.bsp.SimpleTaskScheduler$TaskWorker.call(SimpleTaskScheduler.java:191)
at org.apache.hama.bsp.SimpleTaskScheduler$TaskWorker.call(SimpleTaskScheduler.java:160)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
{noformat}
happens if the grooms are not correctly configured.
{{{
#TaskInProgress->line 145
String location = possibleLocations[i];
GroomServerStatus groom = grooms.get(location);
}}}
This checks the given DFS location, if it does not match the groom it will throw these NPEs.
We should throw a better exception here.
Actually we shouldn't do a reverse DNS in our groom hostnames, just use the configured ones in /etc/hostname just like the datanode does. So this won't give inconsistencies.
> Misc release tasks
> ------------------
>
> Key: HAMA-583
> URL: https://issues.apache.org/jira/browse/HAMA-583
> Project: Hama
> Issue Type: Improvement
> Components: build
> Affects Versions: 0.5.0
> Reporter: Thomas Jungblut
> Priority: Blocker
> Fix For: 0.5.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HAMA-583) Misc release tasks
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284230#comment-13284230 ]
Thomas Jungblut edited comment on HAMA-583 at 5/27/12 8:57 PM:
---------------------------------------------------------------
This
{noformat}
2012-05-27 20:25:28,916 ERROR org.apache.hama.bsp.SimpleTaskScheduler: Error submitting job
java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.hama.bsp.SimpleTaskScheduler$JobProcessor.schedule(SimpleTaskScheduler.java:141)
at org.apache.hama.bsp.SimpleTaskScheduler$JobProcessor.run(SimpleTaskScheduler.java:116)
Caused by: java.lang.NullPointerException
at org.apache.hama.bsp.TaskInProgress.getTaskToRun(TaskInProgress.java:149)
at org.apache.hama.bsp.JobInProgress.obtainNewTask(JobInProgress.java:268)
at org.apache.hama.bsp.SimpleTaskScheduler$TaskWorker.call(SimpleTaskScheduler.java:191)
at org.apache.hama.bsp.SimpleTaskScheduler$TaskWorker.call(SimpleTaskScheduler.java:160)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
{noformat}
happens if the grooms are not correctly configured.
{{{
#TaskInProgress->line 145
String location = possibleLocations[i];
GroomServerStatus groom = grooms.get(location);
}}}
This checks the given DFS location, if it does not match the groom it will throw these NPEs.
We should throw a better exception here.
Actually we shouldn't do a reverse DNS in our groom hostnames, just use the configured ones in /etc/hostname just like the datanode does. So this won't give inconsistencies.
was (Author: thomas.jungblut):
This
{noformat}
2012-05-27 20:25:28,916 ERROR org.apache.hama.bsp.SimpleTaskScheduler: Error submitting job
java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.hama.bsp.SimpleTaskScheduler$JobProcessor.schedule(SimpleTaskScheduler.java:141)
at org.apache.hama.bsp.SimpleTaskScheduler$JobProcessor.run(SimpleTaskScheduler.java:116)
Caused by: java.lang.NullPointerException
at org.apache.hama.bsp.TaskInProgress.getTaskToRun(TaskInProgress.java:149)
at org.apache.hama.bsp.JobInProgress.obtainNewTask(JobInProgress.java:268)
at org.apache.hama.bsp.SimpleTaskScheduler$TaskWorker.call(SimpleTaskScheduler.java:191)
at org.apache.hama.bsp.SimpleTaskScheduler$TaskWorker.call(SimpleTaskScheduler.java:160)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
{noformat}
happens if the grooms are not correctly configured.
{{{
#TaskInProgress->line 145
String location = possibleLocations[i];
GroomServerStatus groom = grooms.get(location);
}}}
This checks the given DFS location, if it does not match the groom it will throw these NPEs.
We should throw a better exception here.
> Misc release tasks
> ------------------
>
> Key: HAMA-583
> URL: https://issues.apache.org/jira/browse/HAMA-583
> Project: Hama
> Issue Type: Improvement
> Components: build
> Affects Versions: 0.5.0
> Reporter: Thomas Jungblut
> Priority: Blocker
> Fix For: 0.5.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira