You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Thomas Jungblut (Created) (JIRA)" <ji...@apache.org> on 2011/11/04 17:59:50 UTC
[jira] [Created] (HAMA-467) Integrate IO system into YARN
Integrate IO system into YARN
-----------------------------
Key: HAMA-467
URL: https://issues.apache.org/jira/browse/HAMA-467
Project: Hama
Issue Type: Sub-task
Reporter: Thomas Jungblut
Fix For: 0.4.0
HAMA-258 adds fancy features to main Hama.
This should be running as well.
TODO
-> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-467) Integrate IO system into YARN
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163632#comment-13163632 ]
Thomas Jungblut commented on HAMA-467:
--------------------------------------
However, current trunk seen to do not work on YARN either?
Did the counters break it? Last time I was working with it, there wasn't a problem.
Containers don't spawn, heartbeats give these logs:
{noformat}
2011-12-06 16:06:14,006 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:getContainerStatus(475)) - Returning container_id {, app_attempt_id {, application_id {, id: 1, cluster_timestamp: 1323183644595, }, attemptId: 1, }, id: 2, }, state: C_RUNNING, diagnostics: "", exit_status: -1000,
Call: protocol=org.apache.hadoop.yarn.proto.LocalizationProtocol.LocalizationProtocolService.BlockingInterface, method=heartbeat
2011-12-06 16:06:14,566 INFO nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:getNodeStatus(210)) - Sending out status for container: container_id {, app_attempt_id {, application_id {, id: 1, cluster_timestamp: 1323183644595, }, attemptId: 1, }, id: 1, }, state: C_RUNNING, diagnostics: "", exit_status: -1000,
2011-12-06 16:06:14,567 INFO nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:getNodeStatus(210)) - Sending out status for container: container_id {, app_attempt_id {, application_id {, id: 1, cluster_timestamp: 1323183644595, }, attemptId: 1, }, id: 2, }, state: C_RUNNING, diagnostics: "", exit_status: -1000,
2011-12-06 16:06:14,572 INFO nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:getNodeStatus(210)) - Sending out status for container: container_id {, app_attempt_id {, application_id {, id: 1, cluster_timestamp: 1323183644595, }, attemptId: 1, }, id: 3, }, state: C_RUNNING, diagnostics: "", exit_status: -1000,
{noformat}
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Attachments: HAMA-467.patch, HAMA-467_1.patch, HAMA-467_2.patch, HAMA-467_3.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-467) Integrate IO system into YARN
Posted by "Edward J. Yoon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161465#comment-13161465 ]
Edward J. Yoon commented on HAMA-467:
-------------------------------------
Let's commit this to trunk and debugging together. What do you think, Thomas?
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Attachments: HAMA-467.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-467) Integrate IO system into YARN
Posted by "Thomas Jungblut (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-467:
---------------------------------
Fix Version/s: (was: 0.4.0)
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-467) Integrate IO system into YARN
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153823#comment-13153823 ]
Thomas Jungblut commented on HAMA-467:
--------------------------------------
Started a bit.
I'm going to extend from our BSPJobClient, so the partitioning and splitting method should be available there.
I try to refactor our task start in the groom (child?) that will be executed on the tasks side after the process has been started.
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-467) Integrate IO system into YARN
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163386#comment-13163386 ]
Thomas Jungblut commented on HAMA-467:
--------------------------------------
I havn't tested 0.23 HDFS yet. I'm still using the local FS.
BTW we don't have 0.23 HDFS as a dependency in maven, should we add this?
The version in trunk should work. It should run the example without failing.
However, I have broken something which I wasn't aware of yet.
bq. So, I'm not sure that our Input Output formatter works well with Hadoop 0.23
I'm currently trying to get the example without I/O running.
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Attachments: HAMA-467.patch, HAMA-467_1.patch, HAMA-467_2.patch, HAMA-467_3.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-467) Integrate IO system into YARN
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163639#comment-13163639 ]
Thomas Jungblut commented on HAMA-467:
--------------------------------------
Oh it does run.
{noformat}
./*:file:/home/thomasjungblut/workspace/hama-trunk/application_1323184094097_0001/app.jar:" org.apache.hama.bsp.BSPRu
nner, appattempt_1323184094097_0001_000001, 1, file:/home/thomasjungblut/workspace/hama-trunk/application_13231840940
97_0001/job.xml, 1><LOG_DIR>/stdout, 2><LOG_DIR>/stderr]
11/12/06 16:13:24 INFO bsp.BSPTaskLauncher: Starting command: [${JAVA_HOME}/bin/java -cp "$CLASSPATH:./*:./package/*:
./*:file:/home/thomasjungblut/workspace/hama-trunk/application_1323184094097_0001/app.jar:" org.apache.hama.bsp.BSPRu
nner, appattempt_1323184094097_0001_000001, 0, file:/home/thomasjungblut/workspace/hama-trunk/application_13231840940
97_0001/job.xml, 1><LOG_DIR>/stdout, 2><LOG_DIR>/stderr]
11/12/06 16:18:41 INFO server.NIOServerCnxn: Accepted socket connection from /127.0.0.1:40611
11/12/06 16:18:41 INFO server.NIOServerCnxn: Client attempting to establish new session at /127.0.0.1:40611
11/12/06 16:18:41 INFO persistence.FileTxnLog: Creating new log file: log.f
11/12/06 16:18:41 INFO server.NIOServerCnxn: Accepted socket connection from /127.0.0.1:40612
11/12/06 16:18:41 INFO server.NIOServerCnxn: Client attempting to establish new session at /127.0.0.1:40612
11/12/06 16:18:41 INFO server.NIOServerCnxn: Established session 0x13413efae560000 with negotiated timeout 1200000 fo
r client /127.0.0.1:40611
11/12/06 16:18:41 INFO server.NIOServerCnxn: Established session 0x13413efae560001 with negotiated timeout 1200000 fo
r client /127.0.0.1:40612
11/12/06 16:18:41 INFO server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x13413e
fae560000 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/job_appattempt_1323184
094097_0001_000001_0000 Error:KeeperErrorCode = NodeExists for /job_appattempt_1323184094097_0001_000001_0000
11/12/06 16:18:41 INFO server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x13413e
fae560000 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/bsp/job_appattempt_132
3184094097_0001_000001_0000/-1/ready Error:KeeperErrorCode = NodeExists for /bsp/job_appattempt_1323184094097_0001_00
0001_0000/-1/ready
11/12/06 16:18:42 INFO server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x13413e
fae560001 type:create cxid:0x18 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/bsp/job_appattempt_13
23184094097_0001_000001_0000/0 Error:KeeperErrorCode = NodeExists for /bsp/job_appattempt_1323184094097_0001_000001_0
000/0
11/12/06 16:18:42 INFO bsp.BSPApplicationMaster: Now in superstep 1
11/12/06 16:18:44 INFO bsp.BSPApplicationMaster: Now in superstep 2
11/12/06 16:18:44 INFO server.PrepRequestProcessor: Processed session termination for sessionid: 0x13413efae560001
11/12/06 16:18:44 INFO server.PrepRequestProcessor: Processed session termination for sessionid: 0x13413efae560000
11/12/06 16:18:44 WARN server.NIOServerCnxn: EndOfStreamException: Unable to read additional data from client session
id 0x13413efae560001, likely client has closed socket
11/12/06 16:18:44 INFO server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:40612 which had sessionid
0x13413efae560001
11/12/06 16:18:44 INFO server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:40611 which had sessionid
0x13413efae560000
11/12/06 16:18:44 INFO bsp.JobImpl: Task "0" sucessfully finished!
{noformat}
I don't want to be offensive, but YARN seems buggy to me.
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Attachments: HAMA-467.patch, HAMA-467_1.patch, HAMA-467_2.patch, HAMA-467_3.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-467) Integrate IO system into YARN
Posted by "Thomas Jungblut (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-467:
---------------------------------
Attachment: HAMA-467_5.patch
I got it running. :)
Please review the changes in core-package, so we can submit this as quickly as possible.
BTW I changed the use of a thread per task to a polling solution, mainly because it should take too much memory for larger number of tasks.
And I excluded the testcase, because the yarn testcluster seems to be not available in the release/snapshot of 23.0 o_o
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Attachments: HAMA-467.patch, HAMA-467_1.patch, HAMA-467_2.patch, HAMA-467_3.patch, HAMA-467_4.patch, HAMA-467_5.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-467) Integrate IO system into YARN
Posted by "Edward J. Yoon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169130#comment-13169130 ]
Edward J. Yoon commented on HAMA-467:
-------------------------------------
+1
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Attachments: HAMA-467.patch, HAMA-467_1.patch, HAMA-467_2.patch, HAMA-467_3.patch, HAMA-467_4.patch, HAMA-467_5.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HAMA-467) Integrate IO system into YARN
Posted by "Thomas Jungblut (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut resolved HAMA-467.
----------------------------------
Resolution: Fixed
Committed :D
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Attachments: HAMA-467.patch, HAMA-467_1.patch, HAMA-467_2.patch, HAMA-467_3.patch, HAMA-467_4.patch, HAMA-467_5.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-467) Integrate IO system into YARN
Posted by "Thomas Jungblut (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-467:
---------------------------------
Attachment: HAMA-467_3.patch
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Attachments: HAMA-467.patch, HAMA-467_1.patch, HAMA-467_2.patch, HAMA-467_3.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-467) Integrate IO system into YARN
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162075#comment-13162075 ]
Thomas Jungblut commented on HAMA-467:
--------------------------------------
{noformat}
11/12/03 09:50:34 INFO bsp.JobImpl: Memory per task: 50m!
11/12/03 09:50:34 INFO bsp.JobImpl: Got response! ID: 1 with num of containers: 0 and following resources: 19456mb
11/12/03 09:50:34 INFO bsp.JobImpl: Waiting to allocate 2 more containers...
11/12/03 09:50:35 INFO bsp.JobImpl: Got response! ID: 2 with num of containers: 2 and following resources: 17408mb
11/12/03 09:50:35 INFO bsp.JobImpl: Waiting to allocate 0 more containers...
11/12/03 09:50:36 INFO bsp.JobImpl: Got 2 containers!
11/12/03 09:50:36 INFO bsp.JobImpl: Launching task on a new container., containerId=container_1322901730652_0003_01_000002, containerNode=localhost.localdomain:47058, containerNodeURI=localhost.localdomain:9999, containerStateNEW, containerResourceMemory1024
11/12/03 09:50:36 INFO bsp.JobImpl: Launching task on a new container., containerId=container_1322901730652_0003_01_000003, containerNode=localhost.localdomain:47058, containerNodeURI=localhost.localdomain:9999, containerStateNEW, containerResourceMemory1024
11/12/03 09:50:36 INFO bsp.BSPTaskLauncher: Spawned task with id: 0 for allocated container id: container_1322901730652_0003_01_000002
11/12/03 09:50:36 INFO bsp.BSPTaskLauncher: Setting up a container for user thomasjungblut with id of 0 and containerID of container_1322901730652_0003_01_000002 as thomasjungblut
11/12/03 09:50:36 INFO bsp.BSPTaskLauncher: Spawned task with id: 1 for allocated container id: container_1322901730652_0003_01_000003
11/12/03 09:50:36 INFO bsp.BSPTaskLauncher: Setting up a container for user thomasjungblut with id of 1 and containerID of container_1322901730652_0003_01_000003 as thomasjungblut
11/12/03 09:50:37 INFO bsp.BSPTaskLauncher: Package resource: port: -1, file: "/tmp/bsp-yarn/submit_qvokf5/job.jar",
11/12/03 09:50:37 INFO bsp.BSPTaskLauncher: Package resource: port: -1, file: "/tmp/bsp-yarn/submit_qvokf5/job.jar",
11/12/03 09:50:37 INFO bsp.BSPTaskLauncher: Starting command: [${JAVA_HOME}/bin/java -cp "$CLASSPATH:./*:./package/*:./*:" org.apache.hama.bsp.BSPRunner, appattempt_1322901730652_0003_000001, 1, file:/tmp/bsp-yarn/submit_qvokf5/job.xml, 1><LOG_DIR>/stdout, 2><LOG_DIR>/stderr]
11/12/03 09:50:37 INFO bsp.BSPTaskLauncher: Starting command: [${JAVA_HOME}/bin/java -cp "$CLASSPATH:./*:./package/*:./*:" org.apache.hama.bsp.BSPRunner, appattempt_1322901730652_0003_000001, 0, file:/tmp/bsp-yarn/submit_qvokf5/job.xml, 1><LOG_DIR>/stdout, 2><LOG_DIR>/stderr]
11/12/03 09:50:37 INFO bsp.BSPTaskLauncher: 0 Last report comes with existatus of -1000 and diagnose string of
11/12/03 09:50:37 ERROR bsp.JobImpl: Task with id "0" failed!
11/12/03 09:50:37 INFO bsp.BSPApplicationMaster: Job "Serialize Printing"'s state after completion: FAILED
{noformat}
Here is my log, first there was a user issue, I provide you with the latest patch.
But now the container won't even start, and BTW what is existatus -1000? No diagnose is left in the last status report.
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Attachments: HAMA-467.patch, HAMA-467_1.patch, HAMA-467_2.patch, HAMA-467_3.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-467) Integrate IO system into YARN
Posted by "Edward J. Yoon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163352#comment-13163352 ]
Edward J. Yoon commented on HAMA-467:
-------------------------------------
Do you test with 0.23 hdfs?
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Attachments: HAMA-467.patch, HAMA-467_1.patch, HAMA-467_2.patch, HAMA-467_3.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HAMA-467) Integrate IO system into YARN
Posted by "Thomas Jungblut (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut reassigned HAMA-467:
------------------------------------
Assignee: Thomas Jungblut
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Attachments: HAMA-467.patch, HAMA-467_1.patch, HAMA-467_2.patch, HAMA-467_3.patch, HAMA-467_4.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-467) Integrate IO system into YARN
Posted by "Thomas Jungblut (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-467:
---------------------------------
Attachment: HAMA-467.patch
First complete scratch, not tested yet. And I want to add a testcase.
Please review if you are okay about the little refactoring in core package.
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Attachments: HAMA-467.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-467) Integrate IO system into YARN
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158209#comment-13158209 ]
Thomas Jungblut commented on HAMA-467:
--------------------------------------
Thanks, but it does not work. I cried tears the whole weekend to get this back running. I haven't got it yet, but I think I'll be complete until friday.
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Attachments: HAMA-467.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-467) Integrate IO system into YARN
Posted by "Edward J. Yoon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163384#comment-13163384 ]
Edward J. Yoon commented on HAMA-467:
-------------------------------------
I didn't look at closely yet but 0.23 hdfs seems have many changes of API and architecture e.g., multiple volume management system. So, I'm not sure that our Input Output formatter works well with Hadoop 0.23.
{quote}
No. But DFS should not be the issue here. It worked before.
{quote}
Do you mean, it worked from among these patches (HAMA-467-*.patch)?
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Attachments: HAMA-467.patch, HAMA-467_1.patch, HAMA-467_2.patch, HAMA-467_3.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-467) Integrate IO system into YARN
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166467#comment-13166467 ]
Thomas Jungblut commented on HAMA-467:
--------------------------------------
BTW I have the issue:
{noformat}
java.net.URISyntaxException: Expected scheme name at index 0: :///tmp/bsp-yarn/submit_2tjgyd/job.jar
at java.net.URI$Parser.fail(URI.java:2809)
at java.net.URI$Parser.failExpecting(URI.java:2815)
at java.net.URI$Parser.parse(URI.java:3007)
at java.net.URI.<init>(URI.java:736)
at org.apache.hadoop.yarn.util.ConverterUtils.getPathFromYarnURL(ConverterUtils.java:63)
at org.apache.hama.bsp.BSPTaskLauncher.setupContainer(BSPTaskLauncher.java:132)
at org.apache.hama.bsp.BSPTaskLauncher.start(BSPTaskLauncher.java:94)
at org.apache.hama.bsp.JobImpl.startJob(JobImpl.java:186)
at org.apache.hama.bsp.BSPApplicationMaster.start(BSPApplicationMaster.java:249)
at org.apache.hama.bsp.BSPApplicationMaster.main(BSPApplicationMaster.java:293)
java.net.URISyntaxException: Expected scheme name at index 0: :///tmp/bsp-yarn/submit_2tjgyd/job.jar
at java.net.URI$Parser.fail(URI.java:2809)
at java.net.URI$Parser.failExpecting(URI.java:2815)
at java.net.URI$Parser.parse(URI.java:3007)
at java.net.URI.<init>(URI.java:736)
at org.apache.hadoop.yarn.util.ConverterUtils.getPathFromYarnURL(ConverterUtils.java:63)
at org.apache.hama.bsp.BSPTaskLauncher.setupContainer(BSPTaskLauncher.java:132)
at org.apache.hama.bsp.BSPTaskLauncher.start(BSPTaskLauncher.java:94)
at org.apache.hama.bsp.JobImpl.startJob(JobImpl.java:186)
at org.apache.hama.bsp.BSPApplicationMaster.start(BSPApplicationMaster.java:249)
at org.apache.hama.bsp.BSPApplicationMaster.main(BSPApplicationMaster.java:293)
{noformat}
It is a bug in Hadoop. It tries to convert a (YARN)URL back to a path on resourcemanager side, but the scheme is missing.
I have a look how to work arround this.
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Attachments: HAMA-467.patch, HAMA-467_1.patch, HAMA-467_2.patch, HAMA-467_3.patch, HAMA-467_4.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-467) Integrate IO system into YARN
Posted by "Thomas Jungblut (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-467:
---------------------------------
Attachment: HAMA-467_1.patch
Still having issues starting containers, but no LOGs so I'm completely in the dark.
I'm on it.
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Attachments: HAMA-467.patch, HAMA-467_1.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-467) Integrate IO system into YARN
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161478#comment-13161478 ]
Thomas Jungblut commented on HAMA-467:
--------------------------------------
There are just a few details that I need to fix. I will provide you with a patch tonight / tomorrow.
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Attachments: HAMA-467.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-467) Integrate IO system into YARN
Posted by "Thomas Jungblut (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-467:
---------------------------------
Attachment: HAMA-467_4.patch
This patch catches up to trunk and adds hdfs to pom.xml.
However Hadoop 23.0 seems to be not in the maven public repository?
Or is there a different?
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Attachments: HAMA-467.patch, HAMA-467_1.patch, HAMA-467_2.patch, HAMA-467_3.patch, HAMA-467_4.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-467) Integrate IO system into YARN
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163379#comment-13163379 ]
Thomas Jungblut commented on HAMA-467:
--------------------------------------
No. But DFS should not be the issue here. It worked before.
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Attachments: HAMA-467.patch, HAMA-467_1.patch, HAMA-467_2.patch, HAMA-467_3.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-467) Integrate IO system into YARN
Posted by "Thomas Jungblut (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-467:
---------------------------------
Attachment: HAMA-467_2.patch
updated version to latest trunk
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Attachments: HAMA-467.patch, HAMA-467_1.patch, HAMA-467_2.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-467) Integrate IO system into YARN
Posted by "Edward J. Yoon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163404#comment-13163404 ]
Edward J. Yoon commented on HAMA-467:
-------------------------------------
I think yes, we have to add 0.23 HDFS to dependency list and figure out whether core's IO system can be shared.
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Attachments: HAMA-467.patch, HAMA-467_1.patch, HAMA-467_2.patch, HAMA-467_3.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-467) Integrate IO system into YARN
Posted by "Edward J. Yoon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158138#comment-13158138 ]
Edward J. Yoon commented on HAMA-467:
-------------------------------------
Patch looks good +1
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Attachments: HAMA-467.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-467) Integrate IO system into YARN
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169886#comment-13169886 ]
Hudson commented on HAMA-467:
-----------------------------
Integrated in Hama-Nightly #387 (See [https://builds.apache.org/job/Hama-Nightly/387/])
[HAMA-467] Add I/O System to YARN
tjungblut :
Files :
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPJob.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPJobClient.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/sync/ZooKeeperSyncServerImpl.java
* /incubator/hama/trunk/yarn/pom.xml
* /incubator/hama/trunk/yarn/src/main/java/org/apache/hama/bsp/BSPApplicationMaster.java
* /incubator/hama/trunk/yarn/src/main/java/org/apache/hama/bsp/BSPRunner.java
* /incubator/hama/trunk/yarn/src/main/java/org/apache/hama/bsp/BSPTaskLauncher.java
* /incubator/hama/trunk/yarn/src/main/java/org/apache/hama/bsp/Job.java
* /incubator/hama/trunk/yarn/src/main/java/org/apache/hama/bsp/JobImpl.java
* /incubator/hama/trunk/yarn/src/main/java/org/apache/hama/bsp/YARNBSPJob.java
* /incubator/hama/trunk/yarn/src/main/java/org/apache/hama/bsp/YARNBSPJobClient.java
> Integrate IO system into YARN
> -----------------------------
>
> Key: HAMA-467
> URL: https://issues.apache.org/jira/browse/HAMA-467
> Project: Hama
> Issue Type: Sub-task
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Attachments: HAMA-467.patch, HAMA-467_1.patch, HAMA-467_2.patch, HAMA-467_3.patch, HAMA-467_4.patch, HAMA-467_5.patch
>
>
> HAMA-258 adds fancy features to main Hama.
> This should be running as well.
> TODO
> -> YARNBSPJob must be refactored, as well as the task that really launches the BSP. Mostly code deduplication must be done here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira