You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by "Maja Kabiljo (JIRA)" <ji...@apache.org> on 2012/10/16 01:41:03 UTC
[jira] [Created] (GIRAPH-372) Write worker addresses to Zookeeper;
move addresses and resolution to NettyClient
Maja Kabiljo created GIRAPH-372:
-----------------------------------
Summary: Write worker addresses to Zookeeper; move addresses and resolution to NettyClient
Key: GIRAPH-372
URL: https://issues.apache.org/jira/browse/GIRAPH-372
Project: Giraph
Issue Type: Improvement
Reporter: Maja Kabiljo
Assignee: Maja Kabiljo
In preparation for GIRAPH-273, we need to have addresses of all workers available, so I write them to Zookeeper along with the master address.
Since address resoultion is needed in both NettyWorkerClient and NettyMasterClient, I moved that to NettyClient, and also added a map taskId->address in there. Now NettyMasterClient and NettyWorkerClient don't need to take care of InetSocketAddresses, just task ids which they want to send messages to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-372) Write worker addresses to
Zookeeper; move addresses and resolution to NettyClient
Posted by "Eugene Koontz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477967#comment-13477967 ]
Eugene Koontz commented on GIRAPH-372:
--------------------------------------
Hi Maja, The master doesn't have a taskId, that's why there's that {{null}} as the second parameter in the code above that you show. Maybe this is going too far afield for this particular JIRA. Let me just make sure your patch works with SASL authentication before I comment further.
> Write worker addresses to Zookeeper; move addresses and resolution to NettyClient
> ---------------------------------------------------------------------------------
>
> Key: GIRAPH-372
> URL: https://issues.apache.org/jira/browse/GIRAPH-372
> Project: Giraph
> Issue Type: Improvement
> Reporter: Maja Kabiljo
> Assignee: Maja Kabiljo
> Attachments: GIRAPH-372.patch
>
>
> In preparation for GIRAPH-273, we need to have addresses of all workers available, so I write them to Zookeeper along with the master address.
> Since address resoultion is needed in both NettyWorkerClient and NettyMasterClient, I moved that to NettyClient, and also added a map taskId->address in there. Now NettyMasterClient and NettyWorkerClient don't need to take care of InetSocketAddresses, just task ids which they want to send messages to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-372) Write worker addresses to Zookeeper;
move addresses and resolution to NettyClient
Posted by "Maja Kabiljo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Maja Kabiljo updated GIRAPH-372:
--------------------------------
Attachment: GIRAPH-372.patch
> Write worker addresses to Zookeeper; move addresses and resolution to NettyClient
> ---------------------------------------------------------------------------------
>
> Key: GIRAPH-372
> URL: https://issues.apache.org/jira/browse/GIRAPH-372
> Project: Giraph
> Issue Type: Improvement
> Reporter: Maja Kabiljo
> Assignee: Maja Kabiljo
> Attachments: GIRAPH-372.patch, GIRAPH-372.patch
>
>
> In preparation for GIRAPH-273, we need to have addresses of all workers available, so I write them to Zookeeper along with the master address.
> Since address resoultion is needed in both NettyWorkerClient and NettyMasterClient, I moved that to NettyClient, and also added a map taskId->address in there. Now NettyMasterClient and NettyWorkerClient don't need to take care of InetSocketAddresses, just task ids which they want to send messages to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-372) Write worker addresses to Zookeeper;
move addresses and resolution to NettyClient
Posted by "Maja Kabiljo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Maja Kabiljo updated GIRAPH-372:
--------------------------------
Attachment: GIRAPH-372.patch
> Write worker addresses to Zookeeper; move addresses and resolution to NettyClient
> ---------------------------------------------------------------------------------
>
> Key: GIRAPH-372
> URL: https://issues.apache.org/jira/browse/GIRAPH-372
> Project: Giraph
> Issue Type: Improvement
> Reporter: Maja Kabiljo
> Assignee: Maja Kabiljo
> Attachments: GIRAPH-372.patch
>
>
> In preparation for GIRAPH-273, we need to have addresses of all workers available, so I write them to Zookeeper along with the master address.
> Since address resoultion is needed in both NettyWorkerClient and NettyMasterClient, I moved that to NettyClient, and also added a map taskId->address in there. Now NettyMasterClient and NettyWorkerClient don't need to take care of InetSocketAddresses, just task ids which they want to send messages to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-372) Write worker addresses to
Zookeeper; move addresses and resolution to NettyClient
Posted by "Maja Kabiljo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477668#comment-13477668 ]
Maja Kabiljo commented on GIRAPH-372:
-------------------------------------
Eugene, I saw GIRAPH-362 but I don't really understand where the issue comes from. Doesn't master also have task id? I see:
{code:title=NettyWorkerClient.java}
addressTaskIdMap.put(service.getMasterInfo().getInetSocketAddress(), null);
{code}
Would the problem be solved if we just put service.getMasterInfo().getTaskId() instead of null? Or am I missing something obvious?
> Write worker addresses to Zookeeper; move addresses and resolution to NettyClient
> ---------------------------------------------------------------------------------
>
> Key: GIRAPH-372
> URL: https://issues.apache.org/jira/browse/GIRAPH-372
> Project: Giraph
> Issue Type: Improvement
> Reporter: Maja Kabiljo
> Assignee: Maja Kabiljo
> Attachments: GIRAPH-372.patch
>
>
> In preparation for GIRAPH-273, we need to have addresses of all workers available, so I write them to Zookeeper along with the master address.
> Since address resoultion is needed in both NettyWorkerClient and NettyMasterClient, I moved that to NettyClient, and also added a map taskId->address in there. Now NettyMasterClient and NettyWorkerClient don't need to take care of InetSocketAddresses, just task ids which they want to send messages to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-372) Write worker addresses to
Zookeeper; move addresses and resolution to NettyClient
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478447#comment-13478447 ]
Hudson commented on GIRAPH-372:
-------------------------------
Integrated in Giraph-trunk-Commit #246 (See [https://builds.apache.org/job/Giraph-trunk-Commit/246/])
GIRAPH-372: Write worker addresses to Zookeeper; move addresses and resolution to NettyClient (Revision 1399453)
Result = SUCCESS
maja : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1399453
Files :
* /giraph/trunk/CHANGELOG
* /giraph/trunk/giraph/src/main/java/org/apache/giraph/bsp/CentralizedService.java
* /giraph/trunk/giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java
* /giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/MasterClient.java
* /giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/WorkerClient.java
* /giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyClient.java
* /giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyMasterClient.java
* /giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyMasterClientServer.java
* /giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClient.java
* /giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java
* /giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/AddressesAndPartitionsWritable.java
* /giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspService.java
* /giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java
* /giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
* /giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/InputSplitsCallable.java
* /giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/WorkerInfo.java
* /giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/WritableUtils.java
* /giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/ConnectionTest.java
* /giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestFailureTest.java
* /giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestTest.java
* /giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java
> Write worker addresses to Zookeeper; move addresses and resolution to NettyClient
> ---------------------------------------------------------------------------------
>
> Key: GIRAPH-372
> URL: https://issues.apache.org/jira/browse/GIRAPH-372
> Project: Giraph
> Issue Type: Improvement
> Reporter: Maja Kabiljo
> Assignee: Maja Kabiljo
> Attachments: GIRAPH-372.patch, GIRAPH-372.patch
>
>
> In preparation for GIRAPH-273, we need to have addresses of all workers available, so I write them to Zookeeper along with the master address.
> Since address resoultion is needed in both NettyWorkerClient and NettyMasterClient, I moved that to NettyClient, and also added a map taskId->address in there. Now NettyMasterClient and NettyWorkerClient don't need to take care of InetSocketAddresses, just task ids which they want to send messages to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-372) Write worker addresses to
Zookeeper; move addresses and resolution to NettyClient
Posted by "Maja Kabiljo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476601#comment-13476601 ]
Maja Kabiljo commented on GIRAPH-372:
-------------------------------------
https://reviews.apache.org/r/7610/
> Write worker addresses to Zookeeper; move addresses and resolution to NettyClient
> ---------------------------------------------------------------------------------
>
> Key: GIRAPH-372
> URL: https://issues.apache.org/jira/browse/GIRAPH-372
> Project: Giraph
> Issue Type: Improvement
> Reporter: Maja Kabiljo
> Assignee: Maja Kabiljo
> Attachments: GIRAPH-372.patch
>
>
> In preparation for GIRAPH-273, we need to have addresses of all workers available, so I write them to Zookeeper along with the master address.
> Since address resoultion is needed in both NettyWorkerClient and NettyMasterClient, I moved that to NettyClient, and also added a map taskId->address in there. Now NettyMasterClient and NettyWorkerClient don't need to take care of InetSocketAddresses, just task ids which they want to send messages to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-372) Write worker addresses to
Zookeeper; move addresses and resolution to NettyClient
Posted by "Maja Kabiljo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478317#comment-13478317 ]
Maja Kabiljo commented on GIRAPH-372:
-------------------------------------
Moving discussion to GIRAPH-362.
> Write worker addresses to Zookeeper; move addresses and resolution to NettyClient
> ---------------------------------------------------------------------------------
>
> Key: GIRAPH-372
> URL: https://issues.apache.org/jira/browse/GIRAPH-372
> Project: Giraph
> Issue Type: Improvement
> Reporter: Maja Kabiljo
> Assignee: Maja Kabiljo
> Attachments: GIRAPH-372.patch
>
>
> In preparation for GIRAPH-273, we need to have addresses of all workers available, so I write them to Zookeeper along with the master address.
> Since address resoultion is needed in both NettyWorkerClient and NettyMasterClient, I moved that to NettyClient, and also added a map taskId->address in there. Now NettyMasterClient and NettyWorkerClient don't need to take care of InetSocketAddresses, just task ids which they want to send messages to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-372) Write worker addresses to
Zookeeper; move addresses and resolution to NettyClient
Posted by "Eugene Koontz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477660#comment-13477660 ]
Eugene Koontz commented on GIRAPH-372:
--------------------------------------
Hi Maja,
This seems like a good idea. I wonder if GIRAPH-372 could lead to a solution to the workaround that mentioned in GIRAPH-362? To summarize, we need a way for workers to both talk to the master (needed for GIRAPH-211) and support accounting of requests (which requires an ID of the master or some kind of handle for clients to use to account for requests to the master). We tried to fix this with GIRAPH-360, but for now, we don't provide for accounting of SASL messages from the client to the server.
Thanks,
Eugene
> Write worker addresses to Zookeeper; move addresses and resolution to NettyClient
> ---------------------------------------------------------------------------------
>
> Key: GIRAPH-372
> URL: https://issues.apache.org/jira/browse/GIRAPH-372
> Project: Giraph
> Issue Type: Improvement
> Reporter: Maja Kabiljo
> Assignee: Maja Kabiljo
> Attachments: GIRAPH-372.patch
>
>
> In preparation for GIRAPH-273, we need to have addresses of all workers available, so I write them to Zookeeper along with the master address.
> Since address resoultion is needed in both NettyWorkerClient and NettyMasterClient, I moved that to NettyClient, and also added a map taskId->address in there. Now NettyMasterClient and NettyWorkerClient don't need to take care of InetSocketAddresses, just task ids which they want to send messages to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira