You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by "Eli Reisman (JIRA)" <ji...@apache.org> on 2012/08/22 23:22:42 UTC
[jira] [Created] (GIRAPH-312) Giraph needs an admin script
Eli Reisman created GIRAPH-312:
----------------------------------
Summary: Giraph needs an admin script
Key: GIRAPH-312
URL: https://issues.apache.org/jira/browse/GIRAPH-312
Project: Giraph
Issue Type: New Feature
Components: conf and scripts, zookeeper
Affects Versions: 0.2.0
Reporter: Eli Reisman
Assignee: Eli Reisman
Priority: Minor
Fix For: 0.2.0
Our zookeeper instances have very long uptimes on our cluster, and failed job trees are never cleaned from memory. There is a separate shell script to do this, but its not picky about which node trees it erases, and on some systems some Giraph users may not have access to it.
This patch will add a shell script to activate a new class which will use Giraph conf file options or our normal -Dgiraph.XYZ command-line opts to get the ZK quorum info, and clean out the remnants from its memory of old failed and killed jobs. They do pile up over time.
This led to the larger idea that Giraph needs a general giraph-admin shell script as a home for stuff like this. Jakob suggested it would be a good idea to put this into such a script since then admin groups can be created so that not every Giraph client can run it. This script currently only has code to start up the zk cleaner, but can have more options added to it as JIRA's to add features crop up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-312) Giraph needs an admin script
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Reisman updated GIRAPH-312:
-------------------------------
Attachment: GIRAPH-312-5.patch
This one is just right, should be good to go. You run it with:
bin/giraph-admin -Dgiraph.zk.whatever=... [-Dgiraph....] -cleanZk
where your -D options are the normal giraph ZK options (CSV hostname list of ZK quorum servers, ZK port, ZK_BASE_NODE, etc.) or else it has defaults. If you don't give it a server list, it quits since this is really meant for ZK quorums that stay up indefinitely on your typical cluster. It pulls this from a Configuration it makes and recursively deletes from the BspService.BASE_DIR (which is "_hadoopBsp") under whatever base node you gave it (if any.)
It reports progress as it goes. This patch also splits off all the shell-level config stuff from bin/giraph into its own bin/giraph-env script which is then sourced in bin/giraph and bin/giraph-admin.
Passes mvn verify and field tests on ZK quorum. The GiraphZooKeeperAdmin class should also provide an easy template for how to add functionality to the giraph-admin script in the future.
> Giraph needs an admin script
> ----------------------------
>
> Key: GIRAPH-312
> URL: https://issues.apache.org/jira/browse/GIRAPH-312
> Project: Giraph
> Issue Type: New Feature
> Components: conf and scripts, zookeeper
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-312-1.patch, GIRAPH-312-2.patch, GIRAPH-312-3.patch, GIRAPH-312-3.patch, GIRAPH-312-4.patch, GIRAPH-312-5.patch
>
>
> Our zookeeper instances have very long uptimes on our cluster, and failed job trees are never cleaned from memory. There is a separate shell script to do this, but its not picky about which node trees it erases, and on some systems some Giraph users may not have access to it.
> This patch will add a shell script to activate a new class which will use Giraph conf file options or our normal -Dgiraph.XYZ command-line opts to get the ZK quorum info, and clean out the remnants from its memory of old failed and killed jobs. They do pile up over time.
> This led to the larger idea that Giraph needs a general giraph-admin shell script as a home for stuff like this. Jakob suggested it would be a good idea to put this into such a script since then admin groups can be created so that not every Giraph client can run it. This script currently only has code to start up the zk cleaner, but can have more options added to it as JIRA's to add features crop up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-312) Giraph needs an admin script
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Reisman updated GIRAPH-312:
-------------------------------
Attachment: GIRAPH-312-6.patch
Sorry the last rebase was the right patch, but the wrong name. This is merged with today's trunk successfully so I'm guessing its still fine as-is. Unless something has changed in bin/giraph script this should still be good to go.
I know this sort of splits up the shell/environment var setup for the "hadoop jar" call in a new way, so someone might want to take a peek and see if this approach conflicts with anything thats happened more recently in the run scripts?
If not I can rebase to just run the Zk cleanup without the giraph-env file etc.
> Giraph needs an admin script
> ----------------------------
>
> Key: GIRAPH-312
> URL: https://issues.apache.org/jira/browse/GIRAPH-312
> Project: Giraph
> Issue Type: New Feature
> Components: conf and scripts, zookeeper
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-312-1.patch, GIRAPH-312-2.patch, GIRAPH-312-3.patch, GIRAPH-312-3.patch, GIRAPH-312-4.patch, GIRAPH-312-5.patch, GIRAPH-312-6.patch, GIRAPH-326-6.patch
>
>
> Our zookeeper instances have very long uptimes on our cluster, and failed job trees are never cleaned from memory. There is a separate shell script to do this, but its not picky about which node trees it erases, and on some systems some Giraph users may not have access to it.
> This patch will add a shell script to activate a new class which will use Giraph conf file options or our normal -Dgiraph.XYZ command-line opts to get the ZK quorum info, and clean out the remnants from its memory of old failed and killed jobs. They do pile up over time.
> This led to the larger idea that Giraph needs a general giraph-admin shell script as a home for stuff like this. Jakob suggested it would be a good idea to put this into such a script since then admin groups can be created so that not every Giraph client can run it. This script currently only has code to start up the zk cleaner, but can have more options added to it as JIRA's to add features crop up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-312) Giraph needs an admin script
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13446084#comment-13446084 ]
Eli Reisman commented on GIRAPH-312:
------------------------------------
Ran this on a cluster full of failed job traces last night (sigh) and it works great, leaves your GiraphJob.ZK_BASE_NODE if you supplied one, zaps everything below and including the _hadoopBsp (BspService.BASE_DIR)
Also verified the giraph-env script works, so now bin/giraph is just a source of giraph-env to get its environment vars and class path stuff set up, then 3 lines to run giraph through "hadoop jar" as before. This should make it easier to create giraph-centric shell scripts in the future if we want more for some reason.
> Giraph needs an admin script
> ----------------------------
>
> Key: GIRAPH-312
> URL: https://issues.apache.org/jira/browse/GIRAPH-312
> Project: Giraph
> Issue Type: New Feature
> Components: conf and scripts, zookeeper
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-312-1.patch, GIRAPH-312-2.patch, GIRAPH-312-3.patch, GIRAPH-312-3.patch, GIRAPH-312-4.patch, GIRAPH-312-5.patch
>
>
> Our zookeeper instances have very long uptimes on our cluster, and failed job trees are never cleaned from memory. There is a separate shell script to do this, but its not picky about which node trees it erases, and on some systems some Giraph users may not have access to it.
> This patch will add a shell script to activate a new class which will use Giraph conf file options or our normal -Dgiraph.XYZ command-line opts to get the ZK quorum info, and clean out the remnants from its memory of old failed and killed jobs. They do pile up over time.
> This led to the larger idea that Giraph needs a general giraph-admin shell script as a home for stuff like this. Jakob suggested it would be a good idea to put this into such a script since then admin groups can be created so that not every Giraph client can run it. This script currently only has code to start up the zk cleaner, but can have more options added to it as JIRA's to add features crop up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-312) Giraph needs an admin script
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Reisman updated GIRAPH-312:
-------------------------------
Attachment: GIRAPH-312-2.patch
Forgot to change shell script name from "giraph-zkadmin" to more general purpose "giraph-admin" even though it currently only has functionality for "-cleanZk" option
> Giraph needs an admin script
> ----------------------------
>
> Key: GIRAPH-312
> URL: https://issues.apache.org/jira/browse/GIRAPH-312
> Project: Giraph
> Issue Type: New Feature
> Components: conf and scripts, zookeeper
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-312-1.patch, GIRAPH-312-2.patch
>
>
> Our zookeeper instances have very long uptimes on our cluster, and failed job trees are never cleaned from memory. There is a separate shell script to do this, but its not picky about which node trees it erases, and on some systems some Giraph users may not have access to it.
> This patch will add a shell script to activate a new class which will use Giraph conf file options or our normal -Dgiraph.XYZ command-line opts to get the ZK quorum info, and clean out the remnants from its memory of old failed and killed jobs. They do pile up over time.
> This led to the larger idea that Giraph needs a general giraph-admin shell script as a home for stuff like this. Jakob suggested it would be a good idea to put this into such a script since then admin groups can be created so that not every Giraph client can run it. This script currently only has code to start up the zk cleaner, but can have more options added to it as JIRA's to add features crop up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-312) Giraph needs an admin script
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Reisman updated GIRAPH-312:
-------------------------------
Attachment: GIRAPH-312-3.patch
This seems to work the best and be the cleanest solution. Still playing with a couple details, so this is a placeholder, but this is almost ready.
> Giraph needs an admin script
> ----------------------------
>
> Key: GIRAPH-312
> URL: https://issues.apache.org/jira/browse/GIRAPH-312
> Project: Giraph
> Issue Type: New Feature
> Components: conf and scripts, zookeeper
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-312-1.patch, GIRAPH-312-2.patch, GIRAPH-312-3.patch, GIRAPH-312-3.patch
>
>
> Our zookeeper instances have very long uptimes on our cluster, and failed job trees are never cleaned from memory. There is a separate shell script to do this, but its not picky about which node trees it erases, and on some systems some Giraph users may not have access to it.
> This patch will add a shell script to activate a new class which will use Giraph conf file options or our normal -Dgiraph.XYZ command-line opts to get the ZK quorum info, and clean out the remnants from its memory of old failed and killed jobs. They do pile up over time.
> This led to the larger idea that Giraph needs a general giraph-admin shell script as a home for stuff like this. Jakob suggested it would be a good idea to put this into such a script since then admin groups can be created so that not every Giraph client can run it. This script currently only has code to start up the zk cleaner, but can have more options added to it as JIRA's to add features crop up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-312) Giraph needs an admin script
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463153#comment-13463153 ]
Avery Ching commented on GIRAPH-312:
------------------------------------
Hi Eli, sorry for the delay on this. It looks good to me, can you please rebase it?
> Giraph needs an admin script
> ----------------------------
>
> Key: GIRAPH-312
> URL: https://issues.apache.org/jira/browse/GIRAPH-312
> Project: Giraph
> Issue Type: New Feature
> Components: conf and scripts, zookeeper
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-312-1.patch, GIRAPH-312-2.patch, GIRAPH-312-3.patch, GIRAPH-312-3.patch, GIRAPH-312-4.patch, GIRAPH-312-5.patch
>
>
> Our zookeeper instances have very long uptimes on our cluster, and failed job trees are never cleaned from memory. There is a separate shell script to do this, but its not picky about which node trees it erases, and on some systems some Giraph users may not have access to it.
> This patch will add a shell script to activate a new class which will use Giraph conf file options or our normal -Dgiraph.XYZ command-line opts to get the ZK quorum info, and clean out the remnants from its memory of old failed and killed jobs. They do pile up over time.
> This led to the larger idea that Giraph needs a general giraph-admin shell script as a home for stuff like this. Jakob suggested it would be a good idea to put this into such a script since then admin groups can be created so that not every Giraph client can run it. This script currently only has code to start up the zk cleaner, but can have more options added to it as JIRA's to add features crop up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-312) Giraph needs an admin script
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Reisman updated GIRAPH-312:
-------------------------------
Attachment: GIRAPH-312-7.patch
Here's the rebase, sorry for the delay.
> Giraph needs an admin script
> ----------------------------
>
> Key: GIRAPH-312
> URL: https://issues.apache.org/jira/browse/GIRAPH-312
> Project: Giraph
> Issue Type: New Feature
> Components: conf and scripts, zookeeper
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-312-1.patch, GIRAPH-312-2.patch, GIRAPH-312-3.patch, GIRAPH-312-3.patch, GIRAPH-312-4.patch, GIRAPH-312-5.patch, GIRAPH-312-6.patch, GIRAPH-312-7.patch, GIRAPH-326-6.patch
>
>
> Our zookeeper instances have very long uptimes on our cluster, and failed job trees are never cleaned from memory. There is a separate shell script to do this, but its not picky about which node trees it erases, and on some systems some Giraph users may not have access to it.
> This patch will add a shell script to activate a new class which will use Giraph conf file options or our normal -Dgiraph.XYZ command-line opts to get the ZK quorum info, and clean out the remnants from its memory of old failed and killed jobs. They do pile up over time.
> This led to the larger idea that Giraph needs a general giraph-admin shell script as a home for stuff like this. Jakob suggested it would be a good idea to put this into such a script since then admin groups can be created so that not every Giraph client can run it. This script currently only has code to start up the zk cleaner, but can have more options added to it as JIRA's to add features crop up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-312) Giraph needs an admin script
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Reisman updated GIRAPH-312:
-------------------------------
Attachment: GIRAPH-312-1.patch
> Giraph needs an admin script
> ----------------------------
>
> Key: GIRAPH-312
> URL: https://issues.apache.org/jira/browse/GIRAPH-312
> Project: Giraph
> Issue Type: New Feature
> Components: conf and scripts, zookeeper
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-312-1.patch
>
>
> Our zookeeper instances have very long uptimes on our cluster, and failed job trees are never cleaned from memory. There is a separate shell script to do this, but its not picky about which node trees it erases, and on some systems some Giraph users may not have access to it.
> This patch will add a shell script to activate a new class which will use Giraph conf file options or our normal -Dgiraph.XYZ command-line opts to get the ZK quorum info, and clean out the remnants from its memory of old failed and killed jobs. They do pile up over time.
> This led to the larger idea that Giraph needs a general giraph-admin shell script as a home for stuff like this. Jakob suggested it would be a good idea to put this into such a script since then admin groups can be created so that not every Giraph client can run it. This script currently only has code to start up the zk cleaner, but can have more options added to it as JIRA's to add features crop up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-312) Giraph needs an admin script
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454241#comment-13454241 ]
Eli Reisman commented on GIRAPH-312:
------------------------------------
Unless its gone stale (should be OK) this is working, tested on 2 clusters, and passes mvn verify (etc.) so I think we're ready for review here.
> Giraph needs an admin script
> ----------------------------
>
> Key: GIRAPH-312
> URL: https://issues.apache.org/jira/browse/GIRAPH-312
> Project: Giraph
> Issue Type: New Feature
> Components: conf and scripts, zookeeper
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-312-1.patch, GIRAPH-312-2.patch, GIRAPH-312-3.patch, GIRAPH-312-3.patch, GIRAPH-312-4.patch, GIRAPH-312-5.patch
>
>
> Our zookeeper instances have very long uptimes on our cluster, and failed job trees are never cleaned from memory. There is a separate shell script to do this, but its not picky about which node trees it erases, and on some systems some Giraph users may not have access to it.
> This patch will add a shell script to activate a new class which will use Giraph conf file options or our normal -Dgiraph.XYZ command-line opts to get the ZK quorum info, and clean out the remnants from its memory of old failed and killed jobs. They do pile up over time.
> This led to the larger idea that Giraph needs a general giraph-admin shell script as a home for stuff like this. Jakob suggested it would be a good idea to put this into such a script since then admin groups can be created so that not every Giraph client can run it. This script currently only has code to start up the zk cleaner, but can have more options added to it as JIRA's to add features crop up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-312) Giraph needs an admin script
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475789#comment-13475789 ]
Eli Reisman commented on GIRAPH-312:
------------------------------------
I will rebase this, sorry its been so long. I ran this a few times on cluster its handy.
> Giraph needs an admin script
> ----------------------------
>
> Key: GIRAPH-312
> URL: https://issues.apache.org/jira/browse/GIRAPH-312
> Project: Giraph
> Issue Type: New Feature
> Components: conf and scripts, zookeeper
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-312-1.patch, GIRAPH-312-2.patch, GIRAPH-312-3.patch, GIRAPH-312-3.patch, GIRAPH-312-4.patch, GIRAPH-312-5.patch
>
>
> Our zookeeper instances have very long uptimes on our cluster, and failed job trees are never cleaned from memory. There is a separate shell script to do this, but its not picky about which node trees it erases, and on some systems some Giraph users may not have access to it.
> This patch will add a shell script to activate a new class which will use Giraph conf file options or our normal -Dgiraph.XYZ command-line opts to get the ZK quorum info, and clean out the remnants from its memory of old failed and killed jobs. They do pile up over time.
> This led to the larger idea that Giraph needs a general giraph-admin shell script as a home for stuff like this. Jakob suggested it would be a good idea to put this into such a script since then admin groups can be created so that not every Giraph client can run it. This script currently only has code to start up the zk cleaner, but can have more options added to it as JIRA's to add features crop up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-312) Giraph needs an admin script
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Reisman updated GIRAPH-312:
-------------------------------
Attachment: GIRAPH-312-4.patch
Uploaded wrong one. Here we go. Otherwise, last msg still applies...
> Giraph needs an admin script
> ----------------------------
>
> Key: GIRAPH-312
> URL: https://issues.apache.org/jira/browse/GIRAPH-312
> Project: Giraph
> Issue Type: New Feature
> Components: conf and scripts, zookeeper
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-312-1.patch, GIRAPH-312-2.patch, GIRAPH-312-3.patch, GIRAPH-312-3.patch, GIRAPH-312-4.patch
>
>
> Our zookeeper instances have very long uptimes on our cluster, and failed job trees are never cleaned from memory. There is a separate shell script to do this, but its not picky about which node trees it erases, and on some systems some Giraph users may not have access to it.
> This patch will add a shell script to activate a new class which will use Giraph conf file options or our normal -Dgiraph.XYZ command-line opts to get the ZK quorum info, and clean out the remnants from its memory of old failed and killed jobs. They do pile up over time.
> This led to the larger idea that Giraph needs a general giraph-admin shell script as a home for stuff like this. Jakob suggested it would be a good idea to put this into such a script since then admin groups can be created so that not every Giraph client can run it. This script currently only has code to start up the zk cleaner, but can have more options added to it as JIRA's to add features crop up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-312) Giraph needs an admin script
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485291#comment-13485291 ]
Avery Ching commented on GIRAPH-312:
------------------------------------
Sorry Eli, but this doesn't merge well since we moved src to giraph/src. Can you please do a rebase?
> Giraph needs an admin script
> ----------------------------
>
> Key: GIRAPH-312
> URL: https://issues.apache.org/jira/browse/GIRAPH-312
> Project: Giraph
> Issue Type: New Feature
> Components: conf and scripts, zookeeper
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-312-1.patch, GIRAPH-312-2.patch, GIRAPH-312-3.patch, GIRAPH-312-3.patch, GIRAPH-312-4.patch, GIRAPH-312-5.patch, GIRAPH-312-6.patch, GIRAPH-326-6.patch
>
>
> Our zookeeper instances have very long uptimes on our cluster, and failed job trees are never cleaned from memory. There is a separate shell script to do this, but its not picky about which node trees it erases, and on some systems some Giraph users may not have access to it.
> This patch will add a shell script to activate a new class which will use Giraph conf file options or our normal -Dgiraph.XYZ command-line opts to get the ZK quorum info, and clean out the remnants from its memory of old failed and killed jobs. They do pile up over time.
> This led to the larger idea that Giraph needs a general giraph-admin shell script as a home for stuff like this. Jakob suggested it would be a good idea to put this into such a script since then admin groups can be created so that not every Giraph client can run it. This script currently only has code to start up the zk cleaner, but can have more options added to it as JIRA's to add features crop up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-312) Giraph needs an admin script
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Reisman updated GIRAPH-312:
-------------------------------
Attachment: GIRAPH-326-6.patch
This seems to work, but I still need to test it in non-pseudo mode. Which will be tricky for me with my current Hadoop rig. Which by tricky I mean it ain't gonna happen. If a reviewer wants to give this a quick run with
{code}
bin/giraph-admin -Dgiraph.zkServerList=... -Dgiraph.zkPort=... -Dgiraph.zkBaseNode=... -cleanZk
{code}
(using the right options, I might have paraphrased above but we need zk host list, zk port, and base znode for all giraph jobs, if your cluster uses one)
and remember it will print results as it goes, but if your quorum has been up a long time, this might take a few hours to run.
Anyway I barely did anything to rebase it, and it was well tested on a cluster before, so I suspect it will work as advertised now. Will play with it more later and if I manage to verify it on my own here I'll let you know.
> Giraph needs an admin script
> ----------------------------
>
> Key: GIRAPH-312
> URL: https://issues.apache.org/jira/browse/GIRAPH-312
> Project: Giraph
> Issue Type: New Feature
> Components: conf and scripts, zookeeper
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-312-1.patch, GIRAPH-312-2.patch, GIRAPH-312-3.patch, GIRAPH-312-3.patch, GIRAPH-312-4.patch, GIRAPH-312-5.patch, GIRAPH-326-6.patch
>
>
> Our zookeeper instances have very long uptimes on our cluster, and failed job trees are never cleaned from memory. There is a separate shell script to do this, but its not picky about which node trees it erases, and on some systems some Giraph users may not have access to it.
> This patch will add a shell script to activate a new class which will use Giraph conf file options or our normal -Dgiraph.XYZ command-line opts to get the ZK quorum info, and clean out the remnants from its memory of old failed and killed jobs. They do pile up over time.
> This led to the larger idea that Giraph needs a general giraph-admin shell script as a home for stuff like this. Jakob suggested it would be a good idea to put this into such a script since then admin groups can be created so that not every Giraph client can run it. This script currently only has code to start up the zk cleaner, but can have more options added to it as JIRA's to add features crop up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-312) Giraph needs an admin script
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Reisman updated GIRAPH-312:
-------------------------------
Attachment: GIRAPH-312-3.patch
Still going to do a bit more testing on this, trying a slightly different approach, will report results ASAP...
> Giraph needs an admin script
> ----------------------------
>
> Key: GIRAPH-312
> URL: https://issues.apache.org/jira/browse/GIRAPH-312
> Project: Giraph
> Issue Type: New Feature
> Components: conf and scripts, zookeeper
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-312-1.patch, GIRAPH-312-2.patch, GIRAPH-312-3.patch
>
>
> Our zookeeper instances have very long uptimes on our cluster, and failed job trees are never cleaned from memory. There is a separate shell script to do this, but its not picky about which node trees it erases, and on some systems some Giraph users may not have access to it.
> This patch will add a shell script to activate a new class which will use Giraph conf file options or our normal -Dgiraph.XYZ command-line opts to get the ZK quorum info, and clean out the remnants from its memory of old failed and killed jobs. They do pile up over time.
> This led to the larger idea that Giraph needs a general giraph-admin shell script as a home for stuff like this. Jakob suggested it would be a good idea to put this into such a script since then admin groups can be created so that not every Giraph client can run it. This script currently only has code to start up the zk cleaner, but can have more options added to it as JIRA's to add features crop up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira