You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Bill Graham (JIRA)" <ji...@apache.org> on 2010/01/12 19:49:54 UTC
[jira] Created: (CHUKWA-446) Refactor tools/init.d scripts
Refactor tools/init.d scripts
-----------------------------
Key: CHUKWA-446
URL: https://issues.apache.org/jira/browse/CHUKWA-446
Project: Hadoop Chukwa
Issue Type: Improvement
Reporter: Bill Graham
1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang reopened CHUKWA-446:
------------------------------
There is a problem with start-* scripts where chukwa command is already in chukwa-daemons.sh.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: chukwa, CHUKWA-446-2.patch, CHUKWA-446.patch
>
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799322#action_12799322 ]
Jerome Boulon commented on CHUKWA-446:
--------------------------------------
CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER are configurable.
It's done when you generate the RPM, look at the ANT RPM target and default.properties file.
To overwrite these values, just create a new file build.properties at the same level.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-446:
-----------------------------
Status: Patch Available (was: Open)
First cut of the clean up.
- Watchdog removed
- Database table management removed
- system metrics moved to ExecAdaptor.
- Rmoved legacy Torque data loader
- Removed legacy Node Activity data loader.
- Removed legacy UserHDFSUsage script. This metering solution is exported by JobSummary log file.
- Single wrapper script for all chukwa commands.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: chukwa, CHUKWA-446.patch
>
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802522#action_12802522 ]
Eric Yang commented on CHUKWA-446:
----------------------------------
+1 for no dbAdmin (database data loader should be deprecated.)
+1 for removing crontab watch dog.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: chukwa
>
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802489#action_12802489 ]
Ari Rabkin commented on CHUKWA-446:
-----------------------------------
I'm okay with this refactoring.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: chukwa
>
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-446:
-----------------------------
Attachment: CHUKWA-446.patch
Clean up shell scripts as agreed in this issue.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: chukwa, CHUKWA-446.patch
>
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-446:
-----------------------------
Fix Version/s: 0.4.0
Status: Patch Available (was: Reopened)
Clean up some of the improper reference in the shell script.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
> Fix For: 0.4.0
>
> Attachments: chukwa, CHUKWA-446-2.patch, CHUKWA-446-3.patch, CHUKWA-446-4.patch, CHUKWA-446.patch
>
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799329#action_12799329 ]
Eric Yang commented on CHUKWA-446:
----------------------------------
start-agent.sh, and stop-agent.sh could be boot trap script for agent.sh and same model applies to jettyCollector.sh. This may be too many wrapper scripts. Could we rename jettyCollector.sh to collector.sh? Hence, for singular use agent.sh and collector.sh, for multiple machines, use start-agents.sh, and start-collectors.sh. I am not sure if this makes sense for everyone. We could vote on this.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-446:
-----------------------------
Status: Patch Available (was: Open)
Please review, thanks. :)
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: chukwa, CHUKWA-446-2.patch, CHUKWA-446.patch
>
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang reassigned CHUKWA-446:
--------------------------------
Assignee: Eric Yang
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-446:
-----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks Guillermo, I just committed the change.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
> Fix For: 0.4.0
>
> Attachments: chukwa, CHUKWA-446-2.patch, CHUKWA-446-4.patch, CHUKWA-446.patch
>
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802356#action_12802356 ]
Eric Yang commented on CHUKWA-446:
----------------------------------
I am keeping all start-* and stop-* commands in the same directory. Hence, the list of files to stay in bin directory are:
{code}
README
VERSION
agent.sh
collector.sh
chukwa
chukwa-config.sh
chukwa-daemon.sh
chukwa-daemons.sh
slaves.sh
start-agents.sh
start-all.sh
start-collectors.sh
start-data-processors.sh
stop-agents.sh
stop-all.sh
stop-collectors.sh
stop-data-processors.sh
{code}
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: chukwa
>
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-446:
-----------------------------
Attachment: CHUKWA-446-4.patch
Stop script handles stop command more gracefully, when there is no command to stop.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: chukwa, CHUKWA-446-2.patch, CHUKWA-446-3.patch, CHUKWA-446-4.patch, CHUKWA-446.patch
>
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799369#action_12799369 ]
Bill Graham commented on CHUKWA-446:
------------------------------------
I'm also fine without using the init.d scripts, since after looking more closely at the bin/stat-*.sh scripts they seem like they'll work better for my needs anyway.
@Eric +1 on renaming jettyCollector.sh to collector.sh for consistency and following the foo.sh, start-foo.sh and stop-foo.sh pattern.
If we're going to refactor the start-*.sh scripts I'd like to suggest we add a feature to slaves.sh, where if no hostnames are configured, the default behavior would be to start the daemon on localhost without ssh'ing. This would basically be the same as the wrapped script, just with pid files managed by start/stop wrapper.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799330#action_12799330 ]
Jerome Boulon commented on CHUKWA-446:
--------------------------------------
My guess is that 80% of these scripts can be merge into 1 script, then it should just be a matter of passing some parameters but most of the logic should be shared.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-446:
-----------------------------
Attachment: CHUKWA-446-3.patch
Correction to startup and stop scripts.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: chukwa, CHUKWA-446-2.patch, CHUKWA-446-3.patch, CHUKWA-446.patch
>
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799353#action_12799353 ]
Eric Yang commented on CHUKWA-446:
----------------------------------
Hadoop has a fat shell script which encapsulated most of the shell script logic. It's possible to mimic the logic:
Usage: chukwa [--config confdir] COMMAND
where COMMAND is one of:
collector run a Chukwa Collector
agent run a Chukwa Agent
dp run the data processors
hicc run the HICC console
version print the version
Most commands print help when invoked w/o parameters.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799327#action_12799327 ]
Ari Rabkin commented on CHUKWA-446:
-----------------------------------
I'm okay just ditching them.
At Berkeley, we use agent.sh and jettyCollector.sh, managed by runit.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799324#action_12799324 ]
Eric Yang commented on CHUKWA-446:
----------------------------------
The scripts were built to run in a specific environment to make it easy for a black magic grown system. My recommendation is completely remove those scripts and use start-*.sh stop-*.sh.
*chukwa-system-metrics, this should be replaced with exec adaptor and configure through initial_adaptors.template.
* chukwa-agent - rename to start-agent.sh and stop-agent.sh.
* chukwa-collector - rename to start-collector.sh and stop-collector.sh
* chukwa-data-processors - use start-data-processors.sh and stop-data-processor.sh
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-446:
-----------------------------
Attachment: (was: CHUKWA-446-3.patch)
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
> Fix For: 0.4.0
>
> Attachments: chukwa, CHUKWA-446-2.patch, CHUKWA-446-4.patch, CHUKWA-446.patch
>
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802056#action_12802056 ]
Ari Rabkin commented on CHUKWA-446:
-----------------------------------
I think there's a typo on "ChukwaArachiveManager"
Should be Archive, I think, not Arachive.
Are we ditching start-agents and start-collectors? I use those...but would be happy if they moved to some auxiliary directory to make bin look less scary.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: chukwa
>
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-446:
-----------------------------
Attachment: chukwa
Consolidate Chukwa shell scripts into a single chukwa command:
{code}
Usage: chukwa [--config confdir] COMMAND
where COMMAND is one of:
agent run a Chukwa Agent
archive run the Archive Manager
collector run a Chukwa Collector
demux run the Demux Manager
dataloader run the Post Demux Data Loaders
hicc run a HICC Webserver
version print the version
Utilities:
backfill run a back fill data loader utility
dumpArchive view an archive file
dumpRecord view a record file
Most command print help when invoked w/o parameters.
{code}
The long list of shell scripts that didn't make to this list will be deleted, unless someone voice their concerns.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: chukwa
>
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-446:
-----------------------------
Status: Open (was: Patch Available)
Have a updated version.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: chukwa, CHUKWA-446.patch
>
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-446:
-----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Ari.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: chukwa, CHUKWA-446-2.patch, CHUKWA-446.patch
>
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850966#action_12850966 ]
Hudson commented on CHUKWA-446:
-------------------------------
Integrated in Chukwa-trunk #330 (See [http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/330/])
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
> Fix For: 0.4.0
>
> Attachments: chukwa, CHUKWA-446-2.patch, CHUKWA-446-4.patch, CHUKWA-446.patch
>
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802515#action_12802515 ]
Bill Graham commented on CHUKWA-446:
------------------------------------
So we're talking about having the general purpose chukwa wrapper script, as well as the start-*, stop-* scripts in bin? If so I'm also good with this refactoring. I'm particularly interested in the start-*, stop-* scripts.
If it's at all feasible, I'd like to be able to able to have options on start-data-processors.sh to 1.) not start dbAdmin; and 2.) to not modify the crontab. We don't use dbAdmin and our rpm install manages crontab entries. Again, a nice to have. Perhaps this is beyond the scope of this refactor.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: chukwa
>
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838639#action_12838639 ]
Ari Rabkin commented on CHUKWA-446:
-----------------------------------
Let's commit it. Passes my smoke test.
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: chukwa, CHUKWA-446-2.patch, CHUKWA-446.patch
>
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-446) Refactor tools/init.d scripts
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-446:
-----------------------------
Attachment: CHUKWA-446-2.patch
The new chukwa command looks like this:
{noformat}
./bin/chukwa
Usage: chukwa [--config confdir] COMMAND
where COMMAND is one of:
agent run a Chukwa Agent
archive run the Archive Manager
collector run a Chukwa Collector
demux run the Demux Manager
dp run the Post Demux data processors
hicc run a HICC Webserver
droll run a daily rolling job (deprecated)
hroll run a hourly rolling job (deprecated)
version print the version
Utilities:
backfill run a back fill data loader utility
dumpArchive view an archive file
dumpRecord view a record file
tail start tailing a file
Most command print help when invoked w/o parameters.
{noformat}
> Refactor tools/init.d scripts
> -----------------------------
>
> Key: CHUKWA-446
> URL: https://issues.apache.org/jira/browse/CHUKWA-446
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: chukwa, CHUKWA-446-2.patch, CHUKWA-446.patch
>
>
> 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.
> 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.
> 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.