You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hemanth Yamijala (JIRA)" <ji...@apache.org> on 2008/09/10 07:17:44 UTC
[jira] Created: (HADOOP-4145) [HOD] Support an accounting plugin
script for HOD
[HOD] Support an accounting plugin script for HOD
-------------------------------------------------
Key: HADOOP-4145
URL: https://issues.apache.org/jira/browse/HADOOP-4145
Project: Hadoop Core
Issue Type: Improvement
Components: contrib/hod
Reporter: Hemanth Yamijala
Assignee: Hemanth Yamijala
Priority: Blocker
Fix For: 0.18.1, 0.19.0
Production environments have accounting packages to track usage of a cluster. HOD should provide a mechanism to plug-in an accounting related script that can be used to verify if the user is using a valid account or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4145) [HOD] Support an accounting plugin
script for HOD
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630081#action_12630081 ]
Hemanth Yamijala commented on HADOOP-4145:
------------------------------------------
The attached patch implements the following changes in HOD. There are slight differences from the proposal listed above, which are also described below.
Changes:
- Hod client looks for a specific file in the {{install-directory/bin}} folder called {{verify-account}}. If this file is found, before cluster allocation is begun, it executes the file passing in {{resource_manager.pbs-account}} as a parameter to the script.
- If the script's exit code is non-zero, Hod client fails allocation. Any output generated by the script (either on stdout or stderr) is printed to the Hod client.
- Hod exits the allocation process with a return value of 5, standing for job execution failure.
- If the verify-account script is not found, or if the script returns a zero exit code, Hod client proceeds allocation as is.
- A dummy script file verify-account is added in the patch which sites can modify with their custom verification script. The dummy script file just exits with 0.
- Documentation changes are done in the Hod admin guide and user guide.
Changes from the proposal:
- There are no configuration options introduced in this patch. This is essentially because Hod cannot control users overriding configuration set up by administrators. And the account script file should not be overridden.
- No support for passing parameters to the account verification script. Again the reason is roughly the same as for the first point above. Also, we don't have enough use cases to see if this additional feature is essential. It should be possible to workaround by generating verification scripts that directly have these arguments.
- All output from the script is displayed to the user, not just stderr output. This is because HOD currently has no good way of capturing only stderr output.
> [HOD] Support an accounting plugin script for HOD
> -------------------------------------------------
>
> Key: HADOOP-4145
> URL: https://issues.apache.org/jira/browse/HADOOP-4145
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HADOOP-4145.patch
>
>
> Production environments have accounting packages to track usage of a cluster. HOD should provide a mechanism to plug-in an accounting related script that can be used to verify if the user is using a valid account or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4145) [HOD] Support an accounting plugin
script for HOD
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630507#action_12630507 ]
Hadoop QA commented on HADOOP-4145:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12389985/HADOOP-4145.patch
against trunk revision 694562.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3256/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3256/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3256/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3256/console
This message is automatically generated.
> [HOD] Support an accounting plugin script for HOD
> -------------------------------------------------
>
> Key: HADOOP-4145
> URL: https://issues.apache.org/jira/browse/HADOOP-4145
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HADOOP-4145.patch, HADOOP-4145.patch, HADOOP-4145.patch
>
>
> Production environments have accounting packages to track usage of a cluster. HOD should provide a mechanism to plug-in an accounting related script that can be used to verify if the user is using a valid account or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4145) [HOD] Support an accounting plugin
script for HOD
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-4145:
-------------------------------------
Attachment: HADOOP-4145.patch
> [HOD] Support an accounting plugin script for HOD
> -------------------------------------------------
>
> Key: HADOOP-4145
> URL: https://issues.apache.org/jira/browse/HADOOP-4145
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HADOOP-4145.patch
>
>
> Production environments have accounting packages to track usage of a cluster. HOD should provide a mechanism to plug-in an accounting related script that can be used to verify if the user is using a valid account or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4145) [HOD] Support an accounting plugin
script for HOD
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630435#action_12630435 ]
Hadoop QA commented on HADOOP-4145:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12389921/HADOOP-4145.patch
against trunk revision 694483.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3251/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3251/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3251/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3251/console
This message is automatically generated.
> [HOD] Support an accounting plugin script for HOD
> -------------------------------------------------
>
> Key: HADOOP-4145
> URL: https://issues.apache.org/jira/browse/HADOOP-4145
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HADOOP-4145.patch, HADOOP-4145.patch
>
>
> Production environments have accounting packages to track usage of a cluster. HOD should provide a mechanism to plug-in an accounting related script that can be used to verify if the user is using a valid account or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4145) [HOD] Support an accounting plugin
script for HOD
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629708#action_12629708 ]
Hemanth Yamijala commented on HADOOP-4145:
------------------------------------------
Proposal:
* We can define an optional hod.account-verification-script in the hod configuration.
* If present, then hod will launch this as:
{code}
${hod.account-verification-script} ${resource_manager.pbs-account} ${hod.account-verification-script-params}
{code}
* The account-verification-script-params is to allow for us to pass other information in the future, for e.g. maybe a URL to a custom accounting package, etc.
* If this script returns a zero exit code, all is fine, and the job proceeds for allocation, passing the account name as before.
* If the script returns a non-zero exit code, allocation fails. The script can write a descriptive message to stderr and that will be printed by HOD to the user.
* If no script option is specified, hod allocation works as it does currently.
> [HOD] Support an accounting plugin script for HOD
> -------------------------------------------------
>
> Key: HADOOP-4145
> URL: https://issues.apache.org/jira/browse/HADOOP-4145
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
>
> Production environments have accounting packages to track usage of a cluster. HOD should provide a mechanism to plug-in an accounting related script that can be used to verify if the user is using a valid account or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4145) [HOD] Support an accounting plugin
script for HOD
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-4145:
-------------------------------------
Status: Open (was: Patch Available)
> [HOD] Support an accounting plugin script for HOD
> -------------------------------------------------
>
> Key: HADOOP-4145
> URL: https://issues.apache.org/jira/browse/HADOOP-4145
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HADOOP-4145.patch, HADOOP-4145.patch, HADOOP-4145.patch
>
>
> Production environments have accounting packages to track usage of a cluster. HOD should provide a mechanism to plug-in an accounting related script that can be used to verify if the user is using a valid account or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4145) [HOD] Support an accounting plugin
script for HOD
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-4145:
-------------------------------------
Attachment: HADOOP-4145.patch
Patch addressing review comments.
> [HOD] Support an accounting plugin script for HOD
> -------------------------------------------------
>
> Key: HADOOP-4145
> URL: https://issues.apache.org/jira/browse/HADOOP-4145
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HADOOP-4145.patch, HADOOP-4145.patch
>
>
> Production environments have accounting packages to track usage of a cluster. HOD should provide a mechanism to plug-in an accounting related script that can be used to verify if the user is using a valid account or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4145) [HOD] Support an accounting plugin
script for HOD
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-4145:
--------------------------------
Resolution: Fixed
Fix Version/s: (was: 0.19.0)
Status: Resolved (was: Patch Available)
I just committed this. Thank Hemanth!
> [HOD] Support an accounting plugin script for HOD
> -------------------------------------------------
>
> Key: HADOOP-4145
> URL: https://issues.apache.org/jira/browse/HADOOP-4145
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.18.1
>
> Attachments: HADOOP-4145.patch, HADOOP-4145.patch, HADOOP-4145.patch
>
>
> Production environments have accounting packages to track usage of a cluster. HOD should provide a mechanism to plug-in an accounting related script that can be used to verify if the user is using a valid account or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4145) [HOD] Support an accounting plugin
script for HOD
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-4145:
-------------------------------------
Hadoop Flags: [Reviewed]
Status: Patch Available (was: Open)
Running through Hudson
> [HOD] Support an accounting plugin script for HOD
> -------------------------------------------------
>
> Key: HADOOP-4145
> URL: https://issues.apache.org/jira/browse/HADOOP-4145
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HADOOP-4145.patch, HADOOP-4145.patch
>
>
> Production environments have accounting packages to track usage of a cluster. HOD should provide a mechanism to plug-in an accounting related script that can be used to verify if the user is using a valid account or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4145) [HOD] Support an accounting plugin
script for HOD
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630098#action_12630098 ]
Vinod K V commented on HADOOP-4145:
-----------------------------------
In the error-code description in the user-guide, it would be good if we have a link to account verification related documentation in the admin-guide.
> [HOD] Support an accounting plugin script for HOD
> -------------------------------------------------
>
> Key: HADOOP-4145
> URL: https://issues.apache.org/jira/browse/HADOOP-4145
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HADOOP-4145.patch
>
>
> Production environments have accounting packages to track usage of a cluster. HOD should provide a mechanism to plug-in an accounting related script that can be used to verify if the user is using a valid account or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4145) [HOD] Support an accounting plugin
script for HOD
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-4145:
-------------------------------------
Status: Patch Available (was: Open)
Running through Hudson again
> [HOD] Support an accounting plugin script for HOD
> -------------------------------------------------
>
> Key: HADOOP-4145
> URL: https://issues.apache.org/jira/browse/HADOOP-4145
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HADOOP-4145.patch, HADOOP-4145.patch, HADOOP-4145.patch
>
>
> Production environments have accounting packages to track usage of a cluster. HOD should provide a mechanism to plug-in an accounting related script that can be used to verify if the user is using a valid account or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4145) [HOD] Support an accounting plugin
script for HOD
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630515#action_12630515 ]
Hemanth Yamijala commented on HADOOP-4145:
------------------------------------------
Spoke to Vinod. Given we have to move this to Hadoop 0.18.1 ASAP, we will live with the additional blank line for the time being, and address in a separate JIRA.
> [HOD] Support an accounting plugin script for HOD
> -------------------------------------------------
>
> Key: HADOOP-4145
> URL: https://issues.apache.org/jira/browse/HADOOP-4145
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HADOOP-4145.patch, HADOOP-4145.patch, HADOOP-4145.patch
>
>
> Production environments have accounting packages to track usage of a cluster. HOD should provide a mechanism to plug-in an accounting related script that can be used to verify if the user is using a valid account or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4145) [HOD] Support an accounting plugin
script for HOD
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630094#action_12630094 ]
Vinod K V commented on HADOOP-4145:
-----------------------------------
Very minor comments, mostly related to logging.
- is_valid_account() documentation string should read as "<work-dir>/verify-account" or "<install-dir>/bin/verify-account" instead of "<install-dir>/verify-account". Actual usage is correct.
- Would be clear if, while logging, the output from the verify-account script file is marked, for e.g. prepended with "verify-account output : ".
- Irrespective of the return code
-- we should just log the output of the script at debug level.
-- we should conclude the operation by a *info* log statement - HOD verify-account succeeded / failed with error code <errCode>, with script output <scriptOut>
- When verify-account returns a non-zero error code, we log "err msg from script: %s' % errMsg" - this should be "output msg from script". This msg will go away if we do what the immediately above point describes.
Other than these, +1 for the patch, it works as described.
> [HOD] Support an accounting plugin script for HOD
> -------------------------------------------------
>
> Key: HADOOP-4145
> URL: https://issues.apache.org/jira/browse/HADOOP-4145
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HADOOP-4145.patch
>
>
> Production environments have accounting packages to track usage of a cluster. HOD should provide a mechanism to plug-in an accounting related script that can be used to verify if the user is using a valid account or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4145) [HOD] Support an accounting plugin
script for HOD
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630487#action_12630487 ]
Hemanth Yamijala commented on HADOOP-4145:
------------------------------------------
bq. The validation script would also require the number of nodes, queue and optionally estimated walltime
Discussed this with Rob. We decided as this was not required for now, and will not require a change to user interface when needed in future, we'll go with the existing approach of only passing in account name. If more parameters are required, I'd actually prefer to think of passing it in some generic way to scripts - because it is a plug-in hardcoding a specific list of params might not be a good idea.
bq. It should fail with "Job Submission Failure"
Modified the exit code to 4, which is the closest we have to job submission failure. Atleast both come in similar circumstances.
> [HOD] Support an accounting plugin script for HOD
> -------------------------------------------------
>
> Key: HADOOP-4145
> URL: https://issues.apache.org/jira/browse/HADOOP-4145
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HADOOP-4145.patch, HADOOP-4145.patch, HADOOP-4145.patch
>
>
> Production environments have accounting packages to track usage of a cluster. HOD should provide a mechanism to plug-in an accounting related script that can be used to verify if the user is using a valid account or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4145) [HOD] Support an accounting plugin
script for HOD
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12633161#action_12633161 ]
Vinod K V commented on HADOOP-4145:
-----------------------------------
bq. [..] we will live with the additional blank line for the time being, and address in a separate JIRA.
Being done as part of HADOOP-4235.
> [HOD] Support an accounting plugin script for HOD
> -------------------------------------------------
>
> Key: HADOOP-4145
> URL: https://issues.apache.org/jira/browse/HADOOP-4145
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.18.1
>
> Attachments: HADOOP-4145.patch, HADOOP-4145.patch, HADOOP-4145.patch
>
>
> Production environments have accounting packages to track usage of a cluster. HOD should provide a mechanism to plug-in an accounting related script that can be used to verify if the user is using a valid account or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4145) [HOD] Support an accounting plugin
script for HOD
Posted by "Rajiv Chittajallu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630275#action_12630275 ]
Rajiv Chittajallu commented on HADOOP-4145:
-------------------------------------------
> - Hod client looks for a specific file in the {{install-directory/bin}} folder called {{verify-account}}. If this file is found, before cluster allocation is begun, it executes the file passing in {{resource_manager.pbs-account}} as a parameter to the script.
The validation script would also require the number of nodes, queue and optionally estimated walltime
Queue name and the number of nodes are mandatory be mandatory.
> - Hod exits the allocation process with a return value of 5, standing for job execution failure.
It should fail with "Job Submission Failure"
> [HOD] Support an accounting plugin script for HOD
> -------------------------------------------------
>
> Key: HADOOP-4145
> URL: https://issues.apache.org/jira/browse/HADOOP-4145
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HADOOP-4145.patch, HADOOP-4145.patch
>
>
> Production environments have accounting packages to track usage of a cluster. HOD should provide a mechanism to plug-in an accounting related script that can be used to verify if the user is using a valid account or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4145) [HOD] Support an accounting plugin
script for HOD
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630498#action_12630498 ]
Vinod K V commented on HADOOP-4145:
-----------------------------------
One very minor comment. When we print the error messages in case of non-zero script exit code, we should strip new lines at the end of each message, otherwise log.critical also appends a new line character to each message and blank lines creep into log output.
> [HOD] Support an accounting plugin script for HOD
> -------------------------------------------------
>
> Key: HADOOP-4145
> URL: https://issues.apache.org/jira/browse/HADOOP-4145
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HADOOP-4145.patch, HADOOP-4145.patch, HADOOP-4145.patch
>
>
> Production environments have accounting packages to track usage of a cluster. HOD should provide a mechanism to plug-in an accounting related script that can be used to verify if the user is using a valid account or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4145) [HOD] Support an accounting plugin
script for HOD
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-4145:
-------------------------------------
Attachment: HADOOP-4145.patch
New patch addressing two defects in the older one:
- Incorporated Rajiv's comments on exit code.
- I'd forgotten to include the verify-install script in the patch.
> [HOD] Support an accounting plugin script for HOD
> -------------------------------------------------
>
> Key: HADOOP-4145
> URL: https://issues.apache.org/jira/browse/HADOOP-4145
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.18.1, 0.19.0
>
> Attachments: HADOOP-4145.patch, HADOOP-4145.patch, HADOOP-4145.patch
>
>
> Production environments have accounting packages to track usage of a cluster. HOD should provide a mechanism to plug-in an accounting related script that can be used to verify if the user is using a valid account or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.