You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ambari.apache.org by "Akhil S Naik (JIRA)" <ji...@apache.org> on 2019/04/26 11:17:00 UTC
[jira] [Updated] (AMBARI-25267) Post user creation hook Operation
is hanging sometimes until the operation is aborted or Ambari server is
restarted
[ https://issues.apache.org/jira/browse/AMBARI-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Akhil S Naik updated AMBARI-25267:
----------------------------------
Description:
i have enabled the automatic Creation of user directory for users of ambari (Post user Creation Hook)
When a new user is added in AD and is synced with ambari sometimes the Post User creation Hook operation is hanging infinitely . This happens very rarely and once in two weeks or so . Once hanged i need to abort the operation or restart the ambari-server for the operation to finish.
from logs i can see :
{code:java}
03 Feb 2018 06:00:23,102 INFO [pool-3-thread-1] UserHookService:131 - Preparing hook execution for event: UserCreatedEvent{eventType=USER_CREATED}
03 Feb 2018 06:00:23,117 ERROR [ambari-action-scheduler] ActionScheduler:783 - Execution command has no timeout parameter{"clusterName":"VWFS_ProdKons","requestId":1894,"stageId":-1,"taskId":8497,"commandId":"1894--1","hostname":"_internal_ambari","role":"AMBARI_SERVER_ACTION","hostLevelParams":{},"roleParams":{"ACTION_USER_NAME":"ambari","ACTION_NAME":"org.apache.ambari.server.serveraction.users.PostUserCreationHookServerAction"},"roleCommand":"EXECUTE","clusterHostInfo":{},"configurations":{},"configuration_attributes":{},"configurationTags":{},"forceRefreshConfigTagsBeforeExecution":false,"commandParams":{"cmd-hdfs-principal":"DKX1S58739@FS01.VWF.VWFS-AD","cmd-input-file":"/var/lib/ambari-server/data/tmp/user_hook_input_1517634023107.csv","cluster-security-type":"KERBEROS","cmd-hdfs-user":"hdfs","cmd-payload":"{\"dkx83au\":[\"de-p-ambariusers-kp-s-g\",\"de-adm-hadoopappops-kp-s-g\"],\"dkx8h9n\":[\"de-p-ambariusers-kp-s-g\"],\"dkx8nwd\":[\"de-p-ambariusers-kp-s-g\"]}","cmd-hdfs-keytab":"/etc/security/keytabs/hdfs.headless.keytab","hook-script":"/var/lib/ambari-server/resources/scripts/
{code}
then the ambari-server logs fills with Actionscheduler logs till the server is restarted or task is aborted.
call doesn't reach's the 'PostUserCreationHookServerAction' at any stage which happens in my local cluster
restarted ambari-server doesnt help
{code:java}
16 Apr 2019 13:25:08,323 WARN [ambari-action-scheduler] ActionScheduler:782 - Host: _internal_ambari, role: AMBARI_SERVER_ACTION, actionId: 506--1 expired and will be failed
16 Apr 2019 13:25:08,326 INFO [ambari-action-scheduler] ActionScheduler:923 - Service name is , component name is AMBARI_SERVER_ACTIONskipping sending ServiceComponentHostOpFailedEvent for AMBARI_SERVER_ACTION
16 Apr 2019 13:25:08,327 INFO [ambari-action-scheduler] ActionScheduler:809 - Removing command from queue, host=_internal_ambari, commandId=506--1
16 Apr 2019 13:25:08,327 ERROR [ambari-action-scheduler] ActionScheduler:754 - Execution command has no timeout parameter{“clusterName”:“TESTCLUSTER”,“requestId”:1894,“stageId”:-1,“taskId”:8497,“commandId”:“1894--1”,“hostname”:“_internal_ambari”,“role”:“AMBARI_SERVER_ACTION”,“hostLevelParams”:{},“roleParams”:{“ACTION_USER_NAME”:“ambari”,“ACTION_NAME”:“org.apache.ambari.server.serveraction.users.PostUserCreationHookServerAction”},“roleCommand”:“EXECUTE”,“clusterHostInfo”:{},“configurations”:{},“configuration_attributes”:{},“configurationTags”:{},“forceRefreshConfigTagsBeforeExecution”:false,“commandParams”:{“cmd-hdfs-principal”:“hdfs-test@EXAMPLE.COM”,“cmd-input-file”:“/var/lib/ambari-server/data/tmp/user_hook_input_1517634023107.csv”,“cluster-security-type”:“KERBEROS”,“cmd-hdfs-user”:“hdfs”,“cmd-payload”:“{\“dqwe3au\“:[\“dere-p-ambariusers-kp-s-g\“,\“de-adm-hadoopappops-kp-s-g\“],\“dkx8123h9n\“:[\“de-p-ambariusers-kp-s-g\“],\“dkx8nwd\“:[\“de-p-ambariusers-kp-s-g\“]}“,”cmd-hdfs-keytab”:“/etc/security/keytabs/hdfs.headless.keytab”,“hook-script”:“/var/lib/ambari-server/resources/scripts/
..
..
{code}
Looks like some issue with ambari action scheduler.
was:
i have enabled the automatic Creation of user directory for users of ambari (Post user Creation Hook)
When a new user is added in AD and is synced with ambari sometimes the Post User creation Hook operation is hanging infinitely . This happens very rarely and once in two weeks or so . Once hanged i need to abort the operation or restart the ambari-server for the operation to finish.
from logs i can see :
{code:java}
03 Feb 2018 06:00:23,102 INFO [pool-3-thread-1] UserHookService:131 - Preparing hook execution for event: UserCreatedEvent{eventType=USER_CREATED}
03 Feb 2018 06:00:23,117 ERROR [ambari-action-scheduler] ActionScheduler:783 - Execution command has no timeout parameter{"clusterName":"VWFS_ProdKons","requestId":1894,"stageId":-1,"taskId":8497,"commandId":"1894--1","hostname":"_internal_ambari","role":"AMBARI_SERVER_ACTION","hostLevelParams":{},"roleParams":{"ACTION_USER_NAME":"ambari","ACTION_NAME":"org.apache.ambari.server.serveraction.users.PostUserCreationHookServerAction"},"roleCommand":"EXECUTE","clusterHostInfo":{},"configurations":{},"configuration_attributes":{},"configurationTags":{},"forceRefreshConfigTagsBeforeExecution":false,"commandParams":{"cmd-hdfs-principal":"DKX1S58739@FS01.VWF.VWFS-AD","cmd-input-file":"/var/lib/ambari-server/data/tmp/user_hook_input_1517634023107.csv","cluster-security-type":"KERBEROS","cmd-hdfs-user":"hdfs","cmd-payload":"{\"dkx83au\":[\"de-p-ambariusers-kp-s-g\",\"de-adm-hadoopappops-kp-s-g\"],\"dkx8h9n\":[\"de-p-ambariusers-kp-s-g\"],\"dkx8nwd\":[\"de-p-ambariusers-kp-s-g\"]}","cmd-hdfs-keytab":"/etc/security/keytabs/hdfs.headless.keytab","hook-script":"/var/lib/ambari-server/resources/scripts/
{code}
then the ambari-server logs fills with Actionscheduler logs till the server is restarted or task is aborted.
call doesn't reach's the 'PostUserCreationHookServerAction' at any stage which happens in my local cluster
restarted ambari-server doesnt help
{code:java}
16 Apr 2019 13:25:08,323 WARN [ambari-action-scheduler] ActionScheduler:782 - Host: _internal_ambari, role: AMBARI_SERVER_ACTION, actionId: 506--1 expired and will be failed
16 Apr 2019 13:25:08,326 INFO [ambari-action-scheduler] ActionScheduler:923 - Service name is , component name is AMBARI_SERVER_ACTIONskipping sending ServiceComponentHostOpFailedEvent for AMBARI_SERVER_ACTION
16 Apr 2019 13:25:08,327 INFO [ambari-action-scheduler] ActionScheduler:809 - Removing command from queue, host=_internal_ambari, commandId=506--1
16 Apr 2019 13:25:08,327 ERROR [ambari-action-scheduler] ActionScheduler:754 - Execution command has no timeout parameter{"clusterName":"XX"
..
..
..
{code}
Looks like some issue with ambari action scheduler.
> Post user creation hook Operation is hanging sometimes until the operation is aborted or Ambari server is restarted
> -------------------------------------------------------------------------------------------------------------------
>
> Key: AMBARI-25267
> URL: https://issues.apache.org/jira/browse/AMBARI-25267
> Project: Ambari
> Issue Type: Bug
> Environment: ambari-2.6.2
> HDP-2.6.5
> Reporter: Akhil S Naik
> Priority: Major
>
> i have enabled the automatic Creation of user directory for users of ambari (Post user Creation Hook)
> When a new user is added in AD and is synced with ambari sometimes the Post User creation Hook operation is hanging infinitely . This happens very rarely and once in two weeks or so . Once hanged i need to abort the operation or restart the ambari-server for the operation to finish.
> from logs i can see :
> {code:java}
> 03 Feb 2018 06:00:23,102 INFO [pool-3-thread-1] UserHookService:131 - Preparing hook execution for event: UserCreatedEvent{eventType=USER_CREATED}
> 03 Feb 2018 06:00:23,117 ERROR [ambari-action-scheduler] ActionScheduler:783 - Execution command has no timeout parameter{"clusterName":"VWFS_ProdKons","requestId":1894,"stageId":-1,"taskId":8497,"commandId":"1894--1","hostname":"_internal_ambari","role":"AMBARI_SERVER_ACTION","hostLevelParams":{},"roleParams":{"ACTION_USER_NAME":"ambari","ACTION_NAME":"org.apache.ambari.server.serveraction.users.PostUserCreationHookServerAction"},"roleCommand":"EXECUTE","clusterHostInfo":{},"configurations":{},"configuration_attributes":{},"configurationTags":{},"forceRefreshConfigTagsBeforeExecution":false,"commandParams":{"cmd-hdfs-principal":"DKX1S58739@FS01.VWF.VWFS-AD","cmd-input-file":"/var/lib/ambari-server/data/tmp/user_hook_input_1517634023107.csv","cluster-security-type":"KERBEROS","cmd-hdfs-user":"hdfs","cmd-payload":"{\"dkx83au\":[\"de-p-ambariusers-kp-s-g\",\"de-adm-hadoopappops-kp-s-g\"],\"dkx8h9n\":[\"de-p-ambariusers-kp-s-g\"],\"dkx8nwd\":[\"de-p-ambariusers-kp-s-g\"]}","cmd-hdfs-keytab":"/etc/security/keytabs/hdfs.headless.keytab","hook-script":"/var/lib/ambari-server/resources/scripts/
> {code}
> then the ambari-server logs fills with Actionscheduler logs till the server is restarted or task is aborted.
> call doesn't reach's the 'PostUserCreationHookServerAction' at any stage which happens in my local cluster
> restarted ambari-server doesnt help
> {code:java}
> 16 Apr 2019 13:25:08,323 WARN [ambari-action-scheduler] ActionScheduler:782 - Host: _internal_ambari, role: AMBARI_SERVER_ACTION, actionId: 506--1 expired and will be failed
> 16 Apr 2019 13:25:08,326 INFO [ambari-action-scheduler] ActionScheduler:923 - Service name is , component name is AMBARI_SERVER_ACTIONskipping sending ServiceComponentHostOpFailedEvent for AMBARI_SERVER_ACTION
> 16 Apr 2019 13:25:08,327 INFO [ambari-action-scheduler] ActionScheduler:809 - Removing command from queue, host=_internal_ambari, commandId=506--1
> 16 Apr 2019 13:25:08,327 ERROR [ambari-action-scheduler] ActionScheduler:754 - Execution command has no timeout parameter{“clusterName”:“TESTCLUSTER”,“requestId”:1894,“stageId”:-1,“taskId”:8497,“commandId”:“1894--1”,“hostname”:“_internal_ambari”,“role”:“AMBARI_SERVER_ACTION”,“hostLevelParams”:{},“roleParams”:{“ACTION_USER_NAME”:“ambari”,“ACTION_NAME”:“org.apache.ambari.server.serveraction.users.PostUserCreationHookServerAction”},“roleCommand”:“EXECUTE”,“clusterHostInfo”:{},“configurations”:{},“configuration_attributes”:{},“configurationTags”:{},“forceRefreshConfigTagsBeforeExecution”:false,“commandParams”:{“cmd-hdfs-principal”:“hdfs-test@EXAMPLE.COM”,“cmd-input-file”:“/var/lib/ambari-server/data/tmp/user_hook_input_1517634023107.csv”,“cluster-security-type”:“KERBEROS”,“cmd-hdfs-user”:“hdfs”,“cmd-payload”:“{\“dqwe3au\“:[\“dere-p-ambariusers-kp-s-g\“,\“de-adm-hadoopappops-kp-s-g\“],\“dkx8123h9n\“:[\“de-p-ambariusers-kp-s-g\“],\“dkx8nwd\“:[\“de-p-ambariusers-kp-s-g\“]}“,”cmd-hdfs-keytab”:“/etc/security/keytabs/hdfs.headless.keytab”,“hook-script”:“/var/lib/ambari-server/resources/scripts/
> ..
> ..
> {code}
> Looks like some issue with ambari action scheduler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)