You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ambari.apache.org by "Brahma Reddy Battula (Jira)" <ji...@apache.org> on 2022/12/01 02:01:00 UTC
[jira] [Commented] (AMBARI-25624) "Creating Kerberos keytabs" takes too long
[ https://issues.apache.org/jira/browse/AMBARI-25624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17641618#comment-17641618 ]
Brahma Reddy Battula commented on AMBARI-25624:
-----------------------------------------------
Committed to trunk also. [~eub] thanks again for your contribution.
> "Creating Kerberos keytabs" takes too long
> ------------------------------------------
>
> Key: AMBARI-25624
> URL: https://issues.apache.org/jira/browse/AMBARI-25624
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Affects Versions: 2.7.1
> Reporter: Jeongin Ju
> Assignee: YUBI LEE
> Priority: Major
> Fix For: 2.8.0, 2.7.7
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> In kerberized cluster, Ambari tried to find out which keytabs are required to be created when adding components.
> During this process, ambari instantiates serviceMapping variable in KerberosKeytabPrincipalEntity class which is not used on creating keytab process.
> More seriously, serviceMapping variable can invoke nested variable instantiation and internal lock acquisitions which takes so much time on large cluster as shown in below callstack.
> h6. {{Thread 129281: (state = IN_JAVA)
> - org.eclipse.persistence.descriptors.FetchGroupManager.isPartialObject(java.lang.Object) @bci=8, line=466 (Compiled frame; information may be imprecise)
> - org.eclipse.persistence.internal.helper.WriteLockManager.traverseRelatedLocks(java.lang.Object, java.util.Map, java.util.Map, org.eclipse.persistence.descriptors.ClassDescriptor, org.eclipse.persistence.internal.sessions.AbstractSession) @bci=23, line=185 (Compiled frame)
> - org.eclipse.persistence.internal.helper.WriteLockManager.acquireLockAndRelatedLocks(java.lang.Object, java.util.Map, java.util.Map, org.eclipse.persistence.internal.identitymaps.CacheKey, org.eclipse.persistence.descriptors.ClassDescriptor, org.eclipse.persistence.internal.sessions.AbstractSession) @bci=96, line=144 (Compiled frame)
> - org.eclipse.persistence.internal.helper.WriteLockManager.checkAndLockObject(java.lang.Object, java.util.Map, java.util.Map, org.eclipse.persistence.mappings.DatabaseMapping, org.eclipse.persistence.internal.sessions.AbstractSession) @bci=148, line=450 (Compiled frame)
> - org.eclipse.persistence.internal.helper.WriteLockManager.traverseRelatedLocks(java.lang.Object, java.util.Map, java.util.Map, org.eclipse.persistence.descriptors.ClassDescriptor, org.eclipse.persistence.internal.sessions.AbstractSession) @bci=237, line=213 (Compiled frame)
> - org.eclipse.persistence.internal.helper.WriteLockManager.acquireLockAndRelatedLocks(java.lang.Object, java.util.Map, java.util.Map, org.eclipse.persistence.internal.identitymaps.CacheKey, org.eclipse.persistence.descriptors.ClassDescriptor, org.eclipse.persistence.internal.sessions.AbstractSession) @bci=96, line=144 (Compiled frame)
> - org.eclipse.persistence.internal.helper.WriteLockManager.checkAndLockObject(java.lang.Object, java.util.Map, java.util.Map, org.eclipse.persistence.mappings.DatabaseMapping, org.eclipse.persistence.internal.sessions.AbstractSession) @bci=148, line=450 (Compiled frame)
> - org.eclipse.persistence.internal.helper.WriteLockManager.traverseRelatedLocks(java.lang.Object, java.util.Map, java.util.Map, org.eclipse.persistence.descriptors.ClassDescriptor, org.eclipse.persistence.internal.sessions.AbstractSession) @bci=237, line=213 (Compiled frame)
> - org.eclipse.persistence.internal.helper.WriteLockManager.acquireLockAndRelatedLocks(java.lang.Object, java.util.Map, java.util.Map, org.eclipse.persistence.internal.identitymaps.CacheKey, org.eclipse.persistence.descriptors.ClassDescriptor, org.eclipse.persistence.internal.sessions.AbstractSession) @bci=96, line=144 (Compiled frame)
> - org.eclipse.persistence.internal.helper.WriteLockManager.checkAndLockObject(java.lang.Object, java.util.Map, java.util.Map, org.eclipse.persistence.mappings.DatabaseMapping, org.eclipse.persistence.internal.sessions.AbstractSession) @bci=148, line=450 (Compiled frame)
> - org.eclipse.persistence.internal.helper.WriteLockManager.traverseRelatedLocks(java.lang.Object, java.util.Map, java.util.Map, org.eclipse.persistence.descriptors.ClassDescriptor, org.eclipse.persistence.internal.sessions.AbstractSession) @bci=172, line=203 (Compiled frame)
> - org.eclipse.persistence.internal.helper.WriteLockManager.acquireLockAndRelatedLocks(java.lang.Object, java.util.Map, java.util.Map, org.eclipse.persistence.internal.identitymaps.CacheKey, org.eclipse.persistence.descriptors.ClassDescriptor, org.eclipse.persistence.internal.sessions.AbstractSession) @bci=96, line=144 (Compiled frame)
> - org.eclipse.persistence.internal.helper.WriteLockManager.checkAndLockObject(java.lang.Object, java.util.Map, java.util.Map, org.eclipse.persistence.mappings.DatabaseMapping, org.eclipse.persistence.internal.sessions.AbstractSession) @bci=148, line=450 (Compiled frame)
> - org.eclipse.persistence.internal.helper.WriteLockManager.traverseRelatedLocks(java.lang.Object, java.util.Map, java.util.Map, org.eclipse.persistence.descriptors.ClassDescriptor, org.eclipse.persistence.internal.sessions.AbstractSession) @bci=237, line=213 (Compiled frame)
> - org.eclipse.persistence.internal.helper.WriteLockManager.acquireLockAndRelatedLocks(java.lang.Object, java.util.Map, java.util.Map, org.eclipse.persistence.internal.identitymaps.CacheKey, org.eclipse.persistence.descriptors.ClassDescriptor, org.eclipse.persistence.internal.sessions.AbstractSession) @bci=96, line=144 (Compiled frame)
> - org.eclipse.persistence.internal.helper.WriteLockManager.checkAndLockObject(java.lang.Object, java.util.Map, java.util.Map, org.eclipse.persistence.mappings.DatabaseMapping, org.eclipse.persistence.internal.sessions.AbstractSession) @bci=148, line=450 (Compiled frame)
> - org.eclipse.persistence.internal.helper.WriteLockManager.traverseRelatedLocks(java.lang.Object, java.util.Map, java.util.Map, org.eclipse.persistence.descriptors.ClassDescriptor, org.eclipse.persistence.internal.sessions.AbstractSession) @bci=237, line=213 (Compiled frame)
> - org.eclipse.persistence.internal.helper.WriteLockManager.acquireLockAndRelatedLocks(java.lang.Object, java.util.Map, java.util.Map, org.eclipse.persistence.internal.identitymaps.CacheKey, org.eclipse.persistence.descriptors.ClassDescriptor, org.eclipse.persistence.internal.sessions.AbstractSession) @bci=96, line=144 (Compiled frame)
> - org.eclipse.persistence.internal.helper.WriteLockManager.acquireLocksForClone(java.lang.Object, org.eclipse.persistence.descriptors.ClassDescriptor, org.eclipse.persistence.internal.identitymaps.CacheKey, org.eclipse.persistence.internal.sessions.AbstractSession) @bci=31, line=75 (Compiled frame)
> - org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(java.lang.Object, org.eclipse.persistence.internal.identitymaps.CacheKey, org.eclipse.persistence.internal.identitymaps.CacheKey, org.eclipse.persistence.descriptors.ClassDescriptor) @bci=140, line=997 (Compiled frame)
> - org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(java.lang.Object, org.eclipse.persistence.internal.identitymaps.CacheKey, org.eclipse.persistence.descriptors.ClassDescriptor) @bci=59, line=955 (Compiled frame)
> - org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getAndCloneCacheKeyFromParent(java.lang.Object, java.lang.Object, java.lang.Class, boolean, org.eclipse.persistence.descriptors.ClassDescriptor) @bci=288, line=209 (Compiled frame)
> - org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getFromIdentityMap(java.lang.Object, java.lang.Object, java.lang.Class, boolean, org.eclipse.persistence.descriptors.ClassDescriptor) @bci=28, line=137 (Compiled frame)
> - org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(java.lang.Object, org.eclipse.persistence.descriptors.ClassDescriptor, java.lang.Object, boolean) @bci=139, line=3940 (Compiled frame)
> - org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(java.lang.Object, boolean) @bci=62, line=3894 (Compiled frame)
> - org.eclipse.persistence.mappings.CollectionMapping.buildElementUnitOfWorkClone(java.lang.Object, java.lang.Object, java.lang.Integer, org.eclipse.persistence.internal.sessions.UnitOfWorkImpl, boolean, boolean) @bci=90, line=308 (Compiled frame)
> - org.eclipse.persistence.mappings.CollectionMapping.buildElementClone(java.lang.Object, java.lang.Object, org.eclipse.persistence.internal.identitymaps.CacheKey, java.lang.Integer, org.eclipse.persistence.internal.sessions.AbstractSession, boolean, boolean) @bci=22, line=321 (Compiled frame)
> - org.eclipse.persistence.internal.queries.ContainerPolicy.addNextValueFromIteratorInto(java.lang.Object, java.lang.Object, org.eclipse.persistence.internal.identitymaps.CacheKey, java.lang.Object, org.eclipse.persistence.mappings.CollectionMapping, java.lang.Integer, org.eclipse.persistence.internal.sessions.AbstractSession, boolean, boolean) @bci=19, line=215 (Compiled frame)
> - org.eclipse.persistence.mappings.CollectionMapping.buildCloneForPartObject(java.lang.Object, java.lang.Object, org.eclipse.persistence.internal.identitymaps.CacheKey, java.lang.Object, org.eclipse.persistence.internal.sessions.AbstractSession, java.lang.Integer, boolean, boolean) @bci=182, line=223 (Compiled frame)
> - org.eclipse.persistence.internal.indirection.UnitOfWorkQueryValueHolder.buildCloneFor(java.lang.Object) @bci=41, line=60 (Compiled frame)
> - org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiateImpl() @bci=154, line=173 (Compiled frame)
> - org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiate() @bci=33, line=234 (Compiled frame)
> - org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue() @bci=20, line=89 (Compiled frame)
> - org.eclipse.persistence.indirection.IndirectList.buildDelegate() @bci=4, line=271 (Compiled frame)
> - org.eclipse.persistence.indirection.IndirectList.getDelegate() @bci=20, line=455 (Compiled frame)
> - org.eclipse.persistence.internal.indirection.jdk8.IndirectList.access$000(org.eclipse.persistence.internal.indirection.jdk8.IndirectList) @bci=1, line=32 (Compiled frame)
> - org.eclipse.persistence.internal.indirection.jdk8.IndirectList$1.<init>(org.eclipse.persistence.internal.indirection.jdk8.IndirectList, int) @bci=19, line=57 (Compiled frame)
> - org.eclipse.persistence.internal.indirection.jdk8.IndirectList.listIterator(int) @bci=6, line=56 (Compiled frame)
> - org.eclipse.persistence.indirection.IndirectList.iterator() @bci=2, line=555 (Compiled frame)
> - org.apache.ambari.server.orm.entities.KerberosKeytabPrincipalEntity.getServiceMappingAsMultimap() @bci=8, line=213 (Compiled frame)
> - org.apache.ambari.server.serveraction.kerberos.stageutils.KerberosKeytabController.fromPrincipalEntities(java.util.Collection) @bci=79, line=221 (Compiled frame)
> - org.apache.ambari.server.serveraction.kerberos.stageutils.KerberosKeytabController.fromKeytabEntity(org.apache.ambari.server.orm.entities.KerberosKeytabEntity, boolean) @bci=9, line=184 (Compiled frame)
> - org.apache.ambari.server.serveraction.kerberos.stageutils.KerberosKeytabController.fromKeytabEntity(org.apache.ambari.server.orm.entities.KerberosKeytabEntity) @bci=3, line=198 (Compiled frame)
> - org.apache.ambari.server.serveraction.kerberos.stageutils.KerberosKeytabController.fromKeytabEntities(java.util.Collection) @bci=35, line=204 (Compiled frame)
> - org.apache.ambari.server.serveraction.kerberos.stageutils.KerberosKeytabController.getFromPrincipal(org.apache.ambari.server.serveraction.kerberos.stageutils.ResolvedKerberosPrincipal) @bci=16, line=94 (Compiled frame)
> - org.apache.ambari.server.serveraction.kerberos.CreateKeytabFilesServerAction.processIdentity(org.apache.ambari.server.serveraction.kerberos.stageutils.ResolvedKerberosPrincipal, org.apache.ambari.server.serveraction.kerberos.KerberosOperationHandler, java.util.Map, boolean, java.util.Map) @bci=86, line=161 (Compiled frame)
> - org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processIdentities(java.util.Map, boolean) @bci=427, line=478 (Interpreted frame)
> - org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processIdentities(java.util.Map) @bci=3, line=427 (Interpreted frame)
> - org.apache.ambari.server.serveraction.kerberos.CreateKeytabFilesServerAction.execute(java.util.concurrent.ConcurrentMap) @bci=2, line=105 (Interpreted frame)
> - org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.execute(org.apache.ambari.server.actionmanager.HostRoleCommand, org.apache.ambari.server.agent.ExecutionCommand) @bci=424, line=550 (Interpreted frame)
> - org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.run() @bci=48, line=466 (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=745 (Compiled frame)}}
>
> In our clusters, which are composed of 4000+ physical machines, adding new 5 hosts (with 7 components, 11 keytabs to be created each) spend about 2.5 hours.
> 2 hours out of 2.5 hours are spended on the above callstack.
> After this patch is applied, adding new 5 hosts with the same requirements spend only half hour as expected.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ambari.apache.org
For additional commands, e-mail: issues-help@ambari.apache.org