[jira] [Created] (IGNITE-4355) Eliminate map threads pauses during startup

["Ivan Veselovsky (JIRA)" <ji...@apache.org>]

Ivan Veselovsky created IGNITE-4355:
---------------------------------------

             Summary: Eliminate map threads pauses during startup
                 Key: IGNITE-4355
                 URL: https://issues.apache.org/jira/browse/IGNITE-4355
             Project: Ignite
          Issue Type: Sub-task
            Reporter: Ivan Veselovsky
            Assignee: Ivan Veselovsky


Pauses in all Map threads but one are observed in the beginning . This is caused by waiting on future.get()  in HadoopV2Job.getTaskContext(HadoopTaskInfo) .

{code}
 at sun.misc.Unsafe.park(boolean, long)
 at java.util.concurrent.locks.LockSupport.park(Object)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int)
 at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(boolean)
 at org.apache.ignite.internal.util.future.GridFutureAdapter.get()
 at org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2Job.getTaskContext(HadoopTaskInfo)
 at org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffleJob.<init>(Object, IgniteLogger, HadoopJob, GridUnsafeMemory, int, int[], int)
 at org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.newJob(HadoopJobId)
 at org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.job(HadoopJobId)
 at org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.output(HadoopTaskContext)
 at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopEmbeddedTaskExecutor$1.createOutput(HadoopTaskContext)
 at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.createOutputInternal(HadoopTaskContext)
 at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopPerformanceCounter)
 at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call0()
 at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call()
 at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call()
 at org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2TaskContext.runAsJobOwner(Callable)
 at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call()
 at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call()
 at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body()
 at org.apache.ignite.internal.util.worker.GridWorker.run()
 at java.lang.Thread.run()
{code}

while the working thread initializes the context:
{code}
Java Monitor Wait
 at java.lang.Object.wait(long)
 at java.lang.Thread.join(long)
 at java.lang.Thread.join()
 at org.apache.hadoop.util.Shell.joinThread(Thread)
 at org.apache.hadoop.util.Shell.runCommand()
 at org.apache.hadoop.util.Shell.run()
 at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute()
 at org.apache.hadoop.util.Shell.isSetsidSupported()
 at org.apache.hadoop.util.Shell.<clinit>()
 at org.apache.hadoop.util.StringUtils.<clinit>()
 at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(Configuration)
 at org.apache.hadoop.security.UserGroupInformation.initialize(Configuration, boolean)
 at org.apache.hadoop.security.UserGroupInformation.ensureInitialized()
 at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(Subject)
 at org.apache.hadoop.security.UserGroupInformation.getLoginUser()
 at org.apache.hadoop.security.UserGroupInformation.getCurrentUser()
 at org.apache.hadoop.mapreduce.task.JobContextImpl.<init>(Configuration, JobID)
 at org.apache.hadoop.mapred.JobContextImpl.<init>(JobConf, JobID, Progressable)
 at org.apache.hadoop.mapred.JobContextImpl.<init>(JobConf, JobID)
 at org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2TaskContext.<init>(HadoopTaskInfo, HadoopJob, HadoopJobId, UUID, DataInput)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Constructor, Object[])
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(Object[])
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Object[])
 at java.lang.reflect.Constructor.newInstance(Object[])
 at org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2Job.getTaskContext(HadoopTaskInfo)
 at org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffleJob.<init>(Object, IgniteLogger, HadoopJob, GridUnsafeMemory, int, int[], int)
 at org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.newJob(HadoopJobId)
 at org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.job(HadoopJobId)
 at org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.output(HadoopTaskContext)
 at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopEmbeddedTaskExecutor$1.createOutput(HadoopTaskContext)
 at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.createOutputInternal(HadoopTaskContext)
 at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopPerformanceCounter)
 at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call0()
 at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call()
 at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call()
 at org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2TaskContext.runAsJobOwner(Callable)
 at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call()
 at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call()
 at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body()
 at org.apache.ignite.internal.util.worker.GridWorker.run()
 at java.lang.Thread.run()

{code}

Experimental solution can be found in branch "ignite-4270-sync-direct-gzip-avoid-setup-pauses". 2 changes are made there:
1) HadoopShuffle.job() avoided "speculative execution"
2) task contexts creation paralleled in HadoopShuffleJob.<init>().

JFR profiles show that there are no pauses any more, however, there is stil a lot of work done in each thread loading classes, parsing XML configs, and initializing Hadoop statics (in case of UserGroupInformation.getCurrentUser() this even involves a shell script execution). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)