You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by Jonathan Leech <jo...@gmail.com> on 2016/02/15 19:35:19 UTC
java core dump
Has anyone else seen this? Happening under load in jdk 1.7.0_80 / phoenix
4.5.2 - cloudera labs. Based on the source code, It seems the JVM is
calling the wrong toObject(), and then dumping. The correct toObject()
method is a couple parent classes away with some generics and Sun / Oracle
must have gotten something wrong with that special case under load.
Thinking I may be able to just patch in a call to the correct method to
stop the bleeding. Anyone else have other ideas, thoughts, or also want the
patch?
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
code)
J 7432 C2
org.apache.phoenix.schema.types.PDecimal.toObject([BIILorg/apache/phoenix/schema/types/PDataType;Lorg/apache/phoenix/schema/SortOrder;Ljava/lang/Integer;Ljava/lang/Integer;)Ljava/lang/Object;
(376 bytes) @ 0x00007f8332f3de52 [0x00007f8332f3d9c0+0x492]
J 14641 C2
org.apache.phoenix.expression.DecimalAddExpression.evaluate(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Z
(178 bytes) @ 0x00007f83349ecde8 [0x00007f83349eca00+0x3e8]
J 11096 C2
org.apache.phoenix.expression.CoerceExpression.evaluate(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Z
(68 bytes) @ 0x00007f8333ed824c [0x00007f8333ed81c0+0x8c]
J 9742 C2
org.apache.phoenix.compile.ExpressionProjector.getValue(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/phoenix/schema/types/PDataType;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Ljava/lang/Object;
(88 bytes) @ 0x00007f83329f57b8 [0x00007f83329f5760+0x58]
J 8783 C2
org.apache.phoenix.compile.UpsertCompiler.upsertSelect(Lorg/apache/phoenix/compile/StatementContext;Lorg/apache/phoenix/schema/TableRef;Lorg/apache/phoenix/compile/RowProjector;Lorg/apache/phoenix/iterate/ResultIterator;[I[I)Lorg/apache/phoenix/execute/MutationState;
(633 bytes) @ 0x00007f833367c448 [0x00007f833367ba00+0xa48]
J 10636 C2
org.apache.phoenix.compile.MutatingParallelIteratorFactory.newIterator(Lorg/apache/phoenix/compile/StatementContext;Lorg/apache/phoenix/iterate/ResultIterator;Lorg/apache/hadoop/hbase/client/Scan;Ljava/lang/String;)Lorg/apache/phoenix/iterate/PeekingResultIterator;
(168 bytes) @ 0x00007f8333d63924 [0x00007f8333d63180+0x7a4]
J 8646 C2
org.apache.phoenix.iterate.ParallelIterators$1.call()Ljava/lang/Object; (5
bytes) @ 0x00007f833358010c [0x00007f833357fe40+0x2cc]
J 9466 C2
org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run()V (12
bytes) @ 0x00007f83338e7240 [0x00007f83338e7180+0xc0]
J 7047 C2
java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
(225 bytes) @ 0x00007f8332e4971c [0x00007f8332e49580+0x19c]
J 12402 C2 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (58 bytes)
@ 0x00007f833420ad2c [0x00007f833420ac80+0xac]
j java.lang.Thread.run()V+26
v ~StubRoutines::call_stub
V [libjvm.so+0x6020a5] JavaCalls::call_helper(JavaValue*, methodHandle*,
JavaCallArguments*, Thread*)+0x365
V [libjvm.so+0x600b08] JavaCalls::call(JavaValue*, methodHandle,
JavaCallArguments*, Thread*)+0x28
V [libjvm.so+0x600dd7] JavaCalls::call_virtual(JavaValue*, KlassHandle,
Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x197
V [libjvm.so+0x600ef7] JavaCalls::call_virtual(JavaValue*, Handle,
KlassHandle, Symbol*, Symbol*, Thread*)+0x47
V [libjvm.so+0x67f8d5] thread_entry(JavaThread*, Thread*)+0xe5
V [libjvm.so+0x95bfff] JavaThread::thread_main_inner()+0xdf
V [libjvm.so+0x95c105] JavaThread::run()+0xf5
V [libjvm.so+0x8238c8] java_start(Thread*)+0x108
Re: java core dump
Posted by Jonathan Leech <jo...@gmail.com>.
As I was seeing sporadic errors similar to that of PHOENIX-2169 in addition to the core dumps, I decided to refactor the upsert select into separate upsert and select statements. So far this has fixed both the errors and the core dumps. I believe there to be a concurrency issue in upsert select between two salted tables, similar to PHOENIX-2169.
- Jonathan
> On Feb 16, 2016, at 10:02 AM, Jonathan Leech <jo...@gmail.com> wrote:
>
> Yeah my fix didn't fix anything; was barking up the wrong tree. The toObject() was the right one I think hotspot just optimized out the intermediate calls. Going to try upgrading to 1.8 before downgrading to u79, will also look at the Phoenix source code with respect to concurrency issues.
>
>
>
>> On Feb 15, 2016, at 12:25 PM, Andrew Purtell <an...@gmail.com> wrote:
>>
>> You might also consider moving back down to 7u79
>>
>>> On Feb 15, 2016, at 10:35 AM, Jonathan Leech <jo...@gmail.com> wrote:
>>>
>>> Has anyone else seen this? Happening under load in jdk 1.7.0_80 / phoenix 4.5.2 - cloudera labs. Based on the source code, It seems the JVM is calling the wrong toObject(), and then dumping. The correct toObject() method is a couple parent classes away with some generics and Sun / Oracle must have gotten something wrong with that special case under load. Thinking I may be able to just patch in a call to the correct method to stop the bleeding. Anyone else have other ideas, thoughts, or also want the patch?
>>>
>>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
>>>
>>> J 7432 C2 org.apache.phoenix.schema.types.PDecimal.toObject([BIILorg/apache/phoenix/schema/types/PDataType;Lorg/apache/phoenix/schema/SortOrder;Ljava/lang/Integer;Ljava/lang/Integer;)Ljava/lang/Object; (376 bytes) @ 0x00007f8332f3de52 [0x00007f8332f3d9c0+0x492]
>>>
>>> J 14641 C2 org.apache.phoenix.expression.DecimalAddExpression.evaluate(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Z (178 bytes) @ 0x00007f83349ecde8 [0x00007f83349eca00+0x3e8]
>>>
>>> J 11096 C2 org.apache.phoenix.expression.CoerceExpression.evaluate(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Z (68 bytes) @ 0x00007f8333ed824c [0x00007f8333ed81c0+0x8c]
>>>
>>> J 9742 C2 org.apache.phoenix.compile.ExpressionProjector.getValue(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/phoenix/schema/types/PDataType;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Ljava/lang/Object; (88 bytes) @ 0x00007f83329f57b8 [0x00007f83329f5760+0x58]
>>>
>>> J 8783 C2 org.apache.phoenix.compile.UpsertCompiler.upsertSelect(Lorg/apache/phoenix/compile/StatementContext;Lorg/apache/phoenix/schema/TableRef;Lorg/apache/phoenix/compile/RowProjector;Lorg/apache/phoenix/iterate/ResultIterator;[I[I)Lorg/apache/phoenix/execute/MutationState; (633 bytes) @ 0x00007f833367c448 [0x00007f833367ba00+0xa48]
>>>
>>> J 10636 C2 org.apache.phoenix.compile.MutatingParallelIteratorFactory.newIterator(Lorg/apache/phoenix/compile/StatementContext;Lorg/apache/phoenix/iterate/ResultIterator;Lorg/apache/hadoop/hbase/client/Scan;Ljava/lang/String;)Lorg/apache/phoenix/iterate/PeekingResultIterator; (168 bytes) @ 0x00007f8333d63924 [0x00007f8333d63180+0x7a4]
>>>
>>> J 8646 C2 org.apache.phoenix.iterate.ParallelIterators$1.call()Ljava/lang/Object; (5 bytes) @ 0x00007f833358010c [0x00007f833357fe40+0x2cc]
>>>
>>> J 9466 C2 org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run()V (12 bytes) @ 0x00007f83338e7240 [0x00007f83338e7180+0xc0]
>>>
>>> J 7047 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007f8332e4971c [0x00007f8332e49580+0x19c]
>>>
>>> J 12402 C2 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (58 bytes) @ 0x00007f833420ad2c [0x00007f833420ac80+0xac]
>>>
>>> j java.lang.Thread.run()V+26
>>>
>>> v ~StubRoutines::call_stub
>>>
>>> V [libjvm.so+0x6020a5] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x365
>>>
>>> V [libjvm.so+0x600b08] JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*)+0x28
>>>
>>> V [libjvm.so+0x600dd7] JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x197
>>>
>>> V [libjvm.so+0x600ef7] JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
>>>
>>> V [libjvm.so+0x67f8d5] thread_entry(JavaThread*, Thread*)+0xe5
>>>
>>> V [libjvm.so+0x95bfff] JavaThread::thread_main_inner()+0xdf
>>>
>>> V [libjvm.so+0x95c105] JavaThread::run()+0xf5
>>>
>>> V [libjvm.so+0x8238c8] java_start(Thread*)+0x108
Re: java core dump
Posted by Jonathan Leech <jo...@gmail.com>.
Yeah my fix didn't fix anything; was barking up the wrong tree. The toObject() was the right one I think hotspot just optimized out the intermediate calls. Going to try upgrading to 1.8 before downgrading to u79, will also look at the Phoenix source code with respect to concurrency issues.
> On Feb 15, 2016, at 12:25 PM, Andrew Purtell <an...@gmail.com> wrote:
>
> You might also consider moving back down to 7u79
>
>> On Feb 15, 2016, at 10:35 AM, Jonathan Leech <jo...@gmail.com> wrote:
>>
>> Has anyone else seen this? Happening under load in jdk 1.7.0_80 / phoenix 4.5.2 - cloudera labs. Based on the source code, It seems the JVM is calling the wrong toObject(), and then dumping. The correct toObject() method is a couple parent classes away with some generics and Sun / Oracle must have gotten something wrong with that special case under load. Thinking I may be able to just patch in a call to the correct method to stop the bleeding. Anyone else have other ideas, thoughts, or also want the patch?
>>
>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
>>
>> J 7432 C2 org.apache.phoenix.schema.types.PDecimal.toObject([BIILorg/apache/phoenix/schema/types/PDataType;Lorg/apache/phoenix/schema/SortOrder;Ljava/lang/Integer;Ljava/lang/Integer;)Ljava/lang/Object; (376 bytes) @ 0x00007f8332f3de52 [0x00007f8332f3d9c0+0x492]
>>
>> J 14641 C2 org.apache.phoenix.expression.DecimalAddExpression.evaluate(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Z (178 bytes) @ 0x00007f83349ecde8 [0x00007f83349eca00+0x3e8]
>>
>> J 11096 C2 org.apache.phoenix.expression.CoerceExpression.evaluate(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Z (68 bytes) @ 0x00007f8333ed824c [0x00007f8333ed81c0+0x8c]
>>
>> J 9742 C2 org.apache.phoenix.compile.ExpressionProjector.getValue(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/phoenix/schema/types/PDataType;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Ljava/lang/Object; (88 bytes) @ 0x00007f83329f57b8 [0x00007f83329f5760+0x58]
>>
>> J 8783 C2 org.apache.phoenix.compile.UpsertCompiler.upsertSelect(Lorg/apache/phoenix/compile/StatementContext;Lorg/apache/phoenix/schema/TableRef;Lorg/apache/phoenix/compile/RowProjector;Lorg/apache/phoenix/iterate/ResultIterator;[I[I)Lorg/apache/phoenix/execute/MutationState; (633 bytes) @ 0x00007f833367c448 [0x00007f833367ba00+0xa48]
>>
>> J 10636 C2 org.apache.phoenix.compile.MutatingParallelIteratorFactory.newIterator(Lorg/apache/phoenix/compile/StatementContext;Lorg/apache/phoenix/iterate/ResultIterator;Lorg/apache/hadoop/hbase/client/Scan;Ljava/lang/String;)Lorg/apache/phoenix/iterate/PeekingResultIterator; (168 bytes) @ 0x00007f8333d63924 [0x00007f8333d63180+0x7a4]
>>
>> J 8646 C2 org.apache.phoenix.iterate.ParallelIterators$1.call()Ljava/lang/Object; (5 bytes) @ 0x00007f833358010c [0x00007f833357fe40+0x2cc]
>>
>> J 9466 C2 org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run()V (12 bytes) @ 0x00007f83338e7240 [0x00007f83338e7180+0xc0]
>>
>> J 7047 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007f8332e4971c [0x00007f8332e49580+0x19c]
>>
>> J 12402 C2 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (58 bytes) @ 0x00007f833420ad2c [0x00007f833420ac80+0xac]
>>
>> j java.lang.Thread.run()V+26
>>
>> v ~StubRoutines::call_stub
>>
>> V [libjvm.so+0x6020a5] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x365
>>
>> V [libjvm.so+0x600b08] JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*)+0x28
>>
>> V [libjvm.so+0x600dd7] JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x197
>>
>> V [libjvm.so+0x600ef7] JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
>>
>> V [libjvm.so+0x67f8d5] thread_entry(JavaThread*, Thread*)+0xe5
>>
>> V [libjvm.so+0x95bfff] JavaThread::thread_main_inner()+0xdf
>>
>> V [libjvm.so+0x95c105] JavaThread::run()+0xf5
>>
>> V [libjvm.so+0x8238c8] java_start(Thread*)+0x108
Re: java core dump
Posted by Andrew Purtell <an...@gmail.com>.
You might also consider moving back down to 7u79
> On Feb 15, 2016, at 10:35 AM, Jonathan Leech <jo...@gmail.com> wrote:
>
> Has anyone else seen this? Happening under load in jdk 1.7.0_80 / phoenix 4.5.2 - cloudera labs. Based on the source code, It seems the JVM is calling the wrong toObject(), and then dumping. The correct toObject() method is a couple parent classes away with some generics and Sun / Oracle must have gotten something wrong with that special case under load. Thinking I may be able to just patch in a call to the correct method to stop the bleeding. Anyone else have other ideas, thoughts, or also want the patch?
>
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
>
> J 7432 C2 org.apache.phoenix.schema.types.PDecimal.toObject([BIILorg/apache/phoenix/schema/types/PDataType;Lorg/apache/phoenix/schema/SortOrder;Ljava/lang/Integer;Ljava/lang/Integer;)Ljava/lang/Object; (376 bytes) @ 0x00007f8332f3de52 [0x00007f8332f3d9c0+0x492]
>
> J 14641 C2 org.apache.phoenix.expression.DecimalAddExpression.evaluate(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Z (178 bytes) @ 0x00007f83349ecde8 [0x00007f83349eca00+0x3e8]
>
> J 11096 C2 org.apache.phoenix.expression.CoerceExpression.evaluate(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Z (68 bytes) @ 0x00007f8333ed824c [0x00007f8333ed81c0+0x8c]
>
> J 9742 C2 org.apache.phoenix.compile.ExpressionProjector.getValue(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/phoenix/schema/types/PDataType;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Ljava/lang/Object; (88 bytes) @ 0x00007f83329f57b8 [0x00007f83329f5760+0x58]
>
> J 8783 C2 org.apache.phoenix.compile.UpsertCompiler.upsertSelect(Lorg/apache/phoenix/compile/StatementContext;Lorg/apache/phoenix/schema/TableRef;Lorg/apache/phoenix/compile/RowProjector;Lorg/apache/phoenix/iterate/ResultIterator;[I[I)Lorg/apache/phoenix/execute/MutationState; (633 bytes) @ 0x00007f833367c448 [0x00007f833367ba00+0xa48]
>
> J 10636 C2 org.apache.phoenix.compile.MutatingParallelIteratorFactory.newIterator(Lorg/apache/phoenix/compile/StatementContext;Lorg/apache/phoenix/iterate/ResultIterator;Lorg/apache/hadoop/hbase/client/Scan;Ljava/lang/String;)Lorg/apache/phoenix/iterate/PeekingResultIterator; (168 bytes) @ 0x00007f8333d63924 [0x00007f8333d63180+0x7a4]
>
> J 8646 C2 org.apache.phoenix.iterate.ParallelIterators$1.call()Ljava/lang/Object; (5 bytes) @ 0x00007f833358010c [0x00007f833357fe40+0x2cc]
>
> J 9466 C2 org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run()V (12 bytes) @ 0x00007f83338e7240 [0x00007f83338e7180+0xc0]
>
> J 7047 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007f8332e4971c [0x00007f8332e49580+0x19c]
>
> J 12402 C2 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (58 bytes) @ 0x00007f833420ad2c [0x00007f833420ac80+0xac]
>
> j java.lang.Thread.run()V+26
>
> v ~StubRoutines::call_stub
>
> V [libjvm.so+0x6020a5] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x365
>
> V [libjvm.so+0x600b08] JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*)+0x28
>
> V [libjvm.so+0x600dd7] JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x197
>
> V [libjvm.so+0x600ef7] JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
>
> V [libjvm.so+0x67f8d5] thread_entry(JavaThread*, Thread*)+0xe5
>
> V [libjvm.so+0x95bfff] JavaThread::thread_main_inner()+0xdf
>
> V [libjvm.so+0x95c105] JavaThread::run()+0xf5
>
> V [libjvm.so+0x8238c8] java_start(Thread*)+0x108
Re: java core dump
Posted by James Taylor <ja...@apache.org>.
Thanks, Jonathan. I haven't seen this, but a patch would be much
appreciated.
James
On Monday, February 15, 2016, Jonathan Leech <jo...@gmail.com> wrote:
> Has anyone else seen this? Happening under load in jdk 1.7.0_80 / phoenix
> 4.5.2 - cloudera labs. Based on the source code, It seems the JVM is
> calling the wrong toObject(), and then dumping. The correct toObject()
> method is a couple parent classes away with some generics and Sun / Oracle
> must have gotten something wrong with that special case under load.
> Thinking I may be able to just patch in a call to the correct method to
> stop the bleeding. Anyone else have other ideas, thoughts, or also want the
> patch?
>
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
>
> J 7432 C2
> org.apache.phoenix.schema.types.PDecimal.toObject([BIILorg/apache/phoenix/schema/types/PDataType;Lorg/apache/phoenix/schema/SortOrder;Ljava/lang/Integer;Ljava/lang/Integer;)Ljava/lang/Object;
> (376 bytes) @ 0x00007f8332f3de52 [0x00007f8332f3d9c0+0x492]
>
> J 14641 C2
> org.apache.phoenix.expression.DecimalAddExpression.evaluate(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Z
> (178 bytes) @ 0x00007f83349ecde8 [0x00007f83349eca00+0x3e8]
>
> J 11096 C2
> org.apache.phoenix.expression.CoerceExpression.evaluate(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Z
> (68 bytes) @ 0x00007f8333ed824c [0x00007f8333ed81c0+0x8c]
>
> J 9742 C2
> org.apache.phoenix.compile.ExpressionProjector.getValue(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/phoenix/schema/types/PDataType;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Ljava/lang/Object;
> (88 bytes) @ 0x00007f83329f57b8 [0x00007f83329f5760+0x58]
>
> J 8783 C2
> org.apache.phoenix.compile.UpsertCompiler.upsertSelect(Lorg/apache/phoenix/compile/StatementContext;Lorg/apache/phoenix/schema/TableRef;Lorg/apache/phoenix/compile/RowProjector;Lorg/apache/phoenix/iterate/ResultIterator;[I[I)Lorg/apache/phoenix/execute/MutationState;
> (633 bytes) @ 0x00007f833367c448 [0x00007f833367ba00+0xa48]
>
> J 10636 C2
> org.apache.phoenix.compile.MutatingParallelIteratorFactory.newIterator(Lorg/apache/phoenix/compile/StatementContext;Lorg/apache/phoenix/iterate/ResultIterator;Lorg/apache/hadoop/hbase/client/Scan;Ljava/lang/String;)Lorg/apache/phoenix/iterate/PeekingResultIterator;
> (168 bytes) @ 0x00007f8333d63924 [0x00007f8333d63180+0x7a4]
>
> J 8646 C2
> org.apache.phoenix.iterate.ParallelIterators$1.call()Ljava/lang/Object; (5
> bytes) @ 0x00007f833358010c [0x00007f833357fe40+0x2cc]
>
> J 9466 C2
> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run()V (12
> bytes) @ 0x00007f83338e7240 [0x00007f83338e7180+0xc0]
>
> J 7047 C2
> java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
> (225 bytes) @ 0x00007f8332e4971c [0x00007f8332e49580+0x19c]
>
> J 12402 C2 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (58
> bytes) @ 0x00007f833420ad2c [0x00007f833420ac80+0xac]
>
> j java.lang.Thread.run()V+26
>
> v ~StubRoutines::call_stub
>
> V [libjvm.so+0x6020a5] JavaCalls::call_helper(JavaValue*, methodHandle*,
> JavaCallArguments*, Thread*)+0x365
>
> V [libjvm.so+0x600b08] JavaCalls::call(JavaValue*, methodHandle,
> JavaCallArguments*, Thread*)+0x28
>
> V [libjvm.so+0x600dd7] JavaCalls::call_virtual(JavaValue*, KlassHandle,
> Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x197
>
> V [libjvm.so+0x600ef7] JavaCalls::call_virtual(JavaValue*, Handle,
> KlassHandle, Symbol*, Symbol*, Thread*)+0x47
>
> V [libjvm.so+0x67f8d5] thread_entry(JavaThread*, Thread*)+0xe5
>
> V [libjvm.so+0x95bfff] JavaThread::thread_main_inner()+0xdf
>
> V [libjvm.so+0x95c105] JavaThread::run()+0xf5
>
> V [libjvm.so+0x8238c8] java_start(Thread*)+0x108
>