Re: java core dump

[Jonathan Leech <jo...@gmail.com>]

As I was seeing sporadic errors similar to that of PHOENIX-2169 in addition to the core dumps, I decided to refactor the upsert select into separate upsert and select statements. So far this has fixed both the errors and the core dumps. I believe there to be a concurrency issue in upsert select between two salted tables, similar to PHOENIX-2169.

- Jonathan



> On Feb 16, 2016, at 10:02 AM, Jonathan Leech <jo...@gmail.com> wrote:
> 
> Yeah my fix didn't fix anything; was barking up the wrong tree. The toObject() was the right one I think hotspot just optimized out the intermediate calls. Going to try upgrading to 1.8 before downgrading to u79, will also look at the Phoenix source code with respect to concurrency issues. 
> 
> 
> 
>> On Feb 15, 2016, at 12:25 PM, Andrew Purtell <an...@gmail.com> wrote:
>> 
>> You might also consider moving back down to 7u79 
>> 
>>> On Feb 15, 2016, at 10:35 AM, Jonathan Leech <jo...@gmail.com> wrote:
>>> 
>>> Has anyone else seen this? Happening under load in jdk 1.7.0_80 / phoenix 4.5.2 - cloudera labs. Based on the source code, It seems the JVM is calling the wrong toObject(), and then dumping. The correct toObject() method is a couple parent classes away with some generics and Sun / Oracle must have gotten something wrong with that special case under load. Thinking I may be able to just patch in a call to the correct method to stop the bleeding. Anyone else have other ideas, thoughts, or also want the patch?
>>> 
>>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
>>> 
>>> J 7432 C2 org.apache.phoenix.schema.types.PDecimal.toObject([BIILorg/apache/phoenix/schema/types/PDataType;Lorg/apache/phoenix/schema/SortOrder;Ljava/lang/Integer;Ljava/lang/Integer;)Ljava/lang/Object; (376 bytes) @ 0x00007f8332f3de52 [0x00007f8332f3d9c0+0x492]
>>> 
>>> J 14641 C2 org.apache.phoenix.expression.DecimalAddExpression.evaluate(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Z (178 bytes) @ 0x00007f83349ecde8 [0x00007f83349eca00+0x3e8]
>>> 
>>> J 11096 C2 org.apache.phoenix.expression.CoerceExpression.evaluate(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Z (68 bytes) @ 0x00007f8333ed824c [0x00007f8333ed81c0+0x8c]
>>> 
>>> J 9742 C2 org.apache.phoenix.compile.ExpressionProjector.getValue(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/phoenix/schema/types/PDataType;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Ljava/lang/Object; (88 bytes) @ 0x00007f83329f57b8 [0x00007f83329f5760+0x58]
>>> 
>>> J 8783 C2 org.apache.phoenix.compile.UpsertCompiler.upsertSelect(Lorg/apache/phoenix/compile/StatementContext;Lorg/apache/phoenix/schema/TableRef;Lorg/apache/phoenix/compile/RowProjector;Lorg/apache/phoenix/iterate/ResultIterator;[I[I)Lorg/apache/phoenix/execute/MutationState; (633 bytes) @ 0x00007f833367c448 [0x00007f833367ba00+0xa48]
>>> 
>>> J 10636 C2 org.apache.phoenix.compile.MutatingParallelIteratorFactory.newIterator(Lorg/apache/phoenix/compile/StatementContext;Lorg/apache/phoenix/iterate/ResultIterator;Lorg/apache/hadoop/hbase/client/Scan;Ljava/lang/String;)Lorg/apache/phoenix/iterate/PeekingResultIterator; (168 bytes) @ 0x00007f8333d63924 [0x00007f8333d63180+0x7a4]
>>> 
>>> J 8646 C2 org.apache.phoenix.iterate.ParallelIterators$1.call()Ljava/lang/Object; (5 bytes) @ 0x00007f833358010c [0x00007f833357fe40+0x2cc]
>>> 
>>> J 9466 C2 org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run()V (12 bytes) @ 0x00007f83338e7240 [0x00007f83338e7180+0xc0]
>>> 
>>> J 7047 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007f8332e4971c [0x00007f8332e49580+0x19c]
>>> 
>>> J 12402 C2 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (58 bytes) @ 0x00007f833420ad2c [0x00007f833420ac80+0xac]
>>> 
>>> j  java.lang.Thread.run()V+26
>>> 
>>> v  ~StubRoutines::call_stub
>>> 
>>> V  [libjvm.so+0x6020a5]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x365
>>> 
>>> V  [libjvm.so+0x600b08]  JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*)+0x28
>>> 
>>> V  [libjvm.so+0x600dd7]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x197
>>> 
>>> V  [libjvm.so+0x600ef7]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
>>> 
>>> V  [libjvm.so+0x67f8d5]  thread_entry(JavaThread*, Thread*)+0xe5
>>> 
>>> V  [libjvm.so+0x95bfff]  JavaThread::thread_main_inner()+0xdf
>>> 
>>> V  [libjvm.so+0x95c105]  JavaThread::run()+0xf5
>>> 
>>> V  [libjvm.so+0x8238c8]  java_start(Thread*)+0x108