You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@orc.apache.org by Telco Phone <te...@yahoo.com> on 2017/10/13 21:02:07 UTC
Error with batch write
What is the best way to troubleshoot this kind of error?
java.lang.NullPointerException at java.lang.System.arraycopy(Native Method) at org.apache.hadoop.io.Text.set(Text.java:225) at org.apache.orc.impl.StringRedBlackTree.add(StringRedBlackTree.java:62) at org.apache.orc.impl.WriterImpl$StringTreeWriter.writeBatch(WriterImpl.java:1459) at org.apache.orc.impl.WriterImpl$StructTreeWriter.writeFields(WriterImpl.java:2106) at org.apache.orc.impl.WriterImpl$StructTreeWriter.writeBatch(WriterImpl.java:2120) at org.apache.orc.impl.WriterImpl$ListTreeWriter.writeBatch(WriterImpl.java:2221) at org.apache.orc.impl.WriterImpl$StructTreeWriter.writeRootBatch(WriterImpl.java:2098) at org.apache.orc.impl.WriterImpl.addRowBatch(WriterImpl.java:2811) at com.sovrn.scratch.Scratch.main(Scratch.java:154)
Line 154 is writer.addRowBatch(batch);
I am not looking to have anyone fixe my current is but more on how do I setup the code base to quickly trouble shoot these kind of issue ?
Is it my array code ? String ?
The logic loop that I have the code in seems to work well but from time to time I will get the above.
Thoughts ?
Re: Error with batch write
Posted by Telco Phone <te...@yahoo.com>.
Owen,
Thanks that is great.
From that error do you have a trick I can do to tear apart this column which is a array of structs to see what I am missing for a null ?
java.lang.NullPointerException at java.lang.String.checkBounds(String.java:384) at java.lang.String.<init>(String.java:545) at org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.stringifyValue(BytesColumnVector.java:445) at org.apache.hadoop.hive.ql.exec.vector.StructColumnVector.stringifyValue(StructColumnVector.java:85) at org.apache.hadoop.hive.ql.exec.vector.ListColumnVector.stringifyValue(ListColumnVector.java:91) at com.sovrn.scratch.Scratch.main(Scratch.java:152)
What a pain to convert avro to orc .....
From: Owen O'Malley <ow...@gmail.com>
To: user@orc.apache.org; Telco Phone <te...@yahoo.com>
Sent: Friday, October 13, 2017 3:57 PM
Subject: Re: Error with batch write
So, the error is a null pointer in a BytesColumnVector.vector[r] down in one of the nested fields.
The relevant path through the schema looks like: struct -> list -> struct -> struct -> string.
You could do something like:
VectorizedRowBatch batch = ...for(int r = 0; r < batch.size; ++r) { System.out.println("Starting row " + r); for(int c=0; c < batch.cols.length; ++c) { System.out.println("Column " + c); // You'll get an exception on the broken field StringBuilder builder = new StringBuilder(); batch.cols[c].stringifyValue(builder, r); }}
.. Owen
On Fri, Oct 13, 2017 at 2:02 PM, Telco Phone <te...@yahoo.com> wrote:
What is the best way to troubleshoot this kind of error?
java.lang.NullPointerException at java.lang.System.arraycopy( Native Method) at org.apache.hadoop.io.Text.set( Text.java:225) at org.apache.orc.impl. StringRedBlackTree.add( StringRedBlackTree.java:62) at org.apache.orc.impl. WriterImpl$StringTreeWriter. writeBatch(WriterImpl.java: 1459) at org.apache.orc.impl. WriterImpl$StructTreeWriter. writeFields(WriterImpl.java: 2106) at org.apache.orc.impl. WriterImpl$StructTreeWriter. writeBatch(WriterImpl.java: 2120) at org.apache.orc.impl. WriterImpl$ListTreeWriter. writeBatch(WriterImpl.java: 2221) at org.apache.orc.impl. WriterImpl$StructTreeWriter. writeRootBatch(WriterImpl. java:2098) at org.apache.orc.impl. WriterImpl.addRowBatch( WriterImpl.java:2811) at com.sovrn.scratch.Scratch. main(Scratch.java:154)
Line 154 is writer.addRowBatch(batch);
I am not looking to have anyone fixe my current is but more on how do I setup the code base to quickly trouble shoot these kind of issue ?
Is it my array code ? String ?
The logic loop that I have the code in seems to work well but from time to time I will get the above.
Thoughts ?
Re: Error with batch write
Posted by Owen O'Malley <ow...@gmail.com>.
So, the error is a null pointer in a BytesColumnVector.vector[r] down in
one of the nested fields.
The relevant path through the schema looks like: struct -> list -> struct
-> struct -> string.
You could do something like:
VectorizedRowBatch batch = ...
for(int r = 0; r < batch.size; ++r) {
System.out.println("Starting row " + r);
for(int c=0; c < batch.cols.length; ++c) {
System.out.println("Column " + c);
// You'll get an exception on the broken field
StringBuilder builder = new StringBuilder();
batch.cols[c].stringifyValue(builder, r);
}
}
.. Owen
On Fri, Oct 13, 2017 at 2:02 PM, Telco Phone <te...@yahoo.com> wrote:
> What is the best way to troubleshoot this kind of error?
>
> java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at org.apache.hadoop.io.Text.set(Text.java:225)
> at org.apache.orc.impl.StringRedBlackTree.add(StringRedBlackTree.java:62)
> at org.apache.orc.impl.WriterImpl$StringTreeWriter.
> writeBatch(WriterImpl.java:1459)
> at org.apache.orc.impl.WriterImpl$StructTreeWriter.
> writeFields(WriterImpl.java:2106)
> at org.apache.orc.impl.WriterImpl$StructTreeWriter.
> writeBatch(WriterImpl.java:2120)
> at org.apache.orc.impl.WriterImpl$ListTreeWriter.
> writeBatch(WriterImpl.java:2221)
> at org.apache.orc.impl.WriterImpl$StructTreeWriter.
> writeRootBatch(WriterImpl.java:2098)
> at org.apache.orc.impl.WriterImpl.addRowBatch(WriterImpl.java:2811)
> at com.sovrn.scratch.Scratch.main(Scratch.java:154)
>
>
> Line 154 is writer.addRowBatch(batch);
>
> I am not looking to have anyone fixe my current is but more on how do I
> setup the code base to quickly trouble shoot these kind of issue ?
>
> Is it my array code ? String ?
>
> The logic loop that I have the code in seems to work well but from time to
> time I will get the above.
>
> Thoughts ?
>
>
>