You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@orc.apache.org by Ashika Umanga Umagiliya <um...@gmail.com> on 2018/09/04 03:16:09 UTC
Write data from JDBC to ORC : NPE when calling WriterImpl.addRowBatch
Greetings,
I am trying to read data from JDBC ResultSet and write to ORC file.
Following is the NPE exception I get . Below the stacktrace, I have posted
my code snippet:
Any tips on what I am doing wrong here ?
-------------------
StackTrack
-------------------
... 33 more
Caused by: java.lang.NullPointerException
at java.lang.System.arraycopy(Native Method)
at org.apache.hadoop.io.Text.set(Text.java:225)
at
org.apache.orc.impl.StringRedBlackTree.add(StringRedBlackTree.java:59)
at
org.apache.orc.impl.writer.StringTreeWriter.writeBatch(StringTreeWriter.java:70)
at
org.apache.orc.impl.writer.StructTreeWriter.writeRootBatch(StructTreeWriter.java:56)
at org.apache.orc.impl.WriterImpl.addRowBatch(WriterImpl.java:546)
at com.xxxx.ingest.write(Hdfs.java:126) #line number
-----------------
Code
-----------------
public long write(ResultSet resultSet) throws SQLException, IOException {
try {
int numCols = resultSet.getMetaData().getColumnCount();
//define ORC
Configuration conf = new Configuration();
TypeDescription schema = TypeDescription.createStruct();
//assume all the columns are String type
for(int i=0;i<numCols;i++) {
String colName = resultSet.getMetaData().getColumnName(i+1);
schema.addField(colName,TypeDescription.createString());
}
Writer writer = OrcFile.createWriter(new Path(stgFilePath),
OrcFile.writerOptions(conf).setSchema(schema));
VectorizedRowBatch batch = schema.createRowBatch();
List<String> colVals = new ArrayList<>();
long rowsTransferred = 0;
//write all rows from JDBC
while (resultSet.next()) {
int row = batch.size++;
for (int index = 1; index <= numCols; index++) {
BytesColumnVector
col=(BytesColumnVector)batch.cols[index-1];
String val = resultSet.getString(index);
col.vector[row] = val.getBytes();
if (batch.size == batch.getMaxSize()) {
writer.addRowBatch(batch); ///**** NPE thrown here
*** ?
batch.reset();
}
}
rowsTransferred++;
}
if (batch.size != 0) {
writer.addRowBatch(batch);
batch.reset();
}
writer.close();
//end write
return rowsTransferred;
}
catch (SQLException | IOException e) {
throw e;
}
finally {
if (resultSet != null) resultSet.close();
}
}