You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@orc.apache.org by Ashika Umanga Umagiliya <um...@gmail.com> on 2018/09/04 03:16:09 UTC

Write data from JDBC to ORC : NPE when calling WriterImpl.addRowBatch

Greetings,

I am trying to read data from JDBC ResultSet and write to ORC file.
Following is the NPE exception I get . Below the stacktrace, I have posted
my code snippet:
Any tips on what I am doing wrong here ?

-------------------
StackTrack
-------------------
      ... 33 more
Caused by: java.lang.NullPointerException
    at java.lang.System.arraycopy(Native Method)
    at org.apache.hadoop.io.Text.set(Text.java:225)
    at
org.apache.orc.impl.StringRedBlackTree.add(StringRedBlackTree.java:59)
    at
org.apache.orc.impl.writer.StringTreeWriter.writeBatch(StringTreeWriter.java:70)
    at
org.apache.orc.impl.writer.StructTreeWriter.writeRootBatch(StructTreeWriter.java:56)
    at org.apache.orc.impl.WriterImpl.addRowBatch(WriterImpl.java:546)
    at com.xxxx.ingest.write(Hdfs.java:126) #line number


-----------------
Code
-----------------
public long write(ResultSet resultSet) throws SQLException, IOException {
        try {
            int numCols = resultSet.getMetaData().getColumnCount();

            //define ORC
            Configuration conf = new Configuration();
            TypeDescription schema = TypeDescription.createStruct();

            //assume all the columns are String type
            for(int i=0;i<numCols;i++) {
                String colName = resultSet.getMetaData().getColumnName(i+1);
                schema.addField(colName,TypeDescription.createString());
            }
            Writer writer = OrcFile.createWriter(new Path(stgFilePath),
OrcFile.writerOptions(conf).setSchema(schema));
            VectorizedRowBatch batch = schema.createRowBatch();

            List<String> colVals = new ArrayList<>();
            long rowsTransferred = 0;

            //write all rows from JDBC
            while (resultSet.next()) {
                int row = batch.size++;
                for (int index = 1; index <= numCols; index++) {
                    BytesColumnVector
col=(BytesColumnVector)batch.cols[index-1];

                    String val = resultSet.getString(index);
                    col.vector[row] = val.getBytes();
                    if (batch.size == batch.getMaxSize()) {
                        writer.addRowBatch(batch); ///**** NPE thrown here
*** ?
                        batch.reset();
                    }
                }
                rowsTransferred++;
            }
            if (batch.size != 0) {
                writer.addRowBatch(batch);
                batch.reset();
            }
            writer.close();
            //end write

            return rowsTransferred;
        }
        catch (SQLException | IOException e) {
            throw e;
        }
        finally {
            if (resultSet != null) resultSet.close();
        }
    }