[jira] [Created] (SQOOP-1315) Cannot export data from hdfs to mysql when record size is rowsPerBatch*N

["ZhongDeyin (JIRA)" <ji...@apache.org>]

ZhongDeyin created SQOOP-1315:
---------------------------------

             Summary: Cannot export data from hdfs to mysql when record size is rowsPerBatch*N
                 Key: SQOOP-1315
                 URL: https://issues.apache.org/jira/browse/SQOOP-1315
             Project: Sqoop
          Issue Type: Bug
          Components: sqoop2-server
    Affects Versions: 1.99.3
            Reporter: ZhongDeyin


When record size is rowsPerBatch*N in HDFS, cannot export data to mysql.(default rowsPerBatch=100, N is positive  Integer)
Source Code [GenericJdbcExportLoader.java]: 
        public static final int DEFAULT_ROWS_PER_BATCH = 100;
	public static final int DEFAULT_BATCHES_PER_TRANSACTION = 100;
	private int rowsPerBatch = DEFAULT_ROWS_PER_BATCH;
	private int batchesPerTransaction = DEFAULT_BATCHES_PER_TRANSACTION;
        ..................................
        ..................................
	 int numberOfRows = 0;
	  int numberOfBatches = 0;
	  Object[] array;
         while ((array = context.getDataReader().readArrayRecord()) != null)    {
               numberOfRows++;
      		executor.addBatch(array);
		
 		if (numberOfRows == rowsPerBatch) {
 			numberOfBatches++;
			 if (numberOfBatches == batchesPerTransaction) {
 					executor.executeBatch(true);
 					numberOfBatches = 0;
 			} else {
 					executor.executeBatch(false); //no commit, only prepare preparedStatement
			 }
 					numberOfRows = 0;
				 }
 			}
			if (numberOfRows != 0) {
			// execute and commit the remaining rows
				executor.executeBatch(true);
			}
			executor.endBatch();

Source Code [GenericJdbcExecutor.java]:
public void endBatch() {
		try {
			if (preparedStatement != null) {
				preparedStatement.close();
			}
		} catch (SQLException e) {
		throw new SqoopException(GenericJdbcConnectorError.GENERIC_JDBC_CONNECTOR_0002, e);
		}
	}
For example：
300 record in HDFS, rowsPerBatch is 100, 300%100=0,execute three times executor.executeBatch(false), without commit transaction, final numberOfRows=0, numberOfBatches=3, execute executor.endBatch(), but no commit in this method, no data will export to mysql.



--
This message was sent by Atlassian JIRA
(v6.2#6252)