LOAD DATA gives Error for multiple files

["Savant, Keshav" <Ke...@fisglobal.com>]

Hi,

 

My setup is

hadoop-0.20.203.0

hive-0.7.1

 

I am using a java program that uses LOAD DATA LOCAL INPATH query to
insert files into a hive table. My program is trying to insert almost
2500+ files using this command one by one. The problem I am facing is
after processing some file (sometimes 180 or 190 files) I get the
following exception

 

java.sql.SQLException: Query returned non-zero code: 9, cause: FAILED:
Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.CopyTask

                at
org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.jav
a:192)

 

I am using Statements as I read somewhere that PreparedStatements can
cause this problem, but the problem is same by both the ways. Once this
problem comes it keeps coming for rest of the files, file permissions
are OK for the files I am inserting in hive table, and the java process
owner has rwx permission for all files.

 

If this exception comes and I force close the java program (or it
completes), restart the hive server, and run my java program again, my
java program again processes (180/190) files and then same problem
comes.

 

>From JIRA a found a new version of CopyTask.java, but that version is
not being compiled as it has dependency on some other
class(CopyWork.java or Task.java) that has isErrorOnSrcEmpty() method.
If I add a new method isErrorOnSrcEmpty() and setErrorOnSrcEmpty(boolean
boolValue) in CopyWork.java then I am not sure that from where the
setter will be called to satisfy the hive CopyTask logic.

 

Any help is appreciated. The source code is attached for ready
reference.

 

Kind regards,

Keshav C Savant 

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.