You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Bobo Vava (Created) (JIRA)" <ji...@apache.org> on 2011/12/27 23:00:31 UTC
[jira] [Created] (HIVE-2683) load data twice from the same hdfs
file without overwrite fails with return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
load data twice from the same hdfs file without overwrite fails with return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
--------------------------------------------------------------------------------------------------------------------------------
Key: HIVE-2683
URL: https://issues.apache.org/jira/browse/HIVE-2683
Project: Hive
Issue Type: Bug
Components: SQL
Affects Versions: 0.7.1
Environment: Cloudera VM for Linux Red Hat
Reporter: Bobo Vava
I try to load data into a hive table twice from the same hdfs file without overwrite option.
I would expect second load to append to existing data or at least replace it.
I get the following error message instead.
Note that, during the operation Hive is renaming the source file to <original_name>_copy_<nbr>, which
makes us believe it is trying to avoid erasing the existing file while moving and both files (original and renamed) would be made available to Hive. But this is not the case.
Example of code end error stack :
load data inpath 'hdfs://0.0.0.0/user/qafiles/SRC_FILE_EMP.dat'
into table EMP_DELIMITED_FILE_TRUNC_TBL
-------------------
Query returned non-zero code: 9, cause: FAILED: Execution Error, return code
1 from org.apache.hadoop.hive.ql.exec.MoveTask
river returned: 9. Errors: Hive history
file=/tmp/hue/hive_job_log_hue_201112132213_439347746.txt
Loading data to table default.emp_delimited_file_trunc_tbl
Failed with exception null
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
Failed with exception null
11/12/13 22:13:55 ERROR exec.MoveTask: Failed with exception null
java.lang.NullPointerException
at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:1738)
at org.apache.hadoop.hive.ql.metadata.Table.copyFiles(Table.java:542)
at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1189)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:197)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:306)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:510)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:499)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:337)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java :1110)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxService Impl.java:499)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java :886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908 )
at java.lang.Thread.run(Thread.java:619)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
11/12/13 22:13:55 ERROR ql.Driver: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.MoveTask
11/12/13 22:13:55 ERROR beeswax.BeeswaxServiceImpl: Exception while
processing query
BeeswaxException(message:Driver returned: 9. Errors: Hive history
file=/tmp/hue/hive_job_log_hue_201112132213_439347746.txt
Loading data to table default.emp_delimited_file_trunc_tbl
Failed with exception null
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
, log_context:8e8ea906-9b5a-4bfe-9946-515249cc523f,
handle:QueryHandle(id:8e8ea906-9b5a-4bfe-9946-515249cc523f,
log_context:8e8ea906-9b5a-4bfe-9946-515249cc523f))
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:313)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:510)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:499)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:337)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java :1110)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxServiceImpl.java:499)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java :886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908 )
at java.lang.Thread.run(Thread.java:619)
11/12/13 22:13:58 ERROR beeswax.BeeswaxServiceImpl: Caught BeeswaxException
BeeswaxException(message:Driver returned: 9. Errors: Hive history
file=/tmp/hue/hive_job_log_hue_201112132213_439347746.txt
Loading data to table default.emp_delimited_file_trunc_tbl
Failed with exception null
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
, log_context:8e8ea906-9b5a-4bfe-9946-515249cc523f,
handle:QueryHandle(id:8e8ea906-9b5a-4bfe-9946-515249cc523f,
log_context:8e8ea906-9b5a-4bfe-9946-515249cc523f))
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:313)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:510)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:499)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:337)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1110)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxServiceImpl.java:499)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908 )
at java.lang.Thread.run(Thread.java:619)
11/12/13 22:30:51 ERROR beeswax.BeeswaxServiceImpl: Caught BeeswaxException
BeeswaxException(message:Driver returned: 9. Errors: Hive history
file=/tmp/hue/hive_job_log_hue_201112132213_439347746.txt
Loading data to table default.emp_delimited_file_trunc_tbl
Failed with exception null
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
, log_context:8e8ea906-9b5a-4bfe-9946-515249cc523f,
handle:QueryHandle(id:8e8ea906-9b5a-4bfe-9946-515249cc523f,
log_context:8e8ea906-9b5a-4bfe-9946-515249cc523f))
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:313)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:510)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:499)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:337)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1110)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxServiceImpl.java:499)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2683) load data twice from the same hdfs
file without overwrite fails with return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
Posted by "Chinna Rao Lalam (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176551#comment-13176551 ]
Chinna Rao Lalam commented on HIVE-2683:
----------------------------------------
Pls check this issue HIVE-1996 it resolves this issue.
> load data twice from the same hdfs file without overwrite fails with return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-2683
> URL: https://issues.apache.org/jira/browse/HIVE-2683
> Project: Hive
> Issue Type: Bug
> Components: SQL
> Affects Versions: 0.7.1
> Environment: Cloudera VM for Linux Red Hat
> Reporter: Bobo Vava
>
> I try to load data into a hive table twice from the same hdfs file without overwrite option.
> I would expect second load to append to existing data or at least replace it.
> I get the following error message instead.
> Note that, during the operation Hive is renaming the source file to <original_name>_copy_<nbr>, which
> makes us believe it is trying to avoid erasing the existing file while moving and both files (original and renamed) would be made available to Hive. But this is not the case.
> Example of code end error stack :
> load data inpath 'hdfs://0.0.0.0/user/qafiles/SRC_FILE_EMP.dat'
> into table EMP_DELIMITED_FILE_TRUNC_TBL
> -------------------
> Query returned non-zero code: 9, cause: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask
> river returned: 9. Errors: Hive history
> file=/tmp/hue/hive_job_log_hue_201112132213_439347746.txt
> Loading data to table default.emp_delimited_file_trunc_tbl
> Failed with exception null
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask
> Failed with exception null
> 11/12/13 22:13:55 ERROR exec.MoveTask: Failed with exception null
> java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:1738)
> at org.apache.hadoop.hive.ql.metadata.Table.copyFiles(Table.java:542)
> at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1189)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:197)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900)
> at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:306)
> at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:510)
> at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:499)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:337)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java :1110)
> at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxService Impl.java:499)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java :886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908 )
> at java.lang.Thread.run(Thread.java:619)
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask
> 11/12/13 22:13:55 ERROR ql.Driver: FAILED: Execution Error, return code 1
> from org.apache.hadoop.hive.ql.exec.MoveTask
> 11/12/13 22:13:55 ERROR beeswax.BeeswaxServiceImpl: Exception while
> processing query
> BeeswaxException(message:Driver returned: 9. Errors: Hive history
> file=/tmp/hue/hive_job_log_hue_201112132213_439347746.txt
> Loading data to table default.emp_delimited_file_trunc_tbl
> Failed with exception null
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask
> , log_context:8e8ea906-9b5a-4bfe-9946-515249cc523f,
> handle:QueryHandle(id:8e8ea906-9b5a-4bfe-9946-515249cc523f,
> log_context:8e8ea906-9b5a-4bfe-9946-515249cc523f))
> at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:313)
> at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:510)
> at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:499)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:337)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java :1110)
> at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxServiceImpl.java:499)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java :886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908 )
> at java.lang.Thread.run(Thread.java:619)
> 11/12/13 22:13:58 ERROR beeswax.BeeswaxServiceImpl: Caught BeeswaxException
> BeeswaxException(message:Driver returned: 9. Errors: Hive history
> file=/tmp/hue/hive_job_log_hue_201112132213_439347746.txt
> Loading data to table default.emp_delimited_file_trunc_tbl
> Failed with exception null
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask
> , log_context:8e8ea906-9b5a-4bfe-9946-515249cc523f,
> handle:QueryHandle(id:8e8ea906-9b5a-4bfe-9946-515249cc523f,
> log_context:8e8ea906-9b5a-4bfe-9946-515249cc523f))
> at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:313)
> at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:510)
> at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:499)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:337)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1110)
> at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxServiceImpl.java:499)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908 )
> at java.lang.Thread.run(Thread.java:619)
> 11/12/13 22:30:51 ERROR beeswax.BeeswaxServiceImpl: Caught BeeswaxException
> BeeswaxException(message:Driver returned: 9. Errors: Hive history
> file=/tmp/hue/hive_job_log_hue_201112132213_439347746.txt
> Loading data to table default.emp_delimited_file_trunc_tbl
> Failed with exception null
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask
> , log_context:8e8ea906-9b5a-4bfe-9946-515249cc523f,
> handle:QueryHandle(id:8e8ea906-9b5a-4bfe-9946-515249cc523f,
> log_context:8e8ea906-9b5a-4bfe-9946-515249cc523f))
> at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:313)
> at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:510)
> at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:499)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:337)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1110)
> at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxServiceImpl.java:499)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira