You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Zhenxiao Luo (JIRA)" <ji...@apache.org> on 2012/07/14 02:33:33 UTC
[jira] [Created] (HIVE-3257) Fix avro_joins.q testcase failure when
building hive on hadoop0.23
Zhenxiao Luo created HIVE-3257:
----------------------------------
Summary: Fix avro_joins.q testcase failure when building hive on hadoop0.23
Key: HIVE-3257
URL: https://issues.apache.org/jira/browse/HIVE-3257
Project: Hive
Issue Type: Bug
Reporter: Zhenxiao Luo
Assignee: Zhenxiao Luo
avro_joins.q is failing when building hive on hadoop0.23 for both MR1 and MR2. It has an execution exception:
This query fails when execution:
SELECT e.title, e.air_date, d.first_name, d.last_name, d.extra_field, e.air_date
FROM doctors4 d JOIN episodes e ON (d.number=e.doctor)
ORDER BY d.last_name, e.title
Execution failed with exit status: 2
Obtaining error information
Task failed!
Task ID:
Stage-1
Logs:
/home/cloudera/Code/hive/build/ql/tmp//hive.log
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3257) Fix avro_joins.q testcase failure
when building hive on hadoop0.23
Posted by "Zhenxiao Luo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414215#comment-13414215 ]
Zhenxiao Luo commented on HIVE-3257:
------------------------------------
Review request submitted at:
https://reviews.facebook.net/D4131
> Fix avro_joins.q testcase failure when building hive on hadoop0.23
> ------------------------------------------------------------------
>
> Key: HIVE-3257
> URL: https://issues.apache.org/jira/browse/HIVE-3257
> Project: Hive
> Issue Type: Bug
> Reporter: Zhenxiao Luo
> Assignee: Zhenxiao Luo
>
> avro_joins.q is failing when building hive on hadoop0.23 for both MR1 and MR2. It has an execution exception:
> This query fails when execution:
> SELECT e.title, e.air_date, d.first_name, d.last_name, d.extra_field, e.air_date
> FROM doctors4 d JOIN episodes e ON (d.number=e.doctor)
> ORDER BY d.last_name, e.title
> Execution failed with exit status: 2
> Obtaining error information
> Task failed!
> Task ID:
> Stage-1
> Logs:
> /home/cloudera/Code/hive/build/ql/tmp//hive.log
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3257) Fix avro_joins.q testcase failure when
building hive on hadoop0.23
Posted by "Zhenxiao Luo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhenxiao Luo updated HIVE-3257:
-------------------------------
Attachment: HIVE-3257.1.patch.txt
> Fix avro_joins.q testcase failure when building hive on hadoop0.23
> ------------------------------------------------------------------
>
> Key: HIVE-3257
> URL: https://issues.apache.org/jira/browse/HIVE-3257
> Project: Hive
> Issue Type: Bug
> Reporter: Zhenxiao Luo
> Assignee: Zhenxiao Luo
> Attachments: HIVE-3257.1.patch.txt
>
>
> avro_joins.q is failing when building hive on hadoop0.23 for both MR1 and MR2. It has an execution exception:
> This query fails when execution:
> SELECT e.title, e.air_date, d.first_name, d.last_name, d.extra_field, e.air_date
> FROM doctors4 d JOIN episodes e ON (d.number=e.doctor)
> ORDER BY d.last_name, e.title
> Execution failed with exit status: 2
> Obtaining error information
> Task failed!
> Task ID:
> Stage-1
> Logs:
> /home/cloudera/Code/hive/build/ql/tmp//hive.log
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3257) Fix avro_joins.q testcase failure
when building hive on hadoop0.23
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414388#comment-13414388 ]
Edward Capriolo commented on HIVE-3257:
---------------------------------------
Tests pass
{noformat}
[edward@tablitha trunk]$ ant test -Dtestcase=TestCliDriver -Dqfile=avro_joins.q,avro_change_schema.q,avro_sanity_test.q,avro_schema_literal.q
[junit] Done query: avro_schema_literal.q elapsedTime=0s
[junit] Cleaning up TestCliDriver
[junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 34.706 sec
{noformat}
> Fix avro_joins.q testcase failure when building hive on hadoop0.23
> ------------------------------------------------------------------
>
> Key: HIVE-3257
> URL: https://issues.apache.org/jira/browse/HIVE-3257
> Project: Hive
> Issue Type: Bug
> Reporter: Zhenxiao Luo
> Assignee: Zhenxiao Luo
> Attachments: HIVE-3257.1.patch.txt
>
>
> avro_joins.q is failing when building hive on hadoop0.23 for both MR1 and MR2. It has an execution exception:
> This query fails when execution:
> SELECT e.title, e.air_date, d.first_name, d.last_name, d.extra_field, e.air_date
> FROM doctors4 d JOIN episodes e ON (d.number=e.doctor)
> ORDER BY d.last_name, e.title
> Execution failed with exit status: 2
> Obtaining error information
> Task failed!
> Task ID:
> Stage-1
> Logs:
> /home/cloudera/Code/hive/build/ql/tmp//hive.log
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3257) Fix avro_joins.q testcase failure
when building hive on hadoop0.23
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414380#comment-13414380 ]
Edward Capriolo commented on HIVE-3257:
---------------------------------------
+1 will test and commit.
> Fix avro_joins.q testcase failure when building hive on hadoop0.23
> ------------------------------------------------------------------
>
> Key: HIVE-3257
> URL: https://issues.apache.org/jira/browse/HIVE-3257
> Project: Hive
> Issue Type: Bug
> Reporter: Zhenxiao Luo
> Assignee: Zhenxiao Luo
> Attachments: HIVE-3257.1.patch.txt
>
>
> avro_joins.q is failing when building hive on hadoop0.23 for both MR1 and MR2. It has an execution exception:
> This query fails when execution:
> SELECT e.title, e.air_date, d.first_name, d.last_name, d.extra_field, e.air_date
> FROM doctors4 d JOIN episodes e ON (d.number=e.doctor)
> ORDER BY d.last_name, e.title
> Execution failed with exit status: 2
> Obtaining error information
> Task failed!
> Task ID:
> Stage-1
> Logs:
> /home/cloudera/Code/hive/build/ql/tmp//hive.log
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3257) Fix avro_joins.q testcase failure
when building hive on hadoop0.23
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414462#comment-13414462 ]
Hudson commented on HIVE-3257:
------------------------------
Integrated in Hive-trunk-h0.21 #1541 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1541/])
HIVE-3257 Fix avro_joins.q testcase failure when building hive on hadoop0.23. Zhenxiao Luo (via egc) (Revision 1361537)
Result = SUCCESS
ecapriolo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361537
Files :
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/avro/AvroGenericRecordReader.java
> Fix avro_joins.q testcase failure when building hive on hadoop0.23
> ------------------------------------------------------------------
>
> Key: HIVE-3257
> URL: https://issues.apache.org/jira/browse/HIVE-3257
> Project: Hive
> Issue Type: Bug
> Reporter: Zhenxiao Luo
> Assignee: Zhenxiao Luo
> Fix For: 0.10.0
>
> Attachments: HIVE-3257.1.patch.txt
>
>
> avro_joins.q is failing when building hive on hadoop0.23 for both MR1 and MR2. It has an execution exception:
> This query fails when execution:
> SELECT e.title, e.air_date, d.first_name, d.last_name, d.extra_field, e.air_date
> FROM doctors4 d JOIN episodes e ON (d.number=e.doctor)
> ORDER BY d.last_name, e.title
> Execution failed with exit status: 2
> Obtaining error information
> Task failed!
> Task ID:
> Stage-1
> Logs:
> /home/cloudera/Code/hive/build/ql/tmp//hive.log
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3257) Fix avro_joins.q testcase failure when
building hive on hadoop0.23
Posted by "Zhenxiao Luo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhenxiao Luo updated HIVE-3257:
-------------------------------
Status: Patch Available (was: Open)
> Fix avro_joins.q testcase failure when building hive on hadoop0.23
> ------------------------------------------------------------------
>
> Key: HIVE-3257
> URL: https://issues.apache.org/jira/browse/HIVE-3257
> Project: Hive
> Issue Type: Bug
> Reporter: Zhenxiao Luo
> Assignee: Zhenxiao Luo
> Attachments: HIVE-3257.1.patch.txt
>
>
> avro_joins.q is failing when building hive on hadoop0.23 for both MR1 and MR2. It has an execution exception:
> This query fails when execution:
> SELECT e.title, e.air_date, d.first_name, d.last_name, d.extra_field, e.air_date
> FROM doctors4 d JOIN episodes e ON (d.number=e.doctor)
> ORDER BY d.last_name, e.title
> Execution failed with exit status: 2
> Obtaining error information
> Task failed!
> Task ID:
> Stage-1
> Logs:
> /home/cloudera/Code/hive/build/ql/tmp//hive.log
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3257) Fix avro_joins.q testcase failure
when building hive on hadoop0.23
Posted by "Zhenxiao Luo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414212#comment-13414212 ]
Zhenxiao Luo commented on HIVE-3257:
------------------------------------
The problem is in
ql/src/java/org/apache/hadoop/hive/ql/io/avro/AvroGenericRecordReader.java:
in getSchema(), the FileSplit does not have the scheme part of path URI, in this case, "pfile:".
The matching function pathIsInPartition() is checking whether the split starts with patitionPath.
In hadoop0.23, partitionPath still holds pfile: prefix, while, FileSplit does not. So, pathIsInPartition() returns false.
In hadoop0.20, both partitionPath and FileSplit hold pfile: prefix. So, pathIsInPartition() returns true.
The root of the problem is in:
shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java
In getSplits(), hadoop0.23 removes scheme part of path URI in the CombineFileInputFormat, in this case "pfile:". This diffs from hadoop0.20 behavior.
The same problem happens in HIVE-2737, HIVE-2778, HIVE-2784.
We already committed patches, which have workaround including checking whether the path is schemeless or not.
Will do the same thing for this AvroGenericRecordReader
> Fix avro_joins.q testcase failure when building hive on hadoop0.23
> ------------------------------------------------------------------
>
> Key: HIVE-3257
> URL: https://issues.apache.org/jira/browse/HIVE-3257
> Project: Hive
> Issue Type: Bug
> Reporter: Zhenxiao Luo
> Assignee: Zhenxiao Luo
>
> avro_joins.q is failing when building hive on hadoop0.23 for both MR1 and MR2. It has an execution exception:
> This query fails when execution:
> SELECT e.title, e.air_date, d.first_name, d.last_name, d.extra_field, e.air_date
> FROM doctors4 d JOIN episodes e ON (d.number=e.doctor)
> ORDER BY d.last_name, e.title
> Execution failed with exit status: 2
> Obtaining error information
> Task failed!
> Task ID:
> Stage-1
> Logs:
> /home/cloudera/Code/hive/build/ql/tmp//hive.log
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3257) Fix avro_joins.q testcase failure when
building hive on hadoop0.23
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward Capriolo updated HIVE-3257:
----------------------------------
Resolution: Fixed
Fix Version/s: 0.10.0
Status: Resolved (was: Patch Available)
Committed. Thanks Zhenxiao.
> Fix avro_joins.q testcase failure when building hive on hadoop0.23
> ------------------------------------------------------------------
>
> Key: HIVE-3257
> URL: https://issues.apache.org/jira/browse/HIVE-3257
> Project: Hive
> Issue Type: Bug
> Reporter: Zhenxiao Luo
> Assignee: Zhenxiao Luo
> Fix For: 0.10.0
>
> Attachments: HIVE-3257.1.patch.txt
>
>
> avro_joins.q is failing when building hive on hadoop0.23 for both MR1 and MR2. It has an execution exception:
> This query fails when execution:
> SELECT e.title, e.air_date, d.first_name, d.last_name, d.extra_field, e.air_date
> FROM doctors4 d JOIN episodes e ON (d.number=e.doctor)
> ORDER BY d.last_name, e.title
> Execution failed with exit status: 2
> Obtaining error information
> Task failed!
> Task ID:
> Stage-1
> Logs:
> /home/cloudera/Code/hive/build/ql/tmp//hive.log
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira