You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "binlijin (JIRA)" <ji...@apache.org> on 2011/09/23 18:40:26 UTC
[jira] [Created] (HIVE-2466) mapjoin_subquery dump two small table
to the same file
mapjoin_subquery dump two small table to the same file
-------------------------------------------------------
Key: HIVE-2466
URL: https://issues.apache.org/jira/browse/HIVE-2466
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.7.1
Reporter: binlijin
Priority: Critical
Fix For: 0.8.0
in mapjoin_subquery.q there is a query：
SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
FROM
(SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
FROM src1 x JOIN src y ON (x.key = y.key)) subq
JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146227#comment-13146227 ]
Hudson commented on HIVE-2466:
------------------------------
Integrated in Hive-trunk-h0.21 #1067 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1067/])
HIVE-2466 mapjoin_subquery dump small table (mapjoin table) to the same file
(binlijin via namit)
namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1199117
Files :
* /hive/trunk/data/files/x.txt
* /hive/trunk/data/files/y.txt
* /hive/trunk/data/files/z.txt
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapredLocalTask.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
* /hive/trunk/ql/src/test/queries/clientpositive/mapjoin_subquery2.q
* /hive/trunk/ql/src/test/results/clientpositive/mapjoin_subquery2.q.out
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Assignee: binlijin
> Priority: Critical
> Attachments: D285.1.patch, D285.2.patch, hive-2466.1.patch, hive-2466.2.patch, hive-2466.3.patch, hive-2466.4.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "binlijin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134767#comment-13134767 ]
binlijin commented on HIVE-2466:
--------------------------------
@Namit Jain,
Thank you very much! I make some changes in hive-2466.3.patch.
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Assignee: binlijin
> Priority: Blocker
> Attachments: hive-2466.1.patch, hive-2466.2.patch, hive-2466.3.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "binlijin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134041#comment-13134041 ]
binlijin commented on HIVE-2466:
--------------------------------
Add a testcase in hive-2466.2.patch
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Assignee: binlijin
> Priority: Blocker
> Attachments: hive-2466.1.patch, hive-2466.2.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "binlijin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
binlijin updated HIVE-2466:
---------------------------
Attachment: hive-2466.1.patch
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Priority: Blocker
> Attachments: hive-2466.1.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "binlijin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
binlijin updated HIVE-2466:
---------------------------
Priority: Blocker (was: Critical)
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Priority: Blocker
> Fix For: 0.8.0
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "Namit Jain (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-2466:
-----------------------------
Status: Open (was: Patch Available)
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Assignee: binlijin
> Priority: Blocker
> Attachments: D285.1.patch, hive-2466.1.patch, hive-2466.2.patch, hive-2466.3.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2466:
------------------------------
Attachment: D285.2.patch
njain updated the revision "HIVE-2466 [jira] mapjoin_subquery dump small table (mapjoin table) to the same file".
Reviewers: JIRA
HIVE-2466
REVISION DETAIL
https://reviews.facebook.net/D285
AFFECTED FILES
data/files/x.txt
data/files/y.txt
data/files/z.txt
ql/src/test/results/clientpositive/mapjoin_subquery2.q.out
ql/src/test/queries/clientpositive/mapjoin_subquery2.q
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/MapredLocalTask.java
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java
ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Assignee: binlijin
> Priority: Critical
> Attachments: D285.1.patch, D285.2.patch, hive-2466.1.patch, hive-2466.2.patch, hive-2466.3.patch, hive-2466.4.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "John Sichi (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi reassigned HIVE-2466:
--------------------------------
Assignee: binlijin
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Assignee: binlijin
> Priority: Blocker
> Attachments: hive-2466.1.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "Namit Jain (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134321#comment-13134321 ]
Namit Jain commented on HIVE-2466:
----------------------------------
reviewing now
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Assignee: binlijin
> Priority: Blocker
> Attachments: hive-2466.1.patch, hive-2466.2.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "binlijin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
binlijin updated HIVE-2466:
---------------------------
Priority: Critical (was: Blocker)
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Assignee: binlijin
> Priority: Critical
> Attachments: D285.1.patch, hive-2466.1.patch, hive-2466.2.patch, hive-2466.3.patch, hive-2466.4.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "Namit Jain (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain resolved HIVE-2466.
------------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Committed. Thanks binlijin
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Assignee: binlijin
> Priority: Critical
> Attachments: D285.1.patch, D285.2.patch, hive-2466.1.patch, hive-2466.2.patch, hive-2466.3.patch, hive-2466.4.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "binlijin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
binlijin updated HIVE-2466:
---------------------------
Summary: mapjoin_subquery dump small table (mapjoin table) to the same file (was: mapjoin_subquery dump two small table to the same file)
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Priority: Critical
> Fix For: 0.8.0
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "binlijin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145263#comment-13145263 ]
binlijin commented on HIVE-2466:
--------------------------------
hive-2466.4.patch is based on the trunk, http://svn.apache.org/repos/asf/hive/trunk, I don't have the compile problem.
how to create a arc diff entry? I use the command "svn diff > hive-2466.4.patch" to get the patch.
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Assignee: binlijin
> Priority: Blocker
> Attachments: D285.1.patch, hive-2466.1.patch, hive-2466.2.patch, hive-2466.3.patch, hive-2466.4.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "Carl Steinbach (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-2466:
---------------------------------
Fix Version/s: (was: 0.8.0)
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Priority: Blocker
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "binlijin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
binlijin updated HIVE-2466:
---------------------------
Status: Patch Available (was: Open)
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Assignee: binlijin
> Priority: Blocker
> Attachments: hive-2466.1.patch, hive-2466.2.patch, hive-2466.3.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "Namit Jain (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145214#comment-13145214 ]
Namit Jain commented on HIVE-2466:
----------------------------------
[javac] Compiling 680 source files to /data/users/njain/hive_commit2/build/ql/classes
[javac] /data/users/njain/hive_commit2/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java:191: cannot find symbol
[javac] symbol : method generatePath(java.lang.String,java.lang.Byte,java.lang.String)
[javac] location: class org.apache.hadoop.hive.ql.exec.Utilities
[javac] String filePath = Utilities.generatePath(baseDir, pos, currentFileName);
Can you refresh ?
I am getting the above error in compiling.
Also, create a arc diff entry for helping reviewing.
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Assignee: binlijin
> Priority: Blocker
> Attachments: D285.1.patch, hive-2466.1.patch, hive-2466.2.patch, hive-2466.3.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "Namit Jain (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134440#comment-13134440 ]
Namit Jain commented on HIVE-2466:
----------------------------------
A few high level comments:
Instead of making the dump prefix optional - why dont you always have it in hashtablesinkdesc and
mapjoindesc.
This way, you can get rid of all the checks : if dumpdescriptor is not null.
The logic will be simpler - the names of map files will be :
mapfile1
mapfile2
..
etc
Also, it might be nicer to add the static function in PlanUtils.java instead of QBJoinTree.java.
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Assignee: binlijin
> Priority: Blocker
> Attachments: hive-2466.1.patch, hive-2466.2.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "He Yongqiang (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
He Yongqiang updated HIVE-2466:
-------------------------------
Status: Open (was: Patch Available)
can you add a testcase?
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Assignee: binlijin
> Priority: Blocker
> Attachments: hive-2466.1.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "binlijin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
binlijin updated HIVE-2466:
---------------------------
Attachment: hive-2466.2.patch
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Assignee: binlijin
> Priority: Blocker
> Attachments: hive-2466.1.patch, hive-2466.2.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2466:
------------------------------
Attachment: D285.1.patch
njain requested code review of "HIVE-2466 [jira] mapjoin_subquery dump small table (mapjoin table) to the same file".
Reviewers: JIRA
HIVE-2466 diff for review
in mapjoin_subquery.q there is a query：
SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
FROM
(SELECT /*+ MAPJOIN<img class="emoticon" src="https://issues.apache.org/jira/images/icons/emoticons/error.gif" height="16" width="16" align="absmiddle" alt="" border="0"/> */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
FROM src1 x JOIN src y ON (x.key = y.key)) subq
JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
when dump x and z to a local file,there all dump to the same file, so we lost the data of x
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D285
AFFECTED FILES
data/files/x.txt
data/files/y.txt
data/files/z.txt
ql/src/test/results/clientpositive/mapjoin_subquery2.q.out
ql/src/test/queries/clientpositive/mapjoin_subquery2.q
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/MapredLocalTask.java.orig
ql/src/java/org/apache/hadoop/hive/ql/exec/MapredLocalTask.java
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java.orig
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java
ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/591/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Assignee: binlijin
> Priority: Blocker
> Attachments: D285.1.patch, hive-2466.1.patch, hive-2466.2.patch, hive-2466.3.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "binlijin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
binlijin updated HIVE-2466:
---------------------------
Attachment: hive-2466.3.patch
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Assignee: binlijin
> Priority: Blocker
> Attachments: hive-2466.1.patch, hive-2466.2.patch, hive-2466.3.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "Namit Jain (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145670#comment-13145670 ]
Namit Jain commented on HIVE-2466:
----------------------------------
+1
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Assignee: binlijin
> Priority: Critical
> Attachments: D285.1.patch, D285.2.patch, hive-2466.1.patch, hive-2466.2.patch, hive-2466.3.patch, hive-2466.4.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "binlijin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
binlijin updated HIVE-2466:
---------------------------
Attachment: hive-2466.4.patch
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Assignee: binlijin
> Priority: Blocker
> Attachments: D285.1.patch, hive-2466.1.patch, hive-2466.2.patch, hive-2466.3.patch, hive-2466.4.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2466) mapjoin_subquery dump small table
(mapjoin table) to the same file
Posted by "binlijin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
binlijin updated HIVE-2466:
---------------------------
Status: Patch Available (was: Open)
> mapjoin_subquery dump small table (mapjoin table) to the same file
> -------------------------------------------------------------------
>
> Key: HIVE-2466
> URL: https://issues.apache.org/jira/browse/HIVE-2466
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.1
> Reporter: binlijin
> Priority: Blocker
> Attachments: hive-2466.1.patch
>
>
> in mapjoin_subquery.q there is a query：
> SELECT /*+ MAPJOIN(z) */ subq.key1, z.value
> FROM
> (SELECT /*+ MAPJOIN(x) */ x.key as key1, x.value as value1, y.key as key2, y.value as value2
> FROM src1 x JOIN src y ON (x.key = y.key)) subq
> JOIN srcpart z ON (subq.key1 = z.key and z.ds='2008-04-08' and z.hr=11);
> when dump x and z to a local file,there all dump to the same file, so we lost the data of x
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira