You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Zheng Shao (JIRA)" <ji...@apache.org> on 2009/11/06 23:26:32 UTC
[jira] Created: (HIVE-917) Bucketed Map Join
Bucketed Map Join
-----------------
Key: HIVE-917
URL: https://issues.apache.org/jira/browse/HIVE-917
Project: Hadoop Hive
Issue Type: New Feature
Reporter: Zheng Shao
Hive already have support for map-join. Map-join treats the big table as job input, and in each mapper, it loads all data from a small table.
In case the big table is already bucketed on the join key, we don't have to load the whole small table in each of the mappers. This will greatly alleviate the memory pressure, and make map-join work with medium-sized tables.
There are 4 steps we can improve:
S0. This is what the user can already do now: create a new bucketed table and insert all data from the small table to it; Submit BUCKETNUM jobs, each doing a map-side join of "bigtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)" with "smallbucketedtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)".
S1. Change the code so that when map-join is loading the small table, we automatically drop the rows with the keys that are NOT in the same bucket as the big table. This should alleviate the problem on memory, but we might still have thousands of mappers reading the whole of the small table.
S2. Let's say the user already bucketed the small table on the join key into exactly the same number of buckets (or a factor of the buckets of the big table), then map-join can choose to load only the buckets that are useful.
S3. Add a new hint (e.g. /*+ MAPBUCKETJOIN(a) */), so that Hive automatically does S2, without the need of asking the user to create temporary bucketed table for the small table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-917) Bucketed Map Join
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832359#action_12832359 ]
Namit Jain commented on HIVE-917:
---------------------------------
Can you generate the patch again - it is not applying cleanly
> Bucketed Map Join
> -----------------
>
> Key: HIVE-917
> URL: https://issues.apache.org/jira/browse/HIVE-917
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Assignee: He Yongqiang
> Attachments: hive-917-2010-2-3.patch, hive-917-2010-2-8.patch
>
>
> Hive already have support for map-join. Map-join treats the big table as job input, and in each mapper, it loads all data from a small table.
> In case the big table is already bucketed on the join key, we don't have to load the whole small table in each of the mappers. This will greatly alleviate the memory pressure, and make map-join work with medium-sized tables.
> There are 4 steps we can improve:
> S0. This is what the user can already do now: create a new bucketed table and insert all data from the small table to it; Submit BUCKETNUM jobs, each doing a map-side join of "bigtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)" with "smallbucketedtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)".
> S1. Change the code so that when map-join is loading the small table, we automatically drop the rows with the keys that are NOT in the same bucket as the big table. This should alleviate the problem on memory, but we might still have thousands of mappers reading the whole of the small table.
> S2. Let's say the user already bucketed the small table on the join key into exactly the same number of buckets (or a factor of the buckets of the big table), then map-join can choose to load only the buckets that are useful.
> S3. Add a new hint (e.g. /*+ MAPBUCKETJOIN(a) */), so that Hive automatically does S2, without the need of asking the user to create temporary bucketed table for the small table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-917) Bucketed Map Join
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
He Yongqiang updated HIVE-917:
------------------------------
Attachment: hive-917-2010-2-16.patch
A new patch:
1) added explain extended
2) break the test to 4 tests.
> Bucketed Map Join
> -----------------
>
> Key: HIVE-917
> URL: https://issues.apache.org/jira/browse/HIVE-917
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Assignee: He Yongqiang
> Attachments: hive-917-2010-2-15.patch, hive-917-2010-2-16.patch, hive-917-2010-2-3.patch, hive-917-2010-2-8.patch
>
>
> Hive already have support for map-join. Map-join treats the big table as job input, and in each mapper, it loads all data from a small table.
> In case the big table is already bucketed on the join key, we don't have to load the whole small table in each of the mappers. This will greatly alleviate the memory pressure, and make map-join work with medium-sized tables.
> There are 4 steps we can improve:
> S0. This is what the user can already do now: create a new bucketed table and insert all data from the small table to it; Submit BUCKETNUM jobs, each doing a map-side join of "bigtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)" with "smallbucketedtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)".
> S1. Change the code so that when map-join is loading the small table, we automatically drop the rows with the keys that are NOT in the same bucket as the big table. This should alleviate the problem on memory, but we might still have thousands of mappers reading the whole of the small table.
> S2. Let's say the user already bucketed the small table on the join key into exactly the same number of buckets (or a factor of the buckets of the big table), then map-join can choose to load only the buckets that are useful.
> S3. Add a new hint (e.g. /*+ MAPBUCKETJOIN(a) */), so that Hive automatically does S2, without the need of asking the user to create temporary bucketed table for the small table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-917) Bucketed Map Join
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830171#action_12830171 ]
Namit Jain commented on HIVE-917:
---------------------------------
Also, filing a new jira for supporting bucketing map-joins where more than 1 partition is being joined for the
big table.
> Bucketed Map Join
> -----------------
>
> Key: HIVE-917
> URL: https://issues.apache.org/jira/browse/HIVE-917
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Attachments: hive-917-2010-2-3.patch
>
>
> Hive already have support for map-join. Map-join treats the big table as job input, and in each mapper, it loads all data from a small table.
> In case the big table is already bucketed on the join key, we don't have to load the whole small table in each of the mappers. This will greatly alleviate the memory pressure, and make map-join work with medium-sized tables.
> There are 4 steps we can improve:
> S0. This is what the user can already do now: create a new bucketed table and insert all data from the small table to it; Submit BUCKETNUM jobs, each doing a map-side join of "bigtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)" with "smallbucketedtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)".
> S1. Change the code so that when map-join is loading the small table, we automatically drop the rows with the keys that are NOT in the same bucket as the big table. This should alleviate the problem on memory, but we might still have thousands of mappers reading the whole of the small table.
> S2. Let's say the user already bucketed the small table on the join key into exactly the same number of buckets (or a factor of the buckets of the big table), then map-join can choose to load only the buckets that are useful.
> S3. Add a new hint (e.g. /*+ MAPBUCKETJOIN(a) */), so that Hive automatically does S2, without the need of asking the user to create temporary bucketed table for the small table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-917) Bucketed Map Join
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
He Yongqiang updated HIVE-917:
------------------------------
Attachment: hive-917-2010-2-8.patch
A new patch integrated with Namit's comments. Thanks Namit.
> Bucketed Map Join
> -----------------
>
> Key: HIVE-917
> URL: https://issues.apache.org/jira/browse/HIVE-917
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Assignee: He Yongqiang
> Attachments: hive-917-2010-2-3.patch, hive-917-2010-2-8.patch
>
>
> Hive already have support for map-join. Map-join treats the big table as job input, and in each mapper, it loads all data from a small table.
> In case the big table is already bucketed on the join key, we don't have to load the whole small table in each of the mappers. This will greatly alleviate the memory pressure, and make map-join work with medium-sized tables.
> There are 4 steps we can improve:
> S0. This is what the user can already do now: create a new bucketed table and insert all data from the small table to it; Submit BUCKETNUM jobs, each doing a map-side join of "bigtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)" with "smallbucketedtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)".
> S1. Change the code so that when map-join is loading the small table, we automatically drop the rows with the keys that are NOT in the same bucket as the big table. This should alleviate the problem on memory, but we might still have thousands of mappers reading the whole of the small table.
> S2. Let's say the user already bucketed the small table on the join key into exactly the same number of buckets (or a factor of the buckets of the big table), then map-join can choose to load only the buckets that are useful.
> S3. Add a new hint (e.g. /*+ MAPBUCKETJOIN(a) */), so that Hive automatically does S2, without the need of asking the user to create temporary bucketed table for the small table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-917) Bucketed Map Join
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834448#action_12834448 ]
Namit Jain commented on HIVE-917:
---------------------------------
Added some more tasks in the follow-up jira after talking to Yongqiang.
Will commit this if the tests pass
> Bucketed Map Join
> -----------------
>
> Key: HIVE-917
> URL: https://issues.apache.org/jira/browse/HIVE-917
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Assignee: He Yongqiang
> Attachments: hive-917-2010-2-15.patch, hive-917-2010-2-16.patch, hive-917-2010-2-3.patch, hive-917-2010-2-8.patch
>
>
> Hive already have support for map-join. Map-join treats the big table as job input, and in each mapper, it loads all data from a small table.
> In case the big table is already bucketed on the join key, we don't have to load the whole small table in each of the mappers. This will greatly alleviate the memory pressure, and make map-join work with medium-sized tables.
> There are 4 steps we can improve:
> S0. This is what the user can already do now: create a new bucketed table and insert all data from the small table to it; Submit BUCKETNUM jobs, each doing a map-side join of "bigtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)" with "smallbucketedtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)".
> S1. Change the code so that when map-join is loading the small table, we automatically drop the rows with the keys that are NOT in the same bucket as the big table. This should alleviate the problem on memory, but we might still have thousands of mappers reading the whole of the small table.
> S2. Let's say the user already bucketed the small table on the join key into exactly the same number of buckets (or a factor of the buckets of the big table), then map-join can choose to load only the buckets that are useful.
> S3. Add a new hint (e.g. /*+ MAPBUCKETJOIN(a) */), so that Hive automatically does S2, without the need of asking the user to create temporary bucketed table for the small table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-917) Bucketed Map Join
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777632#action_12777632 ]
Namit Jain commented on HIVE-917:
---------------------------------
Generalizing it more, we can use bucketing/sorting information to do lot more.
This can be the parent jira out of which we can have various dependent jiras.
I will file one for group by
> Bucketed Map Join
> -----------------
>
> Key: HIVE-917
> URL: https://issues.apache.org/jira/browse/HIVE-917
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
>
> Hive already have support for map-join. Map-join treats the big table as job input, and in each mapper, it loads all data from a small table.
> In case the big table is already bucketed on the join key, we don't have to load the whole small table in each of the mappers. This will greatly alleviate the memory pressure, and make map-join work with medium-sized tables.
> There are 4 steps we can improve:
> S0. This is what the user can already do now: create a new bucketed table and insert all data from the small table to it; Submit BUCKETNUM jobs, each doing a map-side join of "bigtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)" with "smallbucketedtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)".
> S1. Change the code so that when map-join is loading the small table, we automatically drop the rows with the keys that are NOT in the same bucket as the big table. This should alleviate the problem on memory, but we might still have thousands of mappers reading the whole of the small table.
> S2. Let's say the user already bucketed the small table on the join key into exactly the same number of buckets (or a factor of the buckets of the big table), then map-join can choose to load only the buckets that are useful.
> S3. Add a new hint (e.g. /*+ MAPBUCKETJOIN(a) */), so that Hive automatically does S2, without the need of asking the user to create temporary bucketed table for the small table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-917) Bucketed Map Join
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
He Yongqiang updated HIVE-917:
------------------------------
Attachment: hive-917-2010-2-15.patch
Regenerated the patch against trunk. see hive-917...2-15.patch
> Bucketed Map Join
> -----------------
>
> Key: HIVE-917
> URL: https://issues.apache.org/jira/browse/HIVE-917
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Assignee: He Yongqiang
> Attachments: hive-917-2010-2-15.patch, hive-917-2010-2-3.patch, hive-917-2010-2-8.patch
>
>
> Hive already have support for map-join. Map-join treats the big table as job input, and in each mapper, it loads all data from a small table.
> In case the big table is already bucketed on the join key, we don't have to load the whole small table in each of the mappers. This will greatly alleviate the memory pressure, and make map-join work with medium-sized tables.
> There are 4 steps we can improve:
> S0. This is what the user can already do now: create a new bucketed table and insert all data from the small table to it; Submit BUCKETNUM jobs, each doing a map-side join of "bigtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)" with "smallbucketedtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)".
> S1. Change the code so that when map-join is loading the small table, we automatically drop the rows with the keys that are NOT in the same bucket as the big table. This should alleviate the problem on memory, but we might still have thousands of mappers reading the whole of the small table.
> S2. Let's say the user already bucketed the small table on the join key into exactly the same number of buckets (or a factor of the buckets of the big table), then map-join can choose to load only the buckets that are useful.
> S3. Add a new hint (e.g. /*+ MAPBUCKETJOIN(a) */), so that Hive automatically does S2, without the need of asking the user to create temporary bucketed table for the small table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-917) Bucketed Map Join
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830012#action_12830012 ]
Namit Jain commented on HIVE-917:
---------------------------------
BucketMapJoinOptimizer.java: 80
wrong comment:
// process group-by pattern
can you add a correct comment ?
// mapper. That means there is not reducer between the root table scan and
change to:
// mapper. That means there is no reducer between the root table scan and
Add some comments in
private boolean checkBucketColumns(List<String> bucketColumns, MapJoinDesc mjDesc, int index) {
A mapjoin B where A is the big table and partitioned should be optimized
B is not partitioned
(assuming both A and B are bucketed)
if(partNumber == 0) {
Integer num = new Integer(0);
bucketNumbers.add(num);
aliasToBucketNumber.put(alias, num);
aliasToBucketFileNames.put(alias, new ArrayList<String>());
no need to do this - anyway, the results are empty
ExecMapper:
if(bucketMatcherCls == null) {
bucketMatcherCls = org.apache.hadoop.hive.ql.exec.DefaultBucketMatcher.class;
}
Add the class name in mapredlocalwork and initialize it using reflection
Keep file name to file name mapping in mapredlocalwork (only useful for bucketed map join - not for skew join)
MapredLocalWork:
private LinkedHashMap<String, Integer> aliasToBucketNumber;
private LinkedHashMap<String, List<String>> aliasToBucketFileNames;
private String mapJoinBigTableAlias;
private Class<? extends BucketMatcher> bucketMatcker;
create a new class for the above
public Class<? extends BucketMatcher> getBucketMatcker() {
return bucketMatcker;
}
public void setBucketMatcker(Class<? extends BucketMatcher> bucketMatcker) {
this.bucketMatcker = bucketMatcker;
}
spelling: should be Matcher
DefaultBucketMatcher:
public List<Path> getAliasBucketFiles(String refTableInputFile, String refTableAlias, String alias) {
int bigTblBucketNum = aliasToBucketNumber.get(refTableAlias);
int smallTblBucketNum = aliasToBucketNumber.get(alias);
Collections.sort(aliasToBucketFileNames.get(refTableAlias));
Collections.sort(aliasToBucketFileNames.get(alias));
List<Path> resultFileNames = new ArrayList<Path>();
if (bigTblBucketNum >= smallTblBucketNum) {
int temp = bigTblBucketNum / smallTblBucketNum;
int index = aliasToBucketFileNames.get(refTableAlias).indexOf(refTableInputFile);
int toAddSmallIndex = index/temp;
if(toAddSmallIndex < aliasToBucketFileNames.get(alias).size()) {
resultFileNames.add(new Path(aliasToBucketFileNames.get(alias).get(toAddSmallIndex)));
}
} else {
int jump = smallTblBucketNum / bigTblBucketNum;
int index = aliasToBucketFileNames.get(refTableAlias).indexOf(refTableInputFile);
for (int i = index; i < aliasToBucketFileNames.get(alias).size(); i = i + jump) {
if(i <= aliasToBucketFileNames.get(alias).size()) {
resultFileNames.add(new Path(aliasToBucketFileNames.get(alias).get(i)));
}
}
}
return resultFileNames;
}
move this to compile time and add some more comments
FetchOperator.java:
6 boolean ret = false;
267 try {
268 value = currRecReader.createValue();
269 ret = currRecReader.next(key, value);
270 } catch (Exception e) {
271 e.printStackTrace();
272 }
> Bucketed Map Join
> -----------------
>
> Key: HIVE-917
> URL: https://issues.apache.org/jira/browse/HIVE-917
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Attachments: hive-917-2010-2-3.patch
>
>
> Hive already have support for map-join. Map-join treats the big table as job input, and in each mapper, it loads all data from a small table.
> In case the big table is already bucketed on the join key, we don't have to load the whole small table in each of the mappers. This will greatly alleviate the memory pressure, and make map-join work with medium-sized tables.
> There are 4 steps we can improve:
> S0. This is what the user can already do now: create a new bucketed table and insert all data from the small table to it; Submit BUCKETNUM jobs, each doing a map-side join of "bigtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)" with "smallbucketedtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)".
> S1. Change the code so that when map-join is loading the small table, we automatically drop the rows with the keys that are NOT in the same bucket as the big table. This should alleviate the problem on memory, but we might still have thousands of mappers reading the whole of the small table.
> S2. Let's say the user already bucketed the small table on the join key into exactly the same number of buckets (or a factor of the buckets of the big table), then map-join can choose to load only the buckets that are useful.
> S3. Add a new hint (e.g. /*+ MAPBUCKETJOIN(a) */), so that Hive automatically does S2, without the need of asking the user to create temporary bucketed table for the small table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-917) Bucketed Map Join
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain reassigned HIVE-917:
-------------------------------
Assignee: He Yongqiang
> Bucketed Map Join
> -----------------
>
> Key: HIVE-917
> URL: https://issues.apache.org/jira/browse/HIVE-917
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Assignee: He Yongqiang
> Attachments: hive-917-2010-2-3.patch
>
>
> Hive already have support for map-join. Map-join treats the big table as job input, and in each mapper, it loads all data from a small table.
> In case the big table is already bucketed on the join key, we don't have to load the whole small table in each of the mappers. This will greatly alleviate the memory pressure, and make map-join work with medium-sized tables.
> There are 4 steps we can improve:
> S0. This is what the user can already do now: create a new bucketed table and insert all data from the small table to it; Submit BUCKETNUM jobs, each doing a map-side join of "bigtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)" with "smallbucketedtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)".
> S1. Change the code so that when map-join is loading the small table, we automatically drop the rows with the keys that are NOT in the same bucket as the big table. This should alleviate the problem on memory, but we might still have thousands of mappers reading the whole of the small table.
> S2. Let's say the user already bucketed the small table on the join key into exactly the same number of buckets (or a factor of the buckets of the big table), then map-join can choose to load only the buckets that are useful.
> S3. Add a new hint (e.g. /*+ MAPBUCKETJOIN(a) */), so that Hive automatically does S2, without the need of asking the user to create temporary bucketed table for the small table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-917) Bucketed Map Join
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829332#action_12829332 ]
Namit Jain commented on HIVE-917:
---------------------------------
Started looking at the patch:
1. Can you change the test to add a explain for all queries ?
2. Can you change the default value of hive.optimize.bucketmapjoin to true ?
3. Instead of changing QTestUtil.java, can you create the tables/partitions you need in the new test only ?
> Bucketed Map Join
> -----------------
>
> Key: HIVE-917
> URL: https://issues.apache.org/jira/browse/HIVE-917
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Attachments: hive-917-2010-2-3.patch
>
>
> Hive already have support for map-join. Map-join treats the big table as job input, and in each mapper, it loads all data from a small table.
> In case the big table is already bucketed on the join key, we don't have to load the whole small table in each of the mappers. This will greatly alleviate the memory pressure, and make map-join work with medium-sized tables.
> There are 4 steps we can improve:
> S0. This is what the user can already do now: create a new bucketed table and insert all data from the small table to it; Submit BUCKETNUM jobs, each doing a map-side join of "bigtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)" with "smallbucketedtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)".
> S1. Change the code so that when map-join is loading the small table, we automatically drop the rows with the keys that are NOT in the same bucket as the big table. This should alleviate the problem on memory, but we might still have thousands of mappers reading the whole of the small table.
> S2. Let's say the user already bucketed the small table on the join key into exactly the same number of buckets (or a factor of the buckets of the big table), then map-join can choose to load only the buckets that are useful.
> S3. Add a new hint (e.g. /*+ MAPBUCKETJOIN(a) */), so that Hive automatically does S2, without the need of asking the user to create temporary bucketed table for the small table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-917) Bucketed Map Join
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834115#action_12834115 ]
Namit Jain commented on HIVE-917:
---------------------------------
Can you add explain extended to the tests so that it is possible to see which buckets are joining with which buckets ?
> Bucketed Map Join
> -----------------
>
> Key: HIVE-917
> URL: https://issues.apache.org/jira/browse/HIVE-917
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Assignee: He Yongqiang
> Attachments: hive-917-2010-2-15.patch, hive-917-2010-2-3.patch, hive-917-2010-2-8.patch
>
>
> Hive already have support for map-join. Map-join treats the big table as job input, and in each mapper, it loads all data from a small table.
> In case the big table is already bucketed on the join key, we don't have to load the whole small table in each of the mappers. This will greatly alleviate the memory pressure, and make map-join work with medium-sized tables.
> There are 4 steps we can improve:
> S0. This is what the user can already do now: create a new bucketed table and insert all data from the small table to it; Submit BUCKETNUM jobs, each doing a map-side join of "bigtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)" with "smallbucketedtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)".
> S1. Change the code so that when map-join is loading the small table, we automatically drop the rows with the keys that are NOT in the same bucket as the big table. This should alleviate the problem on memory, but we might still have thousands of mappers reading the whole of the small table.
> S2. Let's say the user already bucketed the small table on the join key into exactly the same number of buckets (or a factor of the buckets of the big table), then map-join can choose to load only the buckets that are useful.
> S3. Add a new hint (e.g. /*+ MAPBUCKETJOIN(a) */), so that Hive automatically does S2, without the need of asking the user to create temporary bucketed table for the small table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-917) Bucketed Map Join
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830016#action_12830016 ]
Namit Jain commented on HIVE-917:
---------------------------------
reviewed with Yongqiang
> Bucketed Map Join
> -----------------
>
> Key: HIVE-917
> URL: https://issues.apache.org/jira/browse/HIVE-917
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Attachments: hive-917-2010-2-3.patch
>
>
> Hive already have support for map-join. Map-join treats the big table as job input, and in each mapper, it loads all data from a small table.
> In case the big table is already bucketed on the join key, we don't have to load the whole small table in each of the mappers. This will greatly alleviate the memory pressure, and make map-join work with medium-sized tables.
> There are 4 steps we can improve:
> S0. This is what the user can already do now: create a new bucketed table and insert all data from the small table to it; Submit BUCKETNUM jobs, each doing a map-side join of "bigtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)" with "smallbucketedtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)".
> S1. Change the code so that when map-join is loading the small table, we automatically drop the rows with the keys that are NOT in the same bucket as the big table. This should alleviate the problem on memory, but we might still have thousands of mappers reading the whole of the small table.
> S2. Let's say the user already bucketed the small table on the join key into exactly the same number of buckets (or a factor of the buckets of the big table), then map-join can choose to load only the buckets that are useful.
> S3. Add a new hint (e.g. /*+ MAPBUCKETJOIN(a) */), so that Hive automatically does S2, without the need of asking the user to create temporary bucketed table for the small table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-917) Bucketed Map Join
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain resolved HIVE-917.
-----------------------------
Resolution: Fixed
Fix Version/s: 0.6.0
Hadoop Flags: [Reviewed]
Committed. Thanks Yongqiang
> Bucketed Map Join
> -----------------
>
> Key: HIVE-917
> URL: https://issues.apache.org/jira/browse/HIVE-917
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Assignee: He Yongqiang
> Fix For: 0.6.0
>
> Attachments: hive-917-2010-2-15.patch, hive-917-2010-2-16.patch, hive-917-2010-2-3.patch, hive-917-2010-2-8.patch
>
>
> Hive already have support for map-join. Map-join treats the big table as job input, and in each mapper, it loads all data from a small table.
> In case the big table is already bucketed on the join key, we don't have to load the whole small table in each of the mappers. This will greatly alleviate the memory pressure, and make map-join work with medium-sized tables.
> There are 4 steps we can improve:
> S0. This is what the user can already do now: create a new bucketed table and insert all data from the small table to it; Submit BUCKETNUM jobs, each doing a map-side join of "bigtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)" with "smallbucketedtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)".
> S1. Change the code so that when map-join is loading the small table, we automatically drop the rows with the keys that are NOT in the same bucket as the big table. This should alleviate the problem on memory, but we might still have thousands of mappers reading the whole of the small table.
> S2. Let's say the user already bucketed the small table on the join key into exactly the same number of buckets (or a factor of the buckets of the big table), then map-join can choose to load only the buckets that are useful.
> S3. Add a new hint (e.g. /*+ MAPBUCKETJOIN(a) */), so that Hive automatically does S2, without the need of asking the user to create temporary bucketed table for the small table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-917) Bucketed Map Join
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
He Yongqiang updated HIVE-917:
------------------------------
Attachment: hive-917-2010-2-3.patch
> Bucketed Map Join
> -----------------
>
> Key: HIVE-917
> URL: https://issues.apache.org/jira/browse/HIVE-917
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Attachments: hive-917-2010-2-3.patch
>
>
> Hive already have support for map-join. Map-join treats the big table as job input, and in each mapper, it loads all data from a small table.
> In case the big table is already bucketed on the join key, we don't have to load the whole small table in each of the mappers. This will greatly alleviate the memory pressure, and make map-join work with medium-sized tables.
> There are 4 steps we can improve:
> S0. This is what the user can already do now: create a new bucketed table and insert all data from the small table to it; Submit BUCKETNUM jobs, each doing a map-side join of "bigtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)" with "smallbucketedtable TABLEPARTITION(BUCKET i OUT OF NBUCKETS)".
> S1. Change the code so that when map-join is loading the small table, we automatically drop the rows with the keys that are NOT in the same bucket as the big table. This should alleviate the problem on memory, but we might still have thousands of mappers reading the whole of the small table.
> S2. Let's say the user already bucketed the small table on the join key into exactly the same number of buckets (or a factor of the buckets of the big table), then map-join can choose to load only the buckets that are useful.
> S3. Add a new hint (e.g. /*+ MAPBUCKETJOIN(a) */), so that Hive automatically does S2, without the need of asking the user to create temporary bucketed table for the small table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.