You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Namit Jain (JIRA)" <ji...@apache.org> on 2009/07/13 20:03:14 UTC
[jira] Created: (HIVE-629) concat files needed for map-reduce jobs
also
concat files needed for map-reduce jobs also
--------------------------------------------
Key: HIVE-629
URL: https://issues.apache.org/jira/browse/HIVE-629
Project: Hadoop Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 0.4.0
Reporter: Namit Jain
Assignee: Namit Jain
Currently, hive concatenates files only if the job under consideration is a map-only job.
I got some requests from some users, where they want this behavior for map-reduce jobs also - it may not be a good idea to turn it on by default.
But, we should provide an option to the user where the concatenation can happen even for map-reduce jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-629) concat files needed for map-reduce jobs
also
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-629:
----------------------------
Attachment: hive.629.1.patch
> concat files needed for map-reduce jobs also
> --------------------------------------------
>
> Key: HIVE-629
> URL: https://issues.apache.org/jira/browse/HIVE-629
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.629.1.patch
>
>
> Currently, hive concatenates files only if the job under consideration is a map-only job.
> I got some requests from some users, where they want this behavior for map-reduce jobs also - it may not be a good idea to turn it on by default.
> But, we should provide an option to the user where the concatenation can happen even for map-reduce jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-629) concat files needed for map-reduce jobs
also
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-629:
----------------------------
Resolution: Fixed
Fix Version/s: 0.4.0
Release Note: HIVE-629. Concat files for map-reduce jobs. (Namit Jain via zshao)
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed. Thanks Namit!
> concat files needed for map-reduce jobs also
> --------------------------------------------
>
> Key: HIVE-629
> URL: https://issues.apache.org/jira/browse/HIVE-629
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Namit Jain
> Assignee: Namit Jain
> Fix For: 0.4.0
>
> Attachments: hive.629.1.patch, hive.629.2.patch
>
>
> Currently, hive concatenates files only if the job under consideration is a map-only job.
> I got some requests from some users, where they want this behavior for map-reduce jobs also - it may not be a good idea to turn it on by default.
> But, we should provide an option to the user where the concatenation can happen even for map-reduce jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-629) concat files needed for map-reduce jobs
also
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-629:
----------------------------
Attachment: hive.629.2.patch
> concat files needed for map-reduce jobs also
> --------------------------------------------
>
> Key: HIVE-629
> URL: https://issues.apache.org/jira/browse/HIVE-629
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.629.1.patch, hive.629.2.patch
>
>
> Currently, hive concatenates files only if the job under consideration is a map-only job.
> I got some requests from some users, where they want this behavior for map-reduce jobs also - it may not be a good idea to turn it on by default.
> But, we should provide an option to the user where the concatenation can happen even for map-reduce jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-629) concat files needed for map-reduce
jobs also
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730638#action_12730638 ]
Zheng Shao commented on HIVE-629:
---------------------------------
Agreed. If we turn it on for map-reduce job, the threshold should be much less - maybe 128MB per file or even 64MB per file.
This makes sure it only happens for rare cases (so it does not affect the performance of most queries).
> concat files needed for map-reduce jobs also
> --------------------------------------------
>
> Key: HIVE-629
> URL: https://issues.apache.org/jira/browse/HIVE-629
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Namit Jain
> Assignee: Namit Jain
>
> Currently, hive concatenates files only if the job under consideration is a map-only job.
> I got some requests from some users, where they want this behavior for map-reduce jobs also - it may not be a good idea to turn it on by default.
> But, we should provide an option to the user where the concatenation can happen even for map-reduce jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-629) concat files needed for map-reduce
jobs also
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731591#action_12731591 ]
Namit Jain commented on HIVE-629:
---------------------------------
I will change the name of the parameter and regenerate the patch.
> concat files needed for map-reduce jobs also
> --------------------------------------------
>
> Key: HIVE-629
> URL: https://issues.apache.org/jira/browse/HIVE-629
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.629.1.patch
>
>
> Currently, hive concatenates files only if the job under consideration is a map-only job.
> I got some requests from some users, where they want this behavior for map-reduce jobs also - it may not be a good idea to turn it on by default.
> But, we should provide an option to the user where the concatenation can happen even for map-reduce jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-629) concat files needed for map-reduce
jobs also
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731745#action_12731745 ]
Zheng Shao commented on HIVE-629:
---------------------------------
That makes sense.
If the user is doing a map-reduce job, we'd better not add such a task because the partitioning will be completely changed.
In the future, we can do further optimizations to enable it for map-reduce job, only if the partition key and sort key is still available at the FileSink.
Will test and commit.
> concat files needed for map-reduce jobs also
> --------------------------------------------
>
> Key: HIVE-629
> URL: https://issues.apache.org/jira/browse/HIVE-629
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.629.1.patch, hive.629.2.patch
>
>
> Currently, hive concatenates files only if the job under consideration is a map-only job.
> I got some requests from some users, where they want this behavior for map-reduce jobs also - it may not be a good idea to turn it on by default.
> But, we should provide an option to the user where the concatenation can happen even for map-reduce jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-629) concat files needed for map-reduce
jobs also
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731297#action_12731297 ]
Zheng Shao commented on HIVE-629:
---------------------------------
Three more questions:
1. How do we determine the number of reducers of the merge job? Is that based on "hive.exec.reducers.bytes.per.reducer"?
2. How do we create the additional map-reduce job? Do we copy the cluster key or distribution key in the last job? If so, what if the keys are not available after the reducer?
3. For default value, do we want to enable both (map, map-reduce), but set the threshold to 64MB or smaller like 16MB? So most users won't see a change at all, but people who are producing extremely small files (those are the people who wants this feature) will see the files concatenated?
> concat files needed for map-reduce jobs also
> --------------------------------------------
>
> Key: HIVE-629
> URL: https://issues.apache.org/jira/browse/HIVE-629
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.629.1.patch
>
>
> Currently, hive concatenates files only if the job under consideration is a map-only job.
> I got some requests from some users, where they want this behavior for map-reduce jobs also - it may not be a good idea to turn it on by default.
> But, we should provide an option to the user where the concatenation can happen even for map-reduce jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-629) concat files needed for map-reduce
jobs also
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731282#action_12731282 ]
Zheng Shao commented on HIVE-629:
---------------------------------
The size configuration is still "hive.merge.size.per.mapper".
Do you want to change it now since not many people are using it yet? "hive.merge.size.per.task".
> concat files needed for map-reduce jobs also
> --------------------------------------------
>
> Key: HIVE-629
> URL: https://issues.apache.org/jira/browse/HIVE-629
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.629.1.patch
>
>
> Currently, hive concatenates files only if the job under consideration is a map-only job.
> I got some requests from some users, where they want this behavior for map-reduce jobs also - it may not be a good idea to turn it on by default.
> But, we should provide an option to the user where the concatenation can happen even for map-reduce jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-629) concat files needed for map-reduce jobs
also
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-629:
----------------------------
Status: Patch Available (was: Open)
> concat files needed for map-reduce jobs also
> --------------------------------------------
>
> Key: HIVE-629
> URL: https://issues.apache.org/jira/browse/HIVE-629
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.629.1.patch
>
>
> Currently, hive concatenates files only if the job under consideration is a map-only job.
> I got some requests from some users, where they want this behavior for map-reduce jobs also - it may not be a good idea to turn it on by default.
> But, we should provide an option to the user where the concatenation can happen even for map-reduce jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-629) concat files needed for map-reduce
jobs also
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731589#action_12731589 ]
Namit Jain commented on HIVE-629:
---------------------------------
1. The number of reducers are determined from "hive.merge.size.per.task"
2. We use random()
3. I am not sure about that - by default, it should be disabled in case of a map-reduce job. Only in very specific cases, it is being turned on, where the user knows exactly what is he doing and can set the size appropriately.
> concat files needed for map-reduce jobs also
> --------------------------------------------
>
> Key: HIVE-629
> URL: https://issues.apache.org/jira/browse/HIVE-629
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.629.1.patch
>
>
> Currently, hive concatenates files only if the job under consideration is a map-only job.
> I got some requests from some users, where they want this behavior for map-reduce jobs also - it may not be a good idea to turn it on by default.
> But, we should provide an option to the user where the concatenation can happen even for map-reduce jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.