You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Namit Jain (JIRA)" <ji...@apache.org> on 2009/03/03 19:44:56 UTC
[jira] Created: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
[Hive] union all queries broken - all kinds of problems
-------------------------------------------------------
Key: HIVE-318
URL: https://issues.apache.org/jira/browse/HIVE-318
Project: Hadoop Hive
Issue Type: Bug
Components: Query Processor
Reporter: Namit Jain
Assignee: Namit Jain
1. Map-only job : same input
Hangs because mapper tries to same open twice, and hadoop filesystem complains.
Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
2. Map-only job : different inputs
Loss of data due to rename.
Fix: change rename to move files to the directory.
3. Map-only job in subquery + RedSink: works currently
4. 2 variables: so 4 sub-cases
Number of sub-queries having map-reduce jobs. (1/2)
Operator after Union (RS/FS)
a. Number of sub-queries having map-reduce jobs. 1
Operator after Union: RS
Can be done in 2MR - really difficult with current infrastructure.
Should do with 3 MR jobs now - break on top of UNION.
Future optimization: move operators between Union and RS before Union.
b. Number of sub-queries having map-reduce jobs. 2
Operator after Union: RS
Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
Future optimization: move operators between Union and RS before Union.
c. Number of sub-queries having map-reduce jobs. 1
Operator after Union: FS
Can be done in 1MR - really difficult with current infrastructure.
Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
Should do with 3 MR jobs now - break on top of UNION.
Followup optimization: 2MR should be able to handle
d. Number of sub-queries having map-reduce jobs. 2
Operator after Union: FS
Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
Should do with 3 MR jobs now - break on top of UNION.
Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Status: Open (was: Patch Available)
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.318.2.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Status: Patch Available (was: Open)
resolve conflicts
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Priority: Blocker
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.4.patch, hive.318.5.patch, hive.318.6.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Status: Open (was: Patch Available)
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Priority: Blocker
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.4.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680584#action_12680584 ]
Namit Jain commented on HIVE-318:
---------------------------------
Hi Jeff, all the issues are pretty similar, and their fixes would touch the same code. That's why it would be much easier to merge them in a single fix together. Infact, the fix for all sub-cases
for 4 is the same - I just listed all the issues for clarity. The basic idea was that union is broken currently, and I was just wanted to list all the scenarios that are not working.
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Status: Patch Available (was: Open)
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Status: Open (was: Patch Available)
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Priority: Blocker
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.4.patch, hive.318.5.patch, hive.318.6.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-318:
----------------------------
Resolution: Fixed
Fix Version/s: 0.3.0
Release Note: HIVE-318. Fix union all queries. (Namit Jain via zshao)
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed revision 758494.
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Priority: Blocker
> Fix For: 0.3.0
>
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.4.patch, hive.318.5.patch, hive.318.6.patch, hive.318.7.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Attachment: hive.318.5.patch
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Priority: Blocker
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.4.patch, hive.318.5.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680570#action_12680570 ]
Prasad Chakka commented on HIVE-318:
------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java:325
why are you catching an exception that is never thrown?
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:67
shouldn't you throw an error if the file exists instead of asserting?
convert some of the the info log messages to error
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:134, 153
remove the System.out
ql/src/java/org/apache/hadoop/hive/ql/lib/DefaultRuleDispatcher.java:80
can you make stack to be 3rd parameter to the process function since most process function don't use this param
ql/src/java/org/apache/hadoop/hive/ql/plan/fileSinkDesc.java:35 &47
javadoc?
ql/src/java/org/apache/hadoop/hive/ql/plan/unionDesc.java
empty class? add a comment
unit tests:
can you add comments so that which unit test so that it is easy to figure out what case of union it is testing
will add more comments later..
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Status: Patch Available (was: Open)
resolve conflicts
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Priority: Blocker
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.4.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Attachment: hive.318.4.patch
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Priority: Blocker
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.4.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Status: Open (was: Patch Available)
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Priority: Blocker
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.4.patch, hive.318.5.patch, hive.318.6.patch, hive.318.7.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689301#action_12689301 ]
Zheng Shao commented on HIVE-318:
---------------------------------
+1. Testing now.
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Priority: Blocker
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.4.patch, hive.318.5.patch, hive.318.6.patch, hive.318.7.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689291#action_12689291 ]
Namit Jain commented on HIVE-318:
---------------------------------
Added comments in various places and uploaded the new patch.
Also, filed various follow-up jiras
https://issues.apache.org/jira/browse/HIVE-368
https://issues.apache.org/jira/browse/HIVE-369
https://issues.apache.org/jira/browse/HIVE-370
https://issues.apache.org/jira/browse/HIVE-371
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Priority: Blocker
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.4.patch, hive.318.5.patch, hive.318.6.patch, hive.318.7.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Attachment: hive.318.2.patch
incorporate prasad's comments
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.318.2.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Status: Patch Available (was: Open)
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.318.2.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Status: Open (was: Patch Available)
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Priority: Blocker
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.4.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Status: Patch Available (was: Open)
incorporated Prasad's comments
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680938#action_12680938 ]
Namit Jain commented on HIVE-318:
---------------------------------
Will do 2 and 3 in the list above (use indexof and change PreOrderWalker).
However, 1 cannot be done since ColumnPruner creates the tree again - so state of the union will be lost.
It doesnt matter - predicate pushdown can happen anywhere (either before of after the union). Union does not change the tree,
it just stores some state about which branch is map-side and which branch is root
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.318.2.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Priority: Blocker (was: Major)
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Priority: Blocker
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Attachment: hive.318.patch
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Attachment: hive.318.7.patch
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Priority: Blocker
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.4.patch, hive.318.5.patch, hive.318.6.patch, hive.318.7.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Jeff Hammerbacher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12678466#action_12678466 ]
Jeff Hammerbacher commented on HIVE-318:
----------------------------------------
Hey Namit,
Quick comment: would it be easier to track these issues if they were filed as separate tickets? Perhaps you are preparing a solution as a single patch, or they are all manifestations of the same root issue, but otherwise, this seems like a ticket that could be busted up.
Later,
Jeff
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Status: Open (was: Patch Available)
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Status: Patch Available (was: Open)
incorporated comments
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Priority: Blocker
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.4.patch, hive.318.5.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682418#action_12682418 ]
Ashish Thusoo commented on HIVE-318:
------------------------------------
Looked at this in a lot more detail with Namit. The following are the review comments:
1. The state maintained in the union operator context can be moved to the ParseContext to be consistent with the model that we have today.
2. The init and state code can be moved to Operator.java and the reset logic can be refactored to work on those states. There is no need for another reinit state. Init after close should be transparently allowed.
3. We can change the plan to generate two different file sink operators on the parents of the union operator while breaking the into map/reduce jobs. If we follow that strategy, we can undo the changes to FileSinkOperator
and remove the special case code.
4. Please check indentation in UnionProcessor.java
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Priority: Blocker
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.4.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680651#action_12680651 ]
Prasad Chakka commented on HIVE-318:
------------------------------------
- make the union transform before the column pruner. i will make predicate pushdown to happen after union transform so that it can get the benefit of pushing filters up.
- ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcFactory.java:42
use indexOf() instead of for loop to make it simple
- ql/src/java/org/apache/hadoop/hive/ql/lib/PreOrderWalker.java
should subclass the Default walker and override only the walk() method. all the other code is common to both walkers
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.318.2.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681883#action_12681883 ]
Ashish Thusoo commented on HIVE-318:
------------------------------------
There seems to be a conflict in union.q.xml. Can you regenerate the patch.
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Priority: Blocker
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Status: Patch Available (was: Open)
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Priority: Blocker
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.4.patch, hive.318.5.patch, hive.318.6.patch, hive.318.7.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Attachment: hive.318.6.patch
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Priority: Blocker
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.4.patch, hive.318.5.patch, hive.318.6.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-318) [Hive] union all queries broken - all
kinds of problems
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-318:
----------------------------
Attachment: hive.318.3.patch
> [Hive] union all queries broken - all kinds of problems
> -------------------------------------------------------
>
> Key: HIVE-318
> URL: https://issues.apache.org/jira/browse/HIVE-318
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.318.2.patch, hive.318.3.patch, hive.318.patch
>
>
> 1. Map-only job : same input
> Hangs because mapper tries to same open twice, and hadoop filesystem complains.
> Fix: Only initialize once - keep state at the Operator level for the same. Should do same for Close.
> 2. Map-only job : different inputs
> Loss of data due to rename.
> Fix: change rename to move files to the directory.
> 3. Map-only job in subquery + RedSink: works currently
> 4. 2 variables: so 4 sub-cases
> Number of sub-queries having map-reduce jobs. (1/2)
> Operator after Union (RS/FS)
>
> a. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: RS
> Can be done in 2MR - really difficult with current infrastructure.
> Should do with 3 MR jobs now - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> b. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: RS
> Needs 3MR - Should do with 3 MR jobs - break on top of UNION.
> Future optimization: move operators between Union and RS before Union.
> c. Number of sub-queries having map-reduce jobs. 1
> Operator after Union: FS
> Can be done in 1MR - really difficult with current infrastructure.
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
> d. Number of sub-queries having map-reduce jobs. 2
> Operator after Union: FS
> Can be easily done with 2 MR by removing UNION and cloning operators between Union and FS.
> Should do with 3 MR jobs now - break on top of UNION.
> Followup optimization: 2MR should be able to handle
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.