You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Ruben de Vries (JIRA)" <ji...@apache.org> on 2012/05/01 14:53:48 UTC
[jira] [Created] (HIVE-2992) JOIN + LATERAL VIEW works, but +
MAPJOIN and no longer get any results
Ruben de Vries created HIVE-2992:
------------------------------------
Summary: JOIN + LATERAL VIEW works, but + MAPJOIN and no longer get any results
Key: HIVE-2992
URL: https://issues.apache.org/jira/browse/HIVE-2992
Project: Hive
Issue Type: Bug
Affects Versions: 0.8.1
Environment: ubuntu - hadoop 0.20.2-cdh3u0 - hive 0.8.1
Reporter: Ruben de Vries
Okay first off; I know JOIN + LATERAL VIEW together isn't working so I moved my JOIN into a subquery and that makes the query work properly
However when I added a MAPJOIN hint for the JOIN in the subquery it will also stop doing the reducer for the main query!
This only happens when there's a LATERAL VIEW in there though, if I remove the LATERAL VIEW then the main query still get's a reducer to do grouping
Here's a gist: https://gist.github.com/2499436 Containing the queries and a PHP script which you can run to execute the test case I'm using, which does;
- setup a database called hive_mapjoin
- setup tables
- load some test data
- do the selects
You'll need the https://github.com/rcongiu/Hive-JSON-Serde/downloads json-serde-1.1-jar-with-dependencies.jar with it though and change the path ;)
I guess looking at the queries you guys can probally figure out a better testcase, but maybe it's helpful :-)
Not sure if this is a bug or me doing something that just isn't supposed to be working, but I can't seem to find any pointers that this wouldn't be supported...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2992) JOIN + LATERAL VIEW works, but +
MAPJOIN and no longer get any results
Posted by "Ruben de Vries (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ruben de Vries updated HIVE-2992:
---------------------------------
Description:
Okay first off; I know JOIN + LATERAL VIEW together isn't working so I moved my JOIN into a subquery and that makes the query work properly
However when I added a MAPJOIN hint for the JOIN in the subquery it will also stop doing the reducer for the main query!
This only happens when there's a LATERAL VIEW in there though, if I remove the LATERAL VIEW then the main query still get's a reducer to do grouping
Here's a gist: https://gist.github.com/2499436 Containing the queries and a PHP script which you can run to execute the test case I'm using, which does;
- setup a database called hive_mapjoin
- setup tables
- load some test data
- do the selects
You'll need the https://github.com/rcongiu/Hive-JSON-Serde/downloads json-serde-1.1-jar-with-dependencies.jar with it though and change the path ;)
I guess looking at the queries you guys can probally figure out a better testcase, but maybe it's helpful :-)
Not sure if this is a bug or me doing something that just isn't supposed to be working, but I can't seem to find any pointers that this wouldn't be supported...
Here's another gist with the plan.xml: https://gist.github.com/2499658
was:
Okay first off; I know JOIN + LATERAL VIEW together isn't working so I moved my JOIN into a subquery and that makes the query work properly
However when I added a MAPJOIN hint for the JOIN in the subquery it will also stop doing the reducer for the main query!
This only happens when there's a LATERAL VIEW in there though, if I remove the LATERAL VIEW then the main query still get's a reducer to do grouping
Here's a gist: https://gist.github.com/2499436 Containing the queries and a PHP script which you can run to execute the test case I'm using, which does;
- setup a database called hive_mapjoin
- setup tables
- load some test data
- do the selects
You'll need the https://github.com/rcongiu/Hive-JSON-Serde/downloads json-serde-1.1-jar-with-dependencies.jar with it though and change the path ;)
I guess looking at the queries you guys can probally figure out a better testcase, but maybe it's helpful :-)
Not sure if this is a bug or me doing something that just isn't supposed to be working, but I can't seem to find any pointers that this wouldn't be supported...
> JOIN + LATERAL VIEW works, but + MAPJOIN and no longer get any results
> ----------------------------------------------------------------------
>
> Key: HIVE-2992
> URL: https://issues.apache.org/jira/browse/HIVE-2992
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Environment: ubuntu - hadoop 0.20.2-cdh3u0 - hive 0.8.1
> Reporter: Ruben de Vries
>
> Okay first off; I know JOIN + LATERAL VIEW together isn't working so I moved my JOIN into a subquery and that makes the query work properly
> However when I added a MAPJOIN hint for the JOIN in the subquery it will also stop doing the reducer for the main query!
> This only happens when there's a LATERAL VIEW in there though, if I remove the LATERAL VIEW then the main query still get's a reducer to do grouping
> Here's a gist: https://gist.github.com/2499436 Containing the queries and a PHP script which you can run to execute the test case I'm using, which does;
> - setup a database called hive_mapjoin
> - setup tables
> - load some test data
> - do the selects
> You'll need the https://github.com/rcongiu/Hive-JSON-Serde/downloads json-serde-1.1-jar-with-dependencies.jar with it though and change the path ;)
> I guess looking at the queries you guys can probally figure out a better testcase, but maybe it's helpful :-)
> Not sure if this is a bug or me doing something that just isn't supposed to be working, but I can't seem to find any pointers that this wouldn't be supported...
> Here's another gist with the plan.xml: https://gist.github.com/2499658
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2992) JOIN + LATERAL VIEW + MAPJOIN fails
to return result (seems to stop halfway through and no longer do the final
reduce part)
Posted by "Ruben de Vries (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410115#comment-13410115 ]
Ruben de Vries commented on HIVE-2992:
--------------------------------------
Both test cases pass using hive-0.9
> JOIN + LATERAL VIEW + MAPJOIN fails to return result (seems to stop halfway through and no longer do the final reduce part)
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-2992
> URL: https://issues.apache.org/jira/browse/HIVE-2992
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Environment: ubuntu - hadoop 0.20.2-cdh3u0 - hive 0.8.1
> Reporter: Ruben de Vries
> Fix For: 0.9.0
>
>
> Okay first off; I know JOIN + LATERAL VIEW together isn't working so I moved my JOIN into a subquery and that makes the query work properly
> However when I added a MAPJOIN hint for the JOIN in the subquery it will also stop doing the reducer for the main query!
> This only happens when there's a LATERAL VIEW in there though, if I remove the LATERAL VIEW then the main query still get's a reducer to do grouping
> Here's a gist: https://gist.github.com/2499436 Containing the queries and a PHP script which you can run to execute the test case I'm using, which does;
> - setup a database called hive_mapjoin
> - setup tables
> - load some test data
> - do the selects
> You'll need the https://github.com/rcongiu/Hive-JSON-Serde/downloads json-serde-1.1-jar-with-dependencies.jar with it though and change the path ;)
> I guess looking at the queries you guys can probally figure out a better testcase, but maybe it's helpful :-)
> Not sure if this is a bug or me doing something that just isn't supposed to be working, but I can't seem to find any pointers that this wouldn't be supported...
> Here's another gist with the plan.xml: https://gist.github.com/2499658
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2992) JOIN + LATERAL VIEW + MAPJOIN fails
to return result (seems to stop halfway through and no longer do the final
reduce part)
Posted by "Navis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406840#comment-13406840 ]
Navis commented on HIVE-2992:
-----------------------------
This seemed to be solved by HIVE-2642. Could you consider using hive-0.9?
> JOIN + LATERAL VIEW + MAPJOIN fails to return result (seems to stop halfway through and no longer do the final reduce part)
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-2992
> URL: https://issues.apache.org/jira/browse/HIVE-2992
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Environment: ubuntu - hadoop 0.20.2-cdh3u0 - hive 0.8.1
> Reporter: Ruben de Vries
>
> Okay first off; I know JOIN + LATERAL VIEW together isn't working so I moved my JOIN into a subquery and that makes the query work properly
> However when I added a MAPJOIN hint for the JOIN in the subquery it will also stop doing the reducer for the main query!
> This only happens when there's a LATERAL VIEW in there though, if I remove the LATERAL VIEW then the main query still get's a reducer to do grouping
> Here's a gist: https://gist.github.com/2499436 Containing the queries and a PHP script which you can run to execute the test case I'm using, which does;
> - setup a database called hive_mapjoin
> - setup tables
> - load some test data
> - do the selects
> You'll need the https://github.com/rcongiu/Hive-JSON-Serde/downloads json-serde-1.1-jar-with-dependencies.jar with it though and change the path ;)
> I guess looking at the queries you guys can probally figure out a better testcase, but maybe it's helpful :-)
> Not sure if this is a bug or me doing something that just isn't supposed to be working, but I can't seem to find any pointers that this wouldn't be supported...
> Here's another gist with the plan.xml: https://gist.github.com/2499658
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2992) JOIN + LATERAL VIEW + MAPJOIN fails
to return result (seems to stop halfway through and no longer do the final
reduce part)
Posted by "Navis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406833#comment-13406833 ]
Navis commented on HIVE-2992:
-----------------------------
HIVE-3226 is one of the bugs which caused your problem. I'm working on it.
> JOIN + LATERAL VIEW + MAPJOIN fails to return result (seems to stop halfway through and no longer do the final reduce part)
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-2992
> URL: https://issues.apache.org/jira/browse/HIVE-2992
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Environment: ubuntu - hadoop 0.20.2-cdh3u0 - hive 0.8.1
> Reporter: Ruben de Vries
>
> Okay first off; I know JOIN + LATERAL VIEW together isn't working so I moved my JOIN into a subquery and that makes the query work properly
> However when I added a MAPJOIN hint for the JOIN in the subquery it will also stop doing the reducer for the main query!
> This only happens when there's a LATERAL VIEW in there though, if I remove the LATERAL VIEW then the main query still get's a reducer to do grouping
> Here's a gist: https://gist.github.com/2499436 Containing the queries and a PHP script which you can run to execute the test case I'm using, which does;
> - setup a database called hive_mapjoin
> - setup tables
> - load some test data
> - do the selects
> You'll need the https://github.com/rcongiu/Hive-JSON-Serde/downloads json-serde-1.1-jar-with-dependencies.jar with it though and change the path ;)
> I guess looking at the queries you guys can probally figure out a better testcase, but maybe it's helpful :-)
> Not sure if this is a bug or me doing something that just isn't supposed to be working, but I can't seem to find any pointers that this wouldn't be supported...
> Here's another gist with the plan.xml: https://gist.github.com/2499658
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2992) JOIN + LATERAL VIEW works, but +
MAPJOIN and no longer get any results
Posted by "Luke Forehand (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405965#comment-13405965 ]
Luke Forehand commented on HIVE-2992:
-------------------------------------
I'm experiencing the same issue, thank you for documenting this! Can someone at least explain why the reduce step is never happening? It seems to simply ignore the LATERAL VIEW part of the query and everything after.
> JOIN + LATERAL VIEW works, but + MAPJOIN and no longer get any results
> ----------------------------------------------------------------------
>
> Key: HIVE-2992
> URL: https://issues.apache.org/jira/browse/HIVE-2992
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Environment: ubuntu - hadoop 0.20.2-cdh3u0 - hive 0.8.1
> Reporter: Ruben de Vries
>
> Okay first off; I know JOIN + LATERAL VIEW together isn't working so I moved my JOIN into a subquery and that makes the query work properly
> However when I added a MAPJOIN hint for the JOIN in the subquery it will also stop doing the reducer for the main query!
> This only happens when there's a LATERAL VIEW in there though, if I remove the LATERAL VIEW then the main query still get's a reducer to do grouping
> Here's a gist: https://gist.github.com/2499436 Containing the queries and a PHP script which you can run to execute the test case I'm using, which does;
> - setup a database called hive_mapjoin
> - setup tables
> - load some test data
> - do the selects
> You'll need the https://github.com/rcongiu/Hive-JSON-Serde/downloads json-serde-1.1-jar-with-dependencies.jar with it though and change the path ;)
> I guess looking at the queries you guys can probally figure out a better testcase, but maybe it's helpful :-)
> Not sure if this is a bug or me doing something that just isn't supposed to be working, but I can't seem to find any pointers that this wouldn't be supported...
> Here's another gist with the plan.xml: https://gist.github.com/2499658
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2992) JOIN + LATERAL VIEW + MAPJOIN fails to
return result (seems to stop halfway through and no longer do the final
reduce part)
Posted by "Ruben de Vries (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ruben de Vries updated HIVE-2992:
---------------------------------
Summary: JOIN + LATERAL VIEW + MAPJOIN fails to return result (seems to stop halfway through and no longer do the final reduce part) (was: JOIN + LATERAL VIEW works, but + MAPJOIN and no longer get any results)
> JOIN + LATERAL VIEW + MAPJOIN fails to return result (seems to stop halfway through and no longer do the final reduce part)
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-2992
> URL: https://issues.apache.org/jira/browse/HIVE-2992
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Environment: ubuntu - hadoop 0.20.2-cdh3u0 - hive 0.8.1
> Reporter: Ruben de Vries
>
> Okay first off; I know JOIN + LATERAL VIEW together isn't working so I moved my JOIN into a subquery and that makes the query work properly
> However when I added a MAPJOIN hint for the JOIN in the subquery it will also stop doing the reducer for the main query!
> This only happens when there's a LATERAL VIEW in there though, if I remove the LATERAL VIEW then the main query still get's a reducer to do grouping
> Here's a gist: https://gist.github.com/2499436 Containing the queries and a PHP script which you can run to execute the test case I'm using, which does;
> - setup a database called hive_mapjoin
> - setup tables
> - load some test data
> - do the selects
> You'll need the https://github.com/rcongiu/Hive-JSON-Serde/downloads json-serde-1.1-jar-with-dependencies.jar with it though and change the path ;)
> I guess looking at the queries you guys can probally figure out a better testcase, but maybe it's helpful :-)
> Not sure if this is a bug or me doing something that just isn't supposed to be working, but I can't seem to find any pointers that this wouldn't be supported...
> Here's another gist with the plan.xml: https://gist.github.com/2499658
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2992) JOIN + LATERAL VIEW works, but +
MAPJOIN and no longer get any results
Posted by "Ruben de Vries (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406355#comment-13406355 ]
Ruben de Vries commented on HIVE-2992:
--------------------------------------
tried, didn't help :(
I added a log (with `-hiveconf hive.root.logger=INFO,console -v`) to the gist
> JOIN + LATERAL VIEW works, but + MAPJOIN and no longer get any results
> ----------------------------------------------------------------------
>
> Key: HIVE-2992
> URL: https://issues.apache.org/jira/browse/HIVE-2992
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Environment: ubuntu - hadoop 0.20.2-cdh3u0 - hive 0.8.1
> Reporter: Ruben de Vries
>
> Okay first off; I know JOIN + LATERAL VIEW together isn't working so I moved my JOIN into a subquery and that makes the query work properly
> However when I added a MAPJOIN hint for the JOIN in the subquery it will also stop doing the reducer for the main query!
> This only happens when there's a LATERAL VIEW in there though, if I remove the LATERAL VIEW then the main query still get's a reducer to do grouping
> Here's a gist: https://gist.github.com/2499436 Containing the queries and a PHP script which you can run to execute the test case I'm using, which does;
> - setup a database called hive_mapjoin
> - setup tables
> - load some test data
> - do the selects
> You'll need the https://github.com/rcongiu/Hive-JSON-Serde/downloads json-serde-1.1-jar-with-dependencies.jar with it though and change the path ;)
> I guess looking at the queries you guys can probally figure out a better testcase, but maybe it's helpful :-)
> Not sure if this is a bug or me doing something that just isn't supposed to be working, but I can't seem to find any pointers that this wouldn't be supported...
> Here's another gist with the plan.xml: https://gist.github.com/2499658
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2992) JOIN + LATERAL VIEW + MAPJOIN fails
to return result (seems to stop halfway through and no longer do the final
reduce part)
Posted by "Ruben de Vries (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410101#comment-13410101 ]
Ruben de Vries commented on HIVE-2992:
--------------------------------------
hmm, I'm using hadoop-0.20.2 and can't really upgrade to higher versions of hadoop atm because of jobs running on the same clusters developed and maintained by other teams.
I do have control over the version of hive however, do you know if 0.9 plays nice with hadoop-0.20.2 ?
Anyway, I'll grab 0.9 for my local development cluster and check if it solves the problem :-)
> JOIN + LATERAL VIEW + MAPJOIN fails to return result (seems to stop halfway through and no longer do the final reduce part)
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-2992
> URL: https://issues.apache.org/jira/browse/HIVE-2992
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Environment: ubuntu - hadoop 0.20.2-cdh3u0 - hive 0.8.1
> Reporter: Ruben de Vries
>
> Okay first off; I know JOIN + LATERAL VIEW together isn't working so I moved my JOIN into a subquery and that makes the query work properly
> However when I added a MAPJOIN hint for the JOIN in the subquery it will also stop doing the reducer for the main query!
> This only happens when there's a LATERAL VIEW in there though, if I remove the LATERAL VIEW then the main query still get's a reducer to do grouping
> Here's a gist: https://gist.github.com/2499436 Containing the queries and a PHP script which you can run to execute the test case I'm using, which does;
> - setup a database called hive_mapjoin
> - setup tables
> - load some test data
> - do the selects
> You'll need the https://github.com/rcongiu/Hive-JSON-Serde/downloads json-serde-1.1-jar-with-dependencies.jar with it though and change the path ;)
> I guess looking at the queries you guys can probally figure out a better testcase, but maybe it's helpful :-)
> Not sure if this is a bug or me doing something that just isn't supposed to be working, but I can't seem to find any pointers that this wouldn't be supported...
> Here's another gist with the plan.xml: https://gist.github.com/2499658
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2992) JOIN + LATERAL VIEW + MAPJOIN fails
to return result (seems to stop halfway through and no longer do the final
reduce part)
Posted by "Ruben de Vries (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406584#comment-13406584 ]
Ruben de Vries commented on HIVE-2992:
--------------------------------------
Here are the queries with their explain: https://gist.github.com/3047871
> JOIN + LATERAL VIEW + MAPJOIN fails to return result (seems to stop halfway through and no longer do the final reduce part)
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-2992
> URL: https://issues.apache.org/jira/browse/HIVE-2992
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Environment: ubuntu - hadoop 0.20.2-cdh3u0 - hive 0.8.1
> Reporter: Ruben de Vries
>
> Okay first off; I know JOIN + LATERAL VIEW together isn't working so I moved my JOIN into a subquery and that makes the query work properly
> However when I added a MAPJOIN hint for the JOIN in the subquery it will also stop doing the reducer for the main query!
> This only happens when there's a LATERAL VIEW in there though, if I remove the LATERAL VIEW then the main query still get's a reducer to do grouping
> Here's a gist: https://gist.github.com/2499436 Containing the queries and a PHP script which you can run to execute the test case I'm using, which does;
> - setup a database called hive_mapjoin
> - setup tables
> - load some test data
> - do the selects
> You'll need the https://github.com/rcongiu/Hive-JSON-Serde/downloads json-serde-1.1-jar-with-dependencies.jar with it though and change the path ;)
> I guess looking at the queries you guys can probally figure out a better testcase, but maybe it's helpful :-)
> Not sure if this is a bug or me doing something that just isn't supposed to be working, but I can't seem to find any pointers that this wouldn't be supported...
> Here's another gist with the plan.xml: https://gist.github.com/2499658
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2992) JOIN + LATERAL VIEW + MAPJOIN fails
to return result (seems to stop halfway through and no longer do the final
reduce part)
Posted by "Ruben de Vries (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406570#comment-13406570 ]
Ruben de Vries commented on HIVE-2992:
--------------------------------------
same script but without using the JSON Serde: https://gist.github.com/3047737
I found another query that was failing, but with the `set hive.optimize.cp=false` that one was fixed, but the one that was failing before is still failing!
And with `set hive.optimize.cp=false` the mapjoin seems to fail (crash)
> JOIN + LATERAL VIEW + MAPJOIN fails to return result (seems to stop halfway through and no longer do the final reduce part)
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-2992
> URL: https://issues.apache.org/jira/browse/HIVE-2992
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Environment: ubuntu - hadoop 0.20.2-cdh3u0 - hive 0.8.1
> Reporter: Ruben de Vries
>
> Okay first off; I know JOIN + LATERAL VIEW together isn't working so I moved my JOIN into a subquery and that makes the query work properly
> However when I added a MAPJOIN hint for the JOIN in the subquery it will also stop doing the reducer for the main query!
> This only happens when there's a LATERAL VIEW in there though, if I remove the LATERAL VIEW then the main query still get's a reducer to do grouping
> Here's a gist: https://gist.github.com/2499436 Containing the queries and a PHP script which you can run to execute the test case I'm using, which does;
> - setup a database called hive_mapjoin
> - setup tables
> - load some test data
> - do the selects
> You'll need the https://github.com/rcongiu/Hive-JSON-Serde/downloads json-serde-1.1-jar-with-dependencies.jar with it though and change the path ;)
> I guess looking at the queries you guys can probally figure out a better testcase, but maybe it's helpful :-)
> Not sure if this is a bug or me doing something that just isn't supposed to be working, but I can't seem to find any pointers that this wouldn't be supported...
> Here's another gist with the plan.xml: https://gist.github.com/2499658
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-2992) JOIN + LATERAL VIEW + MAPJOIN fails
to return result (seems to stop halfway through and no longer do the final
reduce part)
Posted by "Ruben de Vries (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ruben de Vries resolved HIVE-2992.
----------------------------------
Resolution: Fixed
Fix Version/s: 0.9.0
> JOIN + LATERAL VIEW + MAPJOIN fails to return result (seems to stop halfway through and no longer do the final reduce part)
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-2992
> URL: https://issues.apache.org/jira/browse/HIVE-2992
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Environment: ubuntu - hadoop 0.20.2-cdh3u0 - hive 0.8.1
> Reporter: Ruben de Vries
> Fix For: 0.9.0
>
>
> Okay first off; I know JOIN + LATERAL VIEW together isn't working so I moved my JOIN into a subquery and that makes the query work properly
> However when I added a MAPJOIN hint for the JOIN in the subquery it will also stop doing the reducer for the main query!
> This only happens when there's a LATERAL VIEW in there though, if I remove the LATERAL VIEW then the main query still get's a reducer to do grouping
> Here's a gist: https://gist.github.com/2499436 Containing the queries and a PHP script which you can run to execute the test case I'm using, which does;
> - setup a database called hive_mapjoin
> - setup tables
> - load some test data
> - do the selects
> You'll need the https://github.com/rcongiu/Hive-JSON-Serde/downloads json-serde-1.1-jar-with-dependencies.jar with it though and change the path ;)
> I guess looking at the queries you guys can probally figure out a better testcase, but maybe it's helpful :-)
> Not sure if this is a bug or me doing something that just isn't supposed to be working, but I can't seem to find any pointers that this wouldn't be supported...
> Here's another gist with the plan.xml: https://gist.github.com/2499658
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2992) JOIN + LATERAL VIEW works, but +
MAPJOIN and no longer get any results
Posted by "Navis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406338#comment-13406338 ]
Navis commented on HIVE-2992:
-----------------------------
Could you try it again with cp disabled? (set hive.optimize.cp=false;)
> JOIN + LATERAL VIEW works, but + MAPJOIN and no longer get any results
> ----------------------------------------------------------------------
>
> Key: HIVE-2992
> URL: https://issues.apache.org/jira/browse/HIVE-2992
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Environment: ubuntu - hadoop 0.20.2-cdh3u0 - hive 0.8.1
> Reporter: Ruben de Vries
>
> Okay first off; I know JOIN + LATERAL VIEW together isn't working so I moved my JOIN into a subquery and that makes the query work properly
> However when I added a MAPJOIN hint for the JOIN in the subquery it will also stop doing the reducer for the main query!
> This only happens when there's a LATERAL VIEW in there though, if I remove the LATERAL VIEW then the main query still get's a reducer to do grouping
> Here's a gist: https://gist.github.com/2499436 Containing the queries and a PHP script which you can run to execute the test case I'm using, which does;
> - setup a database called hive_mapjoin
> - setup tables
> - load some test data
> - do the selects
> You'll need the https://github.com/rcongiu/Hive-JSON-Serde/downloads json-serde-1.1-jar-with-dependencies.jar with it though and change the path ;)
> I guess looking at the queries you guys can probally figure out a better testcase, but maybe it's helpful :-)
> Not sure if this is a bug or me doing something that just isn't supposed to be working, but I can't seem to find any pointers that this wouldn't be supported...
> Here's another gist with the plan.xml: https://gist.github.com/2499658
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2992) JOIN + LATERAL VIEW + MAPJOIN fails
to return result (seems to stop halfway through and no longer do the final
reduce part)
Posted by "Navis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406550#comment-13406550 ]
Navis commented on HIVE-2992:
-----------------------------
Could you upload explain result for that query? It seemed to be a genMR order issue which was thought to be resolved already.
> JOIN + LATERAL VIEW + MAPJOIN fails to return result (seems to stop halfway through and no longer do the final reduce part)
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-2992
> URL: https://issues.apache.org/jira/browse/HIVE-2992
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.8.1
> Environment: ubuntu - hadoop 0.20.2-cdh3u0 - hive 0.8.1
> Reporter: Ruben de Vries
>
> Okay first off; I know JOIN + LATERAL VIEW together isn't working so I moved my JOIN into a subquery and that makes the query work properly
> However when I added a MAPJOIN hint for the JOIN in the subquery it will also stop doing the reducer for the main query!
> This only happens when there's a LATERAL VIEW in there though, if I remove the LATERAL VIEW then the main query still get's a reducer to do grouping
> Here's a gist: https://gist.github.com/2499436 Containing the queries and a PHP script which you can run to execute the test case I'm using, which does;
> - setup a database called hive_mapjoin
> - setup tables
> - load some test data
> - do the selects
> You'll need the https://github.com/rcongiu/Hive-JSON-Serde/downloads json-serde-1.1-jar-with-dependencies.jar with it though and change the path ;)
> I guess looking at the queries you guys can probally figure out a better testcase, but maybe it's helpful :-)
> Not sure if this is a bug or me doing something that just isn't supposed to be working, but I can't seem to find any pointers that this wouldn't be supported...
> Here's another gist with the plan.xml: https://gist.github.com/2499658
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira