You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Adam Kramer (JIRA)" <ji...@apache.org> on 2009/06/28 00:08:47 UTC
[jira] Created: (HIVE-588) LIMIT n is slower than it needs to be
LIMIT n is slower than it needs to be
-------------------------------------
Key: HIVE-588
URL: https://issues.apache.org/jira/browse/HIVE-588
Project: Hadoop Hive
Issue Type: Improvement
Reporter: Adam Kramer
SELECT a FROM t LIMIT 10;
...simply prints the output of the first 10 lines of the first file in the database. That's good.
However,
SELECT function(a) FROM t LIMIT 10;
appears to send all of t to the mappers, runs the function, and and then returns the first 10 rows from whatever mapper(s) finish first. This is very slow in some cases!
Appropriate behavior for LIMIT would be to use ONE mapper, and to push files from the table into that mapper, and then auto-kill the mapper once it has output 10 rows...just take the first 10 rows and kill the whole task if necessary. On dying, throw some informative error message like, "Dying intentionally; LIMIT has been reached." This should be the case even for TRANSFORMs in the mapper...the TRANSFORM could spit out 20 rows, but once it has split out 10, the whole task should die and the 10 should be returned immediately.
The purpose of LIMIT is not just to have "only one response," but it's also to speed up queries a whole lot. Running the function over the entire table is a big waste.
Obviously, when a reduce step is necessary, the whole table will have to be pushed through mappers and then copied and then sorted--but in those cases, whenever 10 total rows have been output by any reducer(s), at which point all reduce tasks should be killed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-588) LIMIT n is slower than it needs to be
Posted by "Adam Kramer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731263#action_12731263 ]
Adam Kramer commented on HIVE-588:
----------------------------------
This is because * allows you to output whole rows at a time, while specifying columns requires that rows be split and then certain indices returned, hence a map job. That's reasonable, but really, this could be optimized as well for straight-up selects with no transform necessary.
But at least, when any mapper has printed 10 rows, Hive should print those 10 rows and kill the rest of the job.
> LIMIT n is slower than it needs to be
> -------------------------------------
>
> Key: HIVE-588
> URL: https://issues.apache.org/jira/browse/HIVE-588
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Adam Kramer
>
> SELECT a FROM t LIMIT 10;
> ...simply prints the output of the first 10 lines of the first file in the database. That's good.
> However,
> SELECT function(a) FROM t LIMIT 10;
> appears to send all of t to the mappers, runs the function, and and then returns the first 10 rows from whatever mapper(s) finish first. This is very slow in some cases!
> Appropriate behavior for LIMIT would be to use ONE mapper, and to push files from the table into that mapper, and then auto-kill the mapper once it has output 10 rows...just take the first 10 rows and kill the whole task if necessary. On dying, throw some informative error message like, "Dying intentionally; LIMIT has been reached." This should be the case even for TRANSFORMs in the mapper...the TRANSFORM could spit out 20 rows, but once it has split out 10, the whole task should die and the 10 should be returned immediately.
> The purpose of LIMIT is not just to have "only one response," but it's also to speed up queries a whole lot. Running the function over the entire table is a big waste.
> Obviously, when a reduce step is necessary, the whole table will have to be pushed through mappers and then copied and then sorted--but in those cases, whenever 10 total rows have been output by any reducer(s), at which point all reduce tasks should be killed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-588) LIMIT n is slower than it needs to be
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731256#action_12731256 ]
Raghotham Murthy commented on HIVE-588:
---------------------------------------
Only queries of the following form dont require a map-reduce job.
SELECT * FROM table WHERE predicate_with_only_partition_columns [LIMIT x]
> LIMIT n is slower than it needs to be
> -------------------------------------
>
> Key: HIVE-588
> URL: https://issues.apache.org/jira/browse/HIVE-588
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Adam Kramer
>
> SELECT a FROM t LIMIT 10;
> ...simply prints the output of the first 10 lines of the first file in the database. That's good.
> However,
> SELECT function(a) FROM t LIMIT 10;
> appears to send all of t to the mappers, runs the function, and and then returns the first 10 rows from whatever mapper(s) finish first. This is very slow in some cases!
> Appropriate behavior for LIMIT would be to use ONE mapper, and to push files from the table into that mapper, and then auto-kill the mapper once it has output 10 rows...just take the first 10 rows and kill the whole task if necessary. On dying, throw some informative error message like, "Dying intentionally; LIMIT has been reached." This should be the case even for TRANSFORMs in the mapper...the TRANSFORM could spit out 20 rows, but once it has split out 10, the whole task should die and the 10 should be returned immediately.
> The purpose of LIMIT is not just to have "only one response," but it's also to speed up queries a whole lot. Running the function over the entire table is a big waste.
> Obviously, when a reduce step is necessary, the whole table will have to be pushed through mappers and then copied and then sorted--but in those cases, whenever 10 total rows have been output by any reducer(s), at which point all reduce tasks should be killed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-588) LIMIT n is slower than it needs to be
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731253#action_12731253 ]
He Yongqiang commented on HIVE-588:
-----------------------------------
>>SELECT a FROM t LIMIT 10;
>>...simply prints the output of the first 10 lines of the first file in the database. That's good.
It seems this also requires a map-reduce job to scan the table.
> LIMIT n is slower than it needs to be
> -------------------------------------
>
> Key: HIVE-588
> URL: https://issues.apache.org/jira/browse/HIVE-588
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Adam Kramer
>
> SELECT a FROM t LIMIT 10;
> ...simply prints the output of the first 10 lines of the first file in the database. That's good.
> However,
> SELECT function(a) FROM t LIMIT 10;
> appears to send all of t to the mappers, runs the function, and and then returns the first 10 rows from whatever mapper(s) finish first. This is very slow in some cases!
> Appropriate behavior for LIMIT would be to use ONE mapper, and to push files from the table into that mapper, and then auto-kill the mapper once it has output 10 rows...just take the first 10 rows and kill the whole task if necessary. On dying, throw some informative error message like, "Dying intentionally; LIMIT has been reached." This should be the case even for TRANSFORMs in the mapper...the TRANSFORM could spit out 20 rows, but once it has split out 10, the whole task should die and the 10 should be returned immediately.
> The purpose of LIMIT is not just to have "only one response," but it's also to speed up queries a whole lot. Running the function over the entire table is a big waste.
> Obviously, when a reduce step is necessary, the whole table will have to be pushed through mappers and then copied and then sorted--but in those cases, whenever 10 total rows have been output by any reducer(s), at which point all reduce tasks should be killed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-588) LIMIT n is slower than it needs to be
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731277#action_12731277 ]
He Yongqiang commented on HIVE-588:
-----------------------------------
>>when any mapper has printed 10 rows, Hive should print those 10 rows and kill the rest of the job
it is not easy to implement with current hadoop map-reduce. You can not kill a whole job inside some mapper.
And hive does not know when enough rows have been collected.
Even though hive use a done flag to note the mapper's job is finished, the split scan for that mapper is not stopped. And new kv pairs are fed in to the mapper and are simply discarded when the done flag in mapper is set.
I think a workaroud would be:
1) file an issue in mapreduce to add the ability to let a mapper and thus maprunner terminate itself when some done flag is set
or
2) extend a maprunner in hive to do the same, because issue in map-reduce may need a long time to review, commit, release etc.
Even though the ability of terminating mapper is added, we can not share a global flag among all mappers. In order to do that, i think one potential method would be to narrow the mappers' number and thus a larger split is fed for each mapper (we will only need to touch a small piece of the whole split).
> LIMIT n is slower than it needs to be
> -------------------------------------
>
> Key: HIVE-588
> URL: https://issues.apache.org/jira/browse/HIVE-588
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Adam Kramer
>
> SELECT a FROM t LIMIT 10;
> ...simply prints the output of the first 10 lines of the first file in the database. That's good.
> However,
> SELECT function(a) FROM t LIMIT 10;
> appears to send all of t to the mappers, runs the function, and and then returns the first 10 rows from whatever mapper(s) finish first. This is very slow in some cases!
> Appropriate behavior for LIMIT would be to use ONE mapper, and to push files from the table into that mapper, and then auto-kill the mapper once it has output 10 rows...just take the first 10 rows and kill the whole task if necessary. On dying, throw some informative error message like, "Dying intentionally; LIMIT has been reached." This should be the case even for TRANSFORMs in the mapper...the TRANSFORM could spit out 20 rows, but once it has split out 10, the whole task should die and the 10 should be returned immediately.
> The purpose of LIMIT is not just to have "only one response," but it's also to speed up queries a whole lot. Running the function over the entire table is a big waste.
> Obviously, when a reduce step is necessary, the whole table will have to be pushed through mappers and then copied and then sorted--but in those cases, whenever 10 total rows have been output by any reducer(s), at which point all reduce tasks should be killed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-588) LIMIT n is slower than it needs to be
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain resolved HIVE-588.
-----------------------------
Resolution: Duplicate
Duplicate of http://issues.apache.org/jira/browse/HIVE-908
> LIMIT n is slower than it needs to be
> -------------------------------------
>
> Key: HIVE-588
> URL: https://issues.apache.org/jira/browse/HIVE-588
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Adam Kramer
>
> SELECT a FROM t LIMIT 10;
> ...simply prints the output of the first 10 lines of the first file in the database. That's good.
> However,
> SELECT function(a) FROM t LIMIT 10;
> appears to send all of t to the mappers, runs the function, and and then returns the first 10 rows from whatever mapper(s) finish first. This is very slow in some cases!
> Appropriate behavior for LIMIT would be to use ONE mapper, and to push files from the table into that mapper, and then auto-kill the mapper once it has output 10 rows...just take the first 10 rows and kill the whole task if necessary. On dying, throw some informative error message like, "Dying intentionally; LIMIT has been reached." This should be the case even for TRANSFORMs in the mapper...the TRANSFORM could spit out 20 rows, but once it has split out 10, the whole task should die and the 10 should be returned immediately.
> The purpose of LIMIT is not just to have "only one response," but it's also to speed up queries a whole lot. Running the function over the entire table is a big waste.
> Obviously, when a reduce step is necessary, the whole table will have to be pushed through mappers and then copied and then sorted--but in those cases, whenever 10 total rows have been output by any reducer(s), at which point all reduce tasks should be killed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-588) LIMIT n is slower than it needs to be
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731274#action_12731274 ]
Namit Jain commented on HIVE-588:
---------------------------------
Different mappers dont communicate with each other - so all mappers are stopped after emitting 10 rows.
> LIMIT n is slower than it needs to be
> -------------------------------------
>
> Key: HIVE-588
> URL: https://issues.apache.org/jira/browse/HIVE-588
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Adam Kramer
>
> SELECT a FROM t LIMIT 10;
> ...simply prints the output of the first 10 lines of the first file in the database. That's good.
> However,
> SELECT function(a) FROM t LIMIT 10;
> appears to send all of t to the mappers, runs the function, and and then returns the first 10 rows from whatever mapper(s) finish first. This is very slow in some cases!
> Appropriate behavior for LIMIT would be to use ONE mapper, and to push files from the table into that mapper, and then auto-kill the mapper once it has output 10 rows...just take the first 10 rows and kill the whole task if necessary. On dying, throw some informative error message like, "Dying intentionally; LIMIT has been reached." This should be the case even for TRANSFORMs in the mapper...the TRANSFORM could spit out 20 rows, but once it has split out 10, the whole task should die and the 10 should be returned immediately.
> The purpose of LIMIT is not just to have "only one response," but it's also to speed up queries a whole lot. Running the function over the entire table is a big waste.
> Obviously, when a reduce step is necessary, the whole table will have to be pushed through mappers and then copied and then sorted--but in those cases, whenever 10 total rows have been output by any reducer(s), at which point all reduce tasks should be killed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.