You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Eric Sun (JIRA)" <ji...@apache.org> on 2009/10/19 18:35:59 UTC
[jira] Created: (HIVE-887) Allow SELECT without a mapreduce
job
Allow SELECT <col> without a mapreduce job
------------------------------------------
Key: HIVE-887
URL: https://issues.apache.org/jira/browse/HIVE-887
Project: Hadoop Hive
Issue Type: New Feature
Environment: All
Reporter: Eric Sun
I often find myself needing to take a quick look at a particular column of a Hive table.
I usually do this by doing a
SELECT * from <table> LIMIT 20;
from the CLI. Doing this is pretty fast since it doesn't require a mapreduce job. However, it's tough to examine just 1 or 2 columns when the table is very wide.
So, I might do
SELECT <col> from <table> LIMIT 20;
but it's much slower since it requires a map-reduce. It'd be really convenient if a map-reduce wasn't necessary.
Currently a good work around is to do
hive -e "select * from table" | cut --key=n
but it'd be more convenient if it were built in.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-887) Allow SELECT without a mapreduce
job
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang reassigned HIVE-887:
-------------------------------
Assignee: Ning Zhang
> Allow SELECT <col> without a mapreduce job
> ------------------------------------------
>
> Key: HIVE-887
> URL: https://issues.apache.org/jira/browse/HIVE-887
> Project: Hadoop Hive
> Issue Type: New Feature
> Environment: All
> Reporter: Eric Sun
> Assignee: Ning Zhang
>
> I often find myself needing to take a quick look at a particular column of a Hive table.
> I usually do this by doing a
> SELECT * from <table> LIMIT 20;
> from the CLI. Doing this is pretty fast since it doesn't require a mapreduce job. However, it's tough to examine just 1 or 2 columns when the table is very wide.
> So, I might do
> SELECT <col> from <table> LIMIT 20;
> but it's much slower since it requires a map-reduce. It'd be really convenient if a map-reduce wasn't necessary.
> Currently a good work around is to do
> hive -e "select * from table" | cut --key=n
> but it'd be more convenient if it were built in since it alleviates the need for column counting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-887) Allow SELECT without a mapreduce
job
Posted by "E. Sammer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802089#action_12802089 ]
E. Sammer commented on HIVE-887:
--------------------------------
I would also like this kind of functionality. I would add WHERE clause support to the request though as there are cases where you know a table will be small. What would be really ideal is to be able to define a projected threshold where, if the query execution engine think there may be many rows, it resorts to a MR job, but if under, performs client side fetch and filter. The expectation is that GROUP BY, joins, ORDER / SORT / CLUSTER and related would always cause a MR job.
Ex:
SELECT a, b FROM t WHERE c = 'foo' FETCH n;
where n is an upper limit for which a fetch should be done based on the projected number of rows. If projection is still not yet on the table in Hive (I haven't looked at the internals), maybe FETCH n acts like a fetch + limit operation. Maybe n is simply some global configuration parameter, although that seems too inflexible.
For me, Hive has been excellent for storing raw parsed log data which can be queried into summary tables of around 1 million rows. These summary tables containing aggregations are then queried by a UI for visualization. This "fetch" functionality would allow for the UI load times to go from minutes to seconds and reduce contention for task slots in a production Hadoop cluster.
> Allow SELECT <col> without a mapreduce job
> ------------------------------------------
>
> Key: HIVE-887
> URL: https://issues.apache.org/jira/browse/HIVE-887
> Project: Hadoop Hive
> Issue Type: New Feature
> Environment: All
> Reporter: Eric Sun
> Assignee: Ning Zhang
>
> I often find myself needing to take a quick look at a particular column of a Hive table.
> I usually do this by doing a
> SELECT * from <table> LIMIT 20;
> from the CLI. Doing this is pretty fast since it doesn't require a mapreduce job. However, it's tough to examine just 1 or 2 columns when the table is very wide.
> So, I might do
> SELECT <col> from <table> LIMIT 20;
> but it's much slower since it requires a map-reduce. It'd be really convenient if a map-reduce wasn't necessary.
> Currently a good work around is to do
> hive -e "select * from table" | cut --key=n
> but it'd be more convenient if it were built in since it alleviates the need for column counting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-887) Allow SELECT without a mapreduce
job
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802105#action_12802105 ]
Ning Zhang commented on HIVE-887:
---------------------------------
Eric, we were discussing the same thing before to allow simple where-clause to the non-MR client side execution. This woud be straightforward after we make fetchOperator extends Operator. Then basically any map-only plan (including map-side joins) could be move to non-MR execution.
The only thing I'm not sure about is the "fetch n" syntax. Since we don't have stats, we don't know the number of rows in the input file. So as a first step we can rely on the file size.
> Allow SELECT <col> without a mapreduce job
> ------------------------------------------
>
> Key: HIVE-887
> URL: https://issues.apache.org/jira/browse/HIVE-887
> Project: Hadoop Hive
> Issue Type: New Feature
> Environment: All
> Reporter: Eric Sun
> Assignee: Ning Zhang
>
> I often find myself needing to take a quick look at a particular column of a Hive table.
> I usually do this by doing a
> SELECT * from <table> LIMIT 20;
> from the CLI. Doing this is pretty fast since it doesn't require a mapreduce job. However, it's tough to examine just 1 or 2 columns when the table is very wide.
> So, I might do
> SELECT <col> from <table> LIMIT 20;
> but it's much slower since it requires a map-reduce. It'd be really convenient if a map-reduce wasn't necessary.
> Currently a good work around is to do
> hive -e "select * from table" | cut --key=n
> but it'd be more convenient if it were built in since it alleviates the need for column counting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-887) Allow SELECT without a mapreduce
job
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767442#action_12767442 ]
Ning Zhang commented on HIVE-887:
---------------------------------
After looking more carefully at the code, the fetch task spawn a fetch operator at its initialization, and the fetch operator is not subclass of Operator. So we probably need to extend it to Operator class and make the select Operator as a child of fetch, or push select operator's logic to fetch operator's implementation. Will have a discussion first and put it as a lower priority in my task list for now.
> Allow SELECT <col> without a mapreduce job
> ------------------------------------------
>
> Key: HIVE-887
> URL: https://issues.apache.org/jira/browse/HIVE-887
> Project: Hadoop Hive
> Issue Type: New Feature
> Environment: All
> Reporter: Eric Sun
> Assignee: Ning Zhang
>
> I often find myself needing to take a quick look at a particular column of a Hive table.
> I usually do this by doing a
> SELECT * from <table> LIMIT 20;
> from the CLI. Doing this is pretty fast since it doesn't require a mapreduce job. However, it's tough to examine just 1 or 2 columns when the table is very wide.
> So, I might do
> SELECT <col> from <table> LIMIT 20;
> but it's much slower since it requires a map-reduce. It'd be really convenient if a map-reduce wasn't necessary.
> Currently a good work around is to do
> hive -e "select * from table" | cut --key=n
> but it'd be more convenient if it were built in since it alleviates the need for column counting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-887) Allow SELECT without a mapreduce
job
Posted by "E. Sammer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802311#action_12802311 ]
E. Sammer commented on HIVE-887:
--------------------------------
Ning:
That sounds great. The syntax isn't really important. I was just trying to think of a way of providing a hint to the query execution layer so the user can indicate when they're willing to take the performance hit in favor of a faster initial response. I would think that in many cases you still probably want to exec as a MR job even for small results; this fetch behavior should be a user requested special case, in my opinion.
> Allow SELECT <col> without a mapreduce job
> ------------------------------------------
>
> Key: HIVE-887
> URL: https://issues.apache.org/jira/browse/HIVE-887
> Project: Hadoop Hive
> Issue Type: New Feature
> Environment: All
> Reporter: Eric Sun
> Assignee: Ning Zhang
>
> I often find myself needing to take a quick look at a particular column of a Hive table.
> I usually do this by doing a
> SELECT * from <table> LIMIT 20;
> from the CLI. Doing this is pretty fast since it doesn't require a mapreduce job. However, it's tough to examine just 1 or 2 columns when the table is very wide.
> So, I might do
> SELECT <col> from <table> LIMIT 20;
> but it's much slower since it requires a map-reduce. It'd be really convenient if a map-reduce wasn't necessary.
> Currently a good work around is to do
> hive -e "select * from table" | cut --key=n
> but it'd be more convenient if it were built in since it alleviates the need for column counting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-887) Allow SELECT without a mapreduce
job
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791452#action_12791452 ]
Namit Jain commented on HIVE-887:
---------------------------------
Based on an offline discussion, this seems very important for the users.
> Allow SELECT <col> without a mapreduce job
> ------------------------------------------
>
> Key: HIVE-887
> URL: https://issues.apache.org/jira/browse/HIVE-887
> Project: Hadoop Hive
> Issue Type: New Feature
> Environment: All
> Reporter: Eric Sun
> Assignee: Ning Zhang
>
> I often find myself needing to take a quick look at a particular column of a Hive table.
> I usually do this by doing a
> SELECT * from <table> LIMIT 20;
> from the CLI. Doing this is pretty fast since it doesn't require a mapreduce job. However, it's tough to examine just 1 or 2 columns when the table is very wide.
> So, I might do
> SELECT <col> from <table> LIMIT 20;
> but it's much slower since it requires a map-reduce. It'd be really convenient if a map-reduce wasn't necessary.
> Currently a good work around is to do
> hive -e "select * from table" | cut --key=n
> but it'd be more convenient if it were built in since it alleviates the need for column counting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-887) Allow SELECT without a mapreduce
job
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767379#action_12767379 ]
Ning Zhang commented on HIVE-887:
---------------------------------
Currently the query "select * from T [limit n]" will generate a fetch task rather than a mapred task. Since fetch task is also an operator, it seems possible to add a select operator on top of it for query 'select a, b, expr(c, d) from T [limit n]".
> Allow SELECT <col> without a mapreduce job
> ------------------------------------------
>
> Key: HIVE-887
> URL: https://issues.apache.org/jira/browse/HIVE-887
> Project: Hadoop Hive
> Issue Type: New Feature
> Environment: All
> Reporter: Eric Sun
> Assignee: Ning Zhang
>
> I often find myself needing to take a quick look at a particular column of a Hive table.
> I usually do this by doing a
> SELECT * from <table> LIMIT 20;
> from the CLI. Doing this is pretty fast since it doesn't require a mapreduce job. However, it's tough to examine just 1 or 2 columns when the table is very wide.
> So, I might do
> SELECT <col> from <table> LIMIT 20;
> but it's much slower since it requires a map-reduce. It'd be really convenient if a map-reduce wasn't necessary.
> Currently a good work around is to do
> hive -e "select * from table" | cut --key=n
> but it'd be more convenient if it were built in since it alleviates the need for column counting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-887) Allow SELECT without a mapreduce
job
Posted by "Eric Sun (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Sun updated HIVE-887:
--------------------------
Description:
I often find myself needing to take a quick look at a particular column of a Hive table.
I usually do this by doing a
SELECT * from <table> LIMIT 20;
from the CLI. Doing this is pretty fast since it doesn't require a mapreduce job. However, it's tough to examine just 1 or 2 columns when the table is very wide.
So, I might do
SELECT <col> from <table> LIMIT 20;
but it's much slower since it requires a map-reduce. It'd be really convenient if a map-reduce wasn't necessary.
Currently a good work around is to do
hive -e "select * from table" | cut --key=n
but it'd be more convenient if it were built in since it alleviates the need for column counting.
was:
I often find myself needing to take a quick look at a particular column of a Hive table.
I usually do this by doing a
SELECT * from <table> LIMIT 20;
from the CLI. Doing this is pretty fast since it doesn't require a mapreduce job. However, it's tough to examine just 1 or 2 columns when the table is very wide.
So, I might do
SELECT <col> from <table> LIMIT 20;
but it's much slower since it requires a map-reduce. It'd be really convenient if a map-reduce wasn't necessary.
Currently a good work around is to do
hive -e "select * from table" | cut --key=n
but it'd be more convenient if it were built in.
> Allow SELECT <col> without a mapreduce job
> ------------------------------------------
>
> Key: HIVE-887
> URL: https://issues.apache.org/jira/browse/HIVE-887
> Project: Hadoop Hive
> Issue Type: New Feature
> Environment: All
> Reporter: Eric Sun
>
> I often find myself needing to take a quick look at a particular column of a Hive table.
> I usually do this by doing a
> SELECT * from <table> LIMIT 20;
> from the CLI. Doing this is pretty fast since it doesn't require a mapreduce job. However, it's tough to examine just 1 or 2 columns when the table is very wide.
> So, I might do
> SELECT <col> from <table> LIMIT 20;
> but it's much slower since it requires a map-reduce. It'd be really convenient if a map-reduce wasn't necessary.
> Currently a good work around is to do
> hive -e "select * from table" | cut --key=n
> but it'd be more convenient if it were built in since it alleviates the need for column counting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.