[jira] [Created] (DRILL-60) multiple query support

["Lisen Mu (JIRA)" <ji...@apache.org>]

Lisen Mu created DRILL-60:
-----------------------------

             Summary: multiple query support
                 Key: DRILL-60
                 URL: https://issues.apache.org/jira/browse/DRILL-60
             Project: Apache Drill
          Issue Type: New Feature
            Reporter: Lisen Mu


parse/optimize several SQL(or other QL)s into one logical plan and execute them once.
 
* use case
If a report generating system is used as a frontend system of drill instead of a command line, There would be a set of similar SQL to execute to generate a report(consider each day's DAU statistics with geoip distribution(groupby) for 7 continuous days). If processed individually, plans for SQLs in this kind of set are similar(scan pv log in these days) or same(scan GeoIP table) in majority. If processed together, duplicated scan(GeoIP) could be eliminated to reduce I/O, or similar scan(pv) could be merged to achieve better throughput, and provides chances of other intermediate result reuse.

* possibility
Query in a QL is represented as a tree. Drill models logical plan and physical plan as DAG, which is more expressive than tree. More precisely, a set of trees(with common nodes or common edges) can be union-ed into a dag. So, the current drill logical/physical plan model supports multiple query already. Each sql would have a corresponding StoreOP as its result.

* problems
To do so, physical operators must support multiple iteration over its output. That would have side effect on mem management.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [jira] [Created] (DRILL-60) multiple query support

[Lisen Mu <im...@gmail.com>]

It's not a feature request for current stage of drill, however I would like
to contribute to it as long as we have a working local query. It's very
useful for us, and I thinks it's also an interesting problem.


Thanks,
Lisen



On Mon, May 13, 2013 at 4:21 PM, Lisen Mu (JIRA) <ji...@apache.org> wrote:

> Lisen Mu created DRILL-60:
> -----------------------------
>
>              Summary: multiple query support
>                  Key: DRILL-60
>                  URL: https://issues.apache.org/jira/browse/DRILL-60
>              Project: Apache Drill
>           Issue Type: New Feature
>             Reporter: Lisen Mu
>
>
> parse/optimize several SQL(or other QL)s into one logical plan and execute
> them once.
>
> * use case
> If a report generating system is used as a frontend system of drill
> instead of a command line, There would be a set of similar SQL to execute
> to generate a report(consider each day's DAU statistics with geoip
> distribution(groupby) for 7 continuous days). If processed individually,
> plans for SQLs in this kind of set are similar(scan pv log in these days)
> or same(scan GeoIP table) in majority. If processed together, duplicated
> scan(GeoIP) could be eliminated to reduce I/O, or similar scan(pv) could be
> merged to achieve better throughput, and provides chances of other
> intermediate result reuse.
>
> * possibility
> Query in a QL is represented as a tree. Drill models logical plan and
> physical plan as DAG, which is more expressive than tree. More precisely, a
> set of trees(with common nodes or common edges) can be union-ed into a dag.
> So, the current drill logical/physical plan model supports multiple query
> already. Each sql would have a corresponding StoreOP as its result.
>
> * problems
> To do so, physical operators must support multiple iteration over its
> output. That would have side effect on mem management.
>
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>