You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by kevin <ke...@gmail.com> on 2015/05/25 15:32:20 UTC
Implement a http service data source plugin
Hi,
I'm working on implementing a new data source plugin (StoragePlugin), to query data from a http service which response json data. At right now, i can query like this:
select * from http.`q=avi&p=2` where count > 100
As you can see, i pass the http service query string by the table name `q=avi&p=2`, and which means the query string is STATIC. But in the real world a dynamic query string based on other query result is more useful. (Am i right ?)
If i want to support dynamic query string, does that mean i must implement StoragePluginOptimizerRule ? (which mongo storage plugin implemented, to transform query to mongodb query)
Say how about my query like this:
select * from http.dummy where reservsed_q='q=avi&p=2' and count > 100
I make a `reserved_q` to store the http query, and this string can be dynamic. And in my StoragePluginOptimizerRule, i must get `reserved_q` value, and consume the expression `reserved_q`.
So my question is, is this a good way (or suggest way) to do ? And can you guys give me some code hints to implement this, since StoragePluginOptimizerRule is a bit complicated to understand ?
Thanks.
Re: Implement a http service data source plugin
Posted by Ted Dunning <te...@gmail.com>.
On Mon, May 25, 2015 at 6:32 AM, kevin <ke...@gmail.com> wrote:
> Say how about my query like this:
>
>
> select * from http.dummy where reservsed_q='q=avi&p=2' and count > 100
>
>
> I make a `reserved_q` to store the http query, and this string can be
> dynamic. And in my StoragePluginOptimizerRule, i must get `reserved_q`
> value, and consume the expression `reserved_q`.
>
This is great stuff!
(but could you use a better name than reserved_q? how about just query?)
It also seems that it would be natural for the query to be represented as a
JSON object rather than URL encoded query args. That would help avoid
quoting bugs and insertion attacks since the JSON to query args could do
careful quoting.
Re: Implement a http service data source plugin
Posted by Kevin Lynx <ke...@gmail.com>.
I tried
select * from `/my/rest/endpoint` where $q = 'avi' and $p = 2 and count
> 0
to make all '$xxx' (starts with character '$') as query string, so i want
to convert the where clause to this:
where count > 0
which means i want to consume "$q='avi' and $p=2". I write my own
FilterBuilder, and PushDownFilterForScan
if (httpFilterBuilder.isAllExpressionsConverted()) {
logger.debug("all expr converted");
call.transformTo(newScanPrel);
} else {
logger.debug("not all expr converted");
call.transformTo(filter.copy(filter.getTraitSet(),
ImmutableList.of((RelNode) newScanPrel)));
}
But drill enters into a infinite loop.
------------------ Original ------------------
*From: * "Jacques Nadeau";<ja...@apache.org>;
*Date: * Tue, May 26, 2015 03:26 AM
*To: * "dev@drill.apache.org"<de...@drill.apache.org>;
*Subject: * Re: Implement a http service data source plugin
What about something like:
use http;
select * from `/my/rest/endpoint` where $q = 'avi' and $p = 2
Then in the storage plugin, we treat all $[name] values as virtual columns
that translate into query parameters?
On Mon, May 25, 2015 at 6:32 AM, kevin <ke...@gmail.com> wrote:
> Hi,
> I'm working on implementing a new data source plugin (StoragePlugin),
> to query data from a http service which response json data. At right now,
i
> can query like this:
>
>
> select * from http.`q=avi&p=2` where count > 100
>
>
> As you can see, i pass the http service query string by the table name
> `q=avi&p=2`, and which means the query string is STATIC. But in the real
> world a dynamic query string based on other query result is more useful.
> (Am i right ?)
> If i want to support dynamic query string, does that mean i must
> implement StoragePluginOptimizerRule ? (which mongo storage plugin
> implemented, to transform query to mongodb query)
> Say how about my query like this:
>
>
> select * from http.dummy where reservsed_q='q=avi&p=2' and count > 100
>
>
> I make a `reserved_q` to store the http query, and this string can be
> dynamic. And in my StoragePluginOptimizerRule, i must get `reserved_q`
> value, and consume the expression `reserved_q`.
>
>
> So my question is, is this a good way (or suggest way) to do ? And can
> you guys give me some code hints to implement this, since
> StoragePluginOptimizerRule is a bit complicated to understand ?
>
>
> Thanks.
2015-05-26 3:26 GMT+08:00 Jacques Nadeau <ja...@apache.org>:
> What about something like:
>
> use http;
> select * from `/my/rest/endpoint` where $q = 'avi' and $p = 2
>
> Then in the storage plugin, we treat all $[name] values as virtual columns
> that translate into query parameters?
>
> On Mon, May 25, 2015 at 6:32 AM, kevin <ke...@gmail.com> wrote:
>
> > Hi,
> > I'm working on implementing a new data source plugin (StoragePlugin),
> > to query data from a http service which response json data. At right
> now, i
> > can query like this:
> >
> >
> > select * from http.`q=avi&p=2` where count > 100
> >
> >
> > As you can see, i pass the http service query string by the table name
> > `q=avi&p=2`, and which means the query string is STATIC. But in the real
> > world a dynamic query string based on other query result is more useful.
> > (Am i right ?)
> > If i want to support dynamic query string, does that mean i must
> > implement StoragePluginOptimizerRule ? (which mongo storage plugin
> > implemented, to transform query to mongodb query)
> > Say how about my query like this:
> >
> >
> > select * from http.dummy where reservsed_q='q=avi&p=2' and count > 100
> >
> >
> > I make a `reserved_q` to store the http query, and this string can be
> > dynamic. And in my StoragePluginOptimizerRule, i must get `reserved_q`
> > value, and consume the expression `reserved_q`.
> >
> >
> > So my question is, is this a good way (or suggest way) to do ? And can
> > you guys give me some code hints to implement this, since
> > StoragePluginOptimizerRule is a bit complicated to understand ?
> >
> >
> > Thanks.
>
Re: Implement a http service data source plugin
Posted by Jacques Nadeau <ja...@apache.org>.
What about something like:
use http;
select * from `/my/rest/endpoint` where $q = 'avi' and $p = 2
Then in the storage plugin, we treat all $[name] values as virtual columns
that translate into query parameters?
On Mon, May 25, 2015 at 6:32 AM, kevin <ke...@gmail.com> wrote:
> Hi,
> I'm working on implementing a new data source plugin (StoragePlugin),
> to query data from a http service which response json data. At right now, i
> can query like this:
>
>
> select * from http.`q=avi&p=2` where count > 100
>
>
> As you can see, i pass the http service query string by the table name
> `q=avi&p=2`, and which means the query string is STATIC. But in the real
> world a dynamic query string based on other query result is more useful.
> (Am i right ?)
> If i want to support dynamic query string, does that mean i must
> implement StoragePluginOptimizerRule ? (which mongo storage plugin
> implemented, to transform query to mongodb query)
> Say how about my query like this:
>
>
> select * from http.dummy where reservsed_q='q=avi&p=2' and count > 100
>
>
> I make a `reserved_q` to store the http query, and this string can be
> dynamic. And in my StoragePluginOptimizerRule, i must get `reserved_q`
> value, and consume the expression `reserved_q`.
>
>
> So my question is, is this a good way (or suggest way) to do ? And can
> you guys give me some code hints to implement this, since
> StoragePluginOptimizerRule is a bit complicated to understand ?
>
>
> Thanks.