You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Srihari Srinivasan <ss...@thoughtworks.com> on 2013/08/07 15:00:03 UTC
Next steps on the REST API
Hi Jacques,
Summarizing a couple of questions and changes needed.
- Is there any overhead to instantiating a new DrillClient, connecting
to a drillbit and closing the connection per http request? Jersey creates a
new instance of the Resource class for each request and the DrillClient
will get instantiated per request unless we make it a static variable.
- I am assuming that the DrillClient which will be instantiated from the
http layer will always connect to the co-located/localhost Drillbit
With respect to the DrillClient we'll need the following methods to support
the async style interaction. If we agree to supporting this design I'll
create a JIRA for this -
- new DrillClient().submitQuery(QueryType, Query String) that returns
the *QueryId* *and continues to execute in the background*
- new DrillClient().getStatus(queryId) that returns a Status object. It
could just return *IN_PROGRESS, SUCCEEDED and FAILED* for now. And some
way to know why it failed in case it did.
- new DrillClient().getResults(queryId) that returns
*QueryResultBatch*as it is doing now.
While the api I've speced out above is the ideal one I guess this will
require more time and effort to implement it as suggested. Especially for
the getStatus and getResults methods as it implies that the state of a
query's lifecycle will be held/persisted by someone. Is the server already
maintaining some state for tracking the lifecycle of a query which can be
exposed?
Hari
Re: Next steps on the REST API
Posted by Jacques Nadeau <ja...@apache.org>.
> You probably should avoid this. It will increase latency as the
> DrillClient is a treadsafe resource that uses ZooKeeper to find
> Drillbits.
The DrillClient is not only treadsafe, it is also threadsafe. My
suggestion would be to maintain a singleton and inject the reference
to it as particular requests come in. The start up and shut it down
could be managed through servlet lifecycle or similar. We're using
Jackson's @JacksonInject for this other places and I'd suggest you
enable Jackson within your http engine and use the same methods there
to minimize different JSON parsers and injection approaches.
J
On Thu, Aug 8, 2013 at 11:03 AM, Jacques Nadeau <ja...@apache.org> wrote:
>> Any thoughts on this?
>
> Sorry, you response was sitting in a draft...
>
>> - Is there any overhead to instantiating a new DrillClient, connecting
>> to a drillbit and closing the connection per http request? Jersey creates a
>> new instance of the Resource class for each request and the DrillClient
>> will get instantiated per request unless we make it a static variable.
>
>
>> - I am assuming that the DrillClient which will be instantiated from the
>> http layer will always connect to the co-located/localhost Drillbit
>
> I don't think you need to assume this. It may run locally or on a
> different node. DrillClient should figure things out and make sure
> that everything runs well.
>
>>
>> With respect to the DrillClient we'll need the following methods to support
>> the async style interaction. If we agree to supporting this design I'll
>> create a JIRA for this -
>>
>> - new DrillClient().submitQuery(QueryType, Query String) that returns
>> the *QueryId* *and continues to execute in the background*
>> - new DrillClient().getStatus(queryId) that returns a Status object. It
>> could just return *IN_PROGRESS, SUCCEEDED and FAILED* for now. And some
>> way to know why it failed in case it did.
>> - new DrillClient().getResults(queryId) that returns
>> *QueryResultBatch*as it is doing now.
>>
>
> I think these make sense. Parts of this are already stubbed out in
> the interface. I don't want to maintain this kind of state on the
> server but within a particular approach to the Drill client is fine.
> I think this would be helpful for multiple consumers. Put together a
> JIRA and we can work through the details further.
>
> One note that we can work out there: I think a paginated or continue
> interface might be better since the results come back in chunks.
>
> thanks,
> Jacques
>
>> While the api I've speced out above is the ideal one I guess this will
>> require more time and effort to implement it as suggested. Especially for
>> the getStatus and getResults methods as it implies that the state of a
>> query's lifecycle will be held/persisted by someone. Is the server already
>> maintaining some state for tracking the lifecycle of a query which can be
>> exposed?
>>
>> Hari
Re: Next steps on the REST API
Posted by Srihari Srinivasan <ss...@thoughtworks.com>.
Hi Jacques,
I've taken the update. Can you point me to the class which has the relevant
changes?
-- Hari
On Fri, Aug 23, 2013 at 6:30 AM, Jacques Nadeau <ja...@apache.org> wrote:
> We'll implement in the DrillClient class soon. I just figured using a fake
> implementation until we get in would allow you to make progress
>
>
>
> On Tue, Aug 20, 2013 at 12:47 AM, Srihari Srinivasan <
> ssriniva@thoughtworks.com> wrote:
>
> > I was thinking on similar lines. I was considering wrapping the
> DrillClient
> > up in a class that provided async semantics.
> >
> > On Tue, Aug 20, 2013 at 7:49 AM, Jacques Nadeau <ja...@apache.org>
> > wrote:
> >
> > > Haven't gotten to it yet. I suggest you create a test version that
> does
> > > what you want it to do and code against that so you aren't blocked. How
> > > does that sound?
> > >
> > > Jacques
> > > On Aug 19, 2013 6:54 AM, "Srihari Srinivasan" <
> > ssriniva@thoughtworks.com>
> > > wrote:
> > >
> > > > Folks,
> > > >
> > > > Any news on this feature? Is there a version of DrillClient I can
> start
> > > > working with?
> > > >
> > > > Hari
> > > >
> > > > On Fri, Aug 9, 2013 at 6:19 PM, Srihari Srinivasan <
> > > > ssriniva@thoughtworks.com> wrote:
> > > >
> > > > > Jacques,
> > > > >
> > > > > - I've created this JIRA for the DrillClient changes<
> > > > https://issues.apache.org/jira/browse/DRILL-164>.
> > > > > We could discuss the design further on JIRA too.
> > > > > - For the Singleton solution I've created a class called
> > > > > DrillClientProxy that wraps a single DrillClient instance.
> > > > > - As far as the localhost assumption goes what I am taking away
> is
> > > the
> > > > > DrillClient will be able to figure things out itself with the
> > > > configuration
> > > > > its been given. The http layer need not do anything special.
> > > > >
> > > > > Hari
> > > > >
> > > > > On Thu, Aug 8, 2013 at 11:33 PM, Jacques Nadeau <
> jacques@apache.org
> > > > >wrote:
> > > > >
> > > > >> > Any thoughts on this?
> > > > >>
> > > > >> Sorry, you response was sitting in a draft...
> > > > >>
> > > > >> > - Is there any overhead to instantiating a new DrillClient,
> > > > >> connecting
> > > > >> > to a drillbit and closing the connection per http request?
> > Jersey
> > > > >> creates a
> > > > >> > new instance of the Resource class for each request and the
> > > > >> DrillClient
> > > > >> > will get instantiated per request unless we make it a static
> > > > >> variable.
> > > > >>
> > > > >> You probably should avoid this. It will increase latency as the
> > > > >> DrillClient is a treadsafe resource that uses ZooKeeper to find
> > > > >> Drillbits.
> > > > >>
> > > > >> > - I am assuming that the DrillClient which will be
> instantiated
> > > > from
> > > > >> the
> > > > >> > http layer will always connect to the co-located/localhost
> > > Drillbit
> > > > >>
> > > > >> I don't think you need to assume this. It may run locally or on a
> > > > >> different node. DrillClient should figure things out and make
> sure
> > > > >> that everything runs well.
> > > > >>
> > > > >> >
> > > > >> > With respect to the DrillClient we'll need the following methods
> > to
> > > > >> support
> > > > >> > the async style interaction. If we agree to supporting this
> design
> > > > I'll
> > > > >> > create a JIRA for this -
> > > > >> >
> > > > >> > - new DrillClient().submitQuery(QueryType, Query String) that
> > > > returns
> > > > >> > the *QueryId* *and continues to execute in the background*
> > > > >> > - new DrillClient().getStatus(queryId) that returns a Status
> > > > object.
> > > > >> It
> > > > >> > could just return *IN_PROGRESS, SUCCEEDED and FAILED* for
> now.
> > > And
> > > > >> some
> > > > >> > way to know why it failed in case it did.
> > > > >> > - new DrillClient().getResults(queryId) that returns
> > > > >> > *QueryResultBatch*as it is doing now.
> > > > >> >
> > > > >>
> > > > >> I think these make sense. Parts of this are already stubbed out
> in
> > > > >> the interface. I don't want to maintain this kind of state on the
> > > > >> server but within a particular approach to the Drill client is
> fine.
> > > > >> I think this would be helpful for multiple consumers. Put
> together
> > a
> > > > >> JIRA and we can work through the details further.
> > > > >>
> > > > >> One note that we can work out there: I think a paginated or
> continue
> > > > >> interface might be better since the results come back in chunks.
> > > > >>
> > > > >> thanks,
> > > > >> Jacques
> > > > >>
> > > > >> > While the api I've speced out above is the ideal one I guess
> this
> > > will
> > > > >> > require more time and effort to implement it as suggested.
> > > Especially
> > > > >> for
> > > > >> > the getStatus and getResults methods as it implies that the
> state
> > > of a
> > > > >> > query's lifecycle will be held/persisted by someone. Is the
> server
> > > > >> already
> > > > >> > maintaining some state for tracking the lifecycle of a query
> which
> > > can
> > > > >> be
> > > > >> > exposed?
> > > > >> >
> > > > >> > Hari
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Next steps on the REST API
Posted by Jacques Nadeau <ja...@apache.org>.
We'll implement in the DrillClient class soon. I just figured using a fake
implementation until we get in would allow you to make progress
On Tue, Aug 20, 2013 at 12:47 AM, Srihari Srinivasan <
ssriniva@thoughtworks.com> wrote:
> I was thinking on similar lines. I was considering wrapping the DrillClient
> up in a class that provided async semantics.
>
> On Tue, Aug 20, 2013 at 7:49 AM, Jacques Nadeau <ja...@apache.org>
> wrote:
>
> > Haven't gotten to it yet. I suggest you create a test version that does
> > what you want it to do and code against that so you aren't blocked. How
> > does that sound?
> >
> > Jacques
> > On Aug 19, 2013 6:54 AM, "Srihari Srinivasan" <
> ssriniva@thoughtworks.com>
> > wrote:
> >
> > > Folks,
> > >
> > > Any news on this feature? Is there a version of DrillClient I can start
> > > working with?
> > >
> > > Hari
> > >
> > > On Fri, Aug 9, 2013 at 6:19 PM, Srihari Srinivasan <
> > > ssriniva@thoughtworks.com> wrote:
> > >
> > > > Jacques,
> > > >
> > > > - I've created this JIRA for the DrillClient changes<
> > > https://issues.apache.org/jira/browse/DRILL-164>.
> > > > We could discuss the design further on JIRA too.
> > > > - For the Singleton solution I've created a class called
> > > > DrillClientProxy that wraps a single DrillClient instance.
> > > > - As far as the localhost assumption goes what I am taking away is
> > the
> > > > DrillClient will be able to figure things out itself with the
> > > configuration
> > > > its been given. The http layer need not do anything special.
> > > >
> > > > Hari
> > > >
> > > > On Thu, Aug 8, 2013 at 11:33 PM, Jacques Nadeau <jacques@apache.org
> > > >wrote:
> > > >
> > > >> > Any thoughts on this?
> > > >>
> > > >> Sorry, you response was sitting in a draft...
> > > >>
> > > >> > - Is there any overhead to instantiating a new DrillClient,
> > > >> connecting
> > > >> > to a drillbit and closing the connection per http request?
> Jersey
> > > >> creates a
> > > >> > new instance of the Resource class for each request and the
> > > >> DrillClient
> > > >> > will get instantiated per request unless we make it a static
> > > >> variable.
> > > >>
> > > >> You probably should avoid this. It will increase latency as the
> > > >> DrillClient is a treadsafe resource that uses ZooKeeper to find
> > > >> Drillbits.
> > > >>
> > > >> > - I am assuming that the DrillClient which will be instantiated
> > > from
> > > >> the
> > > >> > http layer will always connect to the co-located/localhost
> > Drillbit
> > > >>
> > > >> I don't think you need to assume this. It may run locally or on a
> > > >> different node. DrillClient should figure things out and make sure
> > > >> that everything runs well.
> > > >>
> > > >> >
> > > >> > With respect to the DrillClient we'll need the following methods
> to
> > > >> support
> > > >> > the async style interaction. If we agree to supporting this design
> > > I'll
> > > >> > create a JIRA for this -
> > > >> >
> > > >> > - new DrillClient().submitQuery(QueryType, Query String) that
> > > returns
> > > >> > the *QueryId* *and continues to execute in the background*
> > > >> > - new DrillClient().getStatus(queryId) that returns a Status
> > > object.
> > > >> It
> > > >> > could just return *IN_PROGRESS, SUCCEEDED and FAILED* for now.
> > And
> > > >> some
> > > >> > way to know why it failed in case it did.
> > > >> > - new DrillClient().getResults(queryId) that returns
> > > >> > *QueryResultBatch*as it is doing now.
> > > >> >
> > > >>
> > > >> I think these make sense. Parts of this are already stubbed out in
> > > >> the interface. I don't want to maintain this kind of state on the
> > > >> server but within a particular approach to the Drill client is fine.
> > > >> I think this would be helpful for multiple consumers. Put together
> a
> > > >> JIRA and we can work through the details further.
> > > >>
> > > >> One note that we can work out there: I think a paginated or continue
> > > >> interface might be better since the results come back in chunks.
> > > >>
> > > >> thanks,
> > > >> Jacques
> > > >>
> > > >> > While the api I've speced out above is the ideal one I guess this
> > will
> > > >> > require more time and effort to implement it as suggested.
> > Especially
> > > >> for
> > > >> > the getStatus and getResults methods as it implies that the state
> > of a
> > > >> > query's lifecycle will be held/persisted by someone. Is the server
> > > >> already
> > > >> > maintaining some state for tracking the lifecycle of a query which
> > can
> > > >> be
> > > >> > exposed?
> > > >> >
> > > >> > Hari
> > > >>
> > > >
> > > >
> > >
> >
>
Re: Next steps on the REST API
Posted by Srihari Srinivasan <ss...@thoughtworks.com>.
I was thinking on similar lines. I was considering wrapping the DrillClient
up in a class that provided async semantics.
On Tue, Aug 20, 2013 at 7:49 AM, Jacques Nadeau <ja...@apache.org> wrote:
> Haven't gotten to it yet. I suggest you create a test version that does
> what you want it to do and code against that so you aren't blocked. How
> does that sound?
>
> Jacques
> On Aug 19, 2013 6:54 AM, "Srihari Srinivasan" <ss...@thoughtworks.com>
> wrote:
>
> > Folks,
> >
> > Any news on this feature? Is there a version of DrillClient I can start
> > working with?
> >
> > Hari
> >
> > On Fri, Aug 9, 2013 at 6:19 PM, Srihari Srinivasan <
> > ssriniva@thoughtworks.com> wrote:
> >
> > > Jacques,
> > >
> > > - I've created this JIRA for the DrillClient changes<
> > https://issues.apache.org/jira/browse/DRILL-164>.
> > > We could discuss the design further on JIRA too.
> > > - For the Singleton solution I've created a class called
> > > DrillClientProxy that wraps a single DrillClient instance.
> > > - As far as the localhost assumption goes what I am taking away is
> the
> > > DrillClient will be able to figure things out itself with the
> > configuration
> > > its been given. The http layer need not do anything special.
> > >
> > > Hari
> > >
> > > On Thu, Aug 8, 2013 at 11:33 PM, Jacques Nadeau <jacques@apache.org
> > >wrote:
> > >
> > >> > Any thoughts on this?
> > >>
> > >> Sorry, you response was sitting in a draft...
> > >>
> > >> > - Is there any overhead to instantiating a new DrillClient,
> > >> connecting
> > >> > to a drillbit and closing the connection per http request? Jersey
> > >> creates a
> > >> > new instance of the Resource class for each request and the
> > >> DrillClient
> > >> > will get instantiated per request unless we make it a static
> > >> variable.
> > >>
> > >> You probably should avoid this. It will increase latency as the
> > >> DrillClient is a treadsafe resource that uses ZooKeeper to find
> > >> Drillbits.
> > >>
> > >> > - I am assuming that the DrillClient which will be instantiated
> > from
> > >> the
> > >> > http layer will always connect to the co-located/localhost
> Drillbit
> > >>
> > >> I don't think you need to assume this. It may run locally or on a
> > >> different node. DrillClient should figure things out and make sure
> > >> that everything runs well.
> > >>
> > >> >
> > >> > With respect to the DrillClient we'll need the following methods to
> > >> support
> > >> > the async style interaction. If we agree to supporting this design
> > I'll
> > >> > create a JIRA for this -
> > >> >
> > >> > - new DrillClient().submitQuery(QueryType, Query String) that
> > returns
> > >> > the *QueryId* *and continues to execute in the background*
> > >> > - new DrillClient().getStatus(queryId) that returns a Status
> > object.
> > >> It
> > >> > could just return *IN_PROGRESS, SUCCEEDED and FAILED* for now.
> And
> > >> some
> > >> > way to know why it failed in case it did.
> > >> > - new DrillClient().getResults(queryId) that returns
> > >> > *QueryResultBatch*as it is doing now.
> > >> >
> > >>
> > >> I think these make sense. Parts of this are already stubbed out in
> > >> the interface. I don't want to maintain this kind of state on the
> > >> server but within a particular approach to the Drill client is fine.
> > >> I think this would be helpful for multiple consumers. Put together a
> > >> JIRA and we can work through the details further.
> > >>
> > >> One note that we can work out there: I think a paginated or continue
> > >> interface might be better since the results come back in chunks.
> > >>
> > >> thanks,
> > >> Jacques
> > >>
> > >> > While the api I've speced out above is the ideal one I guess this
> will
> > >> > require more time and effort to implement it as suggested.
> Especially
> > >> for
> > >> > the getStatus and getResults methods as it implies that the state
> of a
> > >> > query's lifecycle will be held/persisted by someone. Is the server
> > >> already
> > >> > maintaining some state for tracking the lifecycle of a query which
> can
> > >> be
> > >> > exposed?
> > >> >
> > >> > Hari
> > >>
> > >
> > >
> >
>
Re: Next steps on the REST API
Posted by Jacques Nadeau <ja...@apache.org>.
Haven't gotten to it yet. I suggest you create a test version that does
what you want it to do and code against that so you aren't blocked. How
does that sound?
Jacques
On Aug 19, 2013 6:54 AM, "Srihari Srinivasan" <ss...@thoughtworks.com>
wrote:
> Folks,
>
> Any news on this feature? Is there a version of DrillClient I can start
> working with?
>
> Hari
>
> On Fri, Aug 9, 2013 at 6:19 PM, Srihari Srinivasan <
> ssriniva@thoughtworks.com> wrote:
>
> > Jacques,
> >
> > - I've created this JIRA for the DrillClient changes<
> https://issues.apache.org/jira/browse/DRILL-164>.
> > We could discuss the design further on JIRA too.
> > - For the Singleton solution I've created a class called
> > DrillClientProxy that wraps a single DrillClient instance.
> > - As far as the localhost assumption goes what I am taking away is the
> > DrillClient will be able to figure things out itself with the
> configuration
> > its been given. The http layer need not do anything special.
> >
> > Hari
> >
> > On Thu, Aug 8, 2013 at 11:33 PM, Jacques Nadeau <jacques@apache.org
> >wrote:
> >
> >> > Any thoughts on this?
> >>
> >> Sorry, you response was sitting in a draft...
> >>
> >> > - Is there any overhead to instantiating a new DrillClient,
> >> connecting
> >> > to a drillbit and closing the connection per http request? Jersey
> >> creates a
> >> > new instance of the Resource class for each request and the
> >> DrillClient
> >> > will get instantiated per request unless we make it a static
> >> variable.
> >>
> >> You probably should avoid this. It will increase latency as the
> >> DrillClient is a treadsafe resource that uses ZooKeeper to find
> >> Drillbits.
> >>
> >> > - I am assuming that the DrillClient which will be instantiated
> from
> >> the
> >> > http layer will always connect to the co-located/localhost Drillbit
> >>
> >> I don't think you need to assume this. It may run locally or on a
> >> different node. DrillClient should figure things out and make sure
> >> that everything runs well.
> >>
> >> >
> >> > With respect to the DrillClient we'll need the following methods to
> >> support
> >> > the async style interaction. If we agree to supporting this design
> I'll
> >> > create a JIRA for this -
> >> >
> >> > - new DrillClient().submitQuery(QueryType, Query String) that
> returns
> >> > the *QueryId* *and continues to execute in the background*
> >> > - new DrillClient().getStatus(queryId) that returns a Status
> object.
> >> It
> >> > could just return *IN_PROGRESS, SUCCEEDED and FAILED* for now. And
> >> some
> >> > way to know why it failed in case it did.
> >> > - new DrillClient().getResults(queryId) that returns
> >> > *QueryResultBatch*as it is doing now.
> >> >
> >>
> >> I think these make sense. Parts of this are already stubbed out in
> >> the interface. I don't want to maintain this kind of state on the
> >> server but within a particular approach to the Drill client is fine.
> >> I think this would be helpful for multiple consumers. Put together a
> >> JIRA and we can work through the details further.
> >>
> >> One note that we can work out there: I think a paginated or continue
> >> interface might be better since the results come back in chunks.
> >>
> >> thanks,
> >> Jacques
> >>
> >> > While the api I've speced out above is the ideal one I guess this will
> >> > require more time and effort to implement it as suggested. Especially
> >> for
> >> > the getStatus and getResults methods as it implies that the state of a
> >> > query's lifecycle will be held/persisted by someone. Is the server
> >> already
> >> > maintaining some state for tracking the lifecycle of a query which can
> >> be
> >> > exposed?
> >> >
> >> > Hari
> >>
> >
> >
>
Re: Next steps on the REST API
Posted by Srihari Srinivasan <ss...@thoughtworks.com>.
Folks,
Any news on this feature? Is there a version of DrillClient I can start
working with?
Hari
On Fri, Aug 9, 2013 at 6:19 PM, Srihari Srinivasan <
ssriniva@thoughtworks.com> wrote:
> Jacques,
>
> - I've created this JIRA for the DrillClient changes<https://issues.apache.org/jira/browse/DRILL-164>.
> We could discuss the design further on JIRA too.
> - For the Singleton solution I've created a class called
> DrillClientProxy that wraps a single DrillClient instance.
> - As far as the localhost assumption goes what I am taking away is the
> DrillClient will be able to figure things out itself with the configuration
> its been given. The http layer need not do anything special.
>
> Hari
>
> On Thu, Aug 8, 2013 at 11:33 PM, Jacques Nadeau <ja...@apache.org>wrote:
>
>> > Any thoughts on this?
>>
>> Sorry, you response was sitting in a draft...
>>
>> > - Is there any overhead to instantiating a new DrillClient,
>> connecting
>> > to a drillbit and closing the connection per http request? Jersey
>> creates a
>> > new instance of the Resource class for each request and the
>> DrillClient
>> > will get instantiated per request unless we make it a static
>> variable.
>>
>> You probably should avoid this. It will increase latency as the
>> DrillClient is a treadsafe resource that uses ZooKeeper to find
>> Drillbits.
>>
>> > - I am assuming that the DrillClient which will be instantiated from
>> the
>> > http layer will always connect to the co-located/localhost Drillbit
>>
>> I don't think you need to assume this. It may run locally or on a
>> different node. DrillClient should figure things out and make sure
>> that everything runs well.
>>
>> >
>> > With respect to the DrillClient we'll need the following methods to
>> support
>> > the async style interaction. If we agree to supporting this design I'll
>> > create a JIRA for this -
>> >
>> > - new DrillClient().submitQuery(QueryType, Query String) that returns
>> > the *QueryId* *and continues to execute in the background*
>> > - new DrillClient().getStatus(queryId) that returns a Status object.
>> It
>> > could just return *IN_PROGRESS, SUCCEEDED and FAILED* for now. And
>> some
>> > way to know why it failed in case it did.
>> > - new DrillClient().getResults(queryId) that returns
>> > *QueryResultBatch*as it is doing now.
>> >
>>
>> I think these make sense. Parts of this are already stubbed out in
>> the interface. I don't want to maintain this kind of state on the
>> server but within a particular approach to the Drill client is fine.
>> I think this would be helpful for multiple consumers. Put together a
>> JIRA and we can work through the details further.
>>
>> One note that we can work out there: I think a paginated or continue
>> interface might be better since the results come back in chunks.
>>
>> thanks,
>> Jacques
>>
>> > While the api I've speced out above is the ideal one I guess this will
>> > require more time and effort to implement it as suggested. Especially
>> for
>> > the getStatus and getResults methods as it implies that the state of a
>> > query's lifecycle will be held/persisted by someone. Is the server
>> already
>> > maintaining some state for tracking the lifecycle of a query which can
>> be
>> > exposed?
>> >
>> > Hari
>>
>
>
Re: Next steps on the REST API
Posted by Srihari Srinivasan <ss...@thoughtworks.com>.
Jacques,
- I've created this JIRA for the DrillClient
changes<https://issues.apache.org/jira/browse/DRILL-164>.
We could discuss the design further on JIRA too.
- For the Singleton solution I've created a class called
DrillClientProxy that wraps a single DrillClient instance.
- As far as the localhost assumption goes what I am taking away is the
DrillClient will be able to figure things out itself with the configuration
its been given. The http layer need not do anything special.
Hari
On Thu, Aug 8, 2013 at 11:33 PM, Jacques Nadeau <ja...@apache.org> wrote:
> > Any thoughts on this?
>
> Sorry, you response was sitting in a draft...
>
> > - Is there any overhead to instantiating a new DrillClient, connecting
> > to a drillbit and closing the connection per http request? Jersey
> creates a
> > new instance of the Resource class for each request and the
> DrillClient
> > will get instantiated per request unless we make it a static variable.
>
> You probably should avoid this. It will increase latency as the
> DrillClient is a treadsafe resource that uses ZooKeeper to find
> Drillbits.
>
> > - I am assuming that the DrillClient which will be instantiated from
> the
> > http layer will always connect to the co-located/localhost Drillbit
>
> I don't think you need to assume this. It may run locally or on a
> different node. DrillClient should figure things out and make sure
> that everything runs well.
>
> >
> > With respect to the DrillClient we'll need the following methods to
> support
> > the async style interaction. If we agree to supporting this design I'll
> > create a JIRA for this -
> >
> > - new DrillClient().submitQuery(QueryType, Query String) that returns
> > the *QueryId* *and continues to execute in the background*
> > - new DrillClient().getStatus(queryId) that returns a Status object.
> It
> > could just return *IN_PROGRESS, SUCCEEDED and FAILED* for now. And
> some
> > way to know why it failed in case it did.
> > - new DrillClient().getResults(queryId) that returns
> > *QueryResultBatch*as it is doing now.
> >
>
> I think these make sense. Parts of this are already stubbed out in
> the interface. I don't want to maintain this kind of state on the
> server but within a particular approach to the Drill client is fine.
> I think this would be helpful for multiple consumers. Put together a
> JIRA and we can work through the details further.
>
> One note that we can work out there: I think a paginated or continue
> interface might be better since the results come back in chunks.
>
> thanks,
> Jacques
>
> > While the api I've speced out above is the ideal one I guess this will
> > require more time and effort to implement it as suggested. Especially for
> > the getStatus and getResults methods as it implies that the state of a
> > query's lifecycle will be held/persisted by someone. Is the server
> already
> > maintaining some state for tracking the lifecycle of a query which can be
> > exposed?
> >
> > Hari
>
Re: Next steps on the REST API
Posted by Jacques Nadeau <ja...@apache.org>.
> Any thoughts on this?
Sorry, you response was sitting in a draft...
> - Is there any overhead to instantiating a new DrillClient, connecting
> to a drillbit and closing the connection per http request? Jersey creates a
> new instance of the Resource class for each request and the DrillClient
> will get instantiated per request unless we make it a static variable.
You probably should avoid this. It will increase latency as the
DrillClient is a treadsafe resource that uses ZooKeeper to find
Drillbits.
> - I am assuming that the DrillClient which will be instantiated from the
> http layer will always connect to the co-located/localhost Drillbit
I don't think you need to assume this. It may run locally or on a
different node. DrillClient should figure things out and make sure
that everything runs well.
>
> With respect to the DrillClient we'll need the following methods to support
> the async style interaction. If we agree to supporting this design I'll
> create a JIRA for this -
>
> - new DrillClient().submitQuery(QueryType, Query String) that returns
> the *QueryId* *and continues to execute in the background*
> - new DrillClient().getStatus(queryId) that returns a Status object. It
> could just return *IN_PROGRESS, SUCCEEDED and FAILED* for now. And some
> way to know why it failed in case it did.
> - new DrillClient().getResults(queryId) that returns
> *QueryResultBatch*as it is doing now.
>
I think these make sense. Parts of this are already stubbed out in
the interface. I don't want to maintain this kind of state on the
server but within a particular approach to the Drill client is fine.
I think this would be helpful for multiple consumers. Put together a
JIRA and we can work through the details further.
One note that we can work out there: I think a paginated or continue
interface might be better since the results come back in chunks.
thanks,
Jacques
> While the api I've speced out above is the ideal one I guess this will
> require more time and effort to implement it as suggested. Especially for
> the getStatus and getResults methods as it implies that the state of a
> query's lifecycle will be held/persisted by someone. Is the server already
> maintaining some state for tracking the lifecycle of a query which can be
> exposed?
>
> Hari
Fwd: Next steps on the REST API
Posted by Srihari Srinivasan <ss...@thoughtworks.com>.
Jacques,
Any thoughts on this?
---------- Forwarded message ----------
From: Srihari Srinivasan <ss...@thoughtworks.com>
Date: Wed, Aug 7, 2013 at 6:30 PM
Subject: Next steps on the REST API
To: drill-dev@incubator.apache.org
Hi Jacques,
Summarizing a couple of questions and changes needed.
- Is there any overhead to instantiating a new DrillClient, connecting
to a drillbit and closing the connection per http request? Jersey creates a
new instance of the Resource class for each request and the DrillClient
will get instantiated per request unless we make it a static variable.
- I am assuming that the DrillClient which will be instantiated from the
http layer will always connect to the co-located/localhost Drillbit
With respect to the DrillClient we'll need the following methods to support
the async style interaction. If we agree to supporting this design I'll
create a JIRA for this -
- new DrillClient().submitQuery(QueryType, Query String) that returns
the *QueryId* *and continues to execute in the background*
- new DrillClient().getStatus(queryId) that returns a Status object. It
could just return *IN_PROGRESS, SUCCEEDED and FAILED* for now. And some
way to know why it failed in case it did.
- new DrillClient().getResults(queryId) that returns
*QueryResultBatch*as it is doing now.
While the api I've speced out above is the ideal one I guess this will
require more time and effort to implement it as suggested. Especially for
the getStatus and getResults methods as it implies that the state of a
query's lifecycle will be held/persisted by someone. Is the server already
maintaining some state for tracking the lifecycle of a query which can be
exposed?
Hari