You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Spoutable <jo...@spoutable.com> on 2016/01/28 22:02:38 UTC
CTAS and storage format via REST API
We are using the REST API from nodejs and need to issue some create table statements that are a different format from the default for the system (currently Parquet). We cant issue an alter system command to change the store.format globally.
I tried to issue multiple requests with a alter session set `store.format`='json';
and i tried to have multiple statements in the the same request separated by semi colon. Neither of those worked.
As a last resort i looked at the code in QueryWrapper.java and it looks like its just 1 query per request and no session persistence or anything like that going on between requests to the REST api. I tried to follow the code down through the DrillUserPrincipal and DrillClient and UserClient but ran out of patience. It doesnt look like there is any notion of ‘session’ in the sense of ‘alter session’ statments.
Does anybody have a workaround for this?
If not, does anybody have a suggestion on what change I could make for supporting this in a way that would be acceptable to the project?
Thanks in advance.
Re: CTAS and storage format via REST API
Posted by Spoutable <jo...@spoutable.com>.
Happy to, can you point me in the right direction for the rules & examples for contributing?
> On Jan 29, 2016, at 3:27 PM, Jason Altekruse <al...@gmail.com> wrote:
>
> Glad to hear you got this working, do you want to file a doc fix JIRA with
> your proposed changes?
>
> On Fri, Jan 29, 2016 at 3:20 PM, Spoutable <jo...@spoutable.com> wrote:
>
>> Yes this worked. Thank You!
>>
>> We opted to use a custom authenticator using the java class template
>> instructions here:
>> https://drill.apache.org/docs/configuring-user-authentication/ <
>> https://drill.apache.org/docs/configuring-user-authentication/>
>>
>> The following is an FYI for anybody else that tries this path on the 1.5
>> release.
>>
>> The template didn’t ‘just work’.
>> You will need to correct the imports first.
>> import org.apache.drill.common.config.DrillConfig;
>> import org.apache.drill.exec.exception.DrillbitStartupException;
>> import org.apache.drill.exec.rpc.user.security.UserAuthenticator;
>> import org.apache.drill.exec.rpc.user.security.UserAuthenticatorTemplate;
>> import org.apache.drill.exec.rpc.user.security.UserAuthenticationException;
>> import java.io.IOException;
>>
>> Also you cant just package the class into a jar file and pop the jar into
>> the jars directory. There is a custom classpath scanner that looks for
>> the eligible classes.
>>
>> In order to tell the custom classpath scanner that our new class is
>> eligible to be found we had to put the following config code in a file
>> named drill-module.conf at the root of the jar file with our class.
>>
>> drill {
>> classpath.scanning {
>> packages += "com.spoutable.drill"
>> }
>> }
>>
>> We put our SimpleAuth class in the com.spoutable.drill package. We
>> still placed our jar file in the jars directory.
>>
>> The config in drill-overrides.conf that worked for us is slightly
>> different too.
>>
>> drill.exec: {
>> http: {
>> ssl_enabled: true
>> }
>>
>> security.user.auth: {
>> enabled: true,
>> impl: “SimpleAuth"
>> }
>> }
>>
>> We have the following line annotating our class which matches the ‘impl’
>> property above. Our class is also called SimpleAuth but its really the
>> type parameter in
>> @UserAuthenticatorTemplate(type="SimpleAuth")
>>
>>
>>> On Jan 28, 2016, at 1:09 PM, Jason Altekruse <al...@gmail.com>
>> wrote:
>>>
>>> A few weeks ago a fix for this issue was merged, it requires using the
>> new
>>> web UI security feature, which will hold onto a session while you are
>>> logged in. [1]
>>>
>>> You can try to build the tip of master yourself or wait for the upcoming
>>> 1.5.0 release to try it out.
>>>
>>> [1] - https://issues.apache.org/jira/browse/DRILL-3624
>>>
>>> On Thu, Jan 28, 2016 at 1:02 PM, Spoutable <jo...@spoutable.com> wrote:
>>>
>>>> We are using the REST API from nodejs and need to issue some create
>> table
>>>> statements that are a different format from the default for the system
>>>> (currently Parquet). We cant issue an alter system command to change
>> the
>>>> store.format globally.
>>>>
>>>> I tried to issue multiple requests with a alter session set
>>>> `store.format`='json';
>>>> and i tried to have multiple statements in the the same request
>> separated
>>>> by semi colon. Neither of those worked.
>>>>
>>>> As a last resort i looked at the code in QueryWrapper.java and it looks
>>>> like its just 1 query per request and no session persistence or anything
>>>> like that going on between requests to the REST api. I tried to follow
>> the
>>>> code down through the DrillUserPrincipal and DrillClient and UserClient
>> but
>>>> ran out of patience. It doesnt look like there is any notion of
>> ‘session’
>>>> in the sense of ‘alter session’ statments.
>>>>
>>>> Does anybody have a workaround for this?
>>>> If not, does anybody have a suggestion on what change I could make for
>>>> supporting this in a way that would be acceptable to the project?
>>>>
>>>> Thanks in advance.
>>
>>
Re: CTAS and storage format via REST API
Posted by Jason Altekruse <al...@gmail.com>.
Glad to hear you got this working, do you want to file a doc fix JIRA with
your proposed changes?
On Fri, Jan 29, 2016 at 3:20 PM, Spoutable <jo...@spoutable.com> wrote:
> Yes this worked. Thank You!
>
> We opted to use a custom authenticator using the java class template
> instructions here:
> https://drill.apache.org/docs/configuring-user-authentication/ <
> https://drill.apache.org/docs/configuring-user-authentication/>
>
> The following is an FYI for anybody else that tries this path on the 1.5
> release.
>
> The template didn’t ‘just work’.
> You will need to correct the imports first.
> import org.apache.drill.common.config.DrillConfig;
> import org.apache.drill.exec.exception.DrillbitStartupException;
> import org.apache.drill.exec.rpc.user.security.UserAuthenticator;
> import org.apache.drill.exec.rpc.user.security.UserAuthenticatorTemplate;
> import org.apache.drill.exec.rpc.user.security.UserAuthenticationException;
> import java.io.IOException;
>
> Also you cant just package the class into a jar file and pop the jar into
> the jars directory. There is a custom classpath scanner that looks for
> the eligible classes.
>
> In order to tell the custom classpath scanner that our new class is
> eligible to be found we had to put the following config code in a file
> named drill-module.conf at the root of the jar file with our class.
>
> drill {
> classpath.scanning {
> packages += "com.spoutable.drill"
> }
> }
>
> We put our SimpleAuth class in the com.spoutable.drill package. We
> still placed our jar file in the jars directory.
>
> The config in drill-overrides.conf that worked for us is slightly
> different too.
>
> drill.exec: {
> http: {
> ssl_enabled: true
> }
>
> security.user.auth: {
> enabled: true,
> impl: “SimpleAuth"
> }
> }
>
> We have the following line annotating our class which matches the ‘impl’
> property above. Our class is also called SimpleAuth but its really the
> type parameter in
> @UserAuthenticatorTemplate(type="SimpleAuth")
>
>
> > On Jan 28, 2016, at 1:09 PM, Jason Altekruse <al...@gmail.com>
> wrote:
> >
> > A few weeks ago a fix for this issue was merged, it requires using the
> new
> > web UI security feature, which will hold onto a session while you are
> > logged in. [1]
> >
> > You can try to build the tip of master yourself or wait for the upcoming
> > 1.5.0 release to try it out.
> >
> > [1] - https://issues.apache.org/jira/browse/DRILL-3624
> >
> > On Thu, Jan 28, 2016 at 1:02 PM, Spoutable <jo...@spoutable.com> wrote:
> >
> >> We are using the REST API from nodejs and need to issue some create
> table
> >> statements that are a different format from the default for the system
> >> (currently Parquet). We cant issue an alter system command to change
> the
> >> store.format globally.
> >>
> >> I tried to issue multiple requests with a alter session set
> >> `store.format`='json';
> >> and i tried to have multiple statements in the the same request
> separated
> >> by semi colon. Neither of those worked.
> >>
> >> As a last resort i looked at the code in QueryWrapper.java and it looks
> >> like its just 1 query per request and no session persistence or anything
> >> like that going on between requests to the REST api. I tried to follow
> the
> >> code down through the DrillUserPrincipal and DrillClient and UserClient
> but
> >> ran out of patience. It doesnt look like there is any notion of
> ‘session’
> >> in the sense of ‘alter session’ statments.
> >>
> >> Does anybody have a workaround for this?
> >> If not, does anybody have a suggestion on what change I could make for
> >> supporting this in a way that would be acceptable to the project?
> >>
> >> Thanks in advance.
>
>
Re: CTAS and storage format via REST API
Posted by Spoutable <jo...@spoutable.com>.
Yes this worked. Thank You!
We opted to use a custom authenticator using the java class template instructions here:
https://drill.apache.org/docs/configuring-user-authentication/ <https://drill.apache.org/docs/configuring-user-authentication/>
The following is an FYI for anybody else that tries this path on the 1.5 release.
The template didn’t ‘just work’.
You will need to correct the imports first.
import org.apache.drill.common.config.DrillConfig;
import org.apache.drill.exec.exception.DrillbitStartupException;
import org.apache.drill.exec.rpc.user.security.UserAuthenticator;
import org.apache.drill.exec.rpc.user.security.UserAuthenticatorTemplate;
import org.apache.drill.exec.rpc.user.security.UserAuthenticationException;
import java.io.IOException;
Also you cant just package the class into a jar file and pop the jar into the jars directory. There is a custom classpath scanner that looks for the eligible classes.
In order to tell the custom classpath scanner that our new class is eligible to be found we had to put the following config code in a file named drill-module.conf at the root of the jar file with our class.
drill {
classpath.scanning {
packages += "com.spoutable.drill"
}
}
We put our SimpleAuth class in the com.spoutable.drill package. We still placed our jar file in the jars directory.
The config in drill-overrides.conf that worked for us is slightly different too.
drill.exec: {
http: {
ssl_enabled: true
}
security.user.auth: {
enabled: true,
impl: “SimpleAuth"
}
}
We have the following line annotating our class which matches the ‘impl’ property above. Our class is also called SimpleAuth but its really the type parameter in
@UserAuthenticatorTemplate(type="SimpleAuth")
> On Jan 28, 2016, at 1:09 PM, Jason Altekruse <al...@gmail.com> wrote:
>
> A few weeks ago a fix for this issue was merged, it requires using the new
> web UI security feature, which will hold onto a session while you are
> logged in. [1]
>
> You can try to build the tip of master yourself or wait for the upcoming
> 1.5.0 release to try it out.
>
> [1] - https://issues.apache.org/jira/browse/DRILL-3624
>
> On Thu, Jan 28, 2016 at 1:02 PM, Spoutable <jo...@spoutable.com> wrote:
>
>> We are using the REST API from nodejs and need to issue some create table
>> statements that are a different format from the default for the system
>> (currently Parquet). We cant issue an alter system command to change the
>> store.format globally.
>>
>> I tried to issue multiple requests with a alter session set
>> `store.format`='json';
>> and i tried to have multiple statements in the the same request separated
>> by semi colon. Neither of those worked.
>>
>> As a last resort i looked at the code in QueryWrapper.java and it looks
>> like its just 1 query per request and no session persistence or anything
>> like that going on between requests to the REST api. I tried to follow the
>> code down through the DrillUserPrincipal and DrillClient and UserClient but
>> ran out of patience. It doesnt look like there is any notion of ‘session’
>> in the sense of ‘alter session’ statments.
>>
>> Does anybody have a workaround for this?
>> If not, does anybody have a suggestion on what change I could make for
>> supporting this in a way that would be acceptable to the project?
>>
>> Thanks in advance.
Re: CTAS and storage format via REST API
Posted by Jason Altekruse <al...@gmail.com>.
A few weeks ago a fix for this issue was merged, it requires using the new
web UI security feature, which will hold onto a session while you are
logged in. [1]
You can try to build the tip of master yourself or wait for the upcoming
1.5.0 release to try it out.
[1] - https://issues.apache.org/jira/browse/DRILL-3624
On Thu, Jan 28, 2016 at 1:02 PM, Spoutable <jo...@spoutable.com> wrote:
> We are using the REST API from nodejs and need to issue some create table
> statements that are a different format from the default for the system
> (currently Parquet). We cant issue an alter system command to change the
> store.format globally.
>
> I tried to issue multiple requests with a alter session set
> `store.format`='json';
> and i tried to have multiple statements in the the same request separated
> by semi colon. Neither of those worked.
>
> As a last resort i looked at the code in QueryWrapper.java and it looks
> like its just 1 query per request and no session persistence or anything
> like that going on between requests to the REST api. I tried to follow the
> code down through the DrillUserPrincipal and DrillClient and UserClient but
> ran out of patience. It doesnt look like there is any notion of ‘session’
> in the sense of ‘alter session’ statments.
>
> Does anybody have a workaround for this?
> If not, does anybody have a suggestion on what change I could make for
> supporting this in a way that would be acceptable to the project?
>
> Thanks in advance.