You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@drill.apache.org by Daniil Osipov <da...@shazam.com> on 2015/02/24 22:51:23 UTC

API for storage configuration?

Hello,

I'm working on fixing the EMR bootstrap script for running Drill. I already
made a few PRs that helped get the latest versions of Drill running
properly:
https://github.com/awslabs/emr-bootstrap-actions/pull/56
https://github.com/awslabs/emr-bootstrap-actions/pull/57

I'm now trying to configure Drill to access S3 "out of the box". I'm
following the directions given on
http://drill.apache.org/blog/2014/12/09/running-sql-queries-on-amazon-s3/

EMR already deploys necessary access/secret keys, so Drill configuration
remains. I'd like to configure as much as possible in the bootstrap script
to minimize friction for starting a new cluster.

The problem is the blog post requires one to use the web interface to
configure storage plugin. Is there a programmatic way to accomplish the
same - perhaps by creating a file, issuing a query, or calling a process?

Thanks in advance!
Dan Osipov

Re: API for storage configuration?

Posted by Daniil Osipov <da...@shazam.com>.

Thanks, Jacques - I'll give it a try!

On Tue, Feb 24, 2015 at 1:55 PM, Jacques Nadeau <ja...@apache.org> wrote:

> If you create a new bootstrap-storage-plugins.json file and put it on the
> classpath, it should automatically be loaded when Drill boots for the first
> time.
>
> Example file that adds HBase to the default boot of a new cluster:
>
> https://github.com/apache/drill/blob/master/contrib/storage-hbase/src/main/resources/bootstrap-storage-plugins.json
>
> Content inside the named block is the same as you would add to the web ui.
>
> On Tue, Feb 24, 2015 at 1:51 PM, Daniil Osipov <da...@shazam.com>
> wrote:
>
> > Hello,
> >
> > I'm working on fixing the EMR bootstrap script for running Drill. I
> already
> > made a few PRs that helped get the latest versions of Drill running
> > properly:
> > https://github.com/awslabs/emr-bootstrap-actions/pull/56
> > https://github.com/awslabs/emr-bootstrap-actions/pull/57
> >
> > I'm now trying to configure Drill to access S3 "out of the box". I'm
> > following the directions given on
> >
> http://drill.apache.org/blog/2014/12/09/running-sql-queries-on-amazon-s3/
> >
> > EMR already deploys necessary access/secret keys, so Drill configuration
> > remains. I'd like to configure as much as possible in the bootstrap
> script
> > to minimize friction for starting a new cluster.
> >
> > The problem is the blog post requires one to use the web interface to
> > configure storage plugin. Is there a programmatic way to accomplish the
> > same - perhaps by creating a file, issuing a query, or calling a process?
> >
> > Thanks in advance!
> > Dan Osipov
> >
>

Re: API for storage configuration?

Posted by Jacques Nadeau <ja...@apache.org>.

If you create a new bootstrap-storage-plugins.json file and put it on the
classpath, it should automatically be loaded when Drill boots for the first
time.

Example file that adds HBase to the default boot of a new cluster:
https://github.com/apache/drill/blob/master/contrib/storage-hbase/src/main/resources/bootstrap-storage-plugins.json

Content inside the named block is the same as you would add to the web ui.

On Tue, Feb 24, 2015 at 1:51 PM, Daniil Osipov <da...@shazam.com>
wrote:

> Hello,
>
> I'm working on fixing the EMR bootstrap script for running Drill. I already
> made a few PRs that helped get the latest versions of Drill running
> properly:
> https://github.com/awslabs/emr-bootstrap-actions/pull/56
> https://github.com/awslabs/emr-bootstrap-actions/pull/57
>
> I'm now trying to configure Drill to access S3 "out of the box". I'm
> following the directions given on
> http://drill.apache.org/blog/2014/12/09/running-sql-queries-on-amazon-s3/
>
> EMR already deploys necessary access/secret keys, so Drill configuration
> remains. I'd like to configure as much as possible in the bootstrap script
> to minimize friction for starting a new cluster.
>
> The problem is the blog post requires one to use the web interface to
> configure storage plugin. Is there a programmatic way to accomplish the
> same - perhaps by creating a file, issuing a query, or calling a process?
>
> Thanks in advance!
> Dan Osipov
>