You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by dummy id <du...@gmail.com> on 2018/06/15 06:06:01 UTC
Help with Apache Drill - S3 compatible storage connectivity
*Team, *
*I am not sure who can help me out with this, so just adding both of the
help community..I have followed your documentation on setting up drill and
i am able to query the files from local , meaning class path and dfs, but
not able to do it with s3. I am not using Amazon S3 instead i am using S3
compatible storage from DELL EMC. I would need your help in setting up the
storage plugin file so that it uses path style addressing instead of normal
URL method. I would request you to kindly share an example core-site.xml
file as well as an example storage plugin file where it uses path style
addressing method to connect to s3.*
*I have tried using fs**.**s3a**.**path**.**style**.**access with
value true in both core-site and storage plugin files, but still the path
style addressing is not being read by drill and it again tries to connect
using URL method as it uses in connecting with Amazon S3. Kindly help..*
*Just an FYI, i have followed steps from "Drill in 10 minutes"
documentation to install and connect with my S3 compatible storage.*
*Awaiting for your reply..*
Re: Help with Apache Drill - S3 compatible storage connectivity
Posted by Pritesh Maker <pm...@mapr.com>.
Hi
Can you share any specific error message? Also look at this blog that describes using S3 interface provided by Minio ...
https://blog.minio.io/query-minio-datastore-with-apache-drill-dcaf71d0cee5
Pritesh
On 6/15/18, 8:02 AM, "dummy id" <du...@gmail.com> wrote:
*Team, *
*I am not sure who can help me out with this, so just adding both of the
help community..I have followed your documentation on setting up drill and
i am able to query the files from local , meaning class path and dfs, but
not able to do it with s3. I am not using Amazon S3 instead i am using S3
compatible storage from DELL EMC. I would need your help in setting up the
storage plugin file so that it uses path style addressing instead of normal
URL method. I would request you to kindly share an example core-site.xml
file as well as an example storage plugin file where it uses path style
addressing method to connect to s3.*
*I have tried using fs**.**s3a**.**path**.**style**.**access with
value true in both core-site and storage plugin files, but still the path
style addressing is not being read by drill and it again tries to connect
using URL method as it uses in connecting with Amazon S3. Kindly help..*
*Just an FYI, i have followed steps from "Drill in 10 minutes"
documentation to install and connect with my S3 compatible storage.*
*Awaiting for your reply..*
Re: Help with Apache Drill - S3 compatible storage connectivity
Posted by Saurabh Mahapatra <sa...@gmail.com>.
I mean seriously. Dummy id?
Sent from my iPhone
> On Jun 26, 2018, at 11:30 AM, dummy id <du...@gmail.com> wrote:
>
> Can i get an update on this please?
>
>> On Fri, Jun 15, 2018 at 11:36 AM, dummy id <du...@gmail.com> wrote:
>>
>> *Team, *
>>
>>
>>
>> *I am not sure who can help me out with this, so just adding both of the
>> help community..I have followed your documentation on setting up drill and
>> i am able to query the files from local , meaning class path and dfs, but
>> not able to do it with s3. I am not using Amazon S3 instead i am using S3
>> compatible storage from DELL EMC. I would need your help in setting up the
>> storage plugin file so that it uses path style addressing instead of normal
>> URL method. I would request you to kindly share an example core-site.xml
>> file as well as an example storage plugin file where it uses path style
>> addressing method to connect to s3.*
>>
>>
>>
>> *I have tried using fs**.**s3a**.**path**.**style**.**access with
>> value true in both core-site and storage plugin files, but still the path
>> style addressing is not being read by drill and it again tries to connect
>> using URL method as it uses in connecting with Amazon S3. Kindly help..*
>>
>>
>>
>> *Just an FYI, i have followed steps from "Drill in 10 minutes"
>> documentation to install and connect with my S3 compatible storage.*
>>
>>
>>
>> *Awaiting for your reply..*
>>
>>
>>
Re: Help with Apache Drill - S3 compatible storage connectivity
Posted by Paul Rogers <pa...@yahoo.com.INVALID>.
Hi Dummy ID,
Have you tried getting Drill to work with the real S3? There were a few confusing bits in the S3 docs that Bridget graciously cleaned up. By trying to access the real S3 you'll ensure that you have gotten the config stuff right.
The key missing bit of information was that you had to provide the fs.s3a.endpoint property. The HDFS s3a library is the current version you should use (at least for S3) rather than the older s3n, etc. libraries.
You can also use HDFS to verify that you have proper connectivity to your S3 clone. Configure HDFS to work with your S3 clone, then check if you can access your files using the "hadoop fs" commands. If not, then you've probably got a more fundamental issue than Drill.
If "hadoop fs" works, but Drill does not, then you probably have a Drill config issue since Drill uses the Hadoop libraries for S3 access. Review the docs for what should be in core-site.xml and what should be in your S3 storage plugin config.
Thanks,
- Paul
On Tuesday, June 26, 2018, 2:10:11 PM PDT, dummy id <du...@gmail.com> wrote:
Can i get an update on this please?
On Fri, Jun 15, 2018 at 11:36 AM, dummy id <du...@gmail.com> wrote:
> *Team, *
>
>
>
> *I am not sure who can help me out with this, so just adding both of the
> help community..I have followed your documentation on setting up drill and
> i am able to query the files from local , meaning class path and dfs, but
> not able to do it with s3. I am not using Amazon S3 instead i am using S3
> compatible storage from DELL EMC. I would need your help in setting up the
> storage plugin file so that it uses path style addressing instead of normal
> URL method. I would request you to kindly share an example core-site.xml
> file as well as an example storage plugin file where it uses path style
> addressing method to connect to s3.*
>
>
>
> *I have tried using fs**.**s3a**.**path**.**style**.**access with
> value true in both core-site and storage plugin files, but still the path
> style addressing is not being read by drill and it again tries to connect
> using URL method as it uses in connecting with Amazon S3. Kindly help..*
>
>
>
> *Just an FYI, i have followed steps from "Drill in 10 minutes"
> documentation to install and connect with my S3 compatible storage.*
>
>
>
> *Awaiting for your reply..*
>
>
>
Re: Help with Apache Drill - S3 compatible storage connectivity
Posted by Parth Chandra <pa...@apache.org>.
Drill uses HDFS to access S3, so if you have configured the EMC system to
be usable by hadoop, it will be usable by Drill.
Here's the documentation for an S3 compatible EMC system (
https://www.emc.com/collateral/TechnicalDocument/docu86295.pdf); chapters
6-11 are relevant. I'm not sure if this is the same system you have, but
your system should have similar documentation.
You will probably have to use a different protocol identifier in the URL to
access the storage system.
On Tue, Jun 26, 2018 at 11:30 AM, dummy id <du...@gmail.com> wrote:
> Can i get an update on this please?
>
> On Fri, Jun 15, 2018 at 11:36 AM, dummy id <du...@gmail.com> wrote:
>
> > *Team, *
> >
> >
> >
> > *I am not sure who can help me out with this, so just adding both of the
> > help community..I have followed your documentation on setting up drill
> and
> > i am able to query the files from local , meaning class path and dfs, but
> > not able to do it with s3. I am not using Amazon S3 instead i am using S3
> > compatible storage from DELL EMC. I would need your help in setting up
> the
> > storage plugin file so that it uses path style addressing instead of
> normal
> > URL method. I would request you to kindly share an example core-site.xml
> > file as well as an example storage plugin file where it uses path style
> > addressing method to connect to s3.*
> >
> >
> >
> > *I have tried using fs**.**s3a**.**path**.**style**.**access with
> > value true in both core-site and storage plugin files, but still the path
> > style addressing is not being read by drill and it again tries to connect
> > using URL method as it uses in connecting with Amazon S3. Kindly help..*
> >
> >
> >
> > *Just an FYI, i have followed steps from "Drill in 10 minutes"
> > documentation to install and connect with my S3 compatible storage.*
> >
> >
> >
> > *Awaiting for your reply..*
> >
> >
> >
>
Re: Help with Apache Drill - S3 compatible storage connectivity
Posted by Saurabh Mahapatra <sa...@gmail.com>.
I mean seriously. Dummy id?
Sent from my iPhone
> On Jun 26, 2018, at 11:30 AM, dummy id <du...@gmail.com> wrote:
>
> Can i get an update on this please?
>
>> On Fri, Jun 15, 2018 at 11:36 AM, dummy id <du...@gmail.com> wrote:
>>
>> *Team, *
>>
>>
>>
>> *I am not sure who can help me out with this, so just adding both of the
>> help community..I have followed your documentation on setting up drill and
>> i am able to query the files from local , meaning class path and dfs, but
>> not able to do it with s3. I am not using Amazon S3 instead i am using S3
>> compatible storage from DELL EMC. I would need your help in setting up the
>> storage plugin file so that it uses path style addressing instead of normal
>> URL method. I would request you to kindly share an example core-site.xml
>> file as well as an example storage plugin file where it uses path style
>> addressing method to connect to s3.*
>>
>>
>>
>> *I have tried using fs**.**s3a**.**path**.**style**.**access with
>> value true in both core-site and storage plugin files, but still the path
>> style addressing is not being read by drill and it again tries to connect
>> using URL method as it uses in connecting with Amazon S3. Kindly help..*
>>
>>
>>
>> *Just an FYI, i have followed steps from "Drill in 10 minutes"
>> documentation to install and connect with my S3 compatible storage.*
>>
>>
>>
>> *Awaiting for your reply..*
>>
>>
>>
Re: Help with Apache Drill - S3 compatible storage connectivity
Posted by dummy id <du...@gmail.com>.
Can i get an update on this please?
On Fri, Jun 15, 2018 at 11:36 AM, dummy id <du...@gmail.com> wrote:
> *Team, *
>
>
>
> *I am not sure who can help me out with this, so just adding both of the
> help community..I have followed your documentation on setting up drill and
> i am able to query the files from local , meaning class path and dfs, but
> not able to do it with s3. I am not using Amazon S3 instead i am using S3
> compatible storage from DELL EMC. I would need your help in setting up the
> storage plugin file so that it uses path style addressing instead of normal
> URL method. I would request you to kindly share an example core-site.xml
> file as well as an example storage plugin file where it uses path style
> addressing method to connect to s3.*
>
>
>
> *I have tried using fs**.**s3a**.**path**.**style**.**access with
> value true in both core-site and storage plugin files, but still the path
> style addressing is not being read by drill and it again tries to connect
> using URL method as it uses in connecting with Amazon S3. Kindly help..*
>
>
>
> *Just an FYI, i have followed steps from "Drill in 10 minutes"
> documentation to install and connect with my S3 compatible storage.*
>
>
>
> *Awaiting for your reply..*
>
>
>
Re: Help with Apache Drill - S3 compatible storage connectivity
Posted by dummy id <du...@gmail.com>.
Can i get an update on this please?
On Fri, Jun 15, 2018 at 11:36 AM, dummy id <du...@gmail.com> wrote:
> *Team, *
>
>
>
> *I am not sure who can help me out with this, so just adding both of the
> help community..I have followed your documentation on setting up drill and
> i am able to query the files from local , meaning class path and dfs, but
> not able to do it with s3. I am not using Amazon S3 instead i am using S3
> compatible storage from DELL EMC. I would need your help in setting up the
> storage plugin file so that it uses path style addressing instead of normal
> URL method. I would request you to kindly share an example core-site.xml
> file as well as an example storage plugin file where it uses path style
> addressing method to connect to s3.*
>
>
>
> *I have tried using fs**.**s3a**.**path**.**style**.**access with
> value true in both core-site and storage plugin files, but still the path
> style addressing is not being read by drill and it again tries to connect
> using URL method as it uses in connecting with Amazon S3. Kindly help..*
>
>
>
> *Just an FYI, i have followed steps from "Drill in 10 minutes"
> documentation to install and connect with my S3 compatible storage.*
>
>
>
> *Awaiting for your reply..*
>
>
>
Re: Help with Apache Drill - S3 compatible storage connectivity
Posted by Pritesh Maker <pm...@mapr.com>.
Hi
Can you share any specific error message? Also look at this blog that describes using S3 interface provided by Minio ...
https://blog.minio.io/query-minio-datastore-with-apache-drill-dcaf71d0cee5
Pritesh
On 6/15/18, 8:02 AM, "dummy id" <du...@gmail.com> wrote:
*Team, *
*I am not sure who can help me out with this, so just adding both of the
help community..I have followed your documentation on setting up drill and
i am able to query the files from local , meaning class path and dfs, but
not able to do it with s3. I am not using Amazon S3 instead i am using S3
compatible storage from DELL EMC. I would need your help in setting up the
storage plugin file so that it uses path style addressing instead of normal
URL method. I would request you to kindly share an example core-site.xml
file as well as an example storage plugin file where it uses path style
addressing method to connect to s3.*
*I have tried using fs**.**s3a**.**path**.**style**.**access with
value true in both core-site and storage plugin files, but still the path
style addressing is not being read by drill and it again tries to connect
using URL method as it uses in connecting with Amazon S3. Kindly help..*
*Just an FYI, i have followed steps from "Drill in 10 minutes"
documentation to install and connect with my S3 compatible storage.*
*Awaiting for your reply..*