You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Nirav Shah <ni...@games24x7.com> on 2015/12/11 08:38:27 UTC
Connecting to S3 Drill 1.2
Hi All,
I am trying to query from S3.
I am getting below error.
Error: SYSTEM ERROR: IllegalArgumentException: AWS Access Key ID and Secret
Access Key must be specified as the username or password (respectively) of
a s3 URL, or by setting the fs.s3.awsAccessKeyId or
fs.s3.awsSecretAccessKey properties (respectively).
Configuration :
Step 1: hadoop-2.7.1/etc/hadoop/core-site.xml in
<property> <name>fs.s3.awsAccessKeyId</name><value>key</value>
</property>
<property> <name>fs.s3.awsSecretAccessKey</name> <value>value</value>
</property>
<property> <name>fs.s3n.awsAccessKeyId</name>
<value>key</value> </property>
<property> <name>fs.s3n.awsSecretAccessKey</name>
<value>value</value> </property>
Step 2 : $DRILL_HOME/bin/hadoop_excludes.txt removing the line jets3t
Step 3: copy jets3t-0.9.2.jar to $DRILL_HOME/jars/3rdparty
Step 4: configuring storage plugin
"connection": "s3://g24x7.ana-p", rest is same as file plugin
Still not able to connect. Pls help.
Thanks,
Nirav
Re: Connecting to S3 Drill 1.2
Posted by Nirav Shah <ni...@games24x7.com>.
Hi Kristine,
Thanks for your help. This worked for me.
I was following instruction given on drill documentation but it's not
working.
@Andries
I tried placing core-site.xml in Drill/Conf still it's not working.
Please note that I am using Drill 1.2.
Best,
Nirav
On Fri, Dec 11, 2015 at 8:29 PM, Kristine Hahn <kh...@maprtech.com> wrote:
> Please see the Amazon S3 Help regarding set of the xml:
>
> <property>
> <name>fs.s3.awsAccessKeyId</name>
> <value>ID</value>
> </property>
>
> <property>
> <name>fs.s3.awsSecretAccessKey</name>
> <value>SECRET</value>
> </property>
>
> Your connection in the plugin config needs to include the URL that
> looks something like this:
>
> "connection": "s3n://<accesskey>:<secret>@myname",
> Kristine Hahn
> Sr. Technical Writer
> 415-497-8107 @krishahn skype:krishahn
>
>
>
> On Thu, Dec 10, 2015 at 11:38 PM, Nirav Shah <ni...@games24x7.com>
> wrote:
> > Hi All,
> >
> > I am trying to query from S3.
> > I am getting below error.
> >
> > Error: SYSTEM ERROR: IllegalArgumentException: AWS Access Key ID and
> Secret
> > Access Key must be specified as the username or password (respectively)
> of
> > a s3 URL, or by setting the fs.s3.awsAccessKeyId or
> > fs.s3.awsSecretAccessKey properties (respectively).
> >
> >
> > Configuration :
> > Step 1: hadoop-2.7.1/etc/hadoop/core-site.xml in
> > <property> <name>fs.s3.awsAccessKeyId</name><value>key</value>
> > </property>
> >
> > <property> <name>fs.s3.awsSecretAccessKey</name> <value>value</value>
> > </property>
> > <property> <name>fs.s3n.awsAccessKeyId</name>
> > <value>key</value> </property>
> > <property> <name>fs.s3n.awsSecretAccessKey</name>
> > <value>value</value> </property>
> >
> > Step 2 : $DRILL_HOME/bin/hadoop_excludes.txt removing the line jets3t
> > Step 3: copy jets3t-0.9.2.jar to $DRILL_HOME/jars/3rdparty
> > Step 4: configuring storage plugin
> > "connection": "s3://g24x7.ana-p", rest is same as file plugin
> >
> > Still not able to connect. Pls help.
> >
> > Thanks,
> > Nirav
>
Re: Connecting to S3 Drill 1.2
Posted by Andries Engelbrecht <ae...@maprtech.com>.
I found that placing the core-site.xml with only the S3 properties in the DRILL/CONF directory works best. Seems to get pickup more reliably than when placed in the hadoop directory.
Also this way you don't have to expose the key in the plugin configuration.
--Andries
> On Dec 11, 2015, at 6:59 AM, Kristine Hahn <kh...@maprtech.com> wrote:
>
> Please see the Amazon S3 Help regarding set of the xml:
>
> <property>
> <name>fs.s3.awsAccessKeyId</name>
> <value>ID</value>
> </property>
>
> <property>
> <name>fs.s3.awsSecretAccessKey</name>
> <value>SECRET</value>
> </property>
>
> Your connection in the plugin config needs to include the URL that
> looks something like this:
>
> "connection": "s3n://<accesskey>:<secret>@myname",
> Kristine Hahn
> Sr. Technical Writer
> 415-497-8107 @krishahn skype:krishahn
>
>
>
> On Thu, Dec 10, 2015 at 11:38 PM, Nirav Shah <ni...@games24x7.com> wrote:
>> Hi All,
>>
>> I am trying to query from S3.
>> I am getting below error.
>>
>> Error: SYSTEM ERROR: IllegalArgumentException: AWS Access Key ID and Secret
>> Access Key must be specified as the username or password (respectively) of
>> a s3 URL, or by setting the fs.s3.awsAccessKeyId or
>> fs.s3.awsSecretAccessKey properties (respectively).
>>
>>
>> Configuration :
>> Step 1: hadoop-2.7.1/etc/hadoop/core-site.xml in
>> <property> <name>fs.s3.awsAccessKeyId</name><value>key</value>
>> </property>
>>
>> <property> <name>fs.s3.awsSecretAccessKey</name> <value>value</value>
>> </property>
>> <property> <name>fs.s3n.awsAccessKeyId</name>
>> <value>key</value> </property>
>> <property> <name>fs.s3n.awsSecretAccessKey</name>
>> <value>value</value> </property>
>>
>> Step 2 : $DRILL_HOME/bin/hadoop_excludes.txt removing the line jets3t
>> Step 3: copy jets3t-0.9.2.jar to $DRILL_HOME/jars/3rdparty
>> Step 4: configuring storage plugin
>> "connection": "s3://g24x7.ana-p", rest is same as file plugin
>>
>> Still not able to connect. Pls help.
>>
>> Thanks,
>> Nirav
Re: Connecting to S3 Drill 1.2
Posted by Kristine Hahn <kh...@maprtech.com>.
Please see the Amazon S3 Help regarding set of the xml:
<property>
<name>fs.s3.awsAccessKeyId</name>
<value>ID</value>
</property>
<property>
<name>fs.s3.awsSecretAccessKey</name>
<value>SECRET</value>
</property>
Your connection in the plugin config needs to include the URL that
looks something like this:
"connection": "s3n://<accesskey>:<secret>@myname",
Kristine Hahn
Sr. Technical Writer
415-497-8107 @krishahn skype:krishahn
On Thu, Dec 10, 2015 at 11:38 PM, Nirav Shah <ni...@games24x7.com> wrote:
> Hi All,
>
> I am trying to query from S3.
> I am getting below error.
>
> Error: SYSTEM ERROR: IllegalArgumentException: AWS Access Key ID and Secret
> Access Key must be specified as the username or password (respectively) of
> a s3 URL, or by setting the fs.s3.awsAccessKeyId or
> fs.s3.awsSecretAccessKey properties (respectively).
>
>
> Configuration :
> Step 1: hadoop-2.7.1/etc/hadoop/core-site.xml in
> <property> <name>fs.s3.awsAccessKeyId</name><value>key</value>
> </property>
>
> <property> <name>fs.s3.awsSecretAccessKey</name> <value>value</value>
> </property>
> <property> <name>fs.s3n.awsAccessKeyId</name>
> <value>key</value> </property>
> <property> <name>fs.s3n.awsSecretAccessKey</name>
> <value>value</value> </property>
>
> Step 2 : $DRILL_HOME/bin/hadoop_excludes.txt removing the line jets3t
> Step 3: copy jets3t-0.9.2.jar to $DRILL_HOME/jars/3rdparty
> Step 4: configuring storage plugin
> "connection": "s3://g24x7.ana-p", rest is same as file plugin
>
> Still not able to connect. Pls help.
>
> Thanks,
> Nirav