You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by "Mikhailau, Alex" <Al...@mlb.com> on 2015/11/17 00:21:57 UTC
Problem running queries on EMR -
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
IllegalArgumentException: AWS Access Key ID
Guys,
I keep getting the following exception:
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey properties (respectively). [Error Id: 919f3351-6b9a-4f8b-ac9d-cea5b6a8be1f on ip-172-31-59-234.ec2.internal:31010]
1. I followed the set up at https://www.mapr.com/blog/bootstrap-apache-drill-amazon-emr.
2. I set up /opt/mapr/drill/drill-1.2.0/conf/core-site.xml
<configuration>
<property><name>fs.s3.impl</name><value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value></property>
<property><name>fs.s3n.impl</name><value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value></property>
<property><name>fs.s3.block.size</name><value>67108864</value></property>
<property><name>fs.s3n.block.size</name><value>67108864</value></property>
<property><name>fs.s3.awsAccessKeyId</name><value>XXXXXX</value></property>
<property><name>fs.s3.awsSecretAccessKey</name><value>YYYYYY</value></property>
<property><name>fs.s3n.awsAccessKeyId</name><value>XXXXXX</value></property>
<property><name>fs.s3n.awsSecretAccessKey</name><value>YYYYYY</value></property>
<property>
<name>hbase.table.namespace.mappings</name>
<value>*:/user/${user.name}</value>
</property>
</configuration>
3. I’ve re-started drill > /opt/mapr/bin/maprcli node services -name drill-bits -action restart -nodes ip-172-31-57-10.ec2.internal
4. Running a query like this - SELECT hostname FROM sys.drillbits WHERE `current` = true; - or any other yields the same exception.
5. Running hadoop fs -ls s3://bucket/ lists files just fine.
Can anyone please help figure out what the problem is with Drill not able to access S3 or run any queries for that matter from the sys table?
Greatly appreciated.
Thank you,
Alex
**********************************************************
MLB.com: Where Baseball is Always On
Re: Problem running queries on EMR - org.apache.drill.common.exceptions.UserRemoteException:
SYSTEM ERROR: IllegalArgumentException: AWS Access Key ID
Posted by Nathan Griffith <ng...@dremio.com>.
Hi Alex,
Check out the "Setting up Apache Drill for use with S3" section of
this article I wrote:
http://www.dremio.com/blog/using-sql-to-interface-with-google-analytics-data-stored-on-amazon-s3/
It should handle what you're trying to do. In particular, make sure
your storage plugin for s3 is set up as indicated, since I didn't see
you mention its configuration in your email.
Best of luck!
Nathan
On Mon, Nov 16, 2015 at 3:21 PM, Mikhailau, Alex <Al...@mlb.com> wrote:
> Guys,
>
> I keep getting the following exception:
>
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey properties (respectively). [Error Id: 919f3351-6b9a-4f8b-ac9d-cea5b6a8be1f on ip-172-31-59-234.ec2.internal:31010]
>
>
> 1. I followed the set up at https://www.mapr.com/blog/bootstrap-apache-drill-amazon-emr.
>
>
> 2. I set up /opt/mapr/drill/drill-1.2.0/conf/core-site.xml
>
> <configuration>
>
> <property><name>fs.s3.impl</name><value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value></property>
> <property><name>fs.s3n.impl</name><value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value></property>
>
> <property><name>fs.s3.block.size</name><value>67108864</value></property>
> <property><name>fs.s3n.block.size</name><value>67108864</value></property>
> <property><name>fs.s3.awsAccessKeyId</name><value>XXXXXX</value></property>
> <property><name>fs.s3.awsSecretAccessKey</name><value>YYYYYY</value></property>
> <property><name>fs.s3n.awsAccessKeyId</name><value>XXXXXX</value></property>
> <property><name>fs.s3n.awsSecretAccessKey</name><value>YYYYYY</value></property>
>
> <property>
> <name>hbase.table.namespace.mappings</name>
> <value>*:/user/${user.name}</value>
> </property>
>
> </configuration>
>
> 3. I’ve re-started drill > /opt/mapr/bin/maprcli node services -name drill-bits -action restart -nodes ip-172-31-57-10.ec2.internal
>
> 4. Running a query like this - SELECT hostname FROM sys.drillbits WHERE `current` = true; - or any other yields the same exception.
>
> 5. Running hadoop fs -ls s3://bucket/ lists files just fine.
>
> Can anyone please help figure out what the problem is with Drill not able to access S3 or run any queries for that matter from the sys table?
>
> Greatly appreciated.
>
> Thank you,
>
> Alex
> **********************************************************
>
> MLB.com: Where Baseball is Always On
Re: Problem running queries on EMR -
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
IllegalArgumentException: AWS Access Key ID
Posted by scott cote <sc...@me.com>.
I got tripped up with this by not setting permissions in S3 properly. I gave individual profiles the read permissions on the bucket, but I didn’t give readers permission ….
The act of being able to hit the file on s3 with a get command was not enough of a sanity check. Highly recommend the aws command line tool for sanity check.
SCott
> On Nov 17, 2015, at 1:36 AM, Andries Engelbrecht <ae...@maprtech.com> wrote:
>
> For large cluster I normally use clush (cluster shell), edit the config files on one node and then copy them to all using clush.
>
>
> --Andries
>
>> On Nov 16, 2015, at 6:41 PM, Mikhailau, Alex <Al...@mlb.com> wrote:
>>
>> Thanks, guys. I had to ssh into each node and update core-site.xml on
>> every node.
>>
Re: Problem running queries on EMR - org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalArgumentException: AWS Access Key ID
Posted by Andries Engelbrecht <ae...@maprtech.com>.
For large cluster I normally use clush (cluster shell), edit the config files on one node and then copy them to all using clush.
--Andries
> On Nov 16, 2015, at 6:41 PM, Mikhailau, Alex <Al...@mlb.com> wrote:
>
> Thanks, guys. I had to ssh into each node and update core-site.xml on
> every node.
>
Re: Problem running queries on EMR -
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
IllegalArgumentException: AWS Access Key ID
Posted by "Mikhailau, Alex" <Al...@mlb.com>.
Thanks, guys. I had to ssh into each node and update core-site.xml on
every node.
Re: Problem running queries on EMR - org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalArgumentException: AWS Access Key ID
Posted by Andries Engelbrecht <ae...@maprtech.com>.
Make sure to use s3n and not just s3 in the connection of the storage plugin configuration.
This changed a while back and have tripped some people up.
--Andries
> On Nov 16, 2015, at 3:21 PM, Mikhailau, Alex <Al...@mlb.com> wrote:
>
> Guys,
>
> I keep getting the following exception:
>
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey properties (respectively). [Error Id: 919f3351-6b9a-4f8b-ac9d-cea5b6a8be1f on ip-172-31-59-234.ec2.internal:31010]
>
>
> 1. I followed the set up at https://www.mapr.com/blog/bootstrap-apache-drill-amazon-emr.
>
>
> 2. I set up /opt/mapr/drill/drill-1.2.0/conf/core-site.xml
>
> <configuration>
>
> <property><name>fs.s3.impl</name><value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value></property>
> <property><name>fs.s3n.impl</name><value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value></property>
>
> <property><name>fs.s3.block.size</name><value>67108864</value></property>
> <property><name>fs.s3n.block.size</name><value>67108864</value></property>
> <property><name>fs.s3.awsAccessKeyId</name><value>XXXXXX</value></property>
> <property><name>fs.s3.awsSecretAccessKey</name><value>YYYYYY</value></property>
> <property><name>fs.s3n.awsAccessKeyId</name><value>XXXXXX</value></property>
> <property><name>fs.s3n.awsSecretAccessKey</name><value>YYYYYY</value></property>
>
> <property>
> <name>hbase.table.namespace.mappings</name>
> <value>*:/user/${user.name}</value>
> </property>
>
> </configuration>
>
> 3. I’ve re-started drill > /opt/mapr/bin/maprcli node services -name drill-bits -action restart -nodes ip-172-31-57-10.ec2.internal
>
> 4. Running a query like this - SELECT hostname FROM sys.drillbits WHERE `current` = true; - or any other yields the same exception.
>
> 5. Running hadoop fs -ls s3://bucket/ lists files just fine.
>
> Can anyone please help figure out what the problem is with Drill not able to access S3 or run any queries for that matter from the sys table?
>
> Greatly appreciated.
>
> Thank you,
>
> Alex
> **********************************************************
>
> MLB.com: Where Baseball is Always On