You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Chandrakanth Akkinepalli <ch...@gmail.com> on 2017/05/23 20:34:25 UTC

querying aws s3 objects that are server side encrypted

Hi All,
I have configured aws s3 storage plugin to access data in my s3 bucket.

I have placed both un-encrypted and aws-kms sse objects in the bucket.

I am able to query the un-encrypted objects just fine via apache drill.
But i am getting following error when querying aws-kms encrypted objects.

SQL Error: VALIDATION ERROR: Status Code: 400, AWS Service: Amazon S3, AWS
Request ID: ABCDEFGH12345, AWS Error Code: InvalidArgument, AWS Error
Message: Requests specifying Server Side Encryption with AWS KMS managed
keys require AWS Signature Version 4.

SQL Query null


Did anyone else face this issue? How do i resolve it?

Thanks,
Chandra

Re: querying aws s3 objects that are server side encrypted

Posted by Chandrakanth Akkinepalli <ch...@gmail.com>.
Gautam,
Thanks for your response, S3 plugin config is below.
I am not sure if there is an property in fs.s3a.* to set the signature
version ( in core-site.xml). Any recommendation on how to apply it.

{
  "type": "file",
  "enabled": true,
  "connection": "s3a://abcd1234/",
  "config": null,
  "workspaces": {
    "root": {
      "location": "/EMR/Data/Test/",
      "writable": false,
      "defaultInputFormat": null
    },
    "omniture": {
      "location": "/EMR/Data/Omniture_parquet/Omniture_global_data",
      "writable": false,
      "defaultInputFormat": null
    },
    "tmp": {
      "location": "/tmp",
      "writable": true,
      "defaultInputFormat": null
    }
  },
  "formats": {
    "psv": {
      "type": "text",
      "extensions": [
        "tbl"
      ],
      "delimiter": "|"
    },
    "csv": {
      "type": "text",
      "extensions": [
        "csv"
      ],
      "delimiter": ","
    },
    "tsv": {
      "type": "text",
      "extensions": [
        "tsv"
      ],
      "delimiter": "\t"
    },
    "httpd": {
      "type": "httpd",
      "logFormat": "%h %t \"%r\" %>s %b \"%{Referer}i\"",
      "timestampFormat": null
    },
    "parquet": {
      "type": "parquet"
    },
    "json": {
      "type": "json",
      "extensions": [
        "json"
      ]
    },
    "avro": {
      "type": "avro"
    },
    "sequencefile": {
      "type": "sequencefile",
      "extensions": [
        "seq"
      ]
    },
    "csvh": {
      "type": "text",
      "extensions": [
        "csvh"
      ],
      "extractHeader": true,
      "delimiter": ","
    }
  }
}

On Tue, May 23, 2017 at 7:48 PM, Gautam Parai <gp...@mapr.com> wrote:

> Hi Chandrakanth,
>
>
> A quick google got me this result from AWS:
>
>
> "We are sorry you have trouble getting this to work. The error is
> consistent with KMS requiring SigV4. We are still looking at why, by
> default, the s3 client for node.js doesn't do that. In the interim, you can
> explicitly set the signatureVersion when creating the s3 client object as
> follows:
>
> var s3Obj = new AWS.S3({maxRetries: 10, "signatureVersion":"v4"});
>
> Please let us know if this fixed or didn't fix your problem."
>
> Can you try setting an equivalent S3 storage plugin configuration for
> `signatureVersion`? Please see here for an example
> https://drill.apache.org/docs/s3-storage-plugin/
>
> Also, it would be good if you could share your S3 storage plugin
> configuration here.
>
>
> Thanks,
>
> Gautam
>
> ________________________________
> From: Chandrakanth Akkinepalli <ch...@gmail.com>
> Sent: Tuesday, May 23, 2017 1:34:25 PM
> To: user@drill.apache.org
> Subject: querying aws s3 objects that are server side encrypted
>
> Hi All,
> I have configured aws s3 storage plugin to access data in my s3 bucket.
>
> I have placed both un-encrypted and aws-kms sse objects in the bucket.
>
> I am able to query the un-encrypted objects just fine via apache drill.
> But i am getting following error when querying aws-kms encrypted objects.
>
> SQL Error: VALIDATION ERROR: Status Code: 400, AWS Service: Amazon S3, AWS
> Request ID: ABCDEFGH12345, AWS Error Code: InvalidArgument, AWS Error
> Message: Requests specifying Server Side Encryption with AWS KMS managed
> keys require AWS Signature Version 4.
>
> SQL Query null
>
>
> Did anyone else face this issue? How do i resolve it?
>
> Thanks,
> Chandra
>

Re: querying aws s3 objects that are server side encrypted

Posted by Gautam Parai <gp...@mapr.com>.
Hi Chandrakanth,


A quick google got me this result from AWS:


"We are sorry you have trouble getting this to work. The error is consistent with KMS requiring SigV4. We are still looking at why, by default, the s3 client for node.js doesn't do that. In the interim, you can explicitly set the signatureVersion when creating the s3 client object as follows:

var s3Obj = new AWS.S3({maxRetries: 10, "signatureVersion":"v4"});

Please let us know if this fixed or didn't fix your problem."

Can you try setting an equivalent S3 storage plugin configuration for `signatureVersion`? Please see here for an example https://drill.apache.org/docs/s3-storage-plugin/

Also, it would be good if you could share your S3 storage plugin configuration here.


Thanks,

Gautam

________________________________
From: Chandrakanth Akkinepalli <ch...@gmail.com>
Sent: Tuesday, May 23, 2017 1:34:25 PM
To: user@drill.apache.org
Subject: querying aws s3 objects that are server side encrypted

Hi All,
I have configured aws s3 storage plugin to access data in my s3 bucket.

I have placed both un-encrypted and aws-kms sse objects in the bucket.

I am able to query the un-encrypted objects just fine via apache drill.
But i am getting following error when querying aws-kms encrypted objects.

SQL Error: VALIDATION ERROR: Status Code: 400, AWS Service: Amazon S3, AWS
Request ID: ABCDEFGH12345, AWS Error Code: InvalidArgument, AWS Error
Message: Requests specifying Server Side Encryption with AWS KMS managed
keys require AWS Signature Version 4.

SQL Query null


Did anyone else face this issue? How do i resolve it?

Thanks,
Chandra