You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "jean-baptiste-exorde (via GitHub)" <gi...@apache.org> on 2023/05/15 21:32:24 UTC

[GitHub] [druid] jean-baptiste-exorde opened a new issue, #14284: Historicals fail to download segments on S3 with SSE-C

jean-baptiste-exorde opened a new issue, #14284:
URL: https://github.com/apache/druid/issues/14284

   ## Affected Version
   
   Druid 25.
   
   ## Description
   
   When S3 with SSE-C encryption is used as deep storage, the middlemanager nodes write segments successfully, but the historical nodes fail to download the segments from S3.
   
   Segment files are visible on S3 and can be downloaded manually. But, when historical node retrieves the file, it first checks if the segment file exists by sending a HEAD request to S3 and receives an error 400.
   
   I had the issue with a S3-compatible provider as well as with AWS S3.
   
   Below are an example error logs:
   
   ```
   2023-05-11T21:31:38,373 INFO [SimpleDataSegmentChangeHandler-0] org.apache.druid.storage.s3.S3DataSegmentPuller - Pulling index at path[CloudObjectLocation{bucket='xxx', path='xxx'}] to outDir[/var/druid/segment-cache/xxx]
   2023-05-11T21:31:38,374 DEBUG [SimpleDataSegmentChangeHandler-0] com.amazonaws.request - Sending Request: HEAD https://xxx xxx Headers: (amz-sdk-invocation-id: xxx, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.12.317 Linux/5.4.0-124-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Debian-1deb11u1 java/11.0.16 vendor/Debian cfg/retry-mode/legacy, )
   2023-05-11T21:31:38,375 DEBUG [SimpleDataSegmentChangeHandler-0] com.amazonaws.auth.AWS4Signer - AWS4 Canonical Request: '"HEAD
   xxx
   
   amz-sdk-invocation-id:xxx
   amz-sdk-request:ttl=20230511T213228Z;attempt=1;max=4
   amz-sdk-retry:0/0/500
   content-type:application/octet-stream
   host:xxx
   user-agent:aws-sdk-java/1.12.317 Linux/5.4.0-124-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Debian-1deb11u1 java/11.0.16 vendor/Debian cfg/retry-mode/legacy
   x-amz-content-sha256:UNSIGNED-PAYLOAD
   x-amz-date:20230511T213138Z
   
   amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
   UNSIGNED-PAYLOAD"
   2023-05-11T21:31:38,375 DEBUG [SimpleDataSegmentChangeHandler-0] com.amazonaws.auth.AWS4Signer - AWS4 String to Sign: '"AWS4-HMAC-SHA256
   20230511T213138Z
   20230511/gra/s3/aws4_request
   xxx"
   ...
   2023-05-11T21:31:38,378 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.headers - http-outgoing-0 >> HEAD xxx HTTP/1.1
   2023-05-11T21:31:38,378 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.headers - http-outgoing-0 >> Host: xxx
   2023-05-11T21:31:38,378 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.headers - http-outgoing-0 >> amz-sdk-invocation-id: xxx
   2023-05-11T21:31:38,378 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.headers - http-outgoing-0 >> amz-sdk-request: ttl=20230511T213228Z;attempt=1;max=4
   2023-05-11T21:31:38,378 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.headers - http-outgoing-0 >> amz-sdk-retry: 0/0/500
   2023-05-11T21:31:38,378 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.headers - http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=xxx
   2023-05-11T21:31:38,378 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.headers - http-outgoing-0 >> Content-Type: application/octet-stream
   2023-05-11T21:31:38,378 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.headers - http-outgoing-0 >> User-Agent: aws-sdk-java/1.12.317 Linux/5.4.0-124-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Debian-1deb11u1 java/11.0.16 vendor/Debian cfg/retry-mode/legacy
   2023-05-11T21:31:38,378 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.headers - http-outgoing-0 >> x-amz-content-sha256: UNSIGNED-PAYLOAD
   2023-05-11T21:31:38,378 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.headers - http-outgoing-0 >> X-Amz-Date: 20230511T213138Z
   2023-05-11T21:31:38,378 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.headers - http-outgoing-0 >> Connection: Keep-Alive
   2023-05-11T21:31:38,378 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 >> "HEAD xxx HTTP/1.1[\r][\n]"
   2023-05-11T21:31:38,378 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 >> "Host: xxx[\r][\n]"
   2023-05-11T21:31:38,378 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 >> "amz-sdk-invocation-id: xxx[\r][\n]"
   2023-05-11T21:31:38,378 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 >> "amz-sdk-request: ttl=20230511T213228Z;attempt=1;max=4[\r][\n]"
   2023-05-11T21:31:38,378 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
   2023-05-11T21:31:38,378 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=xxx[\r][\n]"
   2023-05-11T21:31:38,378 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
   2023-05-11T21:31:38,378 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 >> "User-Agent: aws-sdk-java/1.12.317 Linux/5.4.0-124-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Debian-1deb11u1 java/11.0.16 vendor/Debian cfg/retry-mode/legacy[\r][\n]"
   2023-05-11T21:31:38,378 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]"
   2023-05-11T21:31:38,379 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 >> "X-Amz-Date: 20230511T213138Z[\r][\n]"
   2023-05-11T21:31:38,379 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
   2023-05-11T21:31:38,379 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 >> "[\r][\n]"
   2023-05-11T21:31:38,401 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 << "HTTP/1.1 400 Bad Request[\r][\n]"
   2023-05-11T21:31:38,402 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
   2023-05-11T21:31:38,402 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 << "x-amz-id-2: xxx[\r][\n]"
   2023-05-11T21:31:38,402 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 << "x-amz-request-id: xxx[\r][\n]"
   2023-05-11T21:31:38,402 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 << "X-Trans-Id: xxx[\r][\n]"
   2023-05-11T21:31:38,402 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 << "X-Openstack-Request-Id: xxx[\r][\n]"
   2023-05-11T21:31:38,402 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 << "Date: Thu, 11 May 2023 21:31:38 GMT[\r][\n]"
   2023-05-11T21:31:38,402 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 << "Connection: keep-alive[\r][\n]"
   2023-05-11T21:31:38,402 DEBUG [SimpleDataSegmentChangeHandler-0] org.apache.http.wire - http-outgoing-0 << "[\r][\n]"
   ...
   2023-05-11T21:31:38,404 DEBUG [SimpleDataSegmentChangeHandler-0] com.amazonaws.request - Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: xxx; S3 Extended Request ID: xxx; Proxy: null), S3 Extended Request ID: xxx
   2023-05-11T21:31:38,405 ERROR [SimpleDataSegmentChangeHandler-0] org.apache.druid.segment.loading.SegmentLocalCacheManager - Failed to load segment in current location [/var/druid/segment-cache], try next location if any: {class=org.apache.druid.segment.loading.SegmentLocalCacheManager, exceptionType=class org.apache.druid.segment.loading.SegmentLoadingException, exceptionMessage=S3 fail! Key[CloudObjectLocation{bucket='xxx', path='xxx'}], location=/var/druid/segment-cache}
   org.apache.druid.segment.loading.SegmentLoadingException: S3 fail! Key[CloudObjectLocation{bucket='xxx', path='xxx'}]
           at org.apache.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:292) ~[?:?]
           at org.apache.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:80) ~[?:?]
           at org.apache.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:61) ~[?:?]
           at org.apache.druid.segment.loading.SegmentLocalCacheManager.loadInLocation(SegmentLocalCacheManager.java:343) ~[druid-server-25.0.0.jar:25.0.0]
           ...
   Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: xxx; S3 Extended Request ID: xxx; Proxy: null)
           at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879) ~[aws-java-sdk-core-1.12.317.jar:?]
           ...
   ```
   
   ### Steps to reproduce the issue
   
   To reproduce the issue, we can deploy a simple cluster following the [quickstart](https://druid.apache.org/docs/latest/tutorials/index.html).
   
   1. Create a AWS S3 bucket and get credentials.
   2. Download Druid:
   
      ```bash
      wget https://dlcdn.apache.org/druid/25.0.0/apache-druid-25.0.0-bin.tar.gz
      tar -xzf apache-druid-25.0.0-bin.tar.gz
      ```
   
   3. Edit file `apache-druid-25.0.0/conf/druid/auto/_common/common.runtime.properties` to setup S3 deep storage:
   
      ```diff
      #...
   
      -druid.extensions.loadList=["druid-hdfs-storage", "druid-kafka-indexing-service", "druid-datasketches", "druid-multi-stage-query"]
      +druid.extensions.loadList=["druid-s3-extensions"]
   
      #...
   
      -druid.storage.type=local
      -druid.storage.storageDirectory=var/druid/segments
      +druid.storage.type=s3
      +druid.storage.bucket=<bucket-name>
      +druid.storage.baseKey=druid/segments
      +druid.s3.accessKey=<access-key>
      +druid.s3.secretKey=<secret-key>
      +druid.storage.sse.type=custom
      +druid.storage.sse.custom.base64EncodedKey=<encryption-key>
      +druid.s3.endpoint.url=<endpoint>
      +druid.storage.disableAcl=true
   
      # ...
      ```
   
   4. Edit file `apache-druid-25.0.0/conf/druid/auto/_common/log4j2.xml` to enable debug logs:
   
      ```diff
      # ...
      <Loggers>
      -   <Root level="info">
      +   <Root level="debug">
             <AppenderRef ref="FileAppender"/>
          </Root>
      # ...
      ```
   
   5. Start Apache Druid:
   
      ```bash
      cd apache-druid-25.0.0
      AWS_REGION=<REGION> ./bin/start-druid
      ```
   
   6. Load data following the [quickstart](https://druid.apache.org/docs/latest/tutorials/tutorial-batch.html).
   
   Result:
   
   - Ingestion task run successfully and the segment is visible on S3.
   - From the CLI or the [web console](http://localhost:8888/unified-console.html#datasources), we can check the state of the published segment: it is unavailable.
   - The logs of historical node contains the error mentioned above.
   
   ## Possible cause
   
   In logs, we can see a HEAD request to get the object's metadata. This request doesn't contains the encryption key. As per the [AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html#specifying-s3-c-encryption), it's mandatory to provide it when requesting object's metadata. I might be the cause of the error 400.
   
   We have this stack trace telling us that the error occurs when the node checks if the file exists:
   
   ```
   at org.apache.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:292)
   ```
   
   Here are the calls made from this stack trace to the error:
   
   - [S3DataSegmentPuller.isObjectInBucket()](https://github.com/apache/druid/blob/25.0.0/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3DataSegmentPuller.java#L284)
     - [S3Utils.isObjectInBucketIgnoringPermission()](https://github.com/apache/druid/blob/25.0.0/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3Utils.java#L110)
       - [ServerSideEncryptingAmazonS3.doesObjectExist()](https://github.com/apache/druid/blob/25.0.0/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/ServerSideEncryptingAmazonS3.java#L77)
         - [AmazonS3Client.doesObjectExist()](https://github.com/aws/aws-sdk-java/blob/96a0c2f8f11842c7e58498dbf294173fe2356221/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/AmazonS3Client.java#L1423) of the AWS S3 SDK.
           - [AmazonS3Client.getObjectMetadata()](https://github.com/aws/aws-sdk-java/blob/96a0c2f8f11842c7e58498dbf294173fe2356221/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/AmazonS3Client.java#L1345) of the AWS S3 SDK.
   
   The last method sends the HEAD request without SSE parameters and raise the error.
   
   In the class [ServerSideEncryptingAmazonS3](https://github.com/apache/druid/blob/25.0.0/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/ServerSideEncryptingAmazonS3.java), we can see that most AWS S3 SDK calls are wrapped to add the SSE parameters, but the method `doesObjectExist` is not wrapped. So I think that the AWS S3 SDK is called without any information about the SSE parameters and it calls the S3 API like if objects were not encrypted.
   
   Can you tell me if you have the same issue and if it's a bug or a configuration issue?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jean-baptiste-exorde commented on issue #14284: Historicals fail to download segments on S3 with SSE-C

Posted by "jean-baptiste-exorde (via GitHub)" <gi...@apache.org>.
jean-baptiste-exorde commented on issue #14284:
URL: https://github.com/apache/druid/issues/14284#issuecomment-1608162908

   Hi, since PR #14290 is merged, this issue will be resolved in next Druid release. Thanks for your help!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jean-baptiste-exorde commented on issue #14284: Historicals fail to download segments on S3 with SSE-C

Posted by "jean-baptiste-exorde (via GitHub)" <gi...@apache.org>.
jean-baptiste-exorde commented on issue #14284:
URL: https://github.com/apache/druid/issues/14284#issuecomment-1551573877

   Hi, I built Druid from your branch and tried it with my S3 provider: it works with SSE-C!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on issue #14284: Historicals fail to download segments on S3 with SSE-C

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on issue #14284:
URL: https://github.com/apache/druid/issues/14284#issuecomment-1550658590

   Thanks for the report! I think https://github.com/apache/druid/pull/14290 would fix it, but I haven't tested it. If you have a chance to build from source and test with your S3 account, please let us know.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jean-baptiste-exorde closed issue #14284: Historicals fail to download segments on S3 with SSE-C

Posted by "jean-baptiste-exorde (via GitHub)" <gi...@apache.org>.
jean-baptiste-exorde closed issue #14284: Historicals fail to download segments on S3 with SSE-C
URL: https://github.com/apache/druid/issues/14284


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org