You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/02/03 20:17:35 UTC
[GitHub] [pulsar] rangas26 opened a new issue #9461: Encryption support for aws-s3 offloader
rangas26 opened a new issue #9461:
URL: https://github.com/apache/pulsar/issues/9461
#### Expected behavior
aws-s3 offloader should upload the segments even when the target bucket have restricted bucket policy to allow writes of encrypted objects only.
#### Actual behavior
aws-s3 offloaders fail to upload segments when the policy of the target bucket requires all objects to be encrypted.
offloader fails with the error below
```
Caused by: org.jclouds.aws.AWSResponseException:
request POST https://offload-s3-test.s3.amazonaws.com/31548dee-c36c-4186-bdf4-1e969d70f5df-ledger-9?uploads HTTP/1.1 failed with code 403,
error: AWSError{requestId='70DDF12549897X83', requestToken='E4ng+sBnNfT6ssSs+uZ2yTS8nuMbpnKieqvknrjP88rqio2jbhaP9OdZ23o0O7nK5QEtAyfYgJM=',
code='AccessDenied',
message='Access Denied', context='{HostId=E4ng+sBnNfY6ssSs+uZ2yTS8nuMbpnKieqvknrjP88rqio2jbhaP9OdZ23o0O7nK5QEtAyfYgJM=}'}
```
#### Steps to reproduce
Edit the target bucket policy to allow only encrypted objects
```
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "RequiredEncryptedPutObject",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::offload-s3-test/*",
"Condition": {
"StringNotEquals": {
"s3:x-amz-server-side-encryption": [
"AES256",
"aws:kms"
]
}
}
}
]
}
```
Now try to offload the topic with the below command
```pulsar-admin topics offload --size-threshold 10k persistent://tenant/namespace/test-partition-0```
check the status of the upload:
```
pulsar-admin topics offload-status persistent://tenant/namespace/test-partition-0
Warning: Nashorn engine is planned to be removed from a future JDK release
Error in offload
null
Reason: Error offloading: org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: org.jclouds.rest.AuthorizationException: Access Denied
```
#### System configuration
**Pulsar version**: 2.7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Renkai commented on issue #9461: Encryption support for aws-s3 offloader
Posted by GitBox <gi...@apache.org>.
Renkai commented on issue #9461:
URL: https://github.com/apache/pulsar/issues/9461#issuecomment-773064869
@rangas26 AWS has server side encryption and client side encryption, which one would you like to use?
https://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html
https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Renkai commented on issue #9461: Encryption support for aws-s3 offloader
Posted by GitBox <gi...@apache.org>.
Renkai commented on issue #9461:
URL: https://github.com/apache/pulsar/issues/9461#issuecomment-773064869
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Renkai edited a comment on issue #9461: Encryption support for aws-s3 offloader
Posted by GitBox <gi...@apache.org>.
Renkai edited a comment on issue #9461:
URL: https://github.com/apache/pulsar/issues/9461#issuecomment-773301840
https://github.com/apache/jclouds
@rangas26 Could you have a try to use Jclouds to put an object on your bucket use SSE? Pulsar use Jclouds to offload data to S3 now, if Jclouds does not support SSE, Pulsar will not support it until Jclouds do or when Pulsar migrates from Jclouds to Official AWS SDK.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Renkai commented on issue #9461: Encryption support for aws-s3 offloader
Posted by GitBox <gi...@apache.org>.
Renkai commented on issue #9461:
URL: https://github.com/apache/pulsar/issues/9461#issuecomment-773005559
Looks like this part is not in our tests coverage yet, I will try to add some test to reproduce this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] rangas26 commented on issue #9461: Encryption support for aws-s3 offloader
Posted by GitBox <gi...@apache.org>.
rangas26 commented on issue #9461:
URL: https://github.com/apache/pulsar/issues/9461#issuecomment-773290565
@Renkai I was trying to use SSE (server side encryption). I have tested that the offloader works with out the policy in place but our organization policy will not allow to create buckets with out that policy.
Wondering if there are any plans to add this new feature. Ideally it should be configurable in the broker configuration
s3OffloaderEncryptionType=sse
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Renkai removed a comment on issue #9461: Encryption support for aws-s3 offloader
Posted by GitBox <gi...@apache.org>.
Renkai removed a comment on issue #9461:
URL: https://github.com/apache/pulsar/issues/9461#issuecomment-773073653
As far as I know, current offloaders don't support client side encryption,yet. So you may need to make you bucket to allow unencrypted object now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Renkai commented on issue #9461: Encryption support for aws-s3 offloader
Posted by GitBox <gi...@apache.org>.
Renkai commented on issue #9461:
URL: https://github.com/apache/pulsar/issues/9461#issuecomment-773301840
https://github.com/apache/jclouds
@rangas26 Could you have a try to use Jclouds to put an object on your bucket use SSE? Pulsar use Jclouds to offload data to S3 now, if Jclouds does not support SSE, Pulsar will not support it until Jclouds do or when Pulsar migrate from Jclouds to Official AWS SDK.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] rangas26 commented on issue #9461: Encryption support for aws-s3 offloader
Posted by GitBox <gi...@apache.org>.
rangas26 commented on issue #9461:
URL: https://github.com/apache/pulsar/issues/9461#issuecomment-773290565
@Renkai I was trying to use SSE (server side encryption). I have tested that the offloader works with out the policy in place but our organization policy will not allow to create buckets with out that policy.
Wondering if there are any plans to add this new feature. Ideally it should be configurable in the broker configuration
s3OffloaderEncryptionType=sse
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9461: Encryption support for aws-s3 offloader
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9461:
URL: https://github.com/apache/pulsar/issues/9461#issuecomment-1058893544
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Renkai edited a comment on issue #9461: Encryption support for aws-s3 offloader
Posted by GitBox <gi...@apache.org>.
Renkai edited a comment on issue #9461:
URL: https://github.com/apache/pulsar/issues/9461#issuecomment-773301840
https://github.com/apache/jclouds
@rangas26 Could you have a try to use Jclouds to put an object on your bucket use SSE? Pulsar use Jclouds to offload data to S3 now, if Jclouds does not support SSE, Pulsar will not support it until Jclouds do or when Pulsar migrates from Jclouds to Official AWS SDK.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Renkai removed a comment on issue #9461: Encryption support for aws-s3 offloader
Posted by GitBox <gi...@apache.org>.
Renkai removed a comment on issue #9461:
URL: https://github.com/apache/pulsar/issues/9461#issuecomment-773073653
As far as I know, current offloaders don't support client side encryption,yet. So you may need to make you bucket to allow unencrypted object now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Renkai commented on issue #9461: Encryption support for aws-s3 offloader
Posted by GitBox <gi...@apache.org>.
Renkai commented on issue #9461:
URL: https://github.com/apache/pulsar/issues/9461#issuecomment-773073653
As far as I know, current offloaders don't support client side encryption,yet. So you may need to make you bucket to allow unencrypted object now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Renkai commented on issue #9461: Encryption support for aws-s3 offloader
Posted by GitBox <gi...@apache.org>.
Renkai commented on issue #9461:
URL: https://github.com/apache/pulsar/issues/9461#issuecomment-773084273
Pulsar already have an end to end encryption (https://pulsar.apache.org/docs/en/security-encryption/)
If you enabled it, the data is offloaded is encrypted, but it's not encrypted in S3's standard, so S3 will recognize it as unencrypted data, you'd better disable the `RequiredEncryptedPutObject` check in your bucket.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org