You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/05/19 07:40:16 UTC
[GitHub] [pulsar] y00273676 opened a new issue #10636: Within EKS to offload a topic to S3 got AuthorizationException
y00273676 opened a new issue #10636:
URL: https://github.com/apache/pulsar/issues/10636
**Describe the bug**
As deploying Pulsar inside AWS EKS (Kubernetes hosted by AWS), while trying to use the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to assume the AWS credential in order to offload ledgers into S3 Bucket, I've encountered exceptions like:
```
Error in offload
null
Reason: Error offloading: org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: org.jclouds.rest.AuthorizationException: The AWS Access Key Id you provided does not exist in our records.
command terminated with exit code 1
```
Login the eks pod
```
root@pulsar-broker-3:/pulsar# echo $AWS_SECRET_ACCESS_KEY
****************URQQ
root@pulsar-broker-3:/pulsar# echo $AWS_SECRET_ACCESS_KEY
****************I+Gz
root@pulsar-broker-3:/pulsar# aws configure list
Name Value Type Location
---- ----- ---- --------
profile <not set> None None
access_key ****************URQQ env
secret_key ****************I+Gz env
region cn-northwest-1 env ['AWS_REGION', 'AWS_DEFAULT_REGION']
```
So the AWS_SECRET_ACCESS_KEY and AWS_SECRET_ACCESS_KEY is valid
**To Reproduce**
Steps to reproduce the behavior:
precondition： the aws s3 bucket was created
1. Deploy Pulsar Broker within a EKS pod, if that is too costy, you can inject environement variables and broker.conf like this:
```
broker.conf:
managedLedgerOffloadDriver: "aws-s3"
s3ManagedLedgerOffloadBucket: "xxxxxxx"
s3ManagedLedgerOffloadRegion: "cn-northwest-1"
env:
- name: "AWS_ACCESS_KEY_ID"
value: "****************URQQ "
- name: "AWS_SECRET_ACCESS_KEY"
value: "****************I+Gz"
```
2. Set the. offload threshold manually
```
kubectl -n pulsar exec -ti toolset-0 -c pulsar-toolset -- ./bin/pulsar-admin --auth-params "token:xxxxxxxxxx" --auth-plugin "org.apache.pulsar.client.impl.auth.AuthenticationToken" --admin-url http://pulsar-broker.pulsar:8080 topics offload --size-threshold 1K yourTopic
```
3. Send pulsar msg to you topic
4. Get offload status
```
kubectl -n pulsar exec -ti toolset-0 -c pulsar-toolset -- ./bin/pulsar-admin --auth-params "token:xxxxxxxx" --auth-plugin "org.apache.pulsar.client.impl.auth.AuthenticationToken" --admin-url http://pulsar-broker.pulsar:8080 topics offload-status -w yourTopic
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #10636: Within EKS to offload a topic to S3 got AuthorizationException
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #10636:
URL: https://github.com/apache/pulsar/issues/10636#issuecomment-1058890180
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zymap commented on issue #10636: Within EKS to offload a topic to S3 got AuthorizationException
Posted by GitBox <gi...@apache.org>.
zymap commented on issue #10636:
URL: https://github.com/apache/pulsar/issues/10636#issuecomment-845623420
@y00273676 Can you use the same credential to access AWS S3 successfully by AWS CLI? Looks like the secret key missing? You may check if it exists in the AWS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zymap commented on issue #10636: Within EKS to offload a topic to S3 got AuthorizationException
Posted by GitBox <gi...@apache.org>.
zymap commented on issue #10636:
URL: https://github.com/apache/pulsar/issues/10636#issuecomment-845928642
How did you deploy pulsar to EKS? Do you have any configuration files? Do you deploy the cluster with helm charts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] y00273676 commented on issue #10636: Within EKS to offload a topic to S3 got AuthorizationException
Posted by GitBox <gi...@apache.org>.
y00273676 commented on issue #10636:
URL: https://github.com/apache/pulsar/issues/10636#issuecomment-845715599
> @y00273676 Can you use the same credential to access AWS S3 successfully by AWS CLI? Looks like the secret key missing? You may check if it exists in the AWS
The secret key is not missing, I can upload file to s3 and get key list by AWS CLI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org