You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/11/03 13:28:52 UTC
[GitHub] [pulsar] devinbost opened a new issue #8435: GCP Tiered Storage org.jclouds.rest.ResourceNotFoundException - Pulsar 2.6.1
devinbost opened a new issue #8435:
URL: https://github.com/apache/pulsar/issues/8435
I have a heartbeat topic that receives messages on the topic `persistent://public/default/canary`, and I configured retention on this topic to test tiered storage in GCP. When I manually trigger offloading from the topic, I get the error `java.util.concurrent.CompletionException: org.jclouds.rest.ResourceNotFoundException: Not Found`, as shown below:
`bin/pulsar-admin topics offload --size-threshold 10M persistent://public/default/canary`
> Offload triggered for persistent://public/default/canary for messages before 873151:0:-1
`bin/pulsar-admin topics offload-status -w persistent://public/default/canary`
> Error in offload
> null
>
> Reason: Error offloading: org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: org.jclouds.rest.ResourceNotFoundException: Not Found
On the broker and bookie containers, I have the GCP JSON key file located at `conf/auth/tiered-storage.key`
In my broker.conf file, I have the GCS attributes configured as follows:
```
gcsManagedLedgerOffloadRegion=us-west-3
gcsManagedLedgerOffloadBucket=gs://my-bucket-name
gcsManagedLedgerOffloadMaxBlockSizeInBytes=67108864
gcsManagedLedgerOffloadReadBufferSizeInBytes=1048576
gcsManagedLedgerOffloadServiceAccountKeyFile=/pulsar/conf/auth/tiered-storage.key
```
The tiered-storage.key file looks as follows:
```
{
"type": "service_account",
"project_id": "my-project-id",
"private_key_id": "c8myid12345",
"private_key": "-----BEGIN PRIVATE KEY-----\nABCDEFG...XYZ=\n-----END PRIVATE KEY-----\n",
"client_email": "example-email@identity-example.iam.gserviceaccount.com",
"client_id": "123456",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/example-cert-url%40identity-example.iam.gserviceaccount.com"
}
```
It seems that the key file path is incorrect. However, when I tried altering the key file to contain invalid JSON, I got an error message that the JSON was invalid when I triggered a manual offload of the topic, which implies that Pulsar was able to access the key file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] devinbost commented on issue #8435: GCP Tiered Storage org.jclouds.rest.ResourceNotFoundException - Pulsar 2.6.1
Posted by GitBox <gi...@apache.org>.
devinbost commented on issue #8435:
URL: https://github.com/apache/pulsar/issues/8435#issuecomment-720830210
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #8435: GCP Tiered Storage org.jclouds.rest.ResourceNotFoundException - Pulsar 2.6.1
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #8435:
URL: https://github.com/apache/pulsar/issues/8435#issuecomment-722777684
@devinbost Seems we should clarify it in the document, are you interested in fix it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org