You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/06/30 04:11:16 UTC
[GitHub] [pulsar] NiuBlibing opened a new issue, #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
NiuBlibing opened a new issue, #16294:
URL: https://github.com/apache/pulsar/issues/16294
**Describe the bug**
When set s3ManagedLedgerOffloadReadBufferSizeInBytes,managedLedgerOffloadMaxThreads,managedLedgerOffloadPrefetchRounds in the broker.conf,it not work in namespace.
**To Reproduce**
Steps to reproduce the behavior:
1. set the configures in the broker.conf
2. trigger the offload for a namespace `bin/pulsar-admin namespaces set-offload-threshold --size 100M test-tenant2/ns2`
3. check the policy `bin/pulsar-admin namespaces get-offload-policies test-tenant2/ns2` and get the following
```
{
"offloadersDirectory" : "./offloaders",
"managedLedgerOffloadMaxThreads" : 2,
"managedLedgerOffloadPrefetchRounds" : 1,
"managedLedgerOffloadThresholdInBytes" : 104857600,
"managedLedgerOffloadedReadPriority" : "TIERED_STORAGE_FIRST",
"s3ManagedLedgerOffloadMaxBlockSizeInBytes" : 67108864,
"s3ManagedLedgerOffloadReadBufferSizeInBytes" : 1048576,
"s3ManagedLedgerOffloadRoleSessionName" : "pulsar-s3-offload",
"gcsManagedLedgerOffloadMaxBlockSizeInBytes" : 67108864,
"gcsManagedLedgerOffloadReadBufferSizeInBytes" : 1048576,
"s3Driver" : false,
"gcsDriver" : false,
"fileSystemDriver" : false
}
```
**Desktop (please complete the following information):**
- OS: Centos7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] NiuBlibing commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
NiuBlibing commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1171143971
> managedLedgerOffloadReadBufferSizeInBytes
s3,
```
ls offloaders/*
offloaders/tiered-storage-file-system-2.10.0.nar offloaders/tiered-storage-jcloud-2.10.0.nar
```
```
### --- Ledger Offloading --- ###
# The directory for all the offloader implementations
offloadersDirectory=./offloaders
# Driver to use to offload old data to long term storage (Possible values: aws-s3, google-cloud-storage, azureblob, aliyun-oss, filesystem)
# When using google-cloud-storage, Make sure both Google Cloud Storage and Google Cloud Storage JSON API are enabled for
# the project (check from Developers Console -> Api&auth -> APIs).
managedLedgerOffloadDriver=aws-s3
# Maximum number of thread pool threads for ledger offloading
managedLedgerOffloadMaxThreads=20
# Maximum prefetch rounds for ledger reading for offloading
managedLedgerOffloadPrefetchRounds=100
# Use Open Range-Set to cache unacked messages
managedLedgerUnackedRangesOpenCacheSetEnabled=true
# For Amazon S3 ledger offload, AWS region
s3ManagedLedgerOffloadRegion=[xxxx]
# For Amazon S3 ledger offload, Bucket to place offloaded ledger into
s3ManagedLedgerOffloadBucket=[xxxxx]
# For Amazon S3 ledger offload, Alternative endpoint to connect to (useful for testing)
s3ManagedLedgerOffloadServiceEndpoint=http://[xxxx]
# For Amazon S3 ledger offload, Max block size in bytes. (64MB by default, 5MB minimum)
s3ManagedLedgerOffloadMaxBlockSizeInBytes=67108864
# For Amazon S3 ledger offload, Read buffer size in bytes (1MB by default)
#s3ManagedLedgerOffloadReadBufferSizeInBytes=1048576
s3ManagedLedgerOffloadReadBufferSizeInBytes=671088640
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] NiuBlibing commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
NiuBlibing commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1171979861
>  It took effect for me
My version is 2.10.0 with java 11, deploy with cluster, is it related to these?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1170858746
I will take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] NiuBlibing commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
NiuBlibing commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1171023125
> You need to do topic unload to make it take effect.
Thx, I've tried unload the topic, and the `bin/pulsar-admin namespaces set-offload-policies test-tenant/ns1 -rbs 128M -d aws-s3 -b jr-pulsar -oat 100M -e http://[xxxx]` works.
But but the broker.conf was still not work for me, when I removed the policies by `bin/pulsar-admin namespaces remove-offload-policies test-tenant/ns1` and unload the topic, it used the default config rather than the broker's
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] NiuBlibing commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
NiuBlibing commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1171133332
I think it should also be supported for s3, so that it can download parallelly for me.
> `managedLedgerOffloadPrefetchRounds` only apply to tiered-storage of file-system not work with S3 @NiuBlibing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ethqunzhong commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
ethqunzhong commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1175720289
it seems that `s3ManagedLedgerOffloadReadBufferSizeInBytes` and `managedLedgerOffloadReadBufferSizeInBytes` is an invalid configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1170957134
You need do topic unload to make it take effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1171161238
It took effect for me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ethqunzhong commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
ethqunzhong commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1171111988
`managedLedgerOffloadPrefetchRounds` only apply to tiered-storage of file-system not work with S3 @NiuBlibing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ethqunzhong commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
ethqunzhong commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1171140518
>
what is the offloader-driver you set?
and you cloud try this conf `managedLedgerOffloadReadBufferSizeInBytes`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] NiuBlibing commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
NiuBlibing commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1170951522
But when I removed the namespace policies,It didn't use the configurations in the broker.conf, and when I set the namespace policies,it need to restart the cluster brokers to take effect.
> The configuration in broker.conf took effect actually, its value is independent with offload-policies. The only problem is that the set-offload-policies doesn't support `managedLedgerOffloadPrefetchRounds `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] NiuBlibing commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
NiuBlibing commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1170934550
> Are you testing with standalone？ @NiuBlibing
[multi-cluster](https://pulsar.apache.org/docs/next/deploy-bare-metal-multi-cluster)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1170921046
Are you testing with standalone？ @NiuBlibing N
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] NiuBlibing commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
NiuBlibing commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1170868210
my broker config is
```
offloadersDirectory=./offloaders
# Driver to use to offload old data to long term storage (Possible values: aws-s3, google-cloud-storage, azureblob, aliyun-oss, filesystem)
# When using google-cloud-storage, Make sure both Google Cloud Storage and Google Cloud Storage JSON API are enabled for
# the project (check from Developers Console -> Api&auth -> APIs).
managedLedgerOffloadDriver=aws-s3
# Maximum number of thread pool threads for ledger offloading
managedLedgerOffloadMaxThreads=20
# Maximum prefetch rounds for ledger reading for offloading
managedLedgerOffloadPrefetchRounds=100
# Use Open Range-Set to cache unacked messages
managedLedgerUnackedRangesOpenCacheSetEnabled=true
# For Amazon S3 ledger offload, AWS region
s3ManagedLedgerOffloadRegion=[xxxxx]
# For Amazon S3 ledger offload, Bucket to place offloaded ledger into
s3ManagedLedgerOffloadBucket=[xxxxx]
# For Amazon S3 ledger offload, Alternative endpoint to connect to (useful for testing)
s3ManagedLedgerOffloadServiceEndpoint=[http://xxxxx]
# For Amazon S3 ledger offload, Max block size in bytes. (64MB by default, 5MB minimum)
s3ManagedLedgerOffloadMaxBlockSizeInBytes=67108864
# For Amazon S3 ledger offload, Read buffer size in bytes (1MB by default)
#s3ManagedLedgerOffloadReadBufferSizeInBytes=1048576
s3ManagedLedgerOffloadReadBufferSizeInBytes=671088640
```
It's not work, the offloader use the default config which fetch s3 ledgers 1MB without pretch leading the throughtoutput is only about 10Mbit/s
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1171131914
> >
>
> it seems that namespace-offload-policies not merge with broker config. i set `managedLedgerOffloadMaxThreads=10` in broker.conf and test in standalone mode, this argument take effort in PulsarService, but when i check my offloaded-namespace's policies still see `managedLedgerOffloadMaxThreads=2`
Yes, they are independent with each other.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1171135270
I am not familiar with these storage, I cannot be sure `managedLedgerOffloadPrefetchRounds ` `managedLedgerOffloadMaxThreads ` should be supported
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1211489438
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] NiuBlibing commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
NiuBlibing commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1171135770
> > >
> >
> >
> > it seems that namespace-offload-policies not merge with broker config. i set `managedLedgerOffloadMaxThreads=10` in broker.conf and test in standalone mode, this argument take effort in PulsarService, but when i check my offloaded-namespace's policies still see `managedLedgerOffloadMaxThreads=2`
>
> Yes, they are independent with each other.
But the `s3ManagedLedgerOffloadReadBufferSizeInBytes` was not effect for me,I checked the packet by tcpdump, the http range header was the default value 1MB rather than the value in the broker.conf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1170912190
There is a bug, I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1170944713
The configuration in broker.conf took effect actually, its value is independent with offload-policies.
The only problem is that the set-offload-policies doesn't support `managedLedgerOffloadPrefetchRounds `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dave2wave commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
dave2wave commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1180823091
I was able to see an impact on performance using:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] NiuBlibing commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
NiuBlibing commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1170893997
It not works even when set the value by `bin/pulsar-admin namespaces set-offload-policies test-tenant/ns1 -rbs 128M -d aws-s3 -b jr-pulsar -oat 100M -e http://[xxxx]`
It works when set the value by the `bin/pulsar-admin namespaces set-offload-policies test-tenant/ns1 -rbs 128M -d aws-s3 -b jr-pulsar -oat 100M -e http://[xxxx]` command and need to restart the broker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] NiuBlibing commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
NiuBlibing commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1170895054
But I didn't found how to set the managedLedgerOffloadPrefetchRounds config with the command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ethqunzhong commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
ethqunzhong commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1171099083
>
it seems that namespace-offload-policies not merge with broker config.
i set `managedLedgerOffloadMaxThreads=10` in broker.conf and test in standalone mode, this argument take effort in PulsarService, but when i check my offloaded-namespace's policies still see `managedLedgerOffloadMaxThreads=2`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] NiuBlibing commented on issue #16294: s3ManagedLedgerOffloadReadBufferSizeInBytes not work when set in broker
Posted by GitBox <gi...@apache.org>.
NiuBlibing commented on issue #16294:
URL: https://github.com/apache/pulsar/issues/16294#issuecomment-1171217274
Then check the packets with wireshark, found the s3 download request range header is `Range: bytes=681574528-682623103`,is there something wrong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org