You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/07/23 03:47:41 UTC
[GitHub] [pulsar] Shenfeng1011 opened a new issue #7639: offload_policies ineffective
Shenfeng1011 opened a new issue #7639:
URL: https://github.com/apache/pulsar/issues/7639
can not offload data to hdfs?
version:2.6.0
1.broker.conf
managedLedgerOffloadDriver=filesystem
fileSystemProfilePath="../conf/filesystem_offload_core_site.xml"
fileSystemURI="hdfs://10.130.161.123:8020"
2.filesystem_offload_core_site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://10.130.161.123:8020</value>
</property>
3.offloaders directory content
tiered-storage-file-system-2.6.0.nar
tiered-storage-jcloud-2.6.0.nar
operate step
4.pulsar standalone
5.pulsar-admin namespaces set-offload-threshold --size 2K public/default
the log is:
1:28:01.723 [pulsar-web-69-4] INFO org.apache.pulsar.broker.admin.impl.NamespacesBase - [null] Successfully updated offloadThreshold configuration: namespace=public/default, value=2048
11:28:01.723 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x1000d0ae910000a local:/127.0.0.1:44442 remoteserver:localhost/127.0.0.1:2181 lastZxid:5104 xid:80 sent:174 recv:176 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: WatchedEvent state:SyncConnected type:NodeDataChanged path:/admin/policies/public/default
11:28:01.724 [pulsar-web-69-4] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [23/7月/2020:11:28:01 +0800] "PUT /admin/v2/namespaces/public/default/offloadThreshold HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.5.2" 40
11:28:01.725 [ForkJoinPool.commonPool-worker-1] INFO org.apache.pulsar.broker.service.BrokerService - /admin/policies/public/default updating with Policies{auth_policies=org.apache.pulsar.common.policies.data.AuthPolicies@745f, replication_clusters=[standalone], bundles=BundlesData{numBundles=4, boundaries=[0x00000000, 0x40000000, 0x80000000, 0xc0000000, 0xffffffff]}, backlog_quota_map={}, persistence=null, deduplicationEnabled=null, autoTopicCreationOverride=null, autoSubscriptionCreationOverride=null, clusterDispatchRate={}, topicDispatchRate={}, subscriptionDispatchRate={}, replicatorDispatchRate={}, clusterSubscribeRate={}, publishMaxMessageRate={}, latency_stats_sample_rate={}, antiAffinityGroup=null, message_ttl_in_seconds=0, subscription_expiration_time_minutes=0, retention_policies=null, deleted=false, encryption_required=false, delayed_delivery_policies=null, subscription_auth_mode=None, max_producers_per_topic=0, max_consumers_per_topic=0, max_consumers_per_subscription
=0, max_unacked_messages_per_consumer=-1, max_unacked_messages_per_subscription=-1, compaction_threshold=0, offload_threshold=2048, offload_deletion_lag_ms=null, schema_auto_update_compatibility_strategy=Full, schema_validation_enforced=false, schema_compatibility_Strategy=UNDEFINED, is_allow_auto_update_Schema=true, offload_policies=OffloadPolicies{managedLedgerOffloadDriver=null, managedLedgerOffloadMaxThreads=2, managedLedgerOffloadPrefetchRounds=1, managedLedgerOffloadThresholdInBytes=2048, managedLedgerOffloadDeletionLagInMillis=null, s3ManagedLedgerOffloadRegion=null, s3ManagedLedgerOffloadBucket=null, s3ManagedLedgerOffloadServiceEndpoint=null, s3ManagedLedgerOffloadMaxBlockSizeInBytes=67108864, s3ManagedLedgerOffloadReadBufferSizeInBytes=1048576, s3ManagedLedgerOffloadRole=null, s3ManagedLedgerOffloadRoleSessionName=pulsar-s3-offload, gcsManagedLedgerOffloadRegion=null, gcsManagedLedgerOffloadBucket=null, gcsManagedLedgerOffloadMaxBlockSizeInBytes=67108864, gcsManagedLedgerO
ffloadReadBufferSizeInBytes=1048576, gcsManagedLedgerOffloadServiceAccountKeyFile=null, fileSystemProfilePath=null, fileSystemURI=null}}
6.publish some message more than 2k but in hdfs found nothing
7.bin/pulsar-admin topics offload --size-threshold 1K public/default/my-vehicle-string-topic3
11:45:41.081 [pulsar-web-69-11] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/my-vehicle-string-topic3] Starting offload operation at messageId 481:0:-1
11:45:41.081 [pulsar-web-69-11] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/my-vehicle-string-topic3] Start ledgersOffload. ledgers=[481] totalSize=1800
11:45:41.082 [pulsar-web-69-11] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/my-vehicle-string-topic3] No ledgers to offload
11:45:41.082 [pulsar-web-69-11] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/my-vehicle-string-topic3] Completed successfully offload operation at messageId 481:0:-1
still found nothing in hdfs
8.bin/pulsar-admin namespaces policies public/default
found
"offload_policies" : {
"offloadersDirectory" : "./offloaders",
"managedLedgerOffloadMaxThreads" : 2,
"managedLedgerOffloadPrefetchRounds" : 1,
"managedLedgerOffloadThresholdInBytes" : 2048,
"s3ManagedLedgerOffloadMaxBlockSizeInBytes" : 67108864,
"s3ManagedLedgerOffloadReadBufferSizeInBytes" : 1048576,
"s3ManagedLedgerOffloadRoleSessionName" : "pulsar-s3-offload",
"gcsManagedLedgerOffloadMaxBlockSizeInBytes" : 67108864,
"gcsManagedLedgerOffloadReadBufferSizeInBytes" : 1048576,
"gcsDriver" : false,
"fileSystemDriver" : false,
"s3Driver" : false
}
8.it seams that fileSystemDriver is not loaded. where was i wrong?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Shenfeng1011 closed issue #7639: offload_policies ineffective
Posted by GitBox <gi...@apache.org>.
Shenfeng1011 closed issue #7639:
URL: https://github.com/apache/pulsar/issues/7639
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Shenfeng1011 commented on issue #7639: offload_policies ineffective
Posted by GitBox <gi...@apache.org>.
Shenfeng1011 commented on issue #7639:
URL: https://github.com/apache/pulsar/issues/7639#issuecomment-662827439
waiting on line ~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Shenfeng1011 closed issue #7639: offload_policies ineffective
Posted by GitBox <gi...@apache.org>.
Shenfeng1011 closed issue #7639:
URL: https://github.com/apache/pulsar/issues/7639
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org