[GitHub] [pulsar-helm-chart] michaeljmarshall commented on a change in pull request #205: Tiered Storage config

[GitBox <gi...@apache.org>]

michaeljmarshall commented on a change in pull request #205:
URL: https://github.com/apache/pulsar-helm-chart/pull/205#discussion_r787352821



##########
File path: charts/pulsar/templates/broker-statefulset.yaml
##########
@@ -279,6 +279,13 @@ spec:
               path: broker/token
       {{- end}}
       {{- end}}
+      {{- if .Values.storageOffload.driver }}
+      {{- if eq .Values.storageOffload.driver "google-cloud-storage" }}
+      - name: gcp-service-account

Review comment:
       Should we also add this volume to the container's `volumeMounts`?

##########
File path: charts/pulsar/templates/broker-configmap.yaml
##########
@@ -43,6 +43,80 @@ data:
   zooKeeperSessionTimeoutMillis: "30000"
   statusFilePath: "{{ template "pulsar.home" . }}/status"
 
+  # Tiered storage settings
+  {{- if .Values.storageOffload.driver }}
+  {{- if eq .Values.storageOffload.driver "aws-s3" }}
+  managedLedgerOffloadDriver: "{{ .Values.storageOffload.driver }}" 
+  s3ManagedLedgerOffloadBucket: "{{ .Values.storageOffload.bucket }}" 
+  s3ManagedLedgerOffloadRegion: "{{ .Values.storageOffload.region }}" 
+  AWS_ACCESS_KEY_ID: "{{ .Values.storageOffload.accessKey }}" 
+  AWS_SECRET_ACCESS_KEY: "{{ .Values.storageOffload.accessSecret }}"
+  {{- if .Values.storageOffload.managedLedgerOffloadAutoTriggerSizeThresholdBytes }}
+  # Pre 2.6.0 value
+  managedLedgerOffloadAutoTriggerSizeThresholdBytes: "{{ .Values.storageOffload.managedLedgerOffloadAutoTriggerSizeThresholdBytes }}" 

Review comment:
       Given that the Pulsar community just made 2.6 EOL, I don't think we need to support it explicitly here. Note also that users can still make the chart work for <=2.6 by adding values here: `.Values.broker.configData`.

##########
File path: charts/pulsar/values.yaml
##########
@@ -405,6 +405,69 @@ zookeeper:
     usePolicy: true
     maxUnavailable: 1
 
+## Tiered Storage
+##
+storageOffload: {}
+  ## General
+  ## =======
+  # bucket: <bucket>
+  # region: <region>
+  # maxBlockSizeInBytes: "64000000" 
+  # readBufferSizeInBytes: "1000000"
+  ## The following are default values for the cluster. They can be changed
+  ## on each namespace.
+  # managedLedgerOffloadDeletionLagMs: "14400000"
+  # managedLedgerOffloadAutoTriggerSizeThresholdBytes: "-1" # disabled
+
+  # For AWS S3
+  # ======
+  # You must create an IAM account with access to the bucket and 
+  # generate keys for that account.
+  #
+  # driver: aws-s3
+  # accessKey: <access-key>
+  # accessSecret: <secret-key> # pragma: allowlist secret
+
+  # For S3 Compatible
+  # =================
+  # Need to create access and secret key for S3 compatible service
+  #
+  # driver: aws-s3
+  # accessKey: <access-key>
+  # accessSecret: <secret-key> # pragma: allowlist secret 
+  # serviceEndpoint: host:port
+ 
+  # For Aure Blob
+  # =================
+  # Need to create an Azure storage account and a blob containter (bucket) 
+  # To retrieve key, see https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal#code-try-1
+  #
+  # driver: azureblob
+  # storageAccount: <account name>
+  # storageAccountKey: <account key>
+  
+
+  ## For Google Cloud Storage
+  ## ====================
+  ## You must create a service account that has access to the objects in GCP buckets
+  ## and upload its key as a JSON file to a secret.
+  ## 
+  ## 1. Go to https://console.cloud.google.com/iam-admin/serviceaccounts
+  ## 2. Select your project.
+  ## 3. Create a new service account.
+  ## 4. Give the service account permission to access the bucket. For example, 
+  ##    the "Storage Object Admin" role.
+  ## 5. Create a key for the service account and save it as a JSON file.
+  ## 6. Save the JSON file in a secret:
+  ##       kubectl create secret generic pulsar-gcp-sa-secret \
+  ##          --from-file=kafkaesque-223201-f12856532197.json \

Review comment:
       I think we should make this example name generic.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org