You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "chrislusf (via GitHub)" <gi...@apache.org> on 2026/09/12 05:32:50 UTC
[PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
chrislusf opened a new pull request, #14160:
URL: https://github.com/apache/cloudstack/pull/14160
## Summary
Adds SeaweedFS as a first-class object storage provider in CloudStack, alongside the existing MinIO, Ceph RGW, and Cloudian HyperStore providers.
SeaweedFS exposes an S3-compatible API and an AWS IAM-compatible API, so this provider uses the AWS S3 and IAM Java SDKs — the same approach as the Cloudian HyperStore provider. No proprietary admin client is needed.
### Key features
- **Bucket CRUD, policy, versioning, encryption, ACLs** via `AmazonS3` SDK (AWS SDK v1, same as Ceph/Cloudian)
- **Per-account IAM user provisioning** via `AmazonIdentityManagement` SDK (same as Cloudian HyperStore)
- **Per-bucket quota** via the SeaweedFS S3 `?seaweedfs-quota` extension (`PUT /{bucket}?seaweedfs-quota`), authenticated via SigV4 and authorized via the `s3:PutBucketQuota` IAM permission. This requires SeaweedFS PR #11279 (merged).
- **Usage reporting** via S3 `ListObjectsV2` (MVP; Prometheus or SOSAPI `capacity.xml` recommended for production scale)
### Architecture
The plugin follows the Cloudian HyperStore pattern almost line for line:
- Same store-details keys: `s3Url`, `iamUrl`, `accesskey`, `secretkey`
- `iamUrl` defaults to `s3Url` (SeaweedFS registers its IAM API at `POST /` on the same S3 endpoint)
- Same IAM-user-with-restricted-policy pattern
- Same Spring wiring
### Quota management
SeaweedFS enforces bucket quota server-side (read-only flag when usage exceeds the limit). The configuration surface is a narrow S3 subresource — `PUT /{bucket}?seaweedfs-quota` — authenticated via standard S3 SigV4 and authorized via dedicated `s3:PutBucketQuota`/`s3:GetBucketQuota` IAM permissions. This avoids exposing the broad SeaweedFS admin API to CloudStack. The plugin signs the request with `AWSS3V4Signer` and sends it via `java.net.http.HttpClient` (the AWS S3 SDK doesn't natively support custom subresources).
### Comparison with MinIO and Ceph
| Concern | MinIO | Ceph RGW | Cloudian | SeaweedFS |
|---------|-------|----------|----------|-----------|
| Bucket CRUD | `MinioClient` | `AmazonS3` | `AmazonS3` | `AmazonS3` |
| User creation | `MinioAdminClient` | `RgwAdmin` | `AmazonIdentityManagement` | `AmazonIdentityManagement` |
| Per-bucket quota | `MinioAdminClient` | `RgwAdmin` | not supported | S3 extension (SigV4) |
| Usage reporting | `MinioAdminClient` | `RgwAdmin` | Cloudian admin API | S3 `ListObjectsV2` |
### Files
New module under `plugins/storage/object/seaweedfs/`:
- `pom.xml` — Maven module
- `SeaweedFSObjectStoreProviderImpl.java` — Spring provider registration
- `SeaweedFSObjectStoreLifeCycleImpl.java` — Pool add/health-check, URL validation
- `SeaweedFSObjectStoreDriverImpl.java` — Bucket + user ops via S3 + IAM SDK
- `SeaweedFSObjectStoreUtil.java` — S3 + IAM client builders, constants, SigV4 quota request
- Spring/module metadata
- Tests (18 tests, all passing)
### SeaweedFS dependency
Requires SeaweedFS with the `?seaweedfs-quota` S3 extension (PR seaweedfs/seaweedfs#11279, merged). Without it, quota operations will fail with 404; all other operations (bucket CRUD, user provisioning, usage) work with any recent SeaweedFS release.
#### Test plan
- [ ] Unit tests pass: `mvn -pl plugins/storage/object/seaweedfs test` (18 tests, 0 failures)
- [ ] Manual: register a SeaweedFS object store via `addObjectStoragePool` with `s3Url`, `accesskey`, `secretkey`
- [ ] Manual: create a bucket via CloudStack, verify it appears in SeaweedFS
- [ ] Manual: set a bucket quota via CloudStack, verify it appears in SeaweedFS via `weed shell` `s3.bucket.list`
- [ ] Manual: provision a user for a CloudStack account, verify IAM user appears in SeaweedFS
- [ ] Manual: verify usage reporting returns bucket sizes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r4012864842
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,938 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.BucketCrossOriginConfiguration;
+import com.amazonaws.services.s3.model.CORSRule;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketCrossOriginConfigurationRequest;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ /**
+ * DB-backed global lock name prefix for serializing IAM provisioning and
+ * policy refreshes per store+account. Uses {@link GlobalLock} so the
+ * critical section is serialized across management servers in a
+ * clustered deployment, not just within a single JVM.
+ */
+ private static final String IAM_LOCK_PREFIX = "seaweedfs.iam.";
+
+ private static String getIamLockName(long storeId, long accountId) {
+ return IAM_LOCK_PREFIX + storeId + "." + accountId;
+ }
+
+ /**
+ * Acquire a DB-backed global lock for IAM operations on the given
+ * store+account. Returns a {@link GlobalLock} that the caller must
+ * {@link GlobalLock#unlock()} in a {@code finally} block, or {@code null}
+ * if the lock could not be acquired within the timeout.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ */
+ protected GlobalLock acquireIamLock(long storeId, long accountId) {
+ GlobalLock lock = GlobalLock.getInternLock(getIamLockName(storeId, accountId));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire IAM lock for store {} account {}", storeId, accountId);
+ lock.releaseRef();
+ return null;
+ }
+ return lock;
+ }
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account and
+ * store. The store ID is included so that two CloudStack pools pointing
+ * at the same SeaweedFS IAM service do not collide on the same
+ * {@code acs-<uuid>} user and overwrite each other's policy and access
+ * keys.
+ */
+ protected String getUserNameForAccount(Account account, long storeId) {
+ return String.format("%s-%d-%s", ACS_PREFIX, storeId, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Serialize per store+account across management servers so two
+ // concurrent bucket requests do not both rotate credentials and leave
+ // bucket rows with mismatched key pairs.
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ return false;
+ }
+ try {
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach a scoped IAM policy that allows access only to this
+ // account's own buckets (the tenant boundary). Refreshed whenever
+ // buckets are created or deleted. Use the lock-free variant since
+ // createUser already holds the IAM lock.
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, null);
+
+ // Reuse the stored access key only if both the access key id and the
+ // secret key are present and the key is still Active in IAM; otherwise
+ // create a replacement.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ String storedSecretKey = details.get(secretKeyDetailKey);
+ if (storedAccessKeyId != null && storedSecretKey != null
+ && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ return true;
+ }
+
+ // The stored key is missing, inactive, or no longer in IAM. Clean up
+ // ALL keys (including the inactive stored one) before creating a
+ // replacement so we do not accumulate keys and hit IAM limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, null);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Update existing bucket records for this account/store with the new
+ // credentials BEFORE persisting the new key in account details. If a
+ // bucket update fails, the stored key remains the old one and a retry
+ // will re-enter the replacement path; if we persisted first, a retry
+ // would see the new stored key and return without repairing the
+ // remaining buckets.
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ // Persist the credentials in the account details (namespaced by
+ // storeId) with per-key writes. AccountDetailsDao.persist(accountId,
+ // map) expunges every existing detail for the account before inserting
+ // the supplied map, so it would clobber details this snapshot never saw
+ // — including another object store's namespaced credentials, since the
+ // IAM lock is keyed by storeId+accountId and two pools can provision
+ // the same account concurrently. addDetail touches only the named key.
+ details.put(accessKeyDetailKey, key.getAccessKeyId());
+ details.put(secretKeyDetailKey, key.getSecretAccessKey());
+ _accountDetailsDao.addDetail(accountId, accessKeyDetailKey, key.getAccessKeyId(), false);
+ _accountDetailsDao.addDetail(accountId, secretKeyDetailKey, key.getSecretAccessKey(), false);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Update the IAM credentials on all BucketVO rows for this store/account
+ * so previously created buckets reflect the new (rotated) key pair.
+ * Mirrors CloudianHyperStoreObjectStoreDriverImpl.updateAccountBucketCredentials.
+ */
+ private void updateAccountBucketCredentials(long storeId, long accountId, AccessKey iamCredential) {
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ for (BucketVO bucketVO : bucketList) {
+ logger.info("Updating accountId={} bucket {} with new IAM credentials", accountId, bucketVO.getName());
+ bucketVO.setAccessKey(iamCredential.getAccessKeyId());
+ bucketVO.setSecretKey(iamCredential.getSecretAccessKey());
+ _bucketDao.update(bucketVO.getId(), bucketVO);
+ }
+ }
+
+ /**
+ * Refresh the per-account IAM user policy so it grants S3 access only to
+ * the account's current buckets (optionally excluding one, e.g. a bucket
+ * being deleted). This is the tenant boundary: each account's IAM
+ * credentials can only operate on that account's own buckets.
+ *
+ * Acquires the per-store/account IAM lock. Callers that already hold the
+ * lock (e.g. createUser, createBucket post-create) should call
+ * {@link #updateAccountIAMPolicyLocked} instead to avoid re-entrant lock
+ * acquisition warnings from GlobalLock.
+ *
+ * @param iamClient the IAM client
+ * @param storeId the object store
+ * @param accountId the CloudStack account
+ * @param excludeBucket a bucket name to omit (e.g. a bucket being deleted),
+ * or null to include all of the account's buckets
+ */
+ protected void updateAccountIAMPolicy(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, excludeBucket);
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Lock-free variant of {@link #updateAccountIAMPolicy} for callers that
+ * already hold the per-store/account IAM lock. Performs the policy refresh
+ * without reacquiring the lock, avoiding the GlobalLock re-entrant
+ * acquisition warning.
+ */
+ protected void updateAccountIAMPolicyLocked(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ return;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ List<BucketVO> buckets = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ List<String> bucketNames = new ArrayList<>();
+ for (BucketVO bvo : buckets) {
+ if (excludeBucket != null && excludeBucket.equals(bvo.getName())) {
+ continue;
+ }
+ // Skip buckets whose remote counterpart has been deleted but whose
+ // row is still present for resource accounting. Including them
+ // would re-grant a bucket name that is now free for another account
+ // to claim.
+ if (Bucket.State.Destroyed.equals(bvo.getState())) {
+ continue;
+ }
+ bucketNames.add(bvo.getName());
+ }
+ String policy = SeaweedFSObjectStoreUtil.buildAccountIAMPolicy(bucketNames);
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ SeaweedFSObjectStoreUtil.IAM_USER_POLICY_NAME, policy));
+ }
+
+ /**
+ * Check whether the given access key id is still listed and Active in IAM
+ * for the user. Listing failures are propagated rather than swallowed so
+ * a transient IAM outage does not send createUser into the replacement
+ * path (which would overwrite stored credentials and invalidate bucket
+ * records).
+ */
+ private boolean iamAccessKeyExists(AmazonIdentityManagement iamClient, String userName, String accessKeyId) {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ if (accessKeyId.equals(metadata.getAccessKeyId())) {
+ return "Active".equalsIgnoreCase(metadata.getStatus());
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Delete access keys for the user other than the (optionally) preserved
+ * key id. Used to clean up unmanaged leftover keys before creating a
+ * replacement so repeated calls do not hit IAM access-key limits.
+ */
+ private void deleteUnmanagedAccessKeys(AmazonIdentityManagement iamClient, String userName, String preserveAccessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ String keyId = metadata.getAccessKeyId();
+ if (preserveAccessKeyId != null && preserveAccessKeyId.equals(keyId)) {
+ continue;
+ }
+ DeleteAccessKeyRequest deleteReq =
+ new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(keyId);
+ logger.info("Deleting un-managed IAM access key {} for user {}", keyId, userName);
+ iamClient.deleteAccessKey(deleteReq);
+ }
+ } catch (AmazonClientException e) {
+ // Propagate so the caller does not proceed to create a replacement
+ // key while stale unmanaged keys remain (which could hit IAM key
+ // limits or leave orphaned credentials).
+ throw new CloudRuntimeException("Failed to clean up IAM access keys for user " + userName, e);
+ }
+ }
+
+ @Override
+ public Bucket createBucket(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+ long accountId = bucket.getAccountId();
+
+ // Use the store's admin credentials to create the bucket
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Check if the bucket already exists
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket already exists with name " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Create the bucket
+ try {
+ CreateBucketRequest request = new CreateBucketRequest(bucketName);
+ if (objectLock) {
+ request.setObjectLockEnabledForBucket(true);
+ }
+ s3client.createBucket(request);
+ } catch (AmazonClientException e) {
+ logger.error("Create bucket failed", e);
+ throw new CloudRuntimeException(e);
+ }
+
+ // Step 2: update the bucket record with the account's IAM credentials,
+ // configure CORS, and refresh the IAM policy. If any of these fail,
+ // clean up the remote bucket so a retry does not find it already
+ // existing — mirroring the Cloudian createBucket pattern.
+ //
+ // Hold the IAM lock for the account so a concurrent createUser key
+ // rotation does not change the account credentials between reading
+ // them and writing the BucketVO, which would leave the bucket with
+ // a stale key pair.
+ GlobalLock iamLock = acquireIamLock(storeId, accountId);
+ if (iamLock == null) {
+ // The S3 bucket has already been created. Clean it up so a
+ // retry does not find it already existing, then throw.
+ try {
+ s3client.deleteBucket(bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Failed to clean up bucket {} after IAM lock timeout", bucketName, cleanupEx);
+ }
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ // Configure permissive CORS so the CloudStack S3 bucket browser
+ // (which performs list/upload/delete from the browser) can function.
+ // SeaweedFS supports the standard PutBucketCors operation.
+ configureBucketCORS(s3client, bucketName);
+
+ Map<String, String> accountDetails = _accountDetailsDao.findDetails(accountId);
+ String accessKey = accountDetails.get(SeaweedFSObjectStoreUtil.keyAccessKey(storeId));
+ String secretKey = accountDetails.get(SeaweedFSObjectStoreUtil.keySecretKey(storeId));
+ if (accessKey == null || secretKey == null) {
+ throw new CloudRuntimeException("No IAM credentials found for account " + accountId
+ + " on store " + storeId + ". Run createUser before creating a bucket.");
+ }
+
+ String s3Url = getS3Url(storeId);
+ BucketVO bucketVO = _bucketDao.findById(bucket.getId());
+ bucketVO.setAccessKey(accessKey);
+ bucketVO.setSecretKey(secretKey);
+ bucketVO.setBucketURL(s3Url + "/" + bucketName);
+ _bucketDao.update(bucket.getId(), bucketVO);
+
+ // Refresh the account's IAM policy to include the new bucket.
+ // Use the lock-free variant since createBucket already holds the
+ // IAM lock for the post-create section.
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, null);
+
+ // Return the updated BucketVO (not the stale input bucket) so
+ // BucketApiServiceImpl.createBucket does not overwrite the
+ // persisted credentials with the stale values.
+ return bucketVO;
+ } catch (Exception e) {
+ logger.error("Post-create bucket record update failed for {}; cleaning up remote bucket", bucketName, e);
+ CloudRuntimeException primary = new CloudRuntimeException(e);
+ try {
+ s3client.deleteBucket(bucketName);
+ logger.info("Cleanup of bucket {} succeeded", bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Cleanup of bucket {} also failed", bucketName, cleanupEx);
+ primary.addSuppressed(cleanupEx);
+ }
+ // Revoke the IAM policy grant for the new bucket so the account's
+ // credentials cannot access a bucket that no longer exists. If the
+ // policy PUT succeeded before the DB update failed, the grant
+ // would otherwise persist and could be reused if another account
+ // later creates the same bucket name. Use the lock-free variant
+ // since createBucket already holds the IAM lock. Propagate
+ // failures as suppressed exceptions so they are not silently lost.
+ try {
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, bucketName);
+ } catch (Exception policyEx) {
+ logger.warn("Failed to revoke IAM policy for bucket {} after cleanup: {}", bucketName, policyEx.getMessage());
+ primary.addSuppressed(policyEx);
+ }
+ throw primary;
+ } finally {
+ iamLock.unlock();
+ iamLock.releaseRef();
+ }
+ }
+
+ /**
+ * Configure a permissive CORS policy on the bucket so the CloudStack
+ * S3 bucket browser (which performs list/upload/delete from the
+ * browser) can function. Mirrors the Cloudian configureBucketCORS.
+ */
+ private void configureBucketCORS(AmazonS3 s3client, String bucketName) {
+ logger.debug("Configuring CORS for bucket {}", bucketName);
+ List<CORSRule> corsRules = new ArrayList<>();
+ CORSRule allowAnyRule = new CORSRule().withId("AllowAny");
+ allowAnyRule.setAllowedOrigins("*");
+ allowAnyRule.setAllowedHeaders("*");
+ allowAnyRule.setAllowedMethods(
+ CORSRule.AllowedMethods.HEAD,
+ CORSRule.AllowedMethods.GET,
+ CORSRule.AllowedMethods.PUT,
+ CORSRule.AllowedMethods.POST,
+ CORSRule.AllowedMethods.DELETE);
+ corsRules.add(allowAnyRule);
+ BucketCrossOriginConfiguration corsConfig = new BucketCrossOriginConfiguration();
+ corsConfig.setRules(corsRules);
+ SetBucketCrossOriginConfigurationRequest corsRequest = new SetBucketCrossOriginConfigurationRequest(bucketName, corsConfig);
+ s3client.setBucketCrossOriginConfiguration(corsRequest);
+ logger.info("Successfully configured CORS for bucket {}", bucketName);
+ }
+
+ @Override
+ public List<Bucket> listBuckets(long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ List<Bucket> bucketsList = new ArrayList<>();
+ try {
+ List<com.amazonaws.services.s3.model.Bucket> s3Buckets = s3client.listBuckets();
+ for (com.amazonaws.services.s3.model.Bucket s3Bucket : s3Buckets) {
+ Bucket bucket = new BucketObject();
+ bucket.setName(s3Bucket.getName());
+ bucketsList.add(bucket);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ return bucketsList;
+ }
+
+ @Override
+ public boolean deleteBucket(BucketTO bucket, long storeId) {
+ String bucketName = bucket.getName();
+ long accountId = bucket.getAccountId();
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Acquire the per-store/account IAM lock so the S3 delete and the
+ // subsequent policy refresh are atomic with respect to concurrent
+ // createUser/createBucket operations. Without the lock, a concurrent
+ // policy rebuild could re-add the deleted bucket ARN between the
+ // excludeBucket refresh and BucketApiServiceImpl's row removal.
+ GlobalLock iamLock = acquireIamLock(storeId, accountId);
+ if (iamLock == null) {
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ // Delete the S3 bucket first. If this fails (non-empty bucket,
+ // transient error), the IAM policy is still intact so the user
+ // can empty the bucket and retry.
+ //
+ // If the bucket is already gone (e.g. from a previous partial
+ // failure where the S3 delete succeeded but the IAM policy refresh
+ // failed), skip the S3 delete so the retry is idempotent.
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ s3client.deleteBucket(bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Mark the row Destroyed inside the lock. The row itself must
+ // survive so BucketApiServiceImpl.deleteCheckedBucket can still
+ // decrement the resource counts and allocated size, and so a
+ // failure in that cleanup leaves state a retry can reconcile.
+ // Marking it (rather than relying only on excludeBucket) closes the
+ // window where a concurrent createUser/createBucket for this
+ // account rebuilds the policy from the DB and re-adds this bucket's
+ // ARN: buildAccountIAMPolicy skips Destroyed rows, so the grant
+ // cannot come back on a name that is now reusable.
+ markBucketDestroyed(storeId, accountId, bucketName);
Review Comment:
Marking the row `Destroyed` here makes the delete path non-idempotent with respect to CloudStack accounting. If `deleteCheckedBucket` fails after this driver returns (for example while removing the row or updating allocated size), the row remains available for a retry, while the missing S3 bucket makes this method return success again; the retry then decrements the bucket/quota resource counts a second time. Please coordinate this state transition with the accounting cleanup (or record/account the cleanup idempotently) so a retry cannot double-decrement resources.
##########
server/src/main/java/org/apache/cloudstack/storage/object/BucketApiServiceImpl.java:
##########
@@ -313,22 +332,60 @@ private void updateBucketQuota(UpdateBucketCmd cmd, BucketVO bucket, ObjectStore
return;
}
- int quotaDelta = quota - bucket.getQuota();
- objectStore.setQuota(bucketTO, quota);
- bucket.setQuota(quota);
-
+ int previousQuota = ObjectUtils.defaultIfNull(bucket.getQuota(), 0);
+ int quotaDelta = quota - previousQuota;
long diff = quotaDelta * Resource.ResourceType.bytesToGiB;
if (quotaDelta < 0) {
- resourceLimitManager.decrementResourceCount(bucket.getAccountId(), Resource.ResourceType.object_storage, Math.abs(diff));
- _objectStoreDao.updateAllocatedSize(objectStoreVO, diff);
+ // A decrease needs no reservation. Apply it remotely first, then
+ // compensate the backend if the accounting update fails so the two
+ // cannot diverge.
+ objectStore.setQuota(bucketTO, quota);
+ try {
+ resourceLimitManager.decrementResourceCount(bucket.getAccountId(), Resource.ResourceType.object_storage, Math.abs(diff));
+ _objectStoreDao.updateAllocatedSize(objectStoreVO, diff);
Review Comment:
`updateAllocatedSize` returns a `Boolean`, and this branch ignores it; moreover, if it throws after `decrementResourceCount` succeeds, the catch restores only the remote quota. The account's object_storage count can remain decremented while the bucket/store state is restored, so a failed quota decrease can corrupt resource accounting. Check the DAO result and explicitly compensate the committed resource count before rethrowing.
This issue also appears on line 365 of the same file.
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,938 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.BucketCrossOriginConfiguration;
+import com.amazonaws.services.s3.model.CORSRule;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketCrossOriginConfigurationRequest;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ /**
+ * DB-backed global lock name prefix for serializing IAM provisioning and
+ * policy refreshes per store+account. Uses {@link GlobalLock} so the
+ * critical section is serialized across management servers in a
+ * clustered deployment, not just within a single JVM.
+ */
+ private static final String IAM_LOCK_PREFIX = "seaweedfs.iam.";
+
+ private static String getIamLockName(long storeId, long accountId) {
+ return IAM_LOCK_PREFIX + storeId + "." + accountId;
+ }
+
+ /**
+ * Acquire a DB-backed global lock for IAM operations on the given
+ * store+account. Returns a {@link GlobalLock} that the caller must
+ * {@link GlobalLock#unlock()} in a {@code finally} block, or {@code null}
+ * if the lock could not be acquired within the timeout.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ */
+ protected GlobalLock acquireIamLock(long storeId, long accountId) {
+ GlobalLock lock = GlobalLock.getInternLock(getIamLockName(storeId, accountId));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire IAM lock for store {} account {}", storeId, accountId);
+ lock.releaseRef();
+ return null;
+ }
+ return lock;
+ }
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account and
+ * store. The store ID is included so that two CloudStack pools pointing
+ * at the same SeaweedFS IAM service do not collide on the same
+ * {@code acs-<uuid>} user and overwrite each other's policy and access
+ * keys.
+ */
+ protected String getUserNameForAccount(Account account, long storeId) {
+ return String.format("%s-%d-%s", ACS_PREFIX, storeId, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Serialize per store+account across management servers so two
+ // concurrent bucket requests do not both rotate credentials and leave
+ // bucket rows with mismatched key pairs.
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ return false;
+ }
+ try {
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach a scoped IAM policy that allows access only to this
+ // account's own buckets (the tenant boundary). Refreshed whenever
+ // buckets are created or deleted. Use the lock-free variant since
+ // createUser already holds the IAM lock.
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, null);
+
+ // Reuse the stored access key only if both the access key id and the
+ // secret key are present and the key is still Active in IAM; otherwise
+ // create a replacement.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ String storedSecretKey = details.get(secretKeyDetailKey);
+ if (storedAccessKeyId != null && storedSecretKey != null
+ && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ return true;
+ }
+
+ // The stored key is missing, inactive, or no longer in IAM. Clean up
+ // ALL keys (including the inactive stored one) before creating a
+ // replacement so we do not accumulate keys and hit IAM limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, null);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Update existing bucket records for this account/store with the new
+ // credentials BEFORE persisting the new key in account details. If a
+ // bucket update fails, the stored key remains the old one and a retry
+ // will re-enter the replacement path; if we persisted first, a retry
+ // would see the new stored key and return without repairing the
+ // remaining buckets.
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ // Persist the credentials in the account details (namespaced by
+ // storeId) with per-key writes. AccountDetailsDao.persist(accountId,
+ // map) expunges every existing detail for the account before inserting
+ // the supplied map, so it would clobber details this snapshot never saw
+ // — including another object store's namespaced credentials, since the
+ // IAM lock is keyed by storeId+accountId and two pools can provision
+ // the same account concurrently. addDetail touches only the named key.
+ details.put(accessKeyDetailKey, key.getAccessKeyId());
+ details.put(secretKeyDetailKey, key.getSecretAccessKey());
+ _accountDetailsDao.addDetail(accountId, accessKeyDetailKey, key.getAccessKeyId(), false);
+ _accountDetailsDao.addDetail(accountId, secretKeyDetailKey, key.getSecretAccessKey(), false);
Review Comment:
These two `addDetail` calls are independent writes. If the second call fails, the account can retain the new access-key ID with the old or missing secret; on the next `createUser`, the existing non-null pair is treated as reusable, so buckets receive credentials that cannot authenticate. Persist the pair atomically without expunging other store details, or roll back the first write on failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r3995283962
##########
plugins/pom.xml:
##########
@@ -142,6 +142,7 @@
<module>storage/object/minio</module>
<module>storage/object/ceph</module>
<module>storage/object/cloudian</module>
+ <module>storage/object/seaweedfs</module>
Review Comment:
Adding the module only to the plugins reactor does not put it in the management-server/client artifact. `client/pom.xml` explicitly lists the object-storage providers, but has no SeaweedFS dependency, so this provider's classes and Spring metadata will not be shipped or loaded in normal builds. Add the matching dependency to the client packaging POM as well.
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/util/SeaweedFSObjectStoreUtil.java:
##########
@@ -0,0 +1,285 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.util;
+
+import org.apache.commons.lang3.StringUtils;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagementClientBuilder;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3ClientBuilder;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * Utility class for the SeaweedFS object storage provider.
+ *
+ * SeaweedFS exposes both an S3-compatible API and an AWS IAM-compatible API,
+ * so this provider needs no proprietary admin client — only the AWS S3 and IAM
+ * SDKs, the same pair Cloudian HyperStore already uses in this tree.
+ */
+public class SeaweedFSObjectStoreUtil {
+
+ /** The name of our Object Store Provider */
+ public static final String OBJECT_STORE_PROVIDER_NAME = "SeaweedFS";
+
+ public static final String STORE_KEY_PROVIDER_NAME = "providerName";
+ public static final String STORE_KEY_URL = "url";
+ public static final String STORE_KEY_NAME = "name";
+ public static final String STORE_KEY_DETAILS = "details";
+
+ // Store Details Map key names - managed outside of plugin
+ public static final String STORE_DETAILS_KEY_ACCESS_KEY = "accesskey"; // admin/root access key
+ public static final String STORE_DETAILS_KEY_SECRET_KEY = "secretkey"; // admin/root secret key
+ public static final String STORE_DETAILS_KEY_S3_URL = "s3Url"; // S3 endpoint URL
+ public static final String STORE_DETAILS_KEY_IAM_URL = "iamUrl"; // IAM endpoint URL
+
+ // Account Detail Map key names - credentials created per CloudStack account
+ public static final String KEY_ACCESS_KEY = "swfs_AccessKey";
+ public static final String KEY_SECRET_KEY = "swfs_SecretKey";
+
+ /**
+ * IAM user policy applied to each per-account IAM user. Grants full S3
+ * access except bucket creation/deletion, so CloudStack retains control of
+ * bucket lifecycle while the account's IAM credentials can manage objects.
+ */
+ public static final String IAM_USER_POLICY = "{\n" +
+ " \"Version\": \"2012-10-17\",\n" +
+ " \"Statement\": [\n" +
+ " {\n" +
+ " \"Sid\": \"AllowFullS3Access\",\n" +
+ " \"Effect\": \"Allow\",\n" +
+ " \"Action\": [\n" +
+ " \"s3:*\"\n" +
+ " ],\n" +
+ " \"Resource\": \"*\"\n" +
Review Comment:
This policy is attached to every account-specific IAM user, but `s3:*` on `*` permits reading, modifying, deleting, and changing policies/ACLs for every bucket in the store; only bucket creation and deletion are denied. Without a SeaweedFS-side group boundary, one CloudStack account can operate on another account's data. Scope the policy to that account's bucket ARNs and update it as buckets are created.
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/lifecycle/SeaweedFSObjectStoreLifeCycleImpl.java:
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.lifecycle;
+
+import com.cloud.agent.api.StoragePoolInfo;
+import com.cloud.hypervisor.Hypervisor.HypervisorType;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.ClusterScope;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.HostScope;
+import org.apache.cloudstack.engine.subsystem.api.storage.ZoneScope;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.datastore.ObjectStoreHelper;
+import org.apache.cloudstack.storage.object.datastore.ObjectStoreProviderManager;
+import org.apache.cloudstack.storage.object.store.lifecycle.ObjectStoreLifeCycle;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import javax.inject.Inject;
+
+import java.util.HashMap;
+import java.util.Map;
+
+public class SeaweedFSObjectStoreLifeCycleImpl implements ObjectStoreLifeCycle {
+
+ protected Logger logger = LogManager.getLogger(SeaweedFSObjectStoreLifeCycleImpl.class);
+
+ @Inject
+ ObjectStoreHelper objectStoreHelper;
+ @Inject
+ ObjectStoreProviderManager objectStoreMgr;
+
+ public SeaweedFSObjectStoreLifeCycleImpl() {
+ }
+
+ @Override
+ public DataStore initialize(Map<String, Object> dsInfos) {
+
+ String name = (String)dsInfos.get(SeaweedFSObjectStoreUtil.STORE_KEY_NAME);
+ String url = (String)dsInfos.get(SeaweedFSObjectStoreUtil.STORE_KEY_URL);
+ String providerName = (String)dsInfos.get(SeaweedFSObjectStoreUtil.STORE_KEY_PROVIDER_NAME);
+
+ // Check the providerName is what we expect
+ if (! StringUtils.equalsIgnoreCase(providerName, SeaweedFSObjectStoreUtil.OBJECT_STORE_PROVIDER_NAME)) {
+ String msg = String.format("Unexpected providerName \"%s\". Expected \"%s\"", providerName, SeaweedFSObjectStoreUtil.OBJECT_STORE_PROVIDER_NAME);
+ logger.error(msg);
+ throw new CloudRuntimeException(msg);
+ }
+
+ Map<String, Object> objectStoreParameters = new HashMap<String, Object>();
+ objectStoreParameters.put(SeaweedFSObjectStoreUtil.STORE_KEY_NAME, name);
+ objectStoreParameters.put(SeaweedFSObjectStoreUtil.STORE_KEY_URL, url);
+ objectStoreParameters.put(SeaweedFSObjectStoreUtil.STORE_KEY_PROVIDER_NAME, providerName);
Review Comment:
`ObjectStoreHelper.createObjectStore` unconditionally unboxes `params.get("size")`. This map never copies `dsInfos.get("size")`, so every `addObjectStoragePool` for SeaweedFS reaches persistence with a null size and fails before the store is created. Copy the size parameter as the other object-store lifecycles do.
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,478 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account.
+ * Uses the account UUID prefixed with "acs-" for namespacing.
+ */
+ protected String getUserNameForAccount(Account account) {
+ return String.format("%s-%s", ACS_PREFIX, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, create an access key, and persist the
+ * credentials in the account details.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach the restricted S3 policy (idempotent — overwrites if present)
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ "CloudStackPolicy", SeaweedFSObjectStoreUtil.IAM_USER_POLICY));
+
+ // Create a new access key for this user
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
Review Comment:
`createUser` is called by `allocBucket` before each bucket creation, but this path creates a new access key even after `EntityAlreadyExistsException` and overwrites the account's stored key. Repeated bucket requests therefore rotate credentials, leave earlier bucket records with invalid credentials, and eventually hit IAM access-key limits. Reuse and validate the stored key, cleaning up unmanaged keys before creating a replacement.
This issue also appears on line 428 of the same file.
##########
plugins/storage/object/seaweedfs/src/test/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImplTest.java:
##########
@@ -0,0 +1,390 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.lenient;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.Bucket;
+
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.ListObjectsV2Request;
+import com.amazonaws.services.s3.model.ListObjectsV2Result;
+import com.amazonaws.services.s3.model.S3ObjectSummary;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.AccountVO;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mock;
+import org.mockito.MockitoAnnotations;
+import org.mockito.Spy;
+import org.mockito.junit.MockitoJUnitRunner;
+
+@RunWith(MockitoJUnitRunner.Silent.class)
+public class SeaweedFSObjectStoreDriverImplTest {
+
+ @Spy
+ SeaweedFSObjectStoreDriverImpl driver = new SeaweedFSObjectStoreDriverImpl();
+
+ @Mock
+ AmazonS3 s3Client;
+ @Mock
+ AmazonIdentityManagement iamClient;
+ @Mock
+ ObjectStoreDao objectStoreDao;
+ @Mock
+ ObjectStoreVO objectStoreVO;
+ @Mock
+ ObjectStoreDetailsDao objectStoreDetailsDao;
+ @Mock
+ AccountDao accountDao;
+ @Mock
+ BucketDao bucketDao;
+ @Mock
+ AccountDetailsDao accountDetailsDao;
+ @Mock
+ AccountVO account;
+
+ BucketVO bucketVo;
+ Map<String, String> storeDetailsMap;
+ Map<String, String> accountDetailsMap;
+
+ static long TEST_STORE_ID = 1010L;
+ static long TEST_ACCOUNT_ID = 2010L;
+ static long TEST_DOMAIN_ID = 3010L;
+ static String TEST_ACCESS_KEY = "test_access_key";
+ static String TEST_SECRET_KEY = "test_secret_key";
+ static String TEST_BUCKET_NAME = "testbucketname";
+ static String TEST_S3_URL = "http://s3-endpoint";
+ static String TEST_IAM_URL = "http://iam-endpoint";
+ static String TEST_AK = "user_access_key";
+ static String TEST_SK = "user_secret_key";
+ static String TEST_BUCKET_URL = TEST_S3_URL + "/" + TEST_BUCKET_NAME;
+ static String TEST_ACCOUNT_UUID = "account-uuid-1234";
+
+ private AutoCloseable closeable;
+
+ @Before
+ public void setUp() {
+ closeable = MockitoAnnotations.openMocks(this);
+ driver._storeDao = objectStoreDao;
+ driver._storeDetailsDao = objectStoreDetailsDao;
+ driver._accountDao = accountDao;
+ driver._bucketDao = bucketDao;
+ driver._accountDetailsDao = accountDetailsDao;
+
+ lenient().when(objectStoreDao.findById(TEST_STORE_ID)).thenReturn(objectStoreVO);
+ lenient().when(objectStoreVO.getUrl()).thenReturn(TEST_S3_URL);
+
+ storeDetailsMap = new HashMap<>();
+ storeDetailsMap.put(SeaweedFSObjectStoreUtil.STORE_DETAILS_KEY_ACCESS_KEY, TEST_ACCESS_KEY);
+ storeDetailsMap.put(SeaweedFSObjectStoreUtil.STORE_DETAILS_KEY_SECRET_KEY, TEST_SECRET_KEY);
+ storeDetailsMap.put(SeaweedFSObjectStoreUtil.STORE_DETAILS_KEY_S3_URL, TEST_S3_URL);
+ storeDetailsMap.put(SeaweedFSObjectStoreUtil.STORE_DETAILS_KEY_IAM_URL, TEST_IAM_URL);
+ lenient().when(objectStoreDetailsDao.getDetails(TEST_STORE_ID)).thenReturn(storeDetailsMap);
+
+ accountDetailsMap = new HashMap<>();
+ accountDetailsMap.put(SeaweedFSObjectStoreUtil.KEY_ACCESS_KEY, TEST_AK);
+ accountDetailsMap.put(SeaweedFSObjectStoreUtil.KEY_SECRET_KEY, TEST_SK);
+ lenient().when(accountDetailsDao.findDetails(TEST_ACCOUNT_ID)).thenReturn(accountDetailsMap);
+
+ bucketVo = new BucketVO(TEST_ACCOUNT_ID, TEST_DOMAIN_ID, TEST_STORE_ID, TEST_BUCKET_NAME, null, false, false, false, null);
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ closeable.close();
+ }
+
+ @Test
+ public void testGetStoreTO() {
+ assertNull(driver.getStoreTO(null));
+ }
+
+ @Test
+ public void testCreateBucket() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ when(s3Client.doesBucketExistV2(TEST_BUCKET_NAME)).thenReturn(false);
+ when(bucketDao.findById(anyLong())).thenReturn(bucketVo);
+
+ Bucket result = driver.createBucket(bucketVo, false);
+
+ assertEquals(TEST_BUCKET_NAME, result.getName());
+
+ ArgumentCaptor<BucketVO> captor = ArgumentCaptor.forClass(BucketVO.class);
+ verify(bucketDao, times(1)).update(any(), captor.capture());
+ BucketVO updated = captor.getValue();
+ assertEquals(TEST_AK, updated.getAccessKey());
+ assertEquals(TEST_SK, updated.getSecretKey());
+ assertEquals(TEST_BUCKET_URL, updated.getBucketURL());
+
+ verify(s3Client, times(1)).createBucket(any(CreateBucketRequest.class));
+ }
+
+ @Test
+ public void testCreateBucketAlreadyExists() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ when(s3Client.doesBucketExistV2(TEST_BUCKET_NAME)).thenReturn(true);
+
+ assertThrows(CloudRuntimeException.class, () -> driver.createBucket(bucketVo, false));
+ verify(s3Client, never()).createBucket(any(CreateBucketRequest.class));
+ }
+
+ @Test
+ public void testListBuckets() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ List<com.amazonaws.services.s3.model.Bucket> s3Buckets = new ArrayList<>();
+ s3Buckets.add(new com.amazonaws.services.s3.model.Bucket("bucket1"));
+ s3Buckets.add(new com.amazonaws.services.s3.model.Bucket("bucket2"));
+ when(s3Client.listBuckets()).thenReturn(s3Buckets);
+
+ List<Bucket> result = driver.listBuckets(TEST_STORE_ID);
+
+ assertEquals(2, result.size());
+ assertEquals("bucket1", result.get(0).getName());
+ assertEquals("bucket2", result.get(1).getName());
+ }
+
+ @Test
+ public void testDeleteBucket() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ BucketTO bucketTO = mock(BucketTO.class);
+ when(bucketTO.getName()).thenReturn(TEST_BUCKET_NAME);
+ when(s3Client.doesBucketExistV2(TEST_BUCKET_NAME)).thenReturn(true);
+
+ assertTrue(driver.deleteBucket(bucketTO, TEST_STORE_ID));
+ verify(s3Client, times(1)).deleteBucket(TEST_BUCKET_NAME);
+ }
+
+ @Test
+ public void testDeleteBucketNotFound() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ BucketTO bucketTO = mock(BucketTO.class);
+ when(bucketTO.getName()).thenReturn(TEST_BUCKET_NAME);
+ when(s3Client.doesBucketExistV2(TEST_BUCKET_NAME)).thenReturn(false);
+
+ assertThrows(CloudRuntimeException.class, () -> driver.deleteBucket(bucketTO, TEST_STORE_ID));
+ }
+
+ @Test
+ public void testSetBucketVersioning() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ BucketTO bucketTO = mock(BucketTO.class);
+ when(bucketTO.getName()).thenReturn(TEST_BUCKET_NAME);
+
+ assertTrue(driver.setBucketVersioning(bucketTO, TEST_STORE_ID));
+ verify(s3Client, times(1)).setBucketVersioningConfiguration(any(SetBucketVersioningConfigurationRequest.class));
+ }
+
+ @Test
+ public void testDeleteBucketVersioning() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ BucketTO bucketTO = mock(BucketTO.class);
+ when(bucketTO.getName()).thenReturn(TEST_BUCKET_NAME);
+
+ assertTrue(driver.deleteBucketVersioning(bucketTO, TEST_STORE_ID));
+ ArgumentCaptor<SetBucketVersioningConfigurationRequest> captor =
+ ArgumentCaptor.forClass(SetBucketVersioningConfigurationRequest.class);
+ verify(s3Client, times(1)).setBucketVersioningConfiguration(captor.capture());
+ assertEquals(BucketVersioningConfiguration.SUSPENDED, captor.getValue().getVersioningConfiguration().getStatus());
+ }
+
+ @Test
+ public void testSetBucketQuotaZero() throws Exception {
+ BucketTO bucketTO = mock(BucketTO.class);
+ when(bucketTO.getName()).thenReturn(TEST_BUCKET_NAME);
+ // Mock the S3 helpers to return valid values
+ doReturn("http://s3-endpoint").when(driver).getS3Url(TEST_STORE_ID);
+ doReturn("access-key").when(driver).getAccessKey(TEST_STORE_ID);
+ doReturn("secret-key").when(driver).getSecretKey(TEST_STORE_ID);
+ // Should not throw for 0 — uses static method, can't easily mock, but
+ // the test validates the code path doesn't throw before the HTTP call
+ // Since we can't mock the static HTTP call, we expect a CloudRuntimeException
+ // from the HTTP call failing (no real server). That's acceptable — it proves
+ // the code path reaches the S3 extension rather than throwing "not supported".
+ assertThrows(CloudRuntimeException.class, () -> driver.setBucketQuota(bucketTO, TEST_STORE_ID, 0));
Review Comment:
These quota tests call the real `java.net.http.HttpClient` against `http://s3-endpoint` and merely assert `CloudRuntimeException`. They are network/DNS-dependent and would still pass if signing, the request body, or the response handling were broken. Use WireMock or inject the HTTP sender and assert the path, signed headers, and JSON body.
This issue also appears on line 279 of the same file.
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/util/SeaweedFSObjectStoreUtil.java:
##########
@@ -0,0 +1,285 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.util;
+
+import org.apache.commons.lang3.StringUtils;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagementClientBuilder;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3ClientBuilder;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * Utility class for the SeaweedFS object storage provider.
+ *
+ * SeaweedFS exposes both an S3-compatible API and an AWS IAM-compatible API,
+ * so this provider needs no proprietary admin client — only the AWS S3 and IAM
+ * SDKs, the same pair Cloudian HyperStore already uses in this tree.
+ */
+public class SeaweedFSObjectStoreUtil {
+
+ /** The name of our Object Store Provider */
+ public static final String OBJECT_STORE_PROVIDER_NAME = "SeaweedFS";
+
+ public static final String STORE_KEY_PROVIDER_NAME = "providerName";
+ public static final String STORE_KEY_URL = "url";
+ public static final String STORE_KEY_NAME = "name";
+ public static final String STORE_KEY_DETAILS = "details";
+
+ // Store Details Map key names - managed outside of plugin
+ public static final String STORE_DETAILS_KEY_ACCESS_KEY = "accesskey"; // admin/root access key
+ public static final String STORE_DETAILS_KEY_SECRET_KEY = "secretkey"; // admin/root secret key
+ public static final String STORE_DETAILS_KEY_S3_URL = "s3Url"; // S3 endpoint URL
+ public static final String STORE_DETAILS_KEY_IAM_URL = "iamUrl"; // IAM endpoint URL
+
+ // Account Detail Map key names - credentials created per CloudStack account
+ public static final String KEY_ACCESS_KEY = "swfs_AccessKey";
+ public static final String KEY_SECRET_KEY = "swfs_SecretKey";
+
+ /**
+ * IAM user policy applied to each per-account IAM user. Grants full S3
+ * access except bucket creation/deletion, so CloudStack retains control of
+ * bucket lifecycle while the account's IAM credentials can manage objects.
+ */
+ public static final String IAM_USER_POLICY = "{\n" +
+ " \"Version\": \"2012-10-17\",\n" +
+ " \"Statement\": [\n" +
+ " {\n" +
+ " \"Sid\": \"AllowFullS3Access\",\n" +
+ " \"Effect\": \"Allow\",\n" +
+ " \"Action\": [\n" +
+ " \"s3:*\"\n" +
+ " ],\n" +
+ " \"Resource\": \"*\"\n" +
+ " },\n" +
+ " {\n" +
+ " \"Sid\": \"ExceptBucketCreationOrDeletion\",\n" +
+ " \"Effect\": \"Deny\",\n" +
+ " \"Action\": [\n" +
+ " \"s3:CreateBucket\",\n" +
+ " \"s3:DeleteBucket\"\n" +
+ " ],\n" +
+ " \"Resource\": \"*\"\n" +
+ " }\n" +
+ " ]\n" +
+ "}\n";
+
+ // The CloudStack service credential (the accesskey/secretkey configured on
+ // the object store) is the admin credential used for ALL driver operations:
+ // - AmazonS3 client: bucket CRUD, policy, versioning, encryption, listing
+ // - AmazonIdentityManagement client: per-account IAM user provisioning
+ // - setBucketQuotaViaS3Extension: PUT /{bucket}?seaweedfs-quota
+ // It must therefore have broad S3 and IAM permissions. It is NOT scoped
+ // down to only s3:PutBucketQuota/s3:GetBucketQuota — that was an earlier
+ // design idea that does not match the implementation. The per-account IAM
+ // users (created by createUser) are the ones with restricted permissions
+ // (see IAM_USER_POLICY above).
+
+ /**
+ * Returns an S3 connection for the given endpoint and credentials.
+ * Uses path-style access, which SeaweedFS requires.
+ *
+ * @param url the url of the S3 service
+ * @param accessKey the credentials to use for the S3 connection.
+ * @param secretKey the matching secret key.
+ * @return an S3 connection (never null)
+ * @throws CloudRuntimeException on failure.
+ */
+ public static AmazonS3 getS3Client(String url, String accessKey, String secretKey) {
+ AmazonS3 client = AmazonS3ClientBuilder.standard()
+ .enablePathStyleAccess()
+ .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))
+ .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(url, "us-east-1"))
+ .build();
+ if (client == null) {
+ throw new CloudRuntimeException("Error while creating SeaweedFS S3 client");
+ }
+ return client;
+ }
+
+ /**
+ * Returns an IAM connection for the given endpoint and credentials.
+ *
+ * @param url the url of the IAM service
+ * @param accessKey the credentials to use for the iam connection.
+ * @param secretKey the matching secret key.
+ * @return an IAM connection (never null)
+ * @throws CloudRuntimeException on failure.
+ */
+ public static AmazonIdentityManagement getIAMClient(String url, String accessKey, String secretKey) {
+ AmazonIdentityManagement iamClient = AmazonIdentityManagementClientBuilder.standard()
+ .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))
+ .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(url, "us-east-1"))
+ .build();
+ if (iamClient == null) {
+ throw new CloudRuntimeException("Error while creating SeaweedFS IAM client");
+ }
+ return iamClient;
+ }
+
+ /**
+ * Test the S3Url to confirm it behaves like an S3 Service.
+ *
+ * Uses bad credentials and looks for the particular error from S3 that says
+ * InvalidAccessKeyId was used. Quietly returns if we connect and get the
+ * expected error back.
+ *
+ * @param s3Url the url to check
+ * @throws CloudRuntimeException if there is any unexpected issue.
+ */
+ public static void validateS3Url(String s3Url) {
+ try {
+ AmazonS3 s3Client = SeaweedFSObjectStoreUtil.getS3Client(s3Url, "unknown", "unknown");
+ s3Client.listBuckets();
+ } catch (AmazonServiceException e) {
+ if (StringUtils.compareIgnoreCase(e.getErrorCode(), "InvalidAccessKeyId") != 0
+ && StringUtils.compareIgnoreCase(e.getErrorCode(), "SignatureDoesNotMatch") != 0) {
+ throw new CloudRuntimeException("Unexpected response from S3 Endpoint.", e);
+ }
+ }
+ }
+
+ /**
+ * Test the IAMUrl to confirm it behaves like an IAM Service.
+ *
+ * Uses bad credentials and looks for the particular error from IAM that says
+ * InvalidAccessKeyId or InvalidClientTokenId was used. Quietly returns if we
+ * connect and get the expected error back.
+ *
+ * @param iamUrl the url to check
+ * @throws CloudRuntimeException if there is any unexpected issue.
+ */
+ public static void validateIAMUrl(String iamUrl) {
+ try {
+ AmazonIdentityManagement iamClient = SeaweedFSObjectStoreUtil.getIAMClient(iamUrl, "unknown", "unknown");
+ iamClient.listAccessKeys();
+ } catch (AmazonServiceException e) {
+ if (! StringUtils.equalsAnyIgnoreCase(e.getErrorCode(), "InvalidAccessKeyId", "InvalidClientTokenId", "SignatureDoesNotMatch")) {
+ throw new CloudRuntimeException("Unexpected response from IAM Endpoint.", e);
+ }
+ }
+ }
+
+ /**
+ * Set bucket quota via the SeaweedFS S3 extension endpoint.
+ *
+ * SeaweedFS exposes a custom S3 subresource at
+ * PUT /{bucket}?seaweedfs-quota
+ * authenticated via standard S3 SigV4 and authorized via the
+ * s3:PutBucketQuota IAM permission. This avoids the need for a
+ * separate admin API credential.
+ *
+ * The request body is JSON:
+ * {"quota_size": <n>, "quota_unit": "GB", "quota_enabled": true}
+ *
+ * @param s3Url the S3 endpoint URL (e.g. http://host:8333)
+ * @param accessKey the S3 access key (must have s3:PutBucketQuota permission)
+ * @param secretKey the S3 secret key
+ * @param bucketName the bucket name
+ * @param sizeGiB the quota size in GiB (0 to disable quota)
+ * @throws CloudRuntimeException on any failure
+ */
+ public static void setBucketQuotaViaS3Extension(String s3Url, String accessKey, String secretKey, String bucketName, long sizeGiB) {
+ String body;
+ if (sizeGiB <= 0) {
+ body = "{\"quota_size\":0,\"quota_unit\":\"B\",\"quota_enabled\":false}";
+ } else {
+ body = String.format("{\"quota_size\":%d,\"quota_unit\":\"GB\",\"quota_enabled\":true}", sizeGiB);
+ }
+ executeSignedS3Request("PUT", s3Url, "/" + bucketName + "?seaweedfs-quota", accessKey, secretKey, body);
+ }
+
+ /**
+ * Execute a custom S3 request with SigV4 signing.
+ *
+ * Uses the AWS SDK v1 Aws4Signer to sign the request, then sends it via
+ * java.net.http.HttpClient. This allows calling SeaweedFS-specific S3
+ * extensions (like ?seaweedfs-quota) that the AWS SDK doesn't natively
+ * support.
+ *
+ * @param method HTTP method (PUT, GET, etc.)
+ * @param s3Url the S3 endpoint base URL
+ * @param resourcePath the path + query string (e.g. /bucket?seaweedfs-quota)
+ * @param accessKey S3 access key
+ * @param secretKey S3 secret key
+ * @param body the request body (null for GET)
+ * @return the response body as a string
+ * @throws CloudRuntimeException on any failure
+ */
+ private static String executeSignedS3Request(String method, String s3Url, String resourcePath,
+ String accessKey, String secretKey, String body) {
+ try {
+ java.net.URI endpointUri = java.net.URI.create(s3Url);
+ java.net.URL endpointUrl = endpointUri.toURL();
+
+ // Build AWS SDK v1 Request for SigV4 signing
+ com.amazonaws.DefaultRequest<?> request = new com.amazonaws.DefaultRequest<>("s3");
+ request.setEndpoint(endpointUri);
+ request.setHttpMethod(com.amazonaws.http.HttpMethodName.valueOf(method));
+ request.setResourcePath(resourcePath);
Review Comment:
`DefaultRequest` keeps query parameters separately; `setResourcePath` does not parse the `?seaweedfs-quota` suffix. The signer therefore does not sign this as a canonical query parameter, while `fullUri` sends it as one, so SeaweedFS can reject the request with a signature mismatch. Split the path and add `seaweedfs-quota` through `request.addParameter(...)` before signing.
This issue also appears in the following locations of the same file:
- line 258
- line 269
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#issuecomment-6363392364
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r4011606620
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,804 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.BucketCrossOriginConfiguration;
+import com.amazonaws.services.s3.model.CORSRule;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketCrossOriginConfigurationRequest;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ /**
+ * DB-backed global lock name prefix for serializing IAM provisioning and
+ * policy refreshes per store+account. Uses {@link GlobalLock} so the
+ * critical section is serialized across management servers in a
+ * clustered deployment, not just within a single JVM.
+ */
+ private static final String IAM_LOCK_PREFIX = "seaweedfs.iam.";
+
+ private static String getIamLockName(long storeId, long accountId) {
+ return IAM_LOCK_PREFIX + storeId + "." + accountId;
+ }
+
+ /**
+ * Acquire a DB-backed global lock for IAM operations on the given
+ * store+account. Returns a {@link GlobalLock} that the caller must
+ * {@link GlobalLock#unlock()} in a {@code finally} block, or {@code null}
+ * if the lock could not be acquired within the timeout.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ */
+ protected GlobalLock acquireIamLock(long storeId, long accountId) {
+ GlobalLock lock = GlobalLock.getInternLock(getIamLockName(storeId, accountId));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire IAM lock for store {} account {}", storeId, accountId);
+ lock.releaseRef();
+ return null;
+ }
+ return lock;
+ }
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account and
+ * store. The store ID is included so that two CloudStack pools pointing
+ * at the same SeaweedFS IAM service do not collide on the same
+ * {@code acs-<uuid>} user and overwrite each other's policy and access
+ * keys.
+ */
+ protected String getUserNameForAccount(Account account, long storeId) {
+ return String.format("%s-%d-%s", ACS_PREFIX, storeId, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Serialize per store+account across management servers so two
+ // concurrent bucket requests do not both rotate credentials and leave
+ // bucket rows with mismatched key pairs.
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ return false;
+ }
+ try {
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach a scoped IAM policy that allows access only to this
+ // account's own buckets (the tenant boundary). Refreshed whenever
+ // buckets are created or deleted.
+ updateAccountIAMPolicy(iamClient, storeId, accountId, null);
+
+ // Reuse the stored access key only if both the access key id and the
+ // secret key are present and the key is still Active in IAM; otherwise
+ // create a replacement.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ String storedSecretKey = details.get(secretKeyDetailKey);
+ if (storedAccessKeyId != null && storedSecretKey != null
+ && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ return true;
+ }
+
+ // The stored key is missing, inactive, or no longer in IAM. Clean up
+ // ALL keys (including the inactive stored one) before creating a
+ // replacement so we do not accumulate keys and hit IAM limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, null);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Update existing bucket records for this account/store with the new
+ // credentials BEFORE persisting the new key in account details. If a
+ // bucket update fails, the stored key remains the old one and a retry
+ // will re-enter the replacement path; if we persisted first, a retry
+ // would see the new stored key and return without repairing the
+ // remaining buckets.
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ // Persist the credentials in the account details (namespaced by storeId)
+ details.put(accessKeyDetailKey, key.getAccessKeyId());
+ details.put(secretKeyDetailKey, key.getSecretAccessKey());
+ _accountDetailsDao.persist(accountId, details);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Update the IAM credentials on all BucketVO rows for this store/account
+ * so previously created buckets reflect the new (rotated) key pair.
+ * Mirrors CloudianHyperStoreObjectStoreDriverImpl.updateAccountBucketCredentials.
+ */
+ private void updateAccountBucketCredentials(long storeId, long accountId, AccessKey iamCredential) {
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ for (BucketVO bucketVO : bucketList) {
+ logger.info("Updating accountId={} bucket {} with new IAM credentials", accountId, bucketVO.getName());
+ bucketVO.setAccessKey(iamCredential.getAccessKeyId());
+ bucketVO.setSecretKey(iamCredential.getSecretAccessKey());
+ _bucketDao.update(bucketVO.getId(), bucketVO);
+ }
+ }
+
+ /**
+ * Refresh the per-account IAM user policy so it grants S3 access only to
+ * the account's current buckets (optionally excluding one, e.g. a bucket
+ * being deleted). This is the tenant boundary: each account's IAM
+ * credentials can only operate on that account's own buckets.
+ *
+ * @param iamClient the IAM client
+ * @param storeId the object store
+ * @param accountId the CloudStack account
+ * @param excludeBucket a bucket name to omit (e.g. a bucket being deleted),
+ * or null to include all of the account's buckets
+ */
+ protected void updateAccountIAMPolicy(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ return;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ List<BucketVO> buckets = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ List<String> bucketNames = new ArrayList<>();
+ for (BucketVO bvo : buckets) {
+ if (excludeBucket != null && excludeBucket.equals(bvo.getName())) {
+ continue;
+ }
+ bucketNames.add(bvo.getName());
+ }
+ String policy = SeaweedFSObjectStoreUtil.buildAccountIAMPolicy(bucketNames);
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ SeaweedFSObjectStoreUtil.IAM_USER_POLICY_NAME, policy));
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Check whether the given access key id is still listed and Active in IAM
+ * for the user. Listing failures are propagated rather than swallowed so
+ * a transient IAM outage does not send createUser into the replacement
+ * path (which would overwrite stored credentials and invalidate bucket
+ * records).
+ */
+ private boolean iamAccessKeyExists(AmazonIdentityManagement iamClient, String userName, String accessKeyId) {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ if (accessKeyId.equals(metadata.getAccessKeyId())) {
+ return "Active".equalsIgnoreCase(metadata.getStatus());
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Delete access keys for the user other than the (optionally) preserved
+ * key id. Used to clean up unmanaged leftover keys before creating a
+ * replacement so repeated calls do not hit IAM access-key limits.
+ */
+ private void deleteUnmanagedAccessKeys(AmazonIdentityManagement iamClient, String userName, String preserveAccessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ String keyId = metadata.getAccessKeyId();
+ if (preserveAccessKeyId != null && preserveAccessKeyId.equals(keyId)) {
+ continue;
+ }
+ DeleteAccessKeyRequest deleteReq =
+ new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(keyId);
+ logger.info("Deleting un-managed IAM access key {} for user {}", keyId, userName);
+ iamClient.deleteAccessKey(deleteReq);
+ }
+ } catch (AmazonClientException e) {
+ // Propagate so the caller does not proceed to create a replacement
+ // key while stale unmanaged keys remain (which could hit IAM key
+ // limits or leave orphaned credentials).
+ throw new CloudRuntimeException("Failed to clean up IAM access keys for user " + userName, e);
+ }
+ }
+
+ @Override
+ public Bucket createBucket(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+ long accountId = bucket.getAccountId();
+
+ // Use the store's admin credentials to create the bucket
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Check if the bucket already exists
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket already exists with name " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Create the bucket
+ try {
+ CreateBucketRequest request = new CreateBucketRequest(bucketName);
+ if (objectLock) {
+ request.setObjectLockEnabledForBucket(true);
+ }
+ s3client.createBucket(request);
+ } catch (AmazonClientException e) {
+ logger.error("Create bucket failed", e);
+ throw new CloudRuntimeException(e);
+ }
+
+ // Step 2: update the bucket record with the account's IAM credentials,
+ // configure CORS, and refresh the IAM policy. If any of these fail,
+ // clean up the remote bucket so a retry does not find it already
+ // existing — mirroring the Cloudian createBucket pattern.
+ //
+ // Hold the IAM lock for the account so a concurrent createUser key
+ // rotation does not change the account credentials between reading
+ // them and writing the BucketVO, which would leave the bucket with
+ // a stale key pair.
+ GlobalLock iamLock = acquireIamLock(storeId, accountId);
+ if (iamLock == null) {
+ // The S3 bucket has already been created. Clean it up so a
+ // retry does not find it already existing, then throw.
+ try {
+ s3client.deleteBucket(bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Failed to clean up bucket {} after IAM lock timeout", bucketName, cleanupEx);
+ }
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ // Configure permissive CORS so the CloudStack S3 bucket browser
+ // (which performs list/upload/delete from the browser) can function.
+ // SeaweedFS supports the standard PutBucketCors operation.
+ configureBucketCORS(s3client, bucketName);
+
+ Map<String, String> accountDetails = _accountDetailsDao.findDetails(accountId);
+ String accessKey = accountDetails.get(SeaweedFSObjectStoreUtil.keyAccessKey(storeId));
+ String secretKey = accountDetails.get(SeaweedFSObjectStoreUtil.keySecretKey(storeId));
+ if (accessKey == null || secretKey == null) {
+ throw new CloudRuntimeException("No IAM credentials found for account " + accountId
+ + " on store " + storeId + ". Run createUser before creating a bucket.");
+ }
+
+ String s3Url = getS3Url(storeId);
+ BucketVO bucketVO = _bucketDao.findById(bucket.getId());
+ bucketVO.setAccessKey(accessKey);
+ bucketVO.setSecretKey(secretKey);
+ bucketVO.setBucketURL(s3Url + "/" + bucketName);
+ _bucketDao.update(bucket.getId(), bucketVO);
+
+ // Refresh the account's IAM policy to include the new bucket
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicy(iamClient, storeId, accountId, null);
+
+ // Return the updated BucketVO (not the stale input bucket) so
+ // BucketApiServiceImpl.createBucket does not overwrite the
+ // persisted credentials with the stale values.
+ return bucketVO;
+ } catch (Exception e) {
+ logger.error("Post-create bucket record update failed for {}; cleaning up remote bucket", bucketName, e);
+ try {
+ s3client.deleteBucket(bucketName);
+ logger.info("Cleanup of bucket {} succeeded", bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Cleanup of bucket {} also failed", bucketName, cleanupEx);
+ }
+ // Revoke the IAM policy grant for the new bucket so the account's
+ // credentials cannot access a bucket that no longer exists. If the
+ // policy PUT succeeded before the DB update failed, the grant
+ // would otherwise persist and could be reused if another account
+ // later creates the same bucket name.
+ try {
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicy(iamClient, storeId, accountId, bucketName);
+ } catch (Exception policyEx) {
+ logger.warn("Failed to revoke IAM policy for bucket {} after cleanup: {}", bucketName, policyEx.getMessage());
+ }
+ throw new CloudRuntimeException(e);
+ } finally {
+ iamLock.unlock();
+ iamLock.releaseRef();
+ }
+ }
+
+ /**
+ * Configure a permissive CORS policy on the bucket so the CloudStack
+ * S3 bucket browser (which performs list/upload/delete from the
+ * browser) can function. Mirrors the Cloudian configureBucketCORS.
+ */
+ private void configureBucketCORS(AmazonS3 s3client, String bucketName) {
+ logger.debug("Configuring CORS for bucket {}", bucketName);
+ List<CORSRule> corsRules = new ArrayList<>();
+ CORSRule allowAnyRule = new CORSRule().withId("AllowAny");
+ allowAnyRule.setAllowedOrigins("*");
+ allowAnyRule.setAllowedHeaders("*");
+ allowAnyRule.setAllowedMethods(
+ CORSRule.AllowedMethods.HEAD,
+ CORSRule.AllowedMethods.GET,
+ CORSRule.AllowedMethods.PUT,
+ CORSRule.AllowedMethods.POST,
+ CORSRule.AllowedMethods.DELETE);
+ corsRules.add(allowAnyRule);
+ BucketCrossOriginConfiguration corsConfig = new BucketCrossOriginConfiguration();
+ corsConfig.setRules(corsRules);
+ SetBucketCrossOriginConfigurationRequest corsRequest = new SetBucketCrossOriginConfigurationRequest(bucketName, corsConfig);
+ s3client.setBucketCrossOriginConfiguration(corsRequest);
+ logger.info("Successfully configured CORS for bucket {}", bucketName);
+ }
+
+ @Override
+ public List<Bucket> listBuckets(long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ List<Bucket> bucketsList = new ArrayList<>();
+ try {
+ List<com.amazonaws.services.s3.model.Bucket> s3Buckets = s3client.listBuckets();
+ for (com.amazonaws.services.s3.model.Bucket s3Bucket : s3Buckets) {
+ Bucket bucket = new BucketObject();
+ bucket.setName(s3Bucket.getName());
+ bucketsList.add(bucket);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ return bucketsList;
+ }
+
+ @Override
+ public boolean deleteBucket(BucketTO bucket, long storeId) {
+ String bucketName = bucket.getName();
+ long accountId = bucket.getAccountId();
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Delete the S3 bucket first. If this fails (non-empty bucket,
+ // transient error), the IAM policy is still intact so the user
+ // can empty the bucket and retry.
+ //
+ // If the bucket is already gone (e.g. from a previous partial
+ // failure where the S3 delete succeeded but the IAM policy refresh
+ // failed), skip the S3 delete so the retry is idempotent.
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ s3client.deleteBucket(bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Remove the BucketVO row before refreshing the IAM policy so a
+ // concurrent createUser/createBucket policy rebuild (which reads the
+ // bucket list from the DB) cannot re-add the deleted bucket ARN
+ // between this exclusion and BucketApiServiceImpl's row removal.
+ // BucketApiServiceImpl's subsequent _bucketDao.remove is idempotent.
+ for (BucketVO bvo : _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId)) {
+ if (bucketName.equals(bvo.getName())) {
+ _bucketDao.remove(bvo.getId());
+ break;
+ }
Review Comment:
Removing the `BucketVO` before refreshing IAM makes a policy failure unrecoverable. If `updateAccountIAMPolicy` fails after the S3 bucket is deleted, `BucketApiServiceImpl` never reaches its resource-counter decrement; this row is already gone, so a retry cannot find the bucket or reconstruct the policy, while the old account credentials may still access its ARN. Keep deletion pending in retryable state until policy revocation succeeds, then remove the row and release accounting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r4011370492
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,780 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.BucketCrossOriginConfiguration;
+import com.amazonaws.services.s3.model.CORSRule;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketCrossOriginConfigurationRequest;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ /**
+ * DB-backed global lock name prefix for serializing IAM provisioning and
+ * policy refreshes per store+account. Uses {@link GlobalLock} so the
+ * critical section is serialized across management servers in a
+ * clustered deployment, not just within a single JVM.
+ */
+ private static final String IAM_LOCK_PREFIX = "seaweedfs.iam.";
+
+ private static String getIamLockName(long storeId, long accountId) {
+ return IAM_LOCK_PREFIX + storeId + "." + accountId;
+ }
+
+ /**
+ * Acquire a DB-backed global lock for IAM operations on the given
+ * store+account. Returns a {@link GlobalLock} that the caller must
+ * {@link GlobalLock#unlock()} in a {@code finally} block, or {@code null}
+ * if the lock could not be acquired within the timeout.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ */
+ protected GlobalLock acquireIamLock(long storeId, long accountId) {
+ GlobalLock lock = GlobalLock.getInternLock(getIamLockName(storeId, accountId));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire IAM lock for store {} account {}", storeId, accountId);
+ lock.releaseRef();
+ return null;
+ }
+ return lock;
+ }
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account and
+ * store. The store ID is included so that two CloudStack pools pointing
+ * at the same SeaweedFS IAM service do not collide on the same
+ * {@code acs-<uuid>} user and overwrite each other's policy and access
+ * keys.
+ */
+ protected String getUserNameForAccount(Account account, long storeId) {
+ return String.format("%s-%d-%s", ACS_PREFIX, storeId, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Serialize per store+account across management servers so two
+ // concurrent bucket requests do not both rotate credentials and leave
+ // bucket rows with mismatched key pairs.
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ return false;
+ }
+ try {
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach a scoped IAM policy that allows access only to this
+ // account's own buckets (the tenant boundary). Refreshed whenever
+ // buckets are created or deleted.
+ updateAccountIAMPolicy(iamClient, storeId, accountId, null);
+
+ // Reuse the stored access key only if both the access key id and the
+ // secret key are present and the key is still Active in IAM; otherwise
+ // create a replacement.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ String storedSecretKey = details.get(secretKeyDetailKey);
+ if (storedAccessKeyId != null && storedSecretKey != null
+ && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ return true;
+ }
+
+ // The stored key is missing, inactive, or no longer in IAM. Clean up
+ // ALL keys (including the inactive stored one) before creating a
+ // replacement so we do not accumulate keys and hit IAM limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, null);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Update existing bucket records for this account/store with the new
+ // credentials BEFORE persisting the new key in account details. If a
+ // bucket update fails, the stored key remains the old one and a retry
+ // will re-enter the replacement path; if we persisted first, a retry
+ // would see the new stored key and return without repairing the
+ // remaining buckets.
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ // Persist the credentials in the account details (namespaced by storeId)
+ details.put(accessKeyDetailKey, key.getAccessKeyId());
+ details.put(secretKeyDetailKey, key.getSecretAccessKey());
+ _accountDetailsDao.persist(accountId, details);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Update the IAM credentials on all BucketVO rows for this store/account
+ * so previously created buckets reflect the new (rotated) key pair.
+ * Mirrors CloudianHyperStoreObjectStoreDriverImpl.updateAccountBucketCredentials.
+ */
+ private void updateAccountBucketCredentials(long storeId, long accountId, AccessKey iamCredential) {
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ for (BucketVO bucketVO : bucketList) {
+ logger.info("Updating accountId={} bucket {} with new IAM credentials", accountId, bucketVO.getName());
+ bucketVO.setAccessKey(iamCredential.getAccessKeyId());
+ bucketVO.setSecretKey(iamCredential.getSecretAccessKey());
+ _bucketDao.update(bucketVO.getId(), bucketVO);
+ }
+ }
+
+ /**
+ * Refresh the per-account IAM user policy so it grants S3 access only to
+ * the account's current buckets (optionally excluding one, e.g. a bucket
+ * being deleted). This is the tenant boundary: each account's IAM
+ * credentials can only operate on that account's own buckets.
+ *
+ * @param iamClient the IAM client
+ * @param storeId the object store
+ * @param accountId the CloudStack account
+ * @param excludeBucket a bucket name to omit (e.g. a bucket being deleted),
+ * or null to include all of the account's buckets
+ */
+ protected void updateAccountIAMPolicy(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ return;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ List<BucketVO> buckets = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ List<String> bucketNames = new ArrayList<>();
+ for (BucketVO bvo : buckets) {
+ if (excludeBucket != null && excludeBucket.equals(bvo.getName())) {
+ continue;
+ }
+ bucketNames.add(bvo.getName());
+ }
+ String policy = SeaweedFSObjectStoreUtil.buildAccountIAMPolicy(bucketNames);
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ SeaweedFSObjectStoreUtil.IAM_USER_POLICY_NAME, policy));
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Check whether the given access key id is still listed and Active in IAM
+ * for the user. Listing failures are propagated rather than swallowed so
+ * a transient IAM outage does not send createUser into the replacement
+ * path (which would overwrite stored credentials and invalidate bucket
+ * records).
+ */
+ private boolean iamAccessKeyExists(AmazonIdentityManagement iamClient, String userName, String accessKeyId) {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ if (accessKeyId.equals(metadata.getAccessKeyId())) {
+ return "Active".equalsIgnoreCase(metadata.getStatus());
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Delete access keys for the user other than the (optionally) preserved
+ * key id. Used to clean up unmanaged leftover keys before creating a
+ * replacement so repeated calls do not hit IAM access-key limits.
+ */
+ private void deleteUnmanagedAccessKeys(AmazonIdentityManagement iamClient, String userName, String preserveAccessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ String keyId = metadata.getAccessKeyId();
+ if (preserveAccessKeyId != null && preserveAccessKeyId.equals(keyId)) {
+ continue;
+ }
+ DeleteAccessKeyRequest deleteReq =
+ new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(keyId);
+ logger.info("Deleting un-managed IAM access key {} for user {}", keyId, userName);
+ iamClient.deleteAccessKey(deleteReq);
+ }
+ } catch (AmazonClientException e) {
+ // Propagate so the caller does not proceed to create a replacement
+ // key while stale unmanaged keys remain (which could hit IAM key
+ // limits or leave orphaned credentials).
+ throw new CloudRuntimeException("Failed to clean up IAM access keys for user " + userName, e);
+ }
+ }
+
+ @Override
+ public Bucket createBucket(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+ long accountId = bucket.getAccountId();
+
+ // Use the store's admin credentials to create the bucket
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Check if the bucket already exists
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket already exists with name " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Create the bucket
+ try {
+ CreateBucketRequest request = new CreateBucketRequest(bucketName);
+ if (objectLock) {
+ request.setObjectLockEnabledForBucket(true);
+ }
+ s3client.createBucket(request);
+ } catch (AmazonClientException e) {
+ logger.error("Create bucket failed", e);
+ throw new CloudRuntimeException(e);
+ }
+
+ // Step 2: update the bucket record with the account's IAM credentials,
+ // configure CORS, and refresh the IAM policy. If any of these fail,
+ // clean up the remote bucket so a retry does not find it already
+ // existing — mirroring the Cloudian createBucket pattern.
+ //
+ // Hold the IAM lock for the account so a concurrent createUser key
+ // rotation does not change the account credentials between reading
+ // them and writing the BucketVO, which would leave the bucket with
+ // a stale key pair.
+ GlobalLock iamLock = acquireIamLock(storeId, accountId);
+ if (iamLock == null) {
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ // Configure permissive CORS so the CloudStack S3 bucket browser
+ // (which performs list/upload/delete from the browser) can function.
+ // SeaweedFS supports the standard PutBucketCors operation.
+ configureBucketCORS(s3client, bucketName);
+
+ Map<String, String> accountDetails = _accountDetailsDao.findDetails(accountId);
+ String accessKey = accountDetails.get(SeaweedFSObjectStoreUtil.keyAccessKey(storeId));
+ String secretKey = accountDetails.get(SeaweedFSObjectStoreUtil.keySecretKey(storeId));
+ if (accessKey == null || secretKey == null) {
+ throw new CloudRuntimeException("No IAM credentials found for account " + accountId
+ + " on store " + storeId + ". Run createUser before creating a bucket.");
+ }
+
+ String s3Url = getS3Url(storeId);
+ BucketVO bucketVO = _bucketDao.findById(bucket.getId());
+ bucketVO.setAccessKey(accessKey);
+ bucketVO.setSecretKey(secretKey);
+ bucketVO.setBucketURL(s3Url + "/" + bucketName);
+ _bucketDao.update(bucket.getId(), bucketVO);
+
+ // Refresh the account's IAM policy to include the new bucket
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicy(iamClient, storeId, accountId, null);
+
+ return bucket;
+ } catch (Exception e) {
+ logger.error("Post-create bucket record update failed for {}; cleaning up remote bucket", bucketName, e);
+ try {
+ s3client.deleteBucket(bucketName);
+ logger.info("Cleanup of bucket {} succeeded", bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Cleanup of bucket {} also failed", bucketName, cleanupEx);
+ }
+ // Revoke the IAM policy grant for the new bucket so the account's
+ // credentials cannot access a bucket that no longer exists. If the
+ // policy PUT succeeded before the DB update failed, the grant
+ // would otherwise persist and could be reused if another account
+ // later creates the same bucket name.
+ try {
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicy(iamClient, storeId, accountId, bucketName);
+ } catch (Exception policyEx) {
+ logger.warn("Failed to revoke IAM policy for bucket {} after cleanup: {}", bucketName, policyEx.getMessage());
+ }
+ throw new CloudRuntimeException(e);
+ } finally {
+ iamLock.unlock();
+ iamLock.releaseRef();
+ }
+ }
+
+ /**
+ * Configure a permissive CORS policy on the bucket so the CloudStack
+ * S3 bucket browser (which performs list/upload/delete from the
+ * browser) can function. Mirrors the Cloudian configureBucketCORS.
+ */
+ private void configureBucketCORS(AmazonS3 s3client, String bucketName) {
+ logger.debug("Configuring CORS for bucket {}", bucketName);
+ List<CORSRule> corsRules = new ArrayList<>();
+ CORSRule allowAnyRule = new CORSRule().withId("AllowAny");
+ allowAnyRule.setAllowedOrigins("*");
+ allowAnyRule.setAllowedHeaders("*");
+ allowAnyRule.setAllowedMethods(
+ CORSRule.AllowedMethods.HEAD,
+ CORSRule.AllowedMethods.GET,
+ CORSRule.AllowedMethods.PUT,
+ CORSRule.AllowedMethods.POST,
+ CORSRule.AllowedMethods.DELETE);
+ corsRules.add(allowAnyRule);
+ BucketCrossOriginConfiguration corsConfig = new BucketCrossOriginConfiguration();
+ corsConfig.setRules(corsRules);
+ SetBucketCrossOriginConfigurationRequest corsRequest = new SetBucketCrossOriginConfigurationRequest(bucketName, corsConfig);
+ s3client.setBucketCrossOriginConfiguration(corsRequest);
+ logger.info("Successfully configured CORS for bucket {}", bucketName);
+ }
+
+ @Override
+ public List<Bucket> listBuckets(long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ List<Bucket> bucketsList = new ArrayList<>();
+ try {
+ List<com.amazonaws.services.s3.model.Bucket> s3Buckets = s3client.listBuckets();
+ for (com.amazonaws.services.s3.model.Bucket s3Bucket : s3Buckets) {
+ Bucket bucket = new BucketObject();
+ bucket.setName(s3Bucket.getName());
+ bucketsList.add(bucket);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ return bucketsList;
+ }
+
+ @Override
+ public boolean deleteBucket(BucketTO bucket, long storeId) {
+ String bucketName = bucket.getName();
+ long accountId = bucket.getAccountId();
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Refresh the account's IAM policy to drop the deleted bucket BEFORE
+ // the S3 delete. Bucket names are reusable, so revoking the grant
+ // before the name becomes reusable prevents the old account from
+ // accessing a new tenant's bucket with the same name. If this fails,
+ // the bucket still exists and a retry can proceed. If it succeeds but
+ // the S3 delete fails, the grant is already revoked and a retry only
+ // needs to delete the S3 bucket (the policy refresh is idempotent
+ // because excludeBucket still applies).
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicy(iamClient, storeId, accountId, bucketName);
+
+ // If the bucket is already gone (e.g. from a previous partial
+ // failure where the policy refresh succeeded but the S3 delete
+ // failed), skip the S3 delete so the retry is idempotent.
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ s3client.deleteBucket(bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ return true;
+ }
+
+ @Override
+ public AccessControlList getBucketAcl(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ return s3client.getBucketAcl(bucket.getName());
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public void setBucketAcl(BucketTO bucket, AccessControlList acl, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ s3client.setBucketAcl(bucket.getName(), acl);
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public void setBucketPolicy(BucketTO bucket, String policy, long storeId) {
+ if ("private".equalsIgnoreCase(policy)) {
+ deleteBucketPolicy(bucket, storeId);
+ return;
+ }
+
+ StringBuilder sb = new StringBuilder();
+ sb.append("{\n");
+ sb.append(" \"Version\": \"2012-10-17\",\n");
+ sb.append(" \"Statement\": [\n");
+ sb.append(" {\n");
+ sb.append(" \"Sid\": \"PublicReadForObjects\",\n");
+ sb.append(" \"Effect\": \"Allow\",\n");
+ sb.append(" \"Principal\": \"*\",\n");
+ sb.append(" \"Action\": \"s3:GetObject\",\n");
+ sb.append(" \"Resource\": \"arn:aws:s3:::%s/*\"\n");
+ sb.append(" }\n");
+ sb.append(" ]\n");
+ sb.append("}\n");
+
+ String jsonPolicy = String.format(sb.toString(), bucket.getName());
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ s3client.setBucketPolicy(bucket.getName(), jsonPolicy);
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public BucketPolicy getBucketPolicy(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ return s3client.getBucketPolicy(new GetBucketPolicyRequest(bucket.getName()));
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public void deleteBucketPolicy(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ s3client.deleteBucketPolicy(new DeleteBucketPolicyRequest(bucket.getName()));
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public boolean setBucketEncryption(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ SetBucketEncryptionRequest eRequest = new SetBucketEncryptionRequest();
+ eRequest.setBucketName(bucket.getName());
+
+ ServerSideEncryptionByDefault sseByDefault = new ServerSideEncryptionByDefault();
+ sseByDefault.setSSEAlgorithm(SSEAlgorithm.AES256.toString());
+
+ ServerSideEncryptionRule sseRule = new ServerSideEncryptionRule();
+ sseRule.setApplyServerSideEncryptionByDefault(sseByDefault);
+
+ List<ServerSideEncryptionRule> sseRules = new ArrayList<>();
+ sseRules.add(sseRule);
+
+ ServerSideEncryptionConfiguration sseConf = new ServerSideEncryptionConfiguration();
+ sseConf.setRules(sseRules);
+
+ eRequest.setServerSideEncryptionConfiguration(sseConf);
+ s3client.setBucketEncryption(eRequest);
+ return true;
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public boolean deleteBucketEncryption(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ s3client.deleteBucketEncryption(bucket.getName());
+ return true;
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public boolean setBucketVersioning(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ BucketVersioningConfiguration vConf = new BucketVersioningConfiguration(BucketVersioningConfiguration.ENABLED);
+ s3client.setBucketVersioningConfiguration(
+ new SetBucketVersioningConfigurationRequest(bucket.getName(), vConf));
+ return true;
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public boolean deleteBucketVersioning(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ BucketVersioningConfiguration vConf = new BucketVersioningConfiguration(BucketVersioningConfiguration.SUSPENDED);
+ s3client.setBucketVersioningConfiguration(
+ new SetBucketVersioningConfigurationRequest(bucket.getName(), vConf));
+ return true;
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ /**
+ * Set the bucket quota via the SeaweedFS S3 extension
+ * ({@code PUT /{bucket}?seaweedfs-quota}), signed with the store admin
+ * S3 credentials.
+ *
+ * SeaweedFS enforces bucket quota server-side by setting a read-only flag
+ * when usage exceeds the configured limit. The quota is configured via the
+ * SeaweedFS S3 extension endpoint PUT /{bucket}?seaweedfs-quota,
+ * authenticated via standard S3 SigV4 and authorized via the
+ * s3:PutBucketQuota IAM permission.
+ *
+ * @param size the GiB size to set the quota to. 0 disables quota.
+ * @throws CloudRuntimeException if the S3 endpoint or credentials are missing or the request fails.
+ */
+ @Override
+ public void setBucketQuota(BucketTO bucket, long storeId, long size) {
+ String s3Url = getS3Url(storeId);
+ String accessKey = getAccessKey(storeId);
+ String secretKey = getSecretKey(storeId);
+ if (s3Url == null || s3Url.isEmpty() || accessKey == null || accessKey.isEmpty() || secretKey == null || secretKey.isEmpty()) {
+ throw new CloudRuntimeException("SeaweedFS S3 URL and credentials are required to set bucket quota. " +
+ "Configure 's3Url', 'accesskey', and 'secretkey' in the object store details.");
+ }
+ SeaweedFSObjectStoreUtil.setBucketQuotaViaS3Extension(s3Url, accessKey, secretKey, bucket.getName(), size, getS3ExtensionHttpClient());
Review Comment:
`CreateBucketCmd` requires a quota and `BucketApiServiceImpl.createBucket` calls `setQuota` for every create, including quota `0`. Therefore a deployment without the SeaweedFS quota extension receives a 404 here and the create path removes the newly created bucket, so basic bucket CRUD does not work despite the compatibility description saying non-quota operations work without the extension. Either bypass this call for a no-quota create while preserving a way to clear an existing quota, or make the extension a hard provider prerequisite and document it.
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,780 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.BucketCrossOriginConfiguration;
+import com.amazonaws.services.s3.model.CORSRule;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketCrossOriginConfigurationRequest;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ /**
+ * DB-backed global lock name prefix for serializing IAM provisioning and
+ * policy refreshes per store+account. Uses {@link GlobalLock} so the
+ * critical section is serialized across management servers in a
+ * clustered deployment, not just within a single JVM.
+ */
+ private static final String IAM_LOCK_PREFIX = "seaweedfs.iam.";
+
+ private static String getIamLockName(long storeId, long accountId) {
+ return IAM_LOCK_PREFIX + storeId + "." + accountId;
+ }
+
+ /**
+ * Acquire a DB-backed global lock for IAM operations on the given
+ * store+account. Returns a {@link GlobalLock} that the caller must
+ * {@link GlobalLock#unlock()} in a {@code finally} block, or {@code null}
+ * if the lock could not be acquired within the timeout.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ */
+ protected GlobalLock acquireIamLock(long storeId, long accountId) {
+ GlobalLock lock = GlobalLock.getInternLock(getIamLockName(storeId, accountId));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire IAM lock for store {} account {}", storeId, accountId);
+ lock.releaseRef();
+ return null;
+ }
+ return lock;
+ }
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account and
+ * store. The store ID is included so that two CloudStack pools pointing
+ * at the same SeaweedFS IAM service do not collide on the same
+ * {@code acs-<uuid>} user and overwrite each other's policy and access
+ * keys.
+ */
+ protected String getUserNameForAccount(Account account, long storeId) {
+ return String.format("%s-%d-%s", ACS_PREFIX, storeId, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Serialize per store+account across management servers so two
+ // concurrent bucket requests do not both rotate credentials and leave
+ // bucket rows with mismatched key pairs.
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ return false;
+ }
+ try {
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach a scoped IAM policy that allows access only to this
+ // account's own buckets (the tenant boundary). Refreshed whenever
+ // buckets are created or deleted.
+ updateAccountIAMPolicy(iamClient, storeId, accountId, null);
+
+ // Reuse the stored access key only if both the access key id and the
+ // secret key are present and the key is still Active in IAM; otherwise
+ // create a replacement.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ String storedSecretKey = details.get(secretKeyDetailKey);
+ if (storedAccessKeyId != null && storedSecretKey != null
+ && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ return true;
+ }
+
+ // The stored key is missing, inactive, or no longer in IAM. Clean up
+ // ALL keys (including the inactive stored one) before creating a
+ // replacement so we do not accumulate keys and hit IAM limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, null);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Update existing bucket records for this account/store with the new
+ // credentials BEFORE persisting the new key in account details. If a
+ // bucket update fails, the stored key remains the old one and a retry
+ // will re-enter the replacement path; if we persisted first, a retry
+ // would see the new stored key and return without repairing the
+ // remaining buckets.
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ // Persist the credentials in the account details (namespaced by storeId)
+ details.put(accessKeyDetailKey, key.getAccessKeyId());
+ details.put(secretKeyDetailKey, key.getSecretAccessKey());
+ _accountDetailsDao.persist(accountId, details);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Update the IAM credentials on all BucketVO rows for this store/account
+ * so previously created buckets reflect the new (rotated) key pair.
+ * Mirrors CloudianHyperStoreObjectStoreDriverImpl.updateAccountBucketCredentials.
+ */
+ private void updateAccountBucketCredentials(long storeId, long accountId, AccessKey iamCredential) {
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ for (BucketVO bucketVO : bucketList) {
+ logger.info("Updating accountId={} bucket {} with new IAM credentials", accountId, bucketVO.getName());
+ bucketVO.setAccessKey(iamCredential.getAccessKeyId());
+ bucketVO.setSecretKey(iamCredential.getSecretAccessKey());
+ _bucketDao.update(bucketVO.getId(), bucketVO);
+ }
+ }
+
+ /**
+ * Refresh the per-account IAM user policy so it grants S3 access only to
+ * the account's current buckets (optionally excluding one, e.g. a bucket
+ * being deleted). This is the tenant boundary: each account's IAM
+ * credentials can only operate on that account's own buckets.
+ *
+ * @param iamClient the IAM client
+ * @param storeId the object store
+ * @param accountId the CloudStack account
+ * @param excludeBucket a bucket name to omit (e.g. a bucket being deleted),
+ * or null to include all of the account's buckets
+ */
+ protected void updateAccountIAMPolicy(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ return;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ List<BucketVO> buckets = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ List<String> bucketNames = new ArrayList<>();
+ for (BucketVO bvo : buckets) {
+ if (excludeBucket != null && excludeBucket.equals(bvo.getName())) {
+ continue;
+ }
+ bucketNames.add(bvo.getName());
+ }
+ String policy = SeaweedFSObjectStoreUtil.buildAccountIAMPolicy(bucketNames);
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ SeaweedFSObjectStoreUtil.IAM_USER_POLICY_NAME, policy));
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Check whether the given access key id is still listed and Active in IAM
+ * for the user. Listing failures are propagated rather than swallowed so
+ * a transient IAM outage does not send createUser into the replacement
+ * path (which would overwrite stored credentials and invalidate bucket
+ * records).
+ */
+ private boolean iamAccessKeyExists(AmazonIdentityManagement iamClient, String userName, String accessKeyId) {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ if (accessKeyId.equals(metadata.getAccessKeyId())) {
+ return "Active".equalsIgnoreCase(metadata.getStatus());
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Delete access keys for the user other than the (optionally) preserved
+ * key id. Used to clean up unmanaged leftover keys before creating a
+ * replacement so repeated calls do not hit IAM access-key limits.
+ */
+ private void deleteUnmanagedAccessKeys(AmazonIdentityManagement iamClient, String userName, String preserveAccessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ String keyId = metadata.getAccessKeyId();
+ if (preserveAccessKeyId != null && preserveAccessKeyId.equals(keyId)) {
+ continue;
+ }
+ DeleteAccessKeyRequest deleteReq =
+ new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(keyId);
+ logger.info("Deleting un-managed IAM access key {} for user {}", keyId, userName);
+ iamClient.deleteAccessKey(deleteReq);
+ }
+ } catch (AmazonClientException e) {
+ // Propagate so the caller does not proceed to create a replacement
+ // key while stale unmanaged keys remain (which could hit IAM key
+ // limits or leave orphaned credentials).
+ throw new CloudRuntimeException("Failed to clean up IAM access keys for user " + userName, e);
+ }
+ }
+
+ @Override
+ public Bucket createBucket(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+ long accountId = bucket.getAccountId();
+
+ // Use the store's admin credentials to create the bucket
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Check if the bucket already exists
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket already exists with name " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Create the bucket
+ try {
+ CreateBucketRequest request = new CreateBucketRequest(bucketName);
+ if (objectLock) {
+ request.setObjectLockEnabledForBucket(true);
+ }
+ s3client.createBucket(request);
+ } catch (AmazonClientException e) {
+ logger.error("Create bucket failed", e);
+ throw new CloudRuntimeException(e);
+ }
+
+ // Step 2: update the bucket record with the account's IAM credentials,
+ // configure CORS, and refresh the IAM policy. If any of these fail,
+ // clean up the remote bucket so a retry does not find it already
+ // existing — mirroring the Cloudian createBucket pattern.
+ //
+ // Hold the IAM lock for the account so a concurrent createUser key
+ // rotation does not change the account credentials between reading
+ // them and writing the BucketVO, which would leave the bucket with
+ // a stale key pair.
+ GlobalLock iamLock = acquireIamLock(storeId, accountId);
+ if (iamLock == null) {
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ // Configure permissive CORS so the CloudStack S3 bucket browser
+ // (which performs list/upload/delete from the browser) can function.
+ // SeaweedFS supports the standard PutBucketCors operation.
+ configureBucketCORS(s3client, bucketName);
+
+ Map<String, String> accountDetails = _accountDetailsDao.findDetails(accountId);
+ String accessKey = accountDetails.get(SeaweedFSObjectStoreUtil.keyAccessKey(storeId));
+ String secretKey = accountDetails.get(SeaweedFSObjectStoreUtil.keySecretKey(storeId));
+ if (accessKey == null || secretKey == null) {
+ throw new CloudRuntimeException("No IAM credentials found for account " + accountId
+ + " on store " + storeId + ". Run createUser before creating a bucket.");
+ }
+
+ String s3Url = getS3Url(storeId);
+ BucketVO bucketVO = _bucketDao.findById(bucket.getId());
+ bucketVO.setAccessKey(accessKey);
+ bucketVO.setSecretKey(secretKey);
+ bucketVO.setBucketURL(s3Url + "/" + bucketName);
+ _bucketDao.update(bucket.getId(), bucketVO);
+
+ // Refresh the account's IAM policy to include the new bucket
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicy(iamClient, storeId, accountId, null);
+
+ return bucket;
+ } catch (Exception e) {
+ logger.error("Post-create bucket record update failed for {}; cleaning up remote bucket", bucketName, e);
+ try {
+ s3client.deleteBucket(bucketName);
+ logger.info("Cleanup of bucket {} succeeded", bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Cleanup of bucket {} also failed", bucketName, cleanupEx);
+ }
+ // Revoke the IAM policy grant for the new bucket so the account's
+ // credentials cannot access a bucket that no longer exists. If the
+ // policy PUT succeeded before the DB update failed, the grant
+ // would otherwise persist and could be reused if another account
+ // later creates the same bucket name.
+ try {
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicy(iamClient, storeId, accountId, bucketName);
+ } catch (Exception policyEx) {
+ logger.warn("Failed to revoke IAM policy for bucket {} after cleanup: {}", bucketName, policyEx.getMessage());
+ }
+ throw new CloudRuntimeException(e);
+ } finally {
+ iamLock.unlock();
+ iamLock.releaseRef();
+ }
+ }
+
+ /**
+ * Configure a permissive CORS policy on the bucket so the CloudStack
+ * S3 bucket browser (which performs list/upload/delete from the
+ * browser) can function. Mirrors the Cloudian configureBucketCORS.
+ */
+ private void configureBucketCORS(AmazonS3 s3client, String bucketName) {
+ logger.debug("Configuring CORS for bucket {}", bucketName);
+ List<CORSRule> corsRules = new ArrayList<>();
+ CORSRule allowAnyRule = new CORSRule().withId("AllowAny");
+ allowAnyRule.setAllowedOrigins("*");
+ allowAnyRule.setAllowedHeaders("*");
+ allowAnyRule.setAllowedMethods(
+ CORSRule.AllowedMethods.HEAD,
+ CORSRule.AllowedMethods.GET,
+ CORSRule.AllowedMethods.PUT,
+ CORSRule.AllowedMethods.POST,
+ CORSRule.AllowedMethods.DELETE);
+ corsRules.add(allowAnyRule);
+ BucketCrossOriginConfiguration corsConfig = new BucketCrossOriginConfiguration();
+ corsConfig.setRules(corsRules);
+ SetBucketCrossOriginConfigurationRequest corsRequest = new SetBucketCrossOriginConfigurationRequest(bucketName, corsConfig);
+ s3client.setBucketCrossOriginConfiguration(corsRequest);
+ logger.info("Successfully configured CORS for bucket {}", bucketName);
+ }
+
+ @Override
+ public List<Bucket> listBuckets(long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ List<Bucket> bucketsList = new ArrayList<>();
+ try {
+ List<com.amazonaws.services.s3.model.Bucket> s3Buckets = s3client.listBuckets();
+ for (com.amazonaws.services.s3.model.Bucket s3Bucket : s3Buckets) {
+ Bucket bucket = new BucketObject();
+ bucket.setName(s3Bucket.getName());
+ bucketsList.add(bucket);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ return bucketsList;
+ }
+
+ @Override
+ public boolean deleteBucket(BucketTO bucket, long storeId) {
+ String bucketName = bucket.getName();
+ long accountId = bucket.getAccountId();
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Refresh the account's IAM policy to drop the deleted bucket BEFORE
+ // the S3 delete. Bucket names are reusable, so revoking the grant
+ // before the name becomes reusable prevents the old account from
+ // accessing a new tenant's bucket with the same name. If this fails,
+ // the bucket still exists and a retry can proceed. If it succeeds but
+ // the S3 delete fails, the grant is already revoked and a retry only
+ // needs to delete the S3 bucket (the policy refresh is idempotent
+ // because excludeBucket still applies).
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicy(iamClient, storeId, accountId, bucketName);
Review Comment:
Please don't leave the policy revoked when the delete fails. A non-empty bucket or transient S3 error leaves the BucketVO in CloudStack because the API removes it only after this method succeeds, but the account's IAM policy no longer grants access, so the user cannot empty the bucket or retry. Restore the policy on failure, or refresh it only after a successful delete with a rollback path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "chrislusf (via GitHub)" <gi...@apache.org>.
chrislusf commented on PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#issuecomment-5683499726
Addressed Copilot review `5206611018`.
- Credential detail writes: fixed in `932cc3c`; the access/secret pair now rolls back together and cleans up the newly-created IAM key on persistence failure.
- Delete accounting idempotency: fixed in `932cc3c`; bucket count, `object_storage`, allocated size, and row removal are handled together after the driver delete path returns.
- Quota update accounting and `BucketVO` persistence: fixed in `932cc3c`; `updateAllocatedSize` is checked, resource counts are compensated on failure, remote quota is restored, and quota persistence happens inside the quota helper.
- Missing SeaweedFS quota extension: fixed in the branch by allowing missing extension only before the bucket has allocated state, not during update/rollback paths.
- Bucket URL / storage URL trailing slash normalization: fixed in `e76aef2`.
- Store-wide bucket-name locking for create/delete: fixed in `c9eaf62`.
Validated with:
- `mvn -pl plugins/storage/object/seaweedfs -Dtest=SeaweedFSObjectStoreDriverImplTest -Dnet.bytebuddy.experimental=true test`
- `mvn -pl server -Dtest=BucketApiServiceImplTest -Dnet.bytebuddy.experimental=true test`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "chrislusf (via GitHub)" <gi...@apache.org>.
chrislusf commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r4053073055
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,1072 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.BucketCrossOriginConfiguration;
+import com.amazonaws.services.s3.model.CORSRule;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketCrossOriginConfigurationRequest;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ /**
+ * DB-backed global lock name prefix for serializing IAM provisioning and
+ * policy refreshes per store+account. Uses {@link GlobalLock} so the
+ * critical section is serialized across management servers in a
+ * clustered deployment, not just within a single JVM.
+ */
+ private static final String IAM_LOCK_PREFIX = "seaweedfs.iam.";
+ private static final String BUCKET_NAME_LOCK_PREFIX = "seaweedfs.bucket.";
+
+ private static String getIamLockName(long storeId, long accountId) {
+ return IAM_LOCK_PREFIX + storeId + "." + accountId;
+ }
+
+ /**
+ * Acquire a DB-backed global lock for IAM operations on the given
+ * store+account. Returns a {@link GlobalLock} that the caller must
+ * {@link GlobalLock#unlock()} in a {@code finally} block, or {@code null}
+ * if the lock could not be acquired within the timeout.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ */
+ protected GlobalLock acquireIamLock(long storeId, long accountId) {
+ GlobalLock lock = GlobalLock.getInternLock(getIamLockName(storeId, accountId));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire IAM lock for store {} account {}", storeId, accountId);
+ lock.releaseRef();
+ return null;
+ }
+ return lock;
+ }
+
+ private static String getBucketNameLockName(long storeId, String bucketName) {
+ return BUCKET_NAME_LOCK_PREFIX + storeId + "." + bucketName;
+ }
+
+ /**
+ * Acquire a DB-backed global lock covering a bucket <em>name</em> on a store,
+ * independent of the owning account.
+ *
+ * S3 bucket names are globally unique within a store and are reusable after
+ * deletion, while the IAM lock is scoped to (store, account). Without this
+ * lock, once a delete removes the remote bucket but before the old owner's
+ * IAM policy is refreshed, a different account can recreate the same name and
+ * the old owner's credentials would still grant that ARN. Both createBucket
+ * and deleteBucket take this lock so the two never interleave for a name.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ *
+ * @return the held lock, which the caller must {@link GlobalLock#unlock()}
+ * and {@link GlobalLock#releaseRef()}, or {@code null} on timeout
+ */
+ protected GlobalLock acquireBucketNameLock(long storeId, String bucketName) {
+ GlobalLock lock = GlobalLock.getInternLock(getBucketNameLockName(storeId, bucketName));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire bucket name lock for store {} bucket {}", storeId, bucketName);
+ lock.releaseRef();
+ return null;
+ }
+ return lock;
+ }
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account and
+ * store. The store ID is included so that two CloudStack pools pointing
+ * at the same SeaweedFS IAM service do not collide on the same
+ * {@code acs-<uuid>} user and overwrite each other's policy and access
+ * keys.
+ */
+ protected String getUserNameForAccount(Account account, long storeId) {
+ return String.format("%s-%d-%s", ACS_PREFIX, storeId, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Serialize per store+account across management servers so two
+ // concurrent bucket requests do not both rotate credentials and leave
+ // bucket rows with mismatched key pairs.
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ return false;
+ }
+ try {
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach a scoped IAM policy that allows access only to this
+ // account's own buckets (the tenant boundary). Refreshed whenever
+ // buckets are created or deleted. Use the lock-free variant since
+ // createUser already holds the IAM lock.
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, null);
+
+ // Reuse the stored access key only if both the access key id and the
+ // secret key are present and the key is still Active in IAM; otherwise
+ // create a replacement.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ String storedSecretKey = details.get(secretKeyDetailKey);
+ if (storedAccessKeyId != null && storedSecretKey != null
+ && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ updateAccountBucketCredentials(storeId, accountId, storedAccessKeyId, storedSecretKey);
+ return true;
+ }
+
+ // The stored key is missing, inactive, or no longer in IAM. Clean up
+ // ALL keys (including the inactive stored one) before creating a
+ // replacement so we do not accumulate keys and hit IAM limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, null);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Persist the credential pair in account details (namespaced by storeId)
+ // before updating BucketVO rows. The reuse path above reconciles bucket
+ // rows every time, so a later bucket update failure is repairable on
+ // retry. AccountDetailsDao.persist(accountId, map) is deliberately not
+ // used: it expunges every detail for the account and can clobber another
+ // store's namespaced credentials.
+ try {
+ persistAccountCredentialsOrRollback(accountId, accessKeyDetailKey, secretKeyDetailKey,
+ storedAccessKeyId, storedSecretKey, key);
+ } catch (RuntimeException e) {
+ deleteAccessKeyAfterCredentialPersistenceFailure(iamClient, userName, key.getAccessKeyId(), e);
+ throw e;
+ }
+
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Persist a SeaweedFS IAM credential pair without using
+ * AccountDetailsDao.persist(accountId, map), and restore the previous pair if
+ * either single-key write fails so callers never observe a half-new pair.
+ */
+ private void persistAccountCredentialsOrRollback(long accountId, String accessKeyDetailKey, String secretKeyDetailKey,
+ String previousAccessKey, String previousSecretKey, AccessKey key) {
+ try {
+ _accountDetailsDao.addDetail(accountId, accessKeyDetailKey, key.getAccessKeyId(), false);
+ _accountDetailsDao.addDetail(accountId, secretKeyDetailKey, key.getSecretAccessKey(), false);
+ } catch (RuntimeException e) {
+ CloudRuntimeException wrapped = new CloudRuntimeException("Failed to persist SeaweedFS IAM credential pair for account " + accountId, e);
+ restoreAccountCredentialDetail(accountId, accessKeyDetailKey, previousAccessKey, wrapped);
+ restoreAccountCredentialDetail(accountId, secretKeyDetailKey, previousSecretKey, wrapped);
+ throw wrapped;
+ }
+ }
+
+ private void restoreAccountCredentialDetail(long accountId, String detailKey, String previousValue, RuntimeException cause) {
+ try {
+ if (previousValue == null) {
+ _accountDetailsDao.removeDetail(accountId, detailKey);
+ } else {
+ _accountDetailsDao.addDetail(accountId, detailKey, previousValue, false);
+ }
+ } catch (RuntimeException rollbackEx) {
+ logger.error("Failed to restore account detail {} for account {} after SeaweedFS credential persistence failed",
+ detailKey, accountId, rollbackEx);
+ cause.addSuppressed(rollbackEx);
+ }
+ }
+
+ private void deleteAccessKeyAfterCredentialPersistenceFailure(AmazonIdentityManagement iamClient, String userName,
+ String accessKeyId, RuntimeException cause) {
+ try {
+ iamClient.deleteAccessKey(new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(accessKeyId));
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Failed to delete IAM access key {} for user {} after account credential persistence failed",
+ accessKeyId, userName, cleanupEx);
+ cause.addSuppressed(cleanupEx);
+ }
+ }
+
+ private void updateAccountBucketCredentials(long storeId, long accountId, AccessKey iamCredential) {
+ updateAccountBucketCredentials(storeId, accountId, iamCredential.getAccessKeyId(), iamCredential.getSecretAccessKey());
+ }
+
+ /**
+ * Update the IAM credentials on all BucketVO rows for this store/account so
+ * previously created buckets reflect the current key pair.
+ */
+ private void updateAccountBucketCredentials(long storeId, long accountId, String accessKeyId, String secretAccessKey) {
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ for (BucketVO bucketVO : bucketList) {
+ if (accessKeyId.equals(bucketVO.getAccessKey()) && secretAccessKey.equals(bucketVO.getSecretKey())) {
+ continue;
+ }
+ logger.info("Updating accountId={} bucket {} with new IAM credentials", accountId, bucketVO.getName());
+ bucketVO.setAccessKey(accessKeyId);
+ bucketVO.setSecretKey(secretAccessKey);
+ if (!_bucketDao.update(bucketVO.getId(), bucketVO)) {
+ throw new CloudRuntimeException("Failed to update IAM credentials on bucket " + bucketVO.getName());
+ }
+ }
+ }
+
+ /**
+ * Refresh the per-account IAM user policy so it grants S3 access only to
+ * the account's current buckets (optionally excluding one, e.g. a bucket
+ * being deleted). This is the tenant boundary: each account's IAM
+ * credentials can only operate on that account's own buckets.
+ *
+ * Acquires the per-store/account IAM lock. Callers that already hold the
+ * lock (e.g. createUser, createBucket post-create) should call
+ * {@link #updateAccountIAMPolicyLocked} instead to avoid re-entrant lock
+ * acquisition warnings from GlobalLock.
+ *
+ * @param iamClient the IAM client
+ * @param storeId the object store
+ * @param accountId the CloudStack account
+ * @param excludeBucket a bucket name to omit (e.g. a bucket being deleted),
+ * or null to include all of the account's buckets
+ */
+ protected void updateAccountIAMPolicy(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, excludeBucket);
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Lock-free variant of {@link #updateAccountIAMPolicy} for callers that
+ * already hold the per-store/account IAM lock. Performs the policy refresh
+ * without reacquiring the lock, avoiding the GlobalLock re-entrant
+ * acquisition warning.
+ */
+ protected void updateAccountIAMPolicyLocked(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ return;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ List<BucketVO> buckets = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ List<String> bucketNames = new ArrayList<>();
+ for (BucketVO bvo : buckets) {
+ if (excludeBucket != null && excludeBucket.equals(bvo.getName())) {
+ continue;
+ }
+ // Skip buckets whose remote counterpart has been deleted but whose
+ // row is still present for resource accounting. Including them
+ // would re-grant a bucket name that is now free for another account
+ // to claim.
+ if (Bucket.State.Destroyed.equals(bvo.getState())) {
+ continue;
+ }
+ bucketNames.add(bvo.getName());
+ }
+ String policy = SeaweedFSObjectStoreUtil.buildAccountIAMPolicy(bucketNames);
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ SeaweedFSObjectStoreUtil.IAM_USER_POLICY_NAME, policy));
+ }
+
+ /**
+ * Check whether the given access key id is still listed and Active in IAM
+ * for the user. Listing failures are propagated rather than swallowed so
+ * a transient IAM outage does not send createUser into the replacement
+ * path (which would overwrite stored credentials and invalidate bucket
+ * records).
+ */
+ private boolean iamAccessKeyExists(AmazonIdentityManagement iamClient, String userName, String accessKeyId) {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ if (accessKeyId.equals(metadata.getAccessKeyId())) {
+ return "Active".equalsIgnoreCase(metadata.getStatus());
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Delete access keys for the user other than the (optionally) preserved
+ * key id. Used to clean up unmanaged leftover keys before creating a
+ * replacement so repeated calls do not hit IAM access-key limits.
+ */
+ private void deleteUnmanagedAccessKeys(AmazonIdentityManagement iamClient, String userName, String preserveAccessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ String keyId = metadata.getAccessKeyId();
+ if (preserveAccessKeyId != null && preserveAccessKeyId.equals(keyId)) {
+ continue;
+ }
+ DeleteAccessKeyRequest deleteReq =
+ new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(keyId);
+ logger.info("Deleting un-managed IAM access key {} for user {}", keyId, userName);
+ iamClient.deleteAccessKey(deleteReq);
+ }
+ } catch (AmazonClientException e) {
+ // Propagate so the caller does not proceed to create a replacement
+ // key while stale unmanaged keys remain (which could hit IAM key
+ // limits or leave orphaned credentials).
+ throw new CloudRuntimeException("Failed to clean up IAM access keys for user " + userName, e);
+ }
+ }
+
+ @Override
+ public Bucket createBucket(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+
+ // Serialize against a concurrent deleteBucket of the same name by any
+ // account on this store. Bucket names are globally unique per store and
+ // reusable, so without this an account could claim a name while the
+ // previous owner's IAM policy still granted that ARN.
+ GlobalLock nameLock = acquireBucketNameLock(storeId, bucketName);
+ if (nameLock == null) {
+ throw new CloudRuntimeException("Failed to acquire bucket name lock for store " + storeId + " bucket " + bucketName);
+ }
+ try {
+ return createBucketLocked(bucket, objectLock);
+ } finally {
+ nameLock.unlock();
+ nameLock.releaseRef();
+ }
+ }
+
+ private Bucket createBucketLocked(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+ long accountId = bucket.getAccountId();
+
+ // Use the store's admin credentials to create the bucket
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Check if the bucket already exists
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket already exists with name " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Create the bucket
+ try {
+ CreateBucketRequest request = new CreateBucketRequest(bucketName);
+ if (objectLock) {
+ request.setObjectLockEnabledForBucket(true);
+ }
+ s3client.createBucket(request);
+ } catch (AmazonClientException e) {
+ logger.error("Create bucket failed", e);
+ throw new CloudRuntimeException(e);
+ }
+
+ // Step 2: update the bucket record with the account's IAM credentials,
+ // configure CORS, and refresh the IAM policy. If any of these fail,
+ // clean up the remote bucket so a retry does not find it already
+ // existing — mirroring the Cloudian createBucket pattern.
+ //
+ // Hold the IAM lock for the account so a concurrent createUser key
+ // rotation does not change the account credentials between reading
+ // them and writing the BucketVO, which would leave the bucket with
+ // a stale key pair.
+ GlobalLock iamLock = acquireIamLock(storeId, accountId);
+ if (iamLock == null) {
+ // The S3 bucket has already been created. Clean it up so a
+ // retry does not find it already existing, then throw.
+ try {
+ s3client.deleteBucket(bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Failed to clean up bucket {} after IAM lock timeout", bucketName, cleanupEx);
+ }
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ // Configure permissive CORS so the CloudStack S3 bucket browser
+ // (which performs list/upload/delete from the browser) can function.
+ // SeaweedFS supports the standard PutBucketCors operation.
+ configureBucketCORS(s3client, bucketName);
+
+ Map<String, String> accountDetails = _accountDetailsDao.findDetails(accountId);
+ String accessKey = accountDetails.get(SeaweedFSObjectStoreUtil.keyAccessKey(storeId));
+ String secretKey = accountDetails.get(SeaweedFSObjectStoreUtil.keySecretKey(storeId));
+ if (accessKey == null || secretKey == null) {
+ throw new CloudRuntimeException("No IAM credentials found for account " + accountId
+ + " on store " + storeId + ". Run createUser before creating a bucket.");
+ }
+
+ String s3Url = getS3Url(storeId);
+ BucketVO bucketVO = _bucketDao.findById(bucket.getId());
+ bucketVO.setAccessKey(accessKey);
+ bucketVO.setSecretKey(secretKey);
+ // Normalize the endpoint: s3Url is operator-supplied and may carry a
+ // trailing slash, which would persist a broken "...//bucket" URL
+ // that BucketResponse and the object store browser both use.
+ bucketVO.setBucketURL(SeaweedFSObjectStoreUtil.stripTrailingSlashes(s3Url) + "/" + bucketName);
+ _bucketDao.update(bucket.getId(), bucketVO);
Review Comment:
Fixed in c8a267c. Both paths now check the _bucketDao.update result: createBucket fails into the existing cleanup path (policy revoke before remote bucket delete), and markBucketDestroyed propagates the failure before the IAM refresh.
##########
server/src/main/java/com/cloud/storage/StorageManagerImpl.java:
##########
@@ -4812,6 +4813,57 @@ public ObjectStore updateObjectStore(Long id, UpdateObjectStoragePoolCmd cmd) {
return objectStoreVO;
}
+ /**
+ * Object store detail key holding an explicit S3 endpoint override. Providers
+ * that support it (SeaweedFS, Cloudian HyperStore) prefer it over the generic
+ * ObjectStoreVO.url.
+ */
+ private static final String OBJECT_STORE_DETAIL_S3_URL = "s3Url";
+
+ /**
+ * Rewrite the stored bucketURL of every bucket on an object store after the
+ * generic store URL changes.
+ *
+ * BucketVO.bucketURL is written only at bucket creation. BucketResponse
+ * exposes it and the object-store browser builds its S3 client from it, so
+ * without this the browser keeps targeting the old endpoint after a URL
+ * change while the management server uses the new one.
+ *
+ * Skipped when the pool has an explicit s3Url detail: providers that support
+ * that override (SeaweedFS, Cloudian HyperStore) keep using it regardless of
+ * ObjectStoreVO.url, so rewriting the bucket URLs off the generic URL would
+ * point the browser at an endpoint the driver never uses.
+ */
+ private void updateBucketUrls(Long storeId, String oldUrl, String newUrl) {
+ if (oldUrl == null || newUrl == null || oldUrl.equals(newUrl)) {
+ return;
+ }
+ Map<String, String> storeDetails = _objectStoreDetailsDao.getDetails(storeId);
+ if (storeDetails != null && StringUtils.isNotBlank(storeDetails.get(OBJECT_STORE_DETAIL_S3_URL))) {
+ logger.debug("Object store {} has an explicit s3Url; leaving stored bucket URLs unchanged", storeId);
+ return;
+ }
+ // Both URLs are operator-supplied and may carry a trailing slash. Strip
+ // them so the retained suffix (which starts with '/') is not appended to
+ // a base that already ends in one, which would rewrite every bucket URL
+ // to '...//bucket' and break browser access.
+ String oldBase = StringUtils.stripEnd(oldUrl, "/");
+ String newBase = StringUtils.stripEnd(newUrl, "/");
+ if (oldBase.equals(newBase)) {
+ return;
+ }
+ for (BucketVO bucket : _bucketDao.listByObjectStoreId(storeId)) {
+ String bucketUrl = bucket.getBucketURL();
+ if (bucketUrl == null || !bucketUrl.startsWith(oldBase)) {
+ continue;
+ }
+ String suffix = bucketUrl.substring(oldBase.length());
+ bucket.setBucketURL(newBase + (suffix.startsWith("/") ? suffix : "/" + suffix));
+ _bucketDao.update(bucket.getId(), bucket);
Review Comment:
Fixed in c8a267c. updateBucketUrls now checks each _bucketDao.update result; on failure it restores the rows already rewritten and throws, and updateObjectStore reverts the object-store URL so the stored endpoint stays consistent with the bucket URLs.
##########
plugins/pom.xml:
##########
@@ -142,6 +142,7 @@
<module>storage/object/minio</module>
<module>storage/object/ceph</module>
<module>storage/object/cloudian</module>
+ <module>storage/object/seaweedfs</module>
Review Comment:
Fixed in 73bae93 — client/pom.xml now includes the cloud-plugin-storage-object-seaweedfs dependency alongside the other object-storage providers, so the module is packaged into the management-server artifact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r4283950572
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,1125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.BucketCrossOriginConfiguration;
+import com.amazonaws.services.s3.model.CORSRule;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketCrossOriginConfigurationRequest;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ /**
+ * DB-backed global lock name prefix for serializing IAM provisioning and
+ * policy refreshes per store+account. Uses {@link GlobalLock} so the
+ * critical section is serialized across management servers in a
+ * clustered deployment, not just within a single JVM.
+ */
+ private static final String IAM_LOCK_PREFIX = "seaweedfs.iam.";
+ private static final String BUCKET_NAME_LOCK_PREFIX = "seaweedfs.bucket.";
+
+ private static String getIamLockName(long storeId, long accountId) {
+ return IAM_LOCK_PREFIX + storeId + "." + accountId;
+ }
+
+ /**
+ * Acquire a DB-backed global lock for IAM operations on the given
+ * store+account. Returns a {@link GlobalLock} that the caller must
+ * {@link GlobalLock#unlock()} in a {@code finally} block, or {@code null}
+ * if the lock could not be acquired within the timeout.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ */
+ protected GlobalLock acquireIamLock(long storeId, long accountId) {
+ GlobalLock lock = GlobalLock.getInternLock(getIamLockName(storeId, accountId));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire IAM lock for store {} account {}", storeId, accountId);
+ lock.releaseRef();
+ return null;
+ }
+ return lock;
+ }
+
+ private static String getBucketNameLockName(long storeId, String bucketName) {
+ return BUCKET_NAME_LOCK_PREFIX + storeId + "." + bucketName;
+ }
+
+ /**
+ * Acquire a DB-backed global lock covering a bucket <em>name</em> on a store,
+ * independent of the owning account.
+ *
+ * S3 bucket names are globally unique within a store and are reusable after
+ * deletion, while the IAM lock is scoped to (store, account). Without this
+ * lock, once a delete removes the remote bucket but before the old owner's
+ * IAM policy is refreshed, a different account can recreate the same name and
+ * the old owner's credentials would still grant that ARN. Both createBucket
+ * and deleteBucket take this lock so the two never interleave for a name.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ *
+ * @return the held lock, which the caller must {@link GlobalLock#unlock()}
+ * and {@link GlobalLock#releaseRef()}, or {@code null} on timeout
+ */
+ protected GlobalLock acquireBucketNameLock(long storeId, String bucketName) {
+ GlobalLock lock = GlobalLock.getInternLock(getBucketNameLockName(storeId, bucketName));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire bucket name lock for store {} bucket {}", storeId, bucketName);
+ lock.releaseRef();
+ return null;
+ }
+ return lock;
+ }
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account and
+ * store. The store ID is included so that two CloudStack pools pointing
+ * at the same SeaweedFS IAM service do not collide on the same
+ * {@code acs-<uuid>} user and overwrite each other's policy and access
+ * keys.
+ */
+ protected String getUserNameForAccount(Account account, long storeId) {
+ return String.format("%s-%d-%s", ACS_PREFIX, storeId, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Serialize per store+account across management servers so two
+ // concurrent bucket requests do not both rotate credentials and leave
+ // bucket rows with mismatched key pairs.
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ return false;
+ }
+ try {
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach a scoped IAM policy that allows access only to this
+ // account's own buckets (the tenant boundary). Refreshed whenever
+ // buckets are created or deleted. Use the lock-free variant since
+ // createUser already holds the IAM lock.
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, null);
+
+ // Reuse the stored access key only if both the access key id and the
+ // secret key are present and the key is still Active in IAM; otherwise
+ // create a replacement.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ String storedSecretKey = details.get(secretKeyDetailKey);
+ if (storedAccessKeyId != null && storedSecretKey != null
+ && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ updateAccountBucketCredentials(storeId, accountId, storedAccessKeyId, storedSecretKey);
+ return true;
+ }
+
+ // The stored key is missing, inactive, or no longer in IAM. Clean up
+ // ALL keys (including the inactive stored one) before creating a
+ // replacement so we do not accumulate keys and hit IAM limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, null);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Persist the credential pair in account details (namespaced by storeId)
+ // before updating BucketVO rows. The reuse path above reconciles bucket
+ // rows every time, so a later bucket update failure is repairable on
+ // retry. AccountDetailsDao.persist(accountId, map) is deliberately not
+ // used: it expunges every detail for the account and can clobber another
+ // store's namespaced credentials.
+ try {
+ persistAccountCredentialsOrRollback(accountId, accessKeyDetailKey, secretKeyDetailKey,
+ storedAccessKeyId, storedSecretKey, key);
+ } catch (RuntimeException e) {
+ deleteAccessKeyAfterCredentialPersistenceFailure(iamClient, userName, key.getAccessKeyId(), e);
+ throw e;
+ }
+
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Persist a SeaweedFS IAM credential pair without using
+ * AccountDetailsDao.persist(accountId, map), and restore the previous pair if
+ * either single-key write fails so callers never observe a half-new pair.
+ */
+ private void persistAccountCredentialsOrRollback(long accountId, String accessKeyDetailKey, String secretKeyDetailKey,
+ String previousAccessKey, String previousSecretKey, AccessKey key) {
+ try {
+ _accountDetailsDao.addDetail(accountId, accessKeyDetailKey, key.getAccessKeyId(), false);
+ _accountDetailsDao.addDetail(accountId, secretKeyDetailKey, key.getSecretAccessKey(), false);
+ } catch (RuntimeException e) {
+ CloudRuntimeException wrapped = new CloudRuntimeException("Failed to persist SeaweedFS IAM credential pair for account " + accountId, e);
+ restoreAccountCredentialDetail(accountId, accessKeyDetailKey, previousAccessKey, wrapped);
+ restoreAccountCredentialDetail(accountId, secretKeyDetailKey, previousSecretKey, wrapped);
+ throw wrapped;
+ }
+ }
+
+ private void restoreAccountCredentialDetail(long accountId, String detailKey, String previousValue, RuntimeException cause) {
+ try {
+ if (previousValue == null) {
+ _accountDetailsDao.removeDetail(accountId, detailKey);
+ } else {
+ _accountDetailsDao.addDetail(accountId, detailKey, previousValue, false);
+ }
+ } catch (RuntimeException rollbackEx) {
+ logger.error("Failed to restore account detail {} for account {} after SeaweedFS credential persistence failed",
+ detailKey, accountId, rollbackEx);
+ cause.addSuppressed(rollbackEx);
+ }
+ }
+
+ private void deleteAccessKeyAfterCredentialPersistenceFailure(AmazonIdentityManagement iamClient, String userName,
+ String accessKeyId, RuntimeException cause) {
+ try {
+ iamClient.deleteAccessKey(new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(accessKeyId));
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Failed to delete IAM access key {} for user {} after account credential persistence failed",
+ accessKeyId, userName, cleanupEx);
+ cause.addSuppressed(cleanupEx);
+ }
+ }
+
+ private void updateAccountBucketCredentials(long storeId, long accountId, AccessKey iamCredential) {
+ updateAccountBucketCredentials(storeId, accountId, iamCredential.getAccessKeyId(), iamCredential.getSecretAccessKey());
+ }
+
+ /**
+ * Update the IAM credentials on all BucketVO rows for this store/account so
+ * previously created buckets reflect the current key pair.
+ */
+ private void updateAccountBucketCredentials(long storeId, long accountId, String accessKeyId, String secretAccessKey) {
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ for (BucketVO bucketVO : bucketList) {
+ if (accessKeyId.equals(bucketVO.getAccessKey()) && secretAccessKey.equals(bucketVO.getSecretKey())) {
+ continue;
+ }
+ logger.info("Updating accountId={} bucket {} with new IAM credentials", accountId, bucketVO.getName());
+ bucketVO.setAccessKey(accessKeyId);
+ bucketVO.setSecretKey(secretAccessKey);
+ if (!_bucketDao.update(bucketVO.getId(), bucketVO)) {
+ throw new CloudRuntimeException("Failed to update IAM credentials on bucket " + bucketVO.getName());
+ }
+ }
+ }
+
+ /**
+ * Refresh the per-account IAM user policy so it grants S3 access only to
+ * the account's current buckets (optionally excluding one, e.g. a bucket
+ * being deleted). This is the tenant boundary: each account's IAM
+ * credentials can only operate on that account's own buckets.
+ *
+ * Acquires the per-store/account IAM lock. Callers that already hold the
+ * lock (e.g. createUser, createBucket post-create) should call
+ * {@link #updateAccountIAMPolicyLocked} instead to avoid re-entrant lock
+ * acquisition warnings from GlobalLock.
+ *
+ * @param iamClient the IAM client
+ * @param storeId the object store
+ * @param accountId the CloudStack account
+ * @param excludeBucket a bucket name to omit (e.g. a bucket being deleted),
+ * or null to include all of the account's buckets
+ */
+ protected void updateAccountIAMPolicy(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, excludeBucket);
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Lock-free variant of {@link #updateAccountIAMPolicy} for callers that
+ * already hold the per-store/account IAM lock. Performs the policy refresh
+ * without reacquiring the lock, avoiding the GlobalLock re-entrant
+ * acquisition warning.
+ */
+ protected void updateAccountIAMPolicyLocked(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ return;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ List<BucketVO> buckets = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ List<String> bucketNames = new ArrayList<>();
+ for (BucketVO bvo : buckets) {
+ if (excludeBucket != null && excludeBucket.equals(bvo.getName())) {
+ continue;
+ }
+ // Skip buckets whose remote counterpart has been deleted but whose
+ // row is still present for resource accounting. Including them
+ // would re-grant a bucket name that is now free for another account
+ // to claim.
+ if (Bucket.State.Destroyed.equals(bvo.getState())) {
+ continue;
+ }
+ bucketNames.add(bvo.getName());
+ }
+ String policy = SeaweedFSObjectStoreUtil.buildAccountIAMPolicy(bucketNames);
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ SeaweedFSObjectStoreUtil.IAM_USER_POLICY_NAME, policy));
+ }
+
+ /**
+ * Check whether the given access key id is still listed and Active in IAM
+ * for the user. Listing failures are propagated rather than swallowed so
+ * a transient IAM outage does not send createUser into the replacement
+ * path (which would overwrite stored credentials and invalidate bucket
+ * records).
+ */
+ private boolean iamAccessKeyExists(AmazonIdentityManagement iamClient, String userName, String accessKeyId) {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ if (accessKeyId.equals(metadata.getAccessKeyId())) {
+ return "Active".equalsIgnoreCase(metadata.getStatus());
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Delete access keys for the user other than the (optionally) preserved
+ * key id. Used to clean up unmanaged leftover keys before creating a
+ * replacement so repeated calls do not hit IAM access-key limits.
+ */
+ private void deleteUnmanagedAccessKeys(AmazonIdentityManagement iamClient, String userName, String preserveAccessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ String keyId = metadata.getAccessKeyId();
+ if (preserveAccessKeyId != null && preserveAccessKeyId.equals(keyId)) {
+ continue;
+ }
+ DeleteAccessKeyRequest deleteReq =
+ new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(keyId);
+ logger.info("Deleting un-managed IAM access key {} for user {}", keyId, userName);
+ iamClient.deleteAccessKey(deleteReq);
+ }
+ } catch (AmazonClientException e) {
+ // Propagate so the caller does not proceed to create a replacement
+ // key while stale unmanaged keys remain (which could hit IAM key
+ // limits or leave orphaned credentials).
+ throw new CloudRuntimeException("Failed to clean up IAM access keys for user " + userName, e);
+ }
+ }
+
+ @Override
+ public Bucket createBucket(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+
+ // Serialize against a concurrent deleteBucket of the same name by any
+ // account on this store. Bucket names are globally unique per store and
+ // reusable, so without this an account could claim a name while the
+ // previous owner's IAM policy still granted that ARN.
+ GlobalLock nameLock = acquireBucketNameLock(storeId, bucketName);
+ if (nameLock == null) {
+ throw new CloudRuntimeException("Failed to acquire bucket name lock for store " + storeId + " bucket " + bucketName);
+ }
+ try {
+ return createBucketLocked(bucket, objectLock);
+ } finally {
+ nameLock.unlock();
+ nameLock.releaseRef();
+ }
+ }
+
+ private Bucket createBucketLocked(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+ long accountId = bucket.getAccountId();
+
+ // Use the store's admin credentials to create the bucket
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Check if the bucket already exists
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket already exists with name " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Create the bucket
+ try {
+ CreateBucketRequest request = new CreateBucketRequest(bucketName);
+ if (objectLock) {
+ request.setObjectLockEnabledForBucket(true);
+ }
+ s3client.createBucket(request);
+ } catch (AmazonClientException e) {
+ logger.error("Create bucket failed", e);
+ throw new CloudRuntimeException(e);
+ }
+
+ // Step 2: update the bucket record with the account's IAM credentials,
+ // configure CORS, and refresh the IAM policy. If any of these fail,
+ // clean up the remote bucket so a retry does not find it already
+ // existing — mirroring the Cloudian createBucket pattern.
+ //
+ // Hold the IAM lock for the account so a concurrent createUser key
+ // rotation does not change the account credentials between reading
+ // them and writing the BucketVO, which would leave the bucket with
+ // a stale key pair.
+ GlobalLock iamLock = acquireIamLock(storeId, accountId);
+ if (iamLock == null) {
+ // The S3 bucket has already been created. Clean it up so a
+ // retry does not find it already existing, then throw.
+ try {
+ s3client.deleteBucket(bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Failed to clean up bucket {} after IAM lock timeout", bucketName, cleanupEx);
+ }
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ // Configure permissive CORS so the CloudStack S3 bucket browser
+ // (which performs list/upload/delete from the browser) can function.
+ // SeaweedFS supports the standard PutBucketCors operation.
+ configureBucketCORS(s3client, bucketName);
+
+ Map<String, String> accountDetails = _accountDetailsDao.findDetails(accountId);
+ String accessKey = accountDetails.get(SeaweedFSObjectStoreUtil.keyAccessKey(storeId));
+ String secretKey = accountDetails.get(SeaweedFSObjectStoreUtil.keySecretKey(storeId));
+ if (accessKey == null || secretKey == null) {
+ throw new CloudRuntimeException("No IAM credentials found for account " + accountId
+ + " on store " + storeId + ". Run createUser before creating a bucket.");
+ }
+
+ String s3Url = getS3Url(storeId);
+ BucketVO bucketVO = _bucketDao.findById(bucket.getId());
+ if (bucketVO == null) {
+ throw new CloudRuntimeException("Bucket record " + bucket.getId()
+ + " not found for bucket " + bucketName);
+ }
+ bucketVO.setAccessKey(accessKey);
+ bucketVO.setSecretKey(secretKey);
+ // Normalize the endpoint: s3Url is operator-supplied and may carry a
+ // trailing slash, which would persist a broken "...//bucket" URL
+ // that BucketResponse and the object store browser both use.
+ bucketVO.setBucketURL(SeaweedFSObjectStoreUtil.stripTrailingSlashes(s3Url) + "/" + bucketName);
+ if (!_bucketDao.update(bucket.getId(), bucketVO)) {
+ // A failed row update must fail the create: otherwise the IAM
+ // refresh below would grant the new bucket ARN while the
+ // returned BucketVO carries credentials/URL that were never
+ // persisted. The catch block revokes the policy grant and
+ // removes the remote bucket.
+ throw new CloudRuntimeException("Failed to persist credentials and URL for bucket " + bucketName);
+ }
+
+ // Refresh the account's IAM policy to include the new bucket.
+ // Use the lock-free variant since createBucket already holds the
+ // IAM lock for the post-create section.
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, null);
+
+ // Return the updated BucketVO (not the stale input bucket) so
+ // BucketApiServiceImpl.createBucket does not overwrite the
+ // persisted credentials with the stale values.
+ return bucketVO;
+ } catch (Exception e) {
+ logger.error("Post-create bucket record update failed for {}; cleaning up", bucketName, e);
+ CloudRuntimeException primary = new CloudRuntimeException(e);
+ // Revoke the IAM policy grant for the new bucket BEFORE deleting
+ // the remote S3 bucket. If the policy PUT succeeded before the DB
+ // update failed, the grant would otherwise persist and could be
+ // reused if another account later creates the same bucket name.
+ // Use the lock-free variant since createBucket already holds the
+ // IAM lock.
+ //
+ // If the policy revocation fails, the remote S3 bucket is LEFT IN
+ // PLACE: deleting it would free the bucket name for another account
+ // to claim while this account's policy still grants the ARN,
+ // letting the old credentials reach the new tenant's bucket. The
+ // leftover remote bucket keeps the name occupied until an operator
+ // reconciles; the row is removed by the caller so a retry reports
+ // "bucket already exists" rather than silently re-granting access.
+ try {
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, bucketName);
+ } catch (Exception policyEx) {
+ logger.error("Failed to revoke IAM policy for bucket {} after create failure; "
+ + "leaving the remote bucket in place to prevent name reuse", bucketName, policyEx);
+ primary.addSuppressed(policyEx);
+ throw primary;
+ }
+ // Policy revocation succeeded: the stale grant is gone, so it is
+ // safe to delete the remote bucket and free the name.
+ try {
+ s3client.deleteBucket(bucketName);
+ logger.info("Cleanup of bucket {} succeeded", bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Cleanup of bucket {} also failed", bucketName, cleanupEx);
+ primary.addSuppressed(cleanupEx);
+ }
+ throw primary;
+ } finally {
+ iamLock.unlock();
+ iamLock.releaseRef();
+ }
+ }
+
+ /**
+ * Configure a permissive CORS policy on the bucket so the CloudStack
+ * S3 bucket browser (which performs list/upload/delete from the
+ * browser) can function. Mirrors the Cloudian configureBucketCORS.
+ */
+ private void configureBucketCORS(AmazonS3 s3client, String bucketName) {
+ logger.debug("Configuring CORS for bucket {}", bucketName);
+ List<CORSRule> corsRules = new ArrayList<>();
+ CORSRule allowAnyRule = new CORSRule().withId("AllowAny");
+ allowAnyRule.setAllowedOrigins("*");
+ allowAnyRule.setAllowedHeaders("*");
+ allowAnyRule.setAllowedMethods(
+ CORSRule.AllowedMethods.HEAD,
+ CORSRule.AllowedMethods.GET,
+ CORSRule.AllowedMethods.PUT,
+ CORSRule.AllowedMethods.POST,
+ CORSRule.AllowedMethods.DELETE);
Review Comment:
CORSRule#setAllowedOrigins`, `setAllowedHeaders`, and `setAllowedMethods` in the AWS SDK v1 APIs are list-based setters (and typically paired with `withAllowedOrigins(...)`, `withAllowedHeaders(...)`, `withAllowedMethods(...)`). As written, these calls are very likely to be a compile-time error (wrong argument types). Update this to use the SDK’s `withAllowedOrigins("*")`, `withAllowedHeaders("*")`, and `withAllowedMethods(...)` (or set list values explicitly) so the code compiles and the CORS config is applied correctly.
##########
server/src/main/java/org/apache/cloudstack/storage/object/BucketApiServiceImpl.java:
##########
@@ -290,61 +341,238 @@ public boolean updateBucket(UpdateBucketCmd cmd, Account caller) throws Resource
objectStore.deleteBucketVersioning(bucketTO);
}
bucket.setVersioning(cmd.getVersioning());
+ versioningApplied = true;
}
if (cmd.getPolicy() != null) {
objectStore.setBucketPolicy(bucketTO, cmd.getPolicy());
bucket.setPolicy(cmd.getPolicy());
+ policyApplied = true;
}
- updateBucketQuota(cmd, bucket, objectStore, objectStoreVO, bucketTO);
+ boolean bucketPersisted = updateBucketQuota(cmd, bucket, objectStore, objectStoreVO, bucketTO);
- _bucketDao.update(bucket.getId(), bucket);
+ if (!bucketPersisted && !_bucketDao.update(bucket.getId(), bucket)) {
+ throw new CloudRuntimeException("Failed to update bucket " + bucket.getName());
+ }
} catch (Exception e) {
- throw new CloudRuntimeException("Error while updating bucket: " +bucket.getName() +". "+e.getMessage());
+ // Roll back the remote encryption/versioning/policy mutations that
+ // were applied before the failure (e.g. a quota reservation
+ // rejection) so the remote bucket and the BucketVO stay consistent.
+ // The quota path unwinds its own mutations internally.
+ rollbackBucketUpdateMutations(objectStore, bucketTO, bucket,
+ previousEncryption, encryptionApplied,
+ previousVersioning, versioningApplied,
+ previousPolicy, policyApplied, e);
+ if (e instanceof CloudRuntimeException) {
+ throw (CloudRuntimeException) e;
+ }
+ throw new CloudRuntimeException("Error while updating bucket: " + bucket.getName() + ". " + e.getMessage(), e);
}
return true;
}
- private void updateBucketQuota(UpdateBucketCmd cmd, BucketVO bucket, ObjectStoreEntity objectStore, ObjectStoreVO objectStoreVO, BucketTO bucketTO) throws ResourceAllocationException {
+ /**
+ * Best-effort compensation for the remote encryption, versioning, and
+ * policy mutations applied during {@link #updateBucket} when a later step
+ * fails. Each remote mutation is reverted to its pre-update value in
+ * reverse order; the in-memory {@link BucketVO} fields are restored too so
+ * a retry starts from the same state the backend is in. Failures are
+ * logged and attached to the original exception rather than masking it.
+ */
+ private void rollbackBucketUpdateMutations(ObjectStoreEntity objectStore, BucketTO bucketTO, BucketVO bucket,
+ Boolean previousEncryption, boolean encryptionApplied,
+ Boolean previousVersioning, boolean versioningApplied,
+ String previousPolicy, boolean policyApplied, Exception cause) {
+ if (policyApplied) {
+ try {
+ if (previousPolicy == null || "private".equalsIgnoreCase(previousPolicy)) {
+ objectStore.setBucketPolicy(bucketTO, "private");
+ } else {
+ objectStore.setBucketPolicy(bucketTO, previousPolicy);
+ }
+ bucket.setPolicy(previousPolicy);
+ } catch (Exception ex) {
+ logger.error("Failed to roll back bucket policy for {} while compensating a failed update",
+ bucket.getName(), ex);
+ cause.addSuppressed(ex);
+ }
+ }
+ if (versioningApplied) {
+ try {
+ if (Boolean.TRUE.equals(previousVersioning)) {
+ objectStore.setBucketVersioning(bucketTO);
+ } else {
+ objectStore.deleteBucketVersioning(bucketTO);
+ }
+ bucket.setVersioning(previousVersioning);
+ } catch (Exception ex) {
+ logger.error("Failed to roll back bucket versioning for {} while compensating a failed update",
+ bucket.getName(), ex);
+ cause.addSuppressed(ex);
+ }
+ }
+ if (encryptionApplied) {
+ try {
+ if (Boolean.TRUE.equals(previousEncryption)) {
+ objectStore.setBucketEncryption(bucketTO);
+ } else {
+ objectStore.deleteBucketEncryption(bucketTO);
+ }
+ bucket.setEncryption(previousEncryption);
+ } catch (Exception ex) {
+ logger.error("Failed to roll back bucket encryption for {} while compensating a failed update",
+ bucket.getName(), ex);
+ cause.addSuppressed(ex);
+ }
+ }
+ }
+
+ private boolean updateBucketQuota(UpdateBucketCmd cmd, BucketVO bucket, ObjectStoreEntity objectStore, ObjectStoreVO objectStoreVO, BucketTO bucketTO) throws ResourceAllocationException {
Integer quota = cmd.getQuota();
if (quota == null) {
- return;
+ return false;
}
- int quotaDelta = quota - bucket.getQuota();
- objectStore.setQuota(bucketTO, quota);
- bucket.setQuota(quota);
-
+ Integer previousQuota = bucket.getQuota();
+ int previousQuotaValue = ObjectUtils.defaultIfNull(previousQuota, 0);
+ long quotaDelta = (long) quota - previousQuotaValue;
long diff = quotaDelta * Resource.ResourceType.bytesToGiB;
- if (quotaDelta < 0) {
- resourceLimitManager.decrementResourceCount(bucket.getAccountId(), Resource.ResourceType.object_storage, Math.abs(diff));
- _objectStoreDao.updateAllocatedSize(objectStoreVO, diff);
- return;
+ if (diff <= 0) {
+ // A decrease (or no change) cannot exceed a limit, so no reservation
+ // is needed.
+ applyQuotaChange(bucket, objectStore, objectStoreVO, bucketTO, quota, previousQuota, previousQuotaValue, diff);
+ return true;
}
Review Comment:
diff <= 0` treats “no change” (`diff == 0`) the same as a decrease and still calls `applyQuotaChange`, which triggers a remote `setQuota(...)` call and a DAO update even though nothing changed. That increases failure surface area and can make an otherwise no-op update fail. Consider adding an early return when `quota == previousQuotaValue` (skip remote call and persistence; let the caller persist other bucket changes as usual).
##########
plugins/storage/object/seaweedfs/src/test/java/org/apache/cloudstack/storage/datastore/lifecycle/SeaweedFSObjectStoreLifeCycleImplTest.java:
##########
@@ -0,0 +1,306 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.lifecycle;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.ArgumentMatchers.anyMap;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.ObjectStoreEntity;
+import org.apache.cloudstack.storage.object.datastore.ObjectStoreHelper;
+import org.apache.cloudstack.storage.object.datastore.ObjectStoreProviderManager;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mock;
+import org.mockito.MockedStatic;
+import org.mockito.Mockito;
+import org.mockito.MockitoAnnotations;
+import org.mockito.Spy;
+import org.mockito.junit.MockitoJUnitRunner;
+
+import com.cloud.utils.exception.CloudRuntimeException;
+
+@RunWith(MockitoJUnitRunner.class)
+public class SeaweedFSObjectStoreLifeCycleImplTest {
+
+ @Spy
+ SeaweedFSObjectStoreLifeCycleImpl lifecycle = new SeaweedFSObjectStoreLifeCycleImpl();
+
+ @Mock
+ ObjectStoreHelper objectStoreHelper;
+ @Mock
+ ObjectStoreProviderManager objectStoreMgr;
+ @Mock
+ ObjectStoreVO objectStoreVo;
+ @Mock
+ ObjectStoreEntity objectStoreEntity;
+
+ static String TEST_STORE_NAME = "testStore";
+ static String TEST_URL = "http://s3-endpoint";
+ static String TEST_PROVIDER_NAME = "SeaweedFS";
+ static String TEST_ACCESS_KEY = "admin-access-key";
+ static String TEST_SECRET_KEY = "admin-secret-key";
+ static String TEST_S3_URL_OVERRIDE = "http://s3-override:8333";
+ static String TEST_IAM_URL_OVERRIDE = "http://iam-override:8111";
+ static String TEST_METRICS_URL = "http://seaweedfs-s3:9327";
+
+ Map<String, String> detailsMap;
+ Map<String, Object> dsInfos;
+
+ MockedStatic<SeaweedFSObjectStoreUtil> mockStatic;
+
+ private AutoCloseable closeable;
+
+ @Before
+ public void setUp() {
+ closeable = MockitoAnnotations.openMocks(this);
+
+ mockStatic = Mockito.mockStatic(SeaweedFSObjectStoreUtil.class);
Review Comment:
This test class uses `@RunWith(MockitoJUnitRunner.class)` and also calls `MockitoAnnotations.openMocks(this)` in `setUp()`. The runner already initializes `@Mock`/`@Spy` fields; double-initialization is redundant and can lead to confusing behavior. Prefer removing `openMocks` (and the associated `closeable`) when using the Mockito JUnit runner.
##########
plugins/storage/object/seaweedfs/src/test/java/org/apache/cloudstack/storage/datastore/lifecycle/SeaweedFSObjectStoreLifeCycleImplTest.java:
##########
@@ -0,0 +1,306 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.lifecycle;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.ArgumentMatchers.anyMap;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.ObjectStoreEntity;
+import org.apache.cloudstack.storage.object.datastore.ObjectStoreHelper;
+import org.apache.cloudstack.storage.object.datastore.ObjectStoreProviderManager;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mock;
+import org.mockito.MockedStatic;
+import org.mockito.Mockito;
+import org.mockito.MockitoAnnotations;
+import org.mockito.Spy;
+import org.mockito.junit.MockitoJUnitRunner;
+
+import com.cloud.utils.exception.CloudRuntimeException;
+
+@RunWith(MockitoJUnitRunner.class)
+public class SeaweedFSObjectStoreLifeCycleImplTest {
+
+ @Spy
+ SeaweedFSObjectStoreLifeCycleImpl lifecycle = new SeaweedFSObjectStoreLifeCycleImpl();
Review Comment:
This test class uses `@RunWith(MockitoJUnitRunner.class)` and also calls `MockitoAnnotations.openMocks(this)` in `setUp()`. The runner already initializes `@Mock`/`@Spy` fields; double-initialization is redundant and can lead to confusing behavior. Prefer removing `openMocks` (and the associated `closeable`) when using the Mockito JUnit runner.
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/util/SeaweedFSObjectStoreUtil.java:
##########
@@ -0,0 +1,715 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.util;
+
+import org.apache.commons.lang3.StringUtils;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagementClientBuilder;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3ClientBuilder;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * Utility class for the SeaweedFS object storage provider.
+ *
+ * SeaweedFS exposes both an S3-compatible API and an AWS IAM-compatible API,
+ * so this provider needs no proprietary admin client — only the AWS S3 and IAM
+ * SDKs, the same pair Cloudian HyperStore already uses in this tree.
+ */
+public class SeaweedFSObjectStoreUtil {
+
+ /** The name of our Object Store Provider */
+ public static final String OBJECT_STORE_PROVIDER_NAME = "SeaweedFS";
+
+ public static final String STORE_KEY_PROVIDER_NAME = "providerName";
+ public static final String STORE_KEY_URL = "url";
+ public static final String STORE_KEY_NAME = "name";
+ public static final String STORE_KEY_SIZE = "size";
+ public static final String STORE_KEY_DETAILS = "details";
+
+ // Store Details Map key names - managed outside of plugin
+ public static final String STORE_DETAILS_KEY_ACCESS_KEY = "accesskey"; // admin/root access key
+ public static final String STORE_DETAILS_KEY_SECRET_KEY = "secretkey"; // admin/root secret key
+ public static final String STORE_DETAILS_KEY_S3_URL = "s3Url"; // S3 endpoint URL
+ public static final String STORE_DETAILS_KEY_IAM_URL = "iamUrl"; // IAM endpoint URL
+ public static final String STORE_DETAILS_KEY_METRICS_URL = "metricsUrl"; // Prometheus metrics endpoint URL (optional, for scalable usage reporting)
+
+ // Account Detail Map key names - credentials created per CloudStack account.
+ // Namespaced by store ID so one account can use multiple SeaweedFS pools
+ // without the second pool overwriting the first pool's credentials.
+ public static final String KEY_ACCESS_KEY_PREFIX = "swfs_AccessKey_";
+ public static final String KEY_SECRET_KEY_PREFIX = "swfs_SecretKey_";
+
+ /**
+ * Build the account-detail key for the IAM access key of a given store.
+ */
+ public static String keyAccessKey(long storeId) {
+ return KEY_ACCESS_KEY_PREFIX + storeId;
+ }
+
+ /**
+ * Build the account-detail key for the IAM secret key of a given store.
+ */
+ public static String keySecretKey(long storeId) {
+ return KEY_SECRET_KEY_PREFIX + storeId;
+ }
+
+ /**
+ * Strip trailing slashes from a configured endpoint URL so callers can
+ * append a {@code "/" + path} suffix without producing a double slash.
+ * The S3 and IAM endpoint URLs are operator-supplied and are accepted with
+ * or without a trailing slash.
+ *
+ * @param url the URL to normalize, may be null
+ * @return the URL without trailing slashes, or the input if null/empty
+ */
+ public static String stripTrailingSlashes(String url) {
+ if (url == null) {
+ return null;
+ }
+ String normalized = url;
+ while (normalized.endsWith("/")) {
+ normalized = normalized.substring(0, normalized.length() - 1);
+ }
+ return normalized;
+ }
+
+ /**
+ * Returns true when the URL carries a non-empty path component beyond the
+ * authority (e.g. {@code http://host:8333/s3} has the path {@code /s3}).
+ *
+ * The CloudStack object-store browser builds its MinIO client from only the
+ * host and port of the stored bucket URL and drops any path prefix (see
+ * {@code ObjectStoreBrowser.vue#initMinioClient}). The MinIO JS client has
+ * no basePath option, so a SeaweedFS deployment behind a path-prefixed
+ * reverse proxy (e.g. {@code /s3}) works for server-side calls but the
+ * browser's list/upload requests target the wrong endpoint. Rejecting
+ * path-prefixed endpoints at registration prevents a configuration that
+ * would silently break the browser.
+ *
+ * @param url the endpoint URL to check, may be null
+ * @return true if the URL has a path component other than empty or "/"
+ */
+ public static boolean hasPathPrefix(String url) {
+ if (url == null) {
+ return false;
+ }
+ try {
+ java.net.URI uri = java.net.URI.create(url);
+ String path = uri.getPath();
+ return path != null && !path.isEmpty() && !"/".equals(path);
+ } catch (IllegalArgumentException e) {
+ return false;
+ }
+ }
+
+ /**
+ * Connect timeout for the S3 extension HTTP client, in seconds.
+ */
+ public static final int S3_EXTENSION_CONNECT_TIMEOUT_SECONDS = 10;
+ /**
+ * Per-request timeout for the S3 extension HTTP request, in seconds.
+ */
+ public static final int S3_EXTENSION_REQUEST_TIMEOUT_SECONDS = 30;
+
+ /**
+ * IAM user policy name applied to each per-account IAM user.
+ */
+ public static final String IAM_USER_POLICY_NAME = "CloudStackPolicy";
+
+ /**
+ * Build an IAM user policy that grants full S3 access only to the given
+ * buckets (both the bucket and its contents), while denying bucket
+ * creation and deletion everywhere so CloudStack retains control of the
+ * bucket lifecycle. When no buckets are provided, all S3 access is denied.
+ *
+ * <p>This is the tenant boundary: each account's IAM credentials can only
+ * operate on that account's own buckets, not on every bucket in the
+ * SeaweedFS pool. The policy is refreshed whenever buckets are created or
+ * deleted (see
+ * {@code SeaweedFSObjectStoreDriverImpl.updateAccountIAMPolicy}).
+ *
+ * @param bucketNames the bucket names the account is allowed to access
+ * @return a JSON IAM policy document
+ */
+ public static String buildAccountIAMPolicy(java.util.List<String> bucketNames) {
+ StringBuilder sb = new StringBuilder();
+ sb.append("{\n");
+ sb.append(" \"Version\": \"2012-10-17\",\n");
+ sb.append(" \"Statement\": [\n");
+ if (bucketNames == null || bucketNames.isEmpty()) {
+ // No buckets: deny all S3 access. A Resource cannot be empty in
+ // an IAM policy, so deny everything explicitly.
+ sb.append(" {\n");
+ sb.append(" \"Sid\": \"DenyAllS3\",\n");
+ sb.append(" \"Effect\": \"Deny\",\n");
+ sb.append(" \"Action\": [\"s3:*\"],\n");
+ sb.append(" \"Resource\": [\"arn:aws:s3:::*\", \"arn:aws:s3:::*/*\"]\n");
+ sb.append(" }\n");
+ } else {
+ sb.append(" {\n");
+ sb.append(" \"Sid\": \"AllowAccountBuckets\",\n");
+ sb.append(" \"Effect\": \"Allow\",\n");
+ sb.append(" \"Action\": [\"s3:*\"],\n");
+ sb.append(" \"Resource\": [\n");
+ for (int i = 0; i < bucketNames.size(); i++) {
+ String name = bucketNames.get(i);
+ sb.append(" \"arn:aws:s3:::").append(name).append("\",\n");
+ sb.append(" \"arn:aws:s3:::").append(name).append("/*\"");
+ if (i < bucketNames.size() - 1) {
+ sb.append(",");
+ }
+ sb.append("\n");
+ }
+ sb.append(" ]\n");
+ sb.append(" }\n");
+ }
+ // Always deny bucket creation/deletion and quota mutation —
+ // CloudStack controls lifecycle and resource accounting. Denying
+ // s3:PutBucketQuota prevents a tenant from using the credentials
+ // returned in BucketResponse to call the SeaweedFS quota extension
+ // directly and bypass CloudStack's resource accounting.
+ sb.append(" ,{\n");
+ sb.append(" \"Sid\": \"DenyBucketLifecycleAndQuota\",\n");
+ sb.append(" \"Effect\": \"Deny\",\n");
+ sb.append(" \"Action\": [\"s3:CreateBucket\", \"s3:DeleteBucket\", \"s3:PutBucketQuota\"],\n");
+ sb.append(" \"Resource\": \"*\"\n");
+ sb.append(" }\n");
+ sb.append(" ]\n");
+ sb.append("}\n");
+ return sb.toString();
+ }
+
+ /**
+ * Returns an S3 connection for the given endpoint and credentials.
+ * Uses path-style access, which SeaweedFS requires.
+ *
+ * @param url the url of the S3 service
+ * @param accessKey the credentials to use for the S3 connection.
+ * @param secretKey the matching secret key.
+ * @return an S3 connection (never null)
+ * @throws CloudRuntimeException on failure.
+ */
+ public static AmazonS3 getS3Client(String url, String accessKey, String secretKey) {
+ AmazonS3 client = AmazonS3ClientBuilder.standard()
+ .enablePathStyleAccess()
+ .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))
+ .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(url, "us-east-1"))
+ .build();
+ if (client == null) {
+ throw new CloudRuntimeException("Error while creating SeaweedFS S3 client");
+ }
+ return client;
+ }
+
+ /**
+ * Returns an IAM connection for the given endpoint and credentials.
+ *
+ * @param url the url of the IAM service
+ * @param accessKey the credentials to use for the iam connection.
+ * @param secretKey the matching secret key.
+ * @return an IAM connection (never null)
+ * @throws CloudRuntimeException on failure.
+ */
+ public static AmazonIdentityManagement getIAMClient(String url, String accessKey, String secretKey) {
+ AmazonIdentityManagement iamClient = AmazonIdentityManagementClientBuilder.standard()
+ .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))
+ .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(url, "us-east-1"))
+ .build();
+ if (iamClient == null) {
+ throw new CloudRuntimeException("Error while creating SeaweedFS IAM client");
+ }
+ return iamClient;
+ }
+
+ /**
+ * Test the S3Url to confirm it behaves like an S3 Service.
+ *
+ * Uses bad credentials and looks for the particular error from S3 that says
+ * InvalidAccessKeyId was used. Quietly returns if we connect and get the
+ * expected error back.
+ *
+ * @param s3Url the url to check
+ * @throws CloudRuntimeException if there is any unexpected issue.
+ */
+ public static void validateS3Url(String s3Url) {
+ try {
+ AmazonS3 s3Client = SeaweedFSObjectStoreUtil.getS3Client(s3Url, "unknown", "unknown");
+ s3Client.listBuckets();
+ } catch (AmazonServiceException e) {
+ if (StringUtils.compareIgnoreCase(e.getErrorCode(), "InvalidAccessKeyId") != 0
+ && StringUtils.compareIgnoreCase(e.getErrorCode(), "SignatureDoesNotMatch") != 0) {
+ throw new CloudRuntimeException("Unexpected response from S3 Endpoint.", e);
+ }
+ }
+ }
Review Comment:
validateS3Url` (and similarly `validateIAMUrl` / `validateCredentials`) only catches `AmazonServiceException`. Connection failures, DNS errors, TLS failures, etc. typically throw `AmazonClientException` and will currently escape without being wrapped in a `CloudRuntimeException`, despite the method contract/comments implying CloudStack exceptions. Catch `AmazonClientException` and rethrow as `CloudRuntimeException` with a helpful message so registration errors are consistent and user-facing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r4365034192
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,1125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.BucketCrossOriginConfiguration;
+import com.amazonaws.services.s3.model.CORSRule;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketCrossOriginConfigurationRequest;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ /**
+ * DB-backed global lock name prefix for serializing IAM provisioning and
+ * policy refreshes per store+account. Uses {@link GlobalLock} so the
+ * critical section is serialized across management servers in a
+ * clustered deployment, not just within a single JVM.
+ */
+ private static final String IAM_LOCK_PREFIX = "seaweedfs.iam.";
+ private static final String BUCKET_NAME_LOCK_PREFIX = "seaweedfs.bucket.";
+
+ private static String getIamLockName(long storeId, long accountId) {
+ return IAM_LOCK_PREFIX + storeId + "." + accountId;
+ }
+
+ /**
+ * Acquire a DB-backed global lock for IAM operations on the given
+ * store+account. Returns a {@link GlobalLock} that the caller must
+ * {@link GlobalLock#unlock()} in a {@code finally} block, or {@code null}
+ * if the lock could not be acquired within the timeout.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ */
+ protected GlobalLock acquireIamLock(long storeId, long accountId) {
+ GlobalLock lock = GlobalLock.getInternLock(getIamLockName(storeId, accountId));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire IAM lock for store {} account {}", storeId, accountId);
+ lock.releaseRef();
+ return null;
+ }
+ return lock;
+ }
+
+ private static String getBucketNameLockName(long storeId, String bucketName) {
+ return BUCKET_NAME_LOCK_PREFIX + storeId + "." + bucketName;
+ }
+
+ /**
+ * Acquire a DB-backed global lock covering a bucket <em>name</em> on a store,
+ * independent of the owning account.
+ *
+ * S3 bucket names are globally unique within a store and are reusable after
+ * deletion, while the IAM lock is scoped to (store, account). Without this
+ * lock, once a delete removes the remote bucket but before the old owner's
+ * IAM policy is refreshed, a different account can recreate the same name and
+ * the old owner's credentials would still grant that ARN. Both createBucket
+ * and deleteBucket take this lock so the two never interleave for a name.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ *
+ * @return the held lock, which the caller must {@link GlobalLock#unlock()}
+ * and {@link GlobalLock#releaseRef()}, or {@code null} on timeout
+ */
+ protected GlobalLock acquireBucketNameLock(long storeId, String bucketName) {
+ GlobalLock lock = GlobalLock.getInternLock(getBucketNameLockName(storeId, bucketName));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire bucket name lock for store {} bucket {}", storeId, bucketName);
+ lock.releaseRef();
+ return null;
+ }
+ return lock;
+ }
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account and
+ * store. The store ID is included so that two CloudStack pools pointing
+ * at the same SeaweedFS IAM service do not collide on the same
+ * {@code acs-<uuid>} user and overwrite each other's policy and access
+ * keys.
+ */
+ protected String getUserNameForAccount(Account account, long storeId) {
+ return String.format("%s-%d-%s", ACS_PREFIX, storeId, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Serialize per store+account across management servers so two
+ // concurrent bucket requests do not both rotate credentials and leave
+ // bucket rows with mismatched key pairs.
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ return false;
+ }
+ try {
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach a scoped IAM policy that allows access only to this
+ // account's own buckets (the tenant boundary). Refreshed whenever
+ // buckets are created or deleted. Use the lock-free variant since
+ // createUser already holds the IAM lock.
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, null);
+
+ // Reuse the stored access key only if both the access key id and the
+ // secret key are present and the key is still Active in IAM; otherwise
+ // create a replacement.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ String storedSecretKey = details.get(secretKeyDetailKey);
+ if (storedAccessKeyId != null && storedSecretKey != null
+ && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ updateAccountBucketCredentials(storeId, accountId, storedAccessKeyId, storedSecretKey);
+ return true;
+ }
+
+ // The stored key is missing, inactive, or no longer in IAM. Clean up
+ // ALL keys (including the inactive stored one) before creating a
+ // replacement so we do not accumulate keys and hit IAM limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, null);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Persist the credential pair in account details (namespaced by storeId)
+ // before updating BucketVO rows. The reuse path above reconciles bucket
+ // rows every time, so a later bucket update failure is repairable on
+ // retry. AccountDetailsDao.persist(accountId, map) is deliberately not
+ // used: it expunges every detail for the account and can clobber another
+ // store's namespaced credentials.
+ try {
+ persistAccountCredentialsOrRollback(accountId, accessKeyDetailKey, secretKeyDetailKey,
+ storedAccessKeyId, storedSecretKey, key);
+ } catch (RuntimeException e) {
+ deleteAccessKeyAfterCredentialPersistenceFailure(iamClient, userName, key.getAccessKeyId(), e);
+ throw e;
+ }
+
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Persist a SeaweedFS IAM credential pair without using
+ * AccountDetailsDao.persist(accountId, map), and restore the previous pair if
+ * either single-key write fails so callers never observe a half-new pair.
+ */
+ private void persistAccountCredentialsOrRollback(long accountId, String accessKeyDetailKey, String secretKeyDetailKey,
+ String previousAccessKey, String previousSecretKey, AccessKey key) {
+ try {
+ _accountDetailsDao.addDetail(accountId, accessKeyDetailKey, key.getAccessKeyId(), false);
+ _accountDetailsDao.addDetail(accountId, secretKeyDetailKey, key.getSecretAccessKey(), false);
+ } catch (RuntimeException e) {
+ CloudRuntimeException wrapped = new CloudRuntimeException("Failed to persist SeaweedFS IAM credential pair for account " + accountId, e);
+ restoreAccountCredentialDetail(accountId, accessKeyDetailKey, previousAccessKey, wrapped);
+ restoreAccountCredentialDetail(accountId, secretKeyDetailKey, previousSecretKey, wrapped);
+ throw wrapped;
+ }
+ }
+
+ private void restoreAccountCredentialDetail(long accountId, String detailKey, String previousValue, RuntimeException cause) {
+ try {
+ if (previousValue == null) {
+ _accountDetailsDao.removeDetail(accountId, detailKey);
+ } else {
+ _accountDetailsDao.addDetail(accountId, detailKey, previousValue, false);
+ }
+ } catch (RuntimeException rollbackEx) {
+ logger.error("Failed to restore account detail {} for account {} after SeaweedFS credential persistence failed",
+ detailKey, accountId, rollbackEx);
+ cause.addSuppressed(rollbackEx);
+ }
+ }
+
+ private void deleteAccessKeyAfterCredentialPersistenceFailure(AmazonIdentityManagement iamClient, String userName,
+ String accessKeyId, RuntimeException cause) {
+ try {
+ iamClient.deleteAccessKey(new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(accessKeyId));
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Failed to delete IAM access key {} for user {} after account credential persistence failed",
+ accessKeyId, userName, cleanupEx);
+ cause.addSuppressed(cleanupEx);
+ }
+ }
+
+ private void updateAccountBucketCredentials(long storeId, long accountId, AccessKey iamCredential) {
+ updateAccountBucketCredentials(storeId, accountId, iamCredential.getAccessKeyId(), iamCredential.getSecretAccessKey());
+ }
+
+ /**
+ * Update the IAM credentials on all BucketVO rows for this store/account so
+ * previously created buckets reflect the current key pair.
+ */
+ private void updateAccountBucketCredentials(long storeId, long accountId, String accessKeyId, String secretAccessKey) {
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ for (BucketVO bucketVO : bucketList) {
+ if (accessKeyId.equals(bucketVO.getAccessKey()) && secretAccessKey.equals(bucketVO.getSecretKey())) {
+ continue;
+ }
+ logger.info("Updating accountId={} bucket {} with new IAM credentials", accountId, bucketVO.getName());
+ bucketVO.setAccessKey(accessKeyId);
+ bucketVO.setSecretKey(secretAccessKey);
+ if (!_bucketDao.update(bucketVO.getId(), bucketVO)) {
+ throw new CloudRuntimeException("Failed to update IAM credentials on bucket " + bucketVO.getName());
+ }
+ }
+ }
+
+ /**
+ * Refresh the per-account IAM user policy so it grants S3 access only to
+ * the account's current buckets (optionally excluding one, e.g. a bucket
+ * being deleted). This is the tenant boundary: each account's IAM
+ * credentials can only operate on that account's own buckets.
+ *
+ * Acquires the per-store/account IAM lock. Callers that already hold the
+ * lock (e.g. createUser, createBucket post-create) should call
+ * {@link #updateAccountIAMPolicyLocked} instead to avoid re-entrant lock
+ * acquisition warnings from GlobalLock.
+ *
+ * @param iamClient the IAM client
+ * @param storeId the object store
+ * @param accountId the CloudStack account
+ * @param excludeBucket a bucket name to omit (e.g. a bucket being deleted),
+ * or null to include all of the account's buckets
+ */
+ protected void updateAccountIAMPolicy(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, excludeBucket);
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Lock-free variant of {@link #updateAccountIAMPolicy} for callers that
+ * already hold the per-store/account IAM lock. Performs the policy refresh
+ * without reacquiring the lock, avoiding the GlobalLock re-entrant
+ * acquisition warning.
+ */
+ protected void updateAccountIAMPolicyLocked(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ return;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ List<BucketVO> buckets = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ List<String> bucketNames = new ArrayList<>();
+ for (BucketVO bvo : buckets) {
+ if (excludeBucket != null && excludeBucket.equals(bvo.getName())) {
+ continue;
+ }
+ // Skip buckets whose remote counterpart has been deleted but whose
+ // row is still present for resource accounting. Including them
+ // would re-grant a bucket name that is now free for another account
+ // to claim.
+ if (Bucket.State.Destroyed.equals(bvo.getState())) {
+ continue;
+ }
+ bucketNames.add(bvo.getName());
+ }
+ String policy = SeaweedFSObjectStoreUtil.buildAccountIAMPolicy(bucketNames);
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ SeaweedFSObjectStoreUtil.IAM_USER_POLICY_NAME, policy));
+ }
+
+ /**
+ * Check whether the given access key id is still listed and Active in IAM
+ * for the user. Listing failures are propagated rather than swallowed so
+ * a transient IAM outage does not send createUser into the replacement
+ * path (which would overwrite stored credentials and invalidate bucket
+ * records).
+ */
+ private boolean iamAccessKeyExists(AmazonIdentityManagement iamClient, String userName, String accessKeyId) {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ if (accessKeyId.equals(metadata.getAccessKeyId())) {
+ return "Active".equalsIgnoreCase(metadata.getStatus());
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Delete access keys for the user other than the (optionally) preserved
+ * key id. Used to clean up unmanaged leftover keys before creating a
+ * replacement so repeated calls do not hit IAM access-key limits.
+ */
+ private void deleteUnmanagedAccessKeys(AmazonIdentityManagement iamClient, String userName, String preserveAccessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ String keyId = metadata.getAccessKeyId();
+ if (preserveAccessKeyId != null && preserveAccessKeyId.equals(keyId)) {
+ continue;
+ }
+ DeleteAccessKeyRequest deleteReq =
+ new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(keyId);
+ logger.info("Deleting un-managed IAM access key {} for user {}", keyId, userName);
+ iamClient.deleteAccessKey(deleteReq);
+ }
+ } catch (AmazonClientException e) {
+ // Propagate so the caller does not proceed to create a replacement
+ // key while stale unmanaged keys remain (which could hit IAM key
+ // limits or leave orphaned credentials).
+ throw new CloudRuntimeException("Failed to clean up IAM access keys for user " + userName, e);
+ }
+ }
+
+ @Override
+ public Bucket createBucket(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+
+ // Serialize against a concurrent deleteBucket of the same name by any
+ // account on this store. Bucket names are globally unique per store and
+ // reusable, so without this an account could claim a name while the
+ // previous owner's IAM policy still granted that ARN.
+ GlobalLock nameLock = acquireBucketNameLock(storeId, bucketName);
+ if (nameLock == null) {
+ throw new CloudRuntimeException("Failed to acquire bucket name lock for store " + storeId + " bucket " + bucketName);
+ }
+ try {
+ return createBucketLocked(bucket, objectLock);
+ } finally {
+ nameLock.unlock();
+ nameLock.releaseRef();
+ }
+ }
+
+ private Bucket createBucketLocked(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+ long accountId = bucket.getAccountId();
+
+ // Use the store's admin credentials to create the bucket
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Check if the bucket already exists
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket already exists with name " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Create the bucket
+ try {
+ CreateBucketRequest request = new CreateBucketRequest(bucketName);
+ if (objectLock) {
+ request.setObjectLockEnabledForBucket(true);
+ }
+ s3client.createBucket(request);
+ } catch (AmazonClientException e) {
+ logger.error("Create bucket failed", e);
+ throw new CloudRuntimeException(e);
+ }
+
+ // Step 2: update the bucket record with the account's IAM credentials,
+ // configure CORS, and refresh the IAM policy. If any of these fail,
+ // clean up the remote bucket so a retry does not find it already
+ // existing — mirroring the Cloudian createBucket pattern.
+ //
+ // Hold the IAM lock for the account so a concurrent createUser key
+ // rotation does not change the account credentials between reading
+ // them and writing the BucketVO, which would leave the bucket with
+ // a stale key pair.
+ GlobalLock iamLock = acquireIamLock(storeId, accountId);
+ if (iamLock == null) {
+ // The S3 bucket has already been created. Clean it up so a
+ // retry does not find it already existing, then throw.
+ try {
+ s3client.deleteBucket(bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Failed to clean up bucket {} after IAM lock timeout", bucketName, cleanupEx);
+ }
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ // Configure permissive CORS so the CloudStack S3 bucket browser
+ // (which performs list/upload/delete from the browser) can function.
+ // SeaweedFS supports the standard PutBucketCors operation.
+ configureBucketCORS(s3client, bucketName);
+
+ Map<String, String> accountDetails = _accountDetailsDao.findDetails(accountId);
+ String accessKey = accountDetails.get(SeaweedFSObjectStoreUtil.keyAccessKey(storeId));
+ String secretKey = accountDetails.get(SeaweedFSObjectStoreUtil.keySecretKey(storeId));
+ if (accessKey == null || secretKey == null) {
+ throw new CloudRuntimeException("No IAM credentials found for account " + accountId
+ + " on store " + storeId + ". Run createUser before creating a bucket.");
+ }
+
+ String s3Url = getS3Url(storeId);
+ BucketVO bucketVO = _bucketDao.findById(bucket.getId());
+ if (bucketVO == null) {
+ throw new CloudRuntimeException("Bucket record " + bucket.getId()
+ + " not found for bucket " + bucketName);
+ }
+ bucketVO.setAccessKey(accessKey);
+ bucketVO.setSecretKey(secretKey);
+ // Normalize the endpoint: s3Url is operator-supplied and may carry a
+ // trailing slash, which would persist a broken "...//bucket" URL
+ // that BucketResponse and the object store browser both use.
+ bucketVO.setBucketURL(SeaweedFSObjectStoreUtil.stripTrailingSlashes(s3Url) + "/" + bucketName);
+ if (!_bucketDao.update(bucket.getId(), bucketVO)) {
+ // A failed row update must fail the create: otherwise the IAM
+ // refresh below would grant the new bucket ARN while the
+ // returned BucketVO carries credentials/URL that were never
+ // persisted. The catch block revokes the policy grant and
+ // removes the remote bucket.
+ throw new CloudRuntimeException("Failed to persist credentials and URL for bucket " + bucketName);
+ }
+
+ // Refresh the account's IAM policy to include the new bucket.
+ // Use the lock-free variant since createBucket already holds the
+ // IAM lock for the post-create section.
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, null);
+
+ // Return the updated BucketVO (not the stale input bucket) so
+ // BucketApiServiceImpl.createBucket does not overwrite the
+ // persisted credentials with the stale values.
+ return bucketVO;
+ } catch (Exception e) {
+ logger.error("Post-create bucket record update failed for {}; cleaning up", bucketName, e);
+ CloudRuntimeException primary = new CloudRuntimeException(e);
+ // Revoke the IAM policy grant for the new bucket BEFORE deleting
+ // the remote S3 bucket. If the policy PUT succeeded before the DB
+ // update failed, the grant would otherwise persist and could be
+ // reused if another account later creates the same bucket name.
+ // Use the lock-free variant since createBucket already holds the
+ // IAM lock.
+ //
+ // If the policy revocation fails, the remote S3 bucket is LEFT IN
+ // PLACE: deleting it would free the bucket name for another account
+ // to claim while this account's policy still grants the ARN,
+ // letting the old credentials reach the new tenant's bucket. The
+ // leftover remote bucket keeps the name occupied until an operator
+ // reconciles; the row is removed by the caller so a retry reports
+ // "bucket already exists" rather than silently re-granting access.
+ try {
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, bucketName);
+ } catch (Exception policyEx) {
+ logger.error("Failed to revoke IAM policy for bucket {} after create failure; "
+ + "leaving the remote bucket in place to prevent name reuse", bucketName, policyEx);
+ primary.addSuppressed(policyEx);
+ throw primary;
+ }
+ // Policy revocation succeeded: the stale grant is gone, so it is
+ // safe to delete the remote bucket and free the name.
+ try {
+ s3client.deleteBucket(bucketName);
+ logger.info("Cleanup of bucket {} succeeded", bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Cleanup of bucket {} also failed", bucketName, cleanupEx);
+ primary.addSuppressed(cleanupEx);
+ }
+ throw primary;
+ } finally {
+ iamLock.unlock();
+ iamLock.releaseRef();
+ }
+ }
+
+ /**
+ * Configure a permissive CORS policy on the bucket so the CloudStack
+ * S3 bucket browser (which performs list/upload/delete from the
+ * browser) can function. Mirrors the Cloudian configureBucketCORS.
+ */
+ private void configureBucketCORS(AmazonS3 s3client, String bucketName) {
+ logger.debug("Configuring CORS for bucket {}", bucketName);
+ List<CORSRule> corsRules = new ArrayList<>();
+ CORSRule allowAnyRule = new CORSRule().withId("AllowAny");
+ allowAnyRule.setAllowedOrigins("*");
+ allowAnyRule.setAllowedHeaders("*");
+ allowAnyRule.setAllowedMethods(
+ CORSRule.AllowedMethods.HEAD,
+ CORSRule.AllowedMethods.GET,
+ CORSRule.AllowedMethods.PUT,
+ CORSRule.AllowedMethods.POST,
+ CORSRule.AllowedMethods.DELETE);
+ corsRules.add(allowAnyRule);
+ BucketCrossOriginConfiguration corsConfig = new BucketCrossOriginConfiguration();
+ corsConfig.setRules(corsRules);
+ SetBucketCrossOriginConfigurationRequest corsRequest = new SetBucketCrossOriginConfigurationRequest(bucketName, corsConfig);
+ s3client.setBucketCrossOriginConfiguration(corsRequest);
+ logger.info("Successfully configured CORS for bucket {}", bucketName);
+ }
+
+ @Override
+ public List<Bucket> listBuckets(long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ List<Bucket> bucketsList = new ArrayList<>();
+ try {
+ List<com.amazonaws.services.s3.model.Bucket> s3Buckets = s3client.listBuckets();
+ for (com.amazonaws.services.s3.model.Bucket s3Bucket : s3Buckets) {
+ Bucket bucket = new BucketObject();
+ bucket.setName(s3Bucket.getName());
+ bucketsList.add(bucket);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ return bucketsList;
+ }
+
+ @Override
+ public boolean deleteBucket(BucketTO bucket, long storeId) {
+ String bucketName = bucket.getName();
+
+ // Hold the store-wide bucket name lock across the remote delete and the
+ // IAM policy refresh. The IAM lock below is scoped to (store, account)
+ // and therefore does not serialize against createBucket for a *different*
+ // account: once the S3 delete succeeded, that account could claim the now
+ // free name while this account's policy still granted the ARN, letting the
+ // old credentials reach the new tenant's bucket.
+ GlobalLock nameLock = acquireBucketNameLock(storeId, bucketName);
+ if (nameLock == null) {
+ throw new CloudRuntimeException("Failed to acquire bucket name lock for store " + storeId + " bucket " + bucketName);
+ }
+ try {
+ return deleteBucketLocked(bucket, storeId);
+ } finally {
+ nameLock.unlock();
+ nameLock.releaseRef();
+ }
+ }
+
+ private boolean deleteBucketLocked(BucketTO bucket, long storeId) {
+ String bucketName = bucket.getName();
+ long accountId = bucket.getAccountId();
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Acquire the per-store/account IAM lock so the S3 delete and the
+ // subsequent policy refresh are atomic with respect to concurrent
+ // createUser operations for this account.
+ GlobalLock iamLock = acquireIamLock(storeId, accountId);
+ if (iamLock == null) {
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ // Delete the S3 bucket first. If this fails (non-empty bucket,
+ // transient error), the IAM policy is still intact so the user
+ // can empty the bucket and retry.
+ //
+ // If the bucket is already gone (e.g. from a previous partial
+ // failure where the S3 delete succeeded but the IAM policy refresh
+ // failed), skip the S3 delete so the retry is idempotent.
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ s3client.deleteBucket(bucketName);
+ }
Review Comment:
If IAM policy refresh fails after the remote delete, the row remains `Destroyed` and this method returns an error. A retry then reaches this block with the same bucket name, sees a newly-created bucket belonging to another account, and deletes that replacement before attempting the policy refresh. The retry path must distinguish a previously-destroyed row and skip the remote delete (or otherwise verify ownership) so cleanup cannot delete a reused bucket name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r4011008330
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,649 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account.
+ * Uses the account UUID prefixed with "acs-" for namespacing.
+ */
+ protected String getUserNameForAccount(Account account) {
+ return String.format("%s-%s", ACS_PREFIX, account.getUuid());
Review Comment:
Although account-detail keys include `storeId`, the IAM username does not. If two CloudStack pools point to the same SeaweedFS IAM service, both map an account to `acs-<uuid>`; the second pool overwrites the shared policy and deletes or replaces the first pool's access key, breaking isolation and access for the first pool. Namespace the IAM user by store, or reject shared IAM endpoints.
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/util/SeaweedFSObjectStoreUtil.java:
##########
@@ -0,0 +1,447 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.util;
+
+import org.apache.commons.lang3.StringUtils;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagementClientBuilder;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3ClientBuilder;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * Utility class for the SeaweedFS object storage provider.
+ *
+ * SeaweedFS exposes both an S3-compatible API and an AWS IAM-compatible API,
+ * so this provider needs no proprietary admin client — only the AWS S3 and IAM
+ * SDKs, the same pair Cloudian HyperStore already uses in this tree.
+ */
+public class SeaweedFSObjectStoreUtil {
+
+ /** The name of our Object Store Provider */
+ public static final String OBJECT_STORE_PROVIDER_NAME = "SeaweedFS";
+
+ public static final String STORE_KEY_PROVIDER_NAME = "providerName";
+ public static final String STORE_KEY_URL = "url";
+ public static final String STORE_KEY_NAME = "name";
+ public static final String STORE_KEY_SIZE = "size";
+ public static final String STORE_KEY_DETAILS = "details";
+
+ // Store Details Map key names - managed outside of plugin
+ public static final String STORE_DETAILS_KEY_ACCESS_KEY = "accesskey"; // admin/root access key
+ public static final String STORE_DETAILS_KEY_SECRET_KEY = "secretkey"; // admin/root secret key
+ public static final String STORE_DETAILS_KEY_S3_URL = "s3Url"; // S3 endpoint URL
+ public static final String STORE_DETAILS_KEY_IAM_URL = "iamUrl"; // IAM endpoint URL
+
+ // Account Detail Map key names - credentials created per CloudStack account.
+ // Namespaced by store ID so one account can use multiple SeaweedFS pools
+ // without the second pool overwriting the first pool's credentials.
+ public static final String KEY_ACCESS_KEY_PREFIX = "swfs_AccessKey_";
+ public static final String KEY_SECRET_KEY_PREFIX = "swfs_SecretKey_";
+
+ /**
+ * Build the account-detail key for the IAM access key of a given store.
+ */
+ public static String keyAccessKey(long storeId) {
+ return KEY_ACCESS_KEY_PREFIX + storeId;
+ }
+
+ /**
+ * Build the account-detail key for the IAM secret key of a given store.
+ */
+ public static String keySecretKey(long storeId) {
+ return KEY_SECRET_KEY_PREFIX + storeId;
+ }
+
+ /**
+ * Connect timeout for the S3 extension HTTP client, in seconds.
+ */
+ public static final int S3_EXTENSION_CONNECT_TIMEOUT_SECONDS = 10;
+ /**
+ * Per-request timeout for the S3 extension HTTP request, in seconds.
+ */
+ public static final int S3_EXTENSION_REQUEST_TIMEOUT_SECONDS = 30;
+
+ /**
+ * IAM user policy name applied to each per-account IAM user.
+ */
+ public static final String IAM_USER_POLICY_NAME = "CloudStackPolicy";
+
+ /**
+ * Build an IAM user policy that grants full S3 access only to the given
+ * buckets (both the bucket and its contents), while denying bucket
+ * creation and deletion everywhere so CloudStack retains control of the
+ * bucket lifecycle. When no buckets are provided, all S3 access is denied.
+ *
+ * <p>This is the tenant boundary: each account's IAM credentials can only
+ * operate on that account's own buckets, not on every bucket in the
+ * SeaweedFS pool. The policy is refreshed whenever buckets are created or
+ * deleted (see
+ * {@code SeaweedFSObjectStoreDriverImpl.updateAccountIAMPolicy}).
+ *
+ * @param bucketNames the bucket names the account is allowed to access
+ * @return a JSON IAM policy document
+ */
+ public static String buildAccountIAMPolicy(java.util.List<String> bucketNames) {
+ StringBuilder sb = new StringBuilder();
+ sb.append("{\n");
+ sb.append(" \"Version\": \"2012-10-17\",\n");
+ sb.append(" \"Statement\": [\n");
+ if (bucketNames == null || bucketNames.isEmpty()) {
+ // No buckets: deny all S3 access. A Resource cannot be empty in
+ // an IAM policy, so deny everything explicitly.
+ sb.append(" {\n");
+ sb.append(" \"Sid\": \"DenyAllS3\",\n");
+ sb.append(" \"Effect\": \"Deny\",\n");
+ sb.append(" \"Action\": [\"s3:*\"],\n");
+ sb.append(" \"Resource\": [\"arn:aws:s3:::*\", \"arn:aws:s3:::*/*\"]\n");
+ sb.append(" }\n");
+ } else {
+ sb.append(" {\n");
+ sb.append(" \"Sid\": \"AllowAccountBuckets\",\n");
+ sb.append(" \"Effect\": \"Allow\",\n");
+ sb.append(" \"Action\": [\"s3:*\"],\n");
+ sb.append(" \"Resource\": [\n");
+ for (int i = 0; i < bucketNames.size(); i++) {
+ String name = bucketNames.get(i);
+ sb.append(" \"arn:aws:s3:::").append(name).append("\",\n");
+ sb.append(" \"arn:aws:s3:::").append(name).append("/*\"");
+ if (i < bucketNames.size() - 1) {
+ sb.append(",");
+ }
+ sb.append("\n");
+ }
+ sb.append(" ]\n");
+ sb.append(" }\n");
+ }
+ // Always deny bucket creation/deletion — CloudStack controls lifecycle
+ sb.append(" ,{\n");
+ sb.append(" \"Sid\": \"DenyBucketLifecycle\",\n");
+ sb.append(" \"Effect\": \"Deny\",\n");
+ sb.append(" \"Action\": [\"s3:CreateBucket\", \"s3:DeleteBucket\"],\n");
Review Comment:
The `s3:*` allow also grants the client-visible per-account credentials `s3:PutBucketQuota`. Bucket access and secret keys are returned in `BucketResponse`, while this provider uses the store admin credentials for `setBucketQuota`, so a tenant can call the SeaweedFS extension directly to disable or inflate the quota and bypass CloudStack's resource accounting. Add an explicit deny for `s3:PutBucketQuota` to the tenant policy.
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,649 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account.
+ * Uses the account UUID prefixed with "acs-" for namespacing.
+ */
+ protected String getUserNameForAccount(Account account) {
+ return String.format("%s-%s", ACS_PREFIX, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach a scoped IAM policy that allows access only to this
+ // account's own buckets (the tenant boundary). Refreshed whenever
+ // buckets are created or deleted.
+ updateAccountIAMPolicy(iamClient, storeId, accountId, null);
+
+ // Reuse the stored access key only if both the access key id and the
+ // secret key are present and the key is still Active in IAM; otherwise
+ // create a replacement.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ String storedSecretKey = details.get(secretKeyDetailKey);
+ if (storedAccessKeyId != null && storedSecretKey != null
+ && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ return true;
+ }
+
+ // The stored key is missing, inactive, or no longer in IAM. Clean up
+ // ALL keys (including the inactive stored one) before creating a
+ // replacement so we do not accumulate keys and hit IAM limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, null);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Persist the credentials in the account details (namespaced by storeId)
+ details.put(accessKeyDetailKey, key.getAccessKeyId());
+ details.put(secretKeyDetailKey, key.getSecretAccessKey());
+ _accountDetailsDao.persist(accountId, details);
+
+ // Update existing bucket records for this account/store with the new
+ // credentials so previously created buckets don't keep handing out
+ // the old (now invalid) key pair.
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ }
+
+ /**
+ * Update the IAM credentials on all BucketVO rows for this store/account
+ * so previously created buckets reflect the new (rotated) key pair.
+ * Mirrors CloudianHyperStoreObjectStoreDriverImpl.updateAccountBucketCredentials.
+ */
+ private void updateAccountBucketCredentials(long storeId, long accountId, AccessKey iamCredential) {
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ for (BucketVO bucketVO : bucketList) {
+ logger.info("Updating accountId={} bucket {} with new IAM credentials", accountId, bucketVO.getName());
+ bucketVO.setAccessKey(iamCredential.getAccessKeyId());
+ bucketVO.setSecretKey(iamCredential.getSecretAccessKey());
+ _bucketDao.update(bucketVO.getId(), bucketVO);
+ }
+ }
+
+ /**
+ * Refresh the per-account IAM user policy so it grants S3 access only to
+ * the account's current buckets (optionally excluding one, e.g. a bucket
+ * being deleted). This is the tenant boundary: each account's IAM
+ * credentials can only operate on that account's own buckets.
+ *
+ * @param iamClient the IAM client
+ * @param storeId the object store
+ * @param accountId the CloudStack account
+ * @param excludeBucket a bucket name to omit (e.g. a bucket being deleted),
+ * or null to include all of the account's buckets
+ */
+ protected void updateAccountIAMPolicy(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ return;
+ }
+ String userName = getUserNameForAccount(account);
+ List<BucketVO> buckets = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ List<String> bucketNames = new ArrayList<>();
+ for (BucketVO bvo : buckets) {
+ if (excludeBucket != null && excludeBucket.equals(bvo.getName())) {
+ continue;
+ }
+ bucketNames.add(bvo.getName());
+ }
+ String policy = SeaweedFSObjectStoreUtil.buildAccountIAMPolicy(bucketNames);
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ SeaweedFSObjectStoreUtil.IAM_USER_POLICY_NAME, policy));
+ }
+
+ /**
+ * Check whether the given access key id is still listed and Active in IAM
+ * for the user. Listing failures are propagated rather than swallowed so
+ * a transient IAM outage does not send createUser into the replacement
+ * path (which would overwrite stored credentials and invalidate bucket
+ * records).
+ */
+ private boolean iamAccessKeyExists(AmazonIdentityManagement iamClient, String userName, String accessKeyId) {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ if (accessKeyId.equals(metadata.getAccessKeyId())) {
+ return "Active".equalsIgnoreCase(metadata.getStatus());
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Delete access keys for the user other than the (optionally) preserved
+ * key id. Used to clean up unmanaged leftover keys before creating a
+ * replacement so repeated calls do not hit IAM access-key limits.
+ */
+ private void deleteUnmanagedAccessKeys(AmazonIdentityManagement iamClient, String userName, String preserveAccessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ String keyId = metadata.getAccessKeyId();
+ if (preserveAccessKeyId != null && preserveAccessKeyId.equals(keyId)) {
+ continue;
+ }
+ DeleteAccessKeyRequest deleteReq =
+ new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(keyId);
+ logger.info("Deleting un-managed IAM access key {} for user {}", keyId, userName);
+ iamClient.deleteAccessKey(deleteReq);
+ }
+ } catch (AmazonClientException e) {
+ logger.warn("Failed to clean up IAM access keys for user {}: {}", userName, e.getMessage());
+ }
+ }
+
+ @Override
+ public Bucket createBucket(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+ long accountId = bucket.getAccountId();
+
+ // Use the store's admin credentials to create the bucket
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Check if the bucket already exists
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket already exists with name " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Create the bucket
+ try {
+ CreateBucketRequest request = new CreateBucketRequest(bucketName);
+ if (objectLock) {
+ request.setObjectLockEnabledForBucket(true);
+ }
+ s3client.createBucket(request);
+ } catch (AmazonClientException e) {
+ logger.error("Create bucket failed", e);
+ throw new CloudRuntimeException(e);
+ }
+
+ // Step 2: update the bucket record with the account's IAM credentials.
+ // If this fails, clean up the remote bucket so a retry does not find
+ // it already existing — mirroring the Cloudian createBucket pattern.
+ try {
+ Map<String, String> accountDetails = _accountDetailsDao.findDetails(accountId);
+ String accessKey = accountDetails.get(SeaweedFSObjectStoreUtil.keyAccessKey(storeId));
+ String secretKey = accountDetails.get(SeaweedFSObjectStoreUtil.keySecretKey(storeId));
+ if (accessKey == null || secretKey == null) {
+ logger.warn("No IAM credentials found for account {}. Bucket will be created without per-account credentials.", accountId);
+ }
+
+ String s3Url = getS3Url(storeId);
+ BucketVO bucketVO = _bucketDao.findById(bucket.getId());
+ bucketVO.setAccessKey(accessKey);
+ bucketVO.setSecretKey(secretKey);
+ bucketVO.setBucketURL(s3Url + "/" + bucketName);
+ _bucketDao.update(bucket.getId(), bucketVO);
+
+ // Refresh the account's IAM policy to include the new bucket
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicy(iamClient, storeId, accountId, null);
+
+ return bucket;
+ } catch (Exception e) {
+ logger.error("Post-create bucket record update failed for {}; cleaning up remote bucket", bucketName, e);
+ try {
+ s3client.deleteBucket(bucketName);
+ logger.info("Cleanup of bucket {} succeeded", bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Cleanup of bucket {} also failed", bucketName, cleanupEx);
+ }
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public List<Bucket> listBuckets(long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ List<Bucket> bucketsList = new ArrayList<>();
+ try {
+ List<com.amazonaws.services.s3.model.Bucket> s3Buckets = s3client.listBuckets();
+ for (com.amazonaws.services.s3.model.Bucket s3Bucket : s3Buckets) {
+ Bucket bucket = new BucketObject();
+ bucket.setName(s3Bucket.getName());
+ bucketsList.add(bucket);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ return bucketsList;
+ }
+
+ @Override
+ public boolean deleteBucket(BucketTO bucket, long storeId) {
+ String bucketName = bucket.getName();
+ long accountId = bucket.getAccountId();
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ if (! s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket doesn't exist: " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ try {
+ s3client.deleteBucket(bucketName);
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ // Refresh the account's IAM policy to drop the deleted bucket
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicy(iamClient, storeId, accountId, bucketName);
Review Comment:
The S3 deletion has already succeeded before this IAM refresh. If `getIAMClient` or `putUserPolicy` fails, this method throws and `BucketApiServiceImpl` leaves the CloudStack `BucketVO` and resource-accounting state intact even though the remote bucket is gone; retries then hit the not-found path. Make the policy refresh best-effort after a successful deletion (and log or retry it separately) before returning success.
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/lifecycle/SeaweedFSObjectStoreLifeCycleImpl.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.lifecycle;
+
+import com.cloud.agent.api.StoragePoolInfo;
+import com.cloud.hypervisor.Hypervisor.HypervisorType;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.ClusterScope;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.HostScope;
+import org.apache.cloudstack.engine.subsystem.api.storage.ZoneScope;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.datastore.ObjectStoreHelper;
+import org.apache.cloudstack.storage.object.datastore.ObjectStoreProviderManager;
+import org.apache.cloudstack.storage.object.store.lifecycle.ObjectStoreLifeCycle;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import javax.inject.Inject;
+
+import java.util.HashMap;
+import java.util.Map;
+
+public class SeaweedFSObjectStoreLifeCycleImpl implements ObjectStoreLifeCycle {
+
+ protected Logger logger = LogManager.getLogger(SeaweedFSObjectStoreLifeCycleImpl.class);
+
+ @Inject
+ ObjectStoreHelper objectStoreHelper;
+ @Inject
+ ObjectStoreProviderManager objectStoreMgr;
+
+ public SeaweedFSObjectStoreLifeCycleImpl() {
+ }
+
+ @Override
+ public DataStore initialize(Map<String, Object> dsInfos) {
+
+ String name = (String)dsInfos.get(SeaweedFSObjectStoreUtil.STORE_KEY_NAME);
+ String url = (String)dsInfos.get(SeaweedFSObjectStoreUtil.STORE_KEY_URL);
+ String providerName = (String)dsInfos.get(SeaweedFSObjectStoreUtil.STORE_KEY_PROVIDER_NAME);
+ Long size = (Long)dsInfos.get(SeaweedFSObjectStoreUtil.STORE_KEY_SIZE);
+
+ // Check the providerName is what we expect
+ if (! StringUtils.equalsIgnoreCase(providerName, SeaweedFSObjectStoreUtil.OBJECT_STORE_PROVIDER_NAME)) {
+ String msg = String.format("Unexpected providerName \"%s\". Expected \"%s\"", providerName, SeaweedFSObjectStoreUtil.OBJECT_STORE_PROVIDER_NAME);
+ logger.error(msg);
+ throw new CloudRuntimeException(msg);
+ }
+
+ Map<String, Object> objectStoreParameters = new HashMap<String, Object>();
+ objectStoreParameters.put(SeaweedFSObjectStoreUtil.STORE_KEY_NAME, name);
+ objectStoreParameters.put(SeaweedFSObjectStoreUtil.STORE_KEY_URL, url);
+ objectStoreParameters.put(SeaweedFSObjectStoreUtil.STORE_KEY_PROVIDER_NAME, providerName);
+ objectStoreParameters.put(SeaweedFSObjectStoreUtil.STORE_KEY_SIZE, size);
+
+ // Pull out the details map
+ @SuppressWarnings("unchecked")
+ Map<String, String> details = (Map<String, String>) dsInfos.get(SeaweedFSObjectStoreUtil.STORE_KEY_DETAILS);
+ if (details == null) {
+ String msg = String.format("Unexpected null receiving Object Store initialization \"%s\"", SeaweedFSObjectStoreUtil.STORE_KEY_DETAILS);
+ logger.error(msg);
+ throw new CloudRuntimeException(msg);
+ }
+
+ // The admin/root access key and secret key are available as accesskey/secretkey
+ String accessKey = details.get(SeaweedFSObjectStoreUtil.STORE_DETAILS_KEY_ACCESS_KEY);
+ String secretKey = details.get(SeaweedFSObjectStoreUtil.STORE_DETAILS_KEY_SECRET_KEY);
+ String s3Url = details.get(SeaweedFSObjectStoreUtil.STORE_DETAILS_KEY_S3_URL);
+ String iamUrl = details.get(SeaweedFSObjectStoreUtil.STORE_DETAILS_KEY_IAM_URL);
+
+ // If s3Url is not provided, default it to the store url
+ if (StringUtils.isBlank(s3Url)) {
+ s3Url = url;
+ }
+ // If iamUrl is not provided, default it to the s3Url.
+ // SeaweedFS registers its embedded IAM API at POST / on the same S3
+ // endpoint (UnifiedPostHandler), so the IAM endpoint is the same as
+ // the S3 endpoint unless the deployment runs a separate weed iam server.
+ if (StringUtils.isBlank(iamUrl)) {
+ iamUrl = s3Url;
+ }
+
+ if (StringUtils.isAnyBlank(accessKey, secretKey, s3Url, iamUrl)) {
+ final String asteriskPassword = (secretKey == null) ? null : "*".repeat(secretKey.length());
+ logger.error("Required parameters are missing; accessKey={} secretKey={} s3Url={} iamUrl={}",
+ accessKey, asteriskPassword, s3Url, iamUrl);
+ throw new CloudRuntimeException("Required SeaweedFS configuration parameters are missing/empty.");
+ }
+
+ // Update the details map with the resolved URLs so the driver can read them later
+ details.put(SeaweedFSObjectStoreUtil.STORE_DETAILS_KEY_S3_URL, s3Url);
+ details.put(SeaweedFSObjectStoreUtil.STORE_DETAILS_KEY_IAM_URL, iamUrl);
Review Comment:
These writes turn the fallback values into permanent detail overrides. `updateObjectStore` only changes `ObjectStoreVO.url`, while the driver prefers the persisted `s3Url`, so a normal SeaweedFS pool whose `s3Url` defaults to `url` keeps using the old endpoint after a URL update; the update's connectivity check also validates the old endpoint. Persist only explicitly supplied endpoint overrides and let the driver fallback resolve the current store URL.
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/util/SeaweedFSObjectStoreUtil.java:
##########
@@ -0,0 +1,447 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.util;
+
+import org.apache.commons.lang3.StringUtils;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagementClientBuilder;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3ClientBuilder;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * Utility class for the SeaweedFS object storage provider.
+ *
+ * SeaweedFS exposes both an S3-compatible API and an AWS IAM-compatible API,
+ * so this provider needs no proprietary admin client — only the AWS S3 and IAM
+ * SDKs, the same pair Cloudian HyperStore already uses in this tree.
+ */
+public class SeaweedFSObjectStoreUtil {
+
+ /** The name of our Object Store Provider */
+ public static final String OBJECT_STORE_PROVIDER_NAME = "SeaweedFS";
+
+ public static final String STORE_KEY_PROVIDER_NAME = "providerName";
+ public static final String STORE_KEY_URL = "url";
+ public static final String STORE_KEY_NAME = "name";
+ public static final String STORE_KEY_SIZE = "size";
+ public static final String STORE_KEY_DETAILS = "details";
+
+ // Store Details Map key names - managed outside of plugin
+ public static final String STORE_DETAILS_KEY_ACCESS_KEY = "accesskey"; // admin/root access key
+ public static final String STORE_DETAILS_KEY_SECRET_KEY = "secretkey"; // admin/root secret key
+ public static final String STORE_DETAILS_KEY_S3_URL = "s3Url"; // S3 endpoint URL
+ public static final String STORE_DETAILS_KEY_IAM_URL = "iamUrl"; // IAM endpoint URL
+
+ // Account Detail Map key names - credentials created per CloudStack account.
+ // Namespaced by store ID so one account can use multiple SeaweedFS pools
+ // without the second pool overwriting the first pool's credentials.
+ public static final String KEY_ACCESS_KEY_PREFIX = "swfs_AccessKey_";
+ public static final String KEY_SECRET_KEY_PREFIX = "swfs_SecretKey_";
+
+ /**
+ * Build the account-detail key for the IAM access key of a given store.
+ */
+ public static String keyAccessKey(long storeId) {
+ return KEY_ACCESS_KEY_PREFIX + storeId;
+ }
+
+ /**
+ * Build the account-detail key for the IAM secret key of a given store.
+ */
+ public static String keySecretKey(long storeId) {
+ return KEY_SECRET_KEY_PREFIX + storeId;
+ }
+
+ /**
+ * Connect timeout for the S3 extension HTTP client, in seconds.
+ */
+ public static final int S3_EXTENSION_CONNECT_TIMEOUT_SECONDS = 10;
+ /**
+ * Per-request timeout for the S3 extension HTTP request, in seconds.
+ */
+ public static final int S3_EXTENSION_REQUEST_TIMEOUT_SECONDS = 30;
+
+ /**
+ * IAM user policy name applied to each per-account IAM user.
+ */
+ public static final String IAM_USER_POLICY_NAME = "CloudStackPolicy";
+
+ /**
+ * Build an IAM user policy that grants full S3 access only to the given
+ * buckets (both the bucket and its contents), while denying bucket
+ * creation and deletion everywhere so CloudStack retains control of the
+ * bucket lifecycle. When no buckets are provided, all S3 access is denied.
+ *
+ * <p>This is the tenant boundary: each account's IAM credentials can only
+ * operate on that account's own buckets, not on every bucket in the
+ * SeaweedFS pool. The policy is refreshed whenever buckets are created or
+ * deleted (see
+ * {@code SeaweedFSObjectStoreDriverImpl.updateAccountIAMPolicy}).
+ *
+ * @param bucketNames the bucket names the account is allowed to access
+ * @return a JSON IAM policy document
+ */
+ public static String buildAccountIAMPolicy(java.util.List<String> bucketNames) {
+ StringBuilder sb = new StringBuilder();
+ sb.append("{\n");
+ sb.append(" \"Version\": \"2012-10-17\",\n");
+ sb.append(" \"Statement\": [\n");
+ if (bucketNames == null || bucketNames.isEmpty()) {
+ // No buckets: deny all S3 access. A Resource cannot be empty in
+ // an IAM policy, so deny everything explicitly.
+ sb.append(" {\n");
+ sb.append(" \"Sid\": \"DenyAllS3\",\n");
+ sb.append(" \"Effect\": \"Deny\",\n");
+ sb.append(" \"Action\": [\"s3:*\"],\n");
+ sb.append(" \"Resource\": [\"arn:aws:s3:::*\", \"arn:aws:s3:::*/*\"]\n");
+ sb.append(" }\n");
+ } else {
+ sb.append(" {\n");
+ sb.append(" \"Sid\": \"AllowAccountBuckets\",\n");
+ sb.append(" \"Effect\": \"Allow\",\n");
+ sb.append(" \"Action\": [\"s3:*\"],\n");
+ sb.append(" \"Resource\": [\n");
+ for (int i = 0; i < bucketNames.size(); i++) {
+ String name = bucketNames.get(i);
+ sb.append(" \"arn:aws:s3:::").append(name).append("\",\n");
+ sb.append(" \"arn:aws:s3:::").append(name).append("/*\"");
+ if (i < bucketNames.size() - 1) {
+ sb.append(",");
+ }
+ sb.append("\n");
+ }
+ sb.append(" ]\n");
+ sb.append(" }\n");
+ }
+ // Always deny bucket creation/deletion — CloudStack controls lifecycle
+ sb.append(" ,{\n");
+ sb.append(" \"Sid\": \"DenyBucketLifecycle\",\n");
+ sb.append(" \"Effect\": \"Deny\",\n");
+ sb.append(" \"Action\": [\"s3:CreateBucket\", \"s3:DeleteBucket\"],\n");
+ sb.append(" \"Resource\": \"*\"\n");
+ sb.append(" }\n");
+ sb.append(" ]\n");
+ sb.append("}\n");
+ return sb.toString();
+ }
+
+ /**
+ * Returns an S3 connection for the given endpoint and credentials.
+ * Uses path-style access, which SeaweedFS requires.
+ *
+ * @param url the url of the S3 service
+ * @param accessKey the credentials to use for the S3 connection.
+ * @param secretKey the matching secret key.
+ * @return an S3 connection (never null)
+ * @throws CloudRuntimeException on failure.
+ */
+ public static AmazonS3 getS3Client(String url, String accessKey, String secretKey) {
+ AmazonS3 client = AmazonS3ClientBuilder.standard()
+ .enablePathStyleAccess()
+ .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))
+ .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(url, "us-east-1"))
+ .build();
+ if (client == null) {
+ throw new CloudRuntimeException("Error while creating SeaweedFS S3 client");
+ }
+ return client;
+ }
+
+ /**
+ * Returns an IAM connection for the given endpoint and credentials.
+ *
+ * @param url the url of the IAM service
+ * @param accessKey the credentials to use for the iam connection.
+ * @param secretKey the matching secret key.
+ * @return an IAM connection (never null)
+ * @throws CloudRuntimeException on failure.
+ */
+ public static AmazonIdentityManagement getIAMClient(String url, String accessKey, String secretKey) {
+ AmazonIdentityManagement iamClient = AmazonIdentityManagementClientBuilder.standard()
+ .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))
+ .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(url, "us-east-1"))
+ .build();
+ if (iamClient == null) {
+ throw new CloudRuntimeException("Error while creating SeaweedFS IAM client");
+ }
+ return iamClient;
+ }
+
+ /**
+ * Test the S3Url to confirm it behaves like an S3 Service.
+ *
+ * Uses bad credentials and looks for the particular error from S3 that says
+ * InvalidAccessKeyId was used. Quietly returns if we connect and get the
+ * expected error back.
+ *
+ * @param s3Url the url to check
+ * @throws CloudRuntimeException if there is any unexpected issue.
+ */
+ public static void validateS3Url(String s3Url) {
+ try {
+ AmazonS3 s3Client = SeaweedFSObjectStoreUtil.getS3Client(s3Url, "unknown", "unknown");
+ s3Client.listBuckets();
+ } catch (AmazonServiceException e) {
+ if (StringUtils.compareIgnoreCase(e.getErrorCode(), "InvalidAccessKeyId") != 0
+ && StringUtils.compareIgnoreCase(e.getErrorCode(), "SignatureDoesNotMatch") != 0) {
+ throw new CloudRuntimeException("Unexpected response from S3 Endpoint.", e);
+ }
+ }
+ }
+
+ /**
+ * Test the IAMUrl to confirm it behaves like an IAM Service.
+ *
+ * Uses bad credentials and looks for the particular error from IAM that says
+ * InvalidAccessKeyId or InvalidClientTokenId was used. Quietly returns if we
+ * connect and get the expected error back.
+ *
+ * @param iamUrl the url to check
+ * @throws CloudRuntimeException if there is any unexpected issue.
+ */
+ public static void validateIAMUrl(String iamUrl) {
+ try {
+ AmazonIdentityManagement iamClient = SeaweedFSObjectStoreUtil.getIAMClient(iamUrl, "unknown", "unknown");
+ iamClient.listAccessKeys();
+ } catch (AmazonServiceException e) {
+ if (! StringUtils.equalsAnyIgnoreCase(e.getErrorCode(), "InvalidAccessKeyId", "InvalidClientTokenId", "SignatureDoesNotMatch")) {
+ throw new CloudRuntimeException("Unexpected response from IAM Endpoint.", e);
+ }
+ }
+ }
+
+ /**
+ * Set bucket quota via the SeaweedFS S3 extension endpoint.
+ *
+ * SeaweedFS exposes a custom S3 subresource at
+ * PUT /{bucket}?seaweedfs-quota
+ * authenticated via standard S3 SigV4 and authorized via the
+ * s3:PutBucketQuota IAM permission. This avoids the need for a
+ * separate admin API credential.
+ *
+ * The request body is JSON:
+ * {"quota_size": <n>, "quota_unit": "GB", "quota_enabled": true}
+ *
+ * @param s3Url the S3 endpoint URL (e.g. http://host:8333)
+ * @param accessKey the S3 access key (must have s3:PutBucketQuota permission)
+ * @param secretKey the S3 secret key
+ * @param bucketName the bucket name
+ * @param sizeGiB the quota size in GiB (0 to disable quota)
+ * @throws CloudRuntimeException on any failure
+ */
+ public static void setBucketQuotaViaS3Extension(String s3Url, String accessKey, String secretKey, String bucketName, long sizeGiB) {
+ setBucketQuotaViaS3Extension(s3Url, accessKey, secretKey, bucketName, sizeGiB, newS3ExtensionHttpClient());
+ }
+
+ /**
+ * Build a bounded HTTP client for SeaweedFS S3 extension requests with a
+ * connect timeout so a stalled endpoint cannot block the management-server
+ * API thread indefinitely.
+ */
+ public static java.net.http.HttpClient newS3ExtensionHttpClient() {
+ return java.net.http.HttpClient.newBuilder()
+ .connectTimeout(java.time.Duration.ofSeconds(S3_EXTENSION_CONNECT_TIMEOUT_SECONDS))
+ .build();
+ }
+
+ /**
+ * Set bucket quota via the SeaweedFS S3 extension endpoint using the
+ * supplied HTTP client. The client is injected so tests can assert the
+ * signed request without hitting the network.
+ */
+ public static void setBucketQuotaViaS3Extension(String s3Url, String accessKey, String secretKey,
+ String bucketName, long sizeGiB, java.net.http.HttpClient httpClient) {
+ if (sizeGiB < 0) {
+ // Only zero disables a quota; a negative value would corrupt
+ // resource accounting (BucketApiServiceImpl persists the requested
+ // value and computes deltas from it), so reject it outright.
+ throw new CloudRuntimeException("Bucket quota cannot be negative: " + sizeGiB);
+ }
+ String body;
+ if (sizeGiB == 0) {
+ body = "{\"quota_size\":0,\"quota_unit\":\"B\",\"quota_enabled\":false}";
+ } else {
+ body = String.format("{\"quota_size\":%d,\"quota_unit\":\"GB\",\"quota_enabled\":true}", sizeGiB);
+ }
+ executeSignedS3Request("PUT", s3Url, "/" + bucketName + "?seaweedfs-quota", accessKey, secretKey, body, httpClient);
+ }
+
+ /**
+ * Execute a custom S3 request with SigV4 signing.
+ *
+ * Uses the AWS SDK v1 Aws4Signer to sign the request, then sends it via
+ * java.net.http.HttpClient. This allows calling SeaweedFS-specific S3
+ * extensions (like ?seaweedfs-quota) that the AWS SDK doesn't natively
+ * support.
+ *
+ * The query string portion of {@code resourcePath} (e.g.
+ * {@code /bucket?seaweedfs-quota}) is split off and added to the request
+ * via {@code addParameter(...)} before signing, so the signer includes it
+ * in the canonical query string. {@code DefaultRequest.setResourcePath}
+ * does not parse an embedded query string, so passing it verbatim would
+ * leave the subresource unsigned while the outgoing URI would still carry
+ * it, causing a signature mismatch on the server.
+ *
+ * @param method HTTP method (PUT, GET, etc.)
+ * @param s3Url the S3 endpoint base URL
+ * @param resourcePath the path + optional query string (e.g. /bucket?seaweedfs-quota)
+ * @param accessKey S3 access key
+ * @param secretKey S3 secret key
+ * @param body the request body (null for GET)
+ * @param httpClient the HTTP client used to send the request
+ * @return the response body as a string
+ * @throws CloudRuntimeException on any failure
+ */
+ protected static String executeSignedS3Request(String method, String s3Url, String resourcePath,
+ String accessKey, String secretKey, String body,
+ java.net.http.HttpClient httpClient) {
+ try {
+ java.net.URI endpointUri = java.net.URI.create(s3Url);
+
+ // Split the resource path into a path and a query string so the
+ // query parameters are signed as canonical query parameters.
+ String path = resourcePath;
+ String queryString = "";
+ int q = resourcePath.indexOf('?');
+ if (q >= 0) {
+ path = resourcePath.substring(0, q);
+ queryString = resourcePath.substring(q + 1);
+ }
+
+ // Build AWS SDK v1 Request for SigV4 signing
+ com.amazonaws.DefaultRequest<?> request = new com.amazonaws.DefaultRequest<>("s3");
+ request.setEndpoint(endpointUri);
+ request.setHttpMethod(com.amazonaws.http.HttpMethodName.valueOf(method));
+ request.setResourcePath(path);
Review Comment:
The outgoing URI below includes `endpointPath`, but the AWS SigV4 signer canonicalizes the request resource path set above. For an endpoint such as `https://host/object-s3`, the signature is generated for `/bucket` while the request is sent to `/object-s3/bucket`, which causes `SignatureDoesNotMatch`. Include the endpoint path in the signed resource path or reject path-prefixed endpoints.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r4011076720
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,662 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account and
+ * store. The store ID is included so that two CloudStack pools pointing
+ * at the same SeaweedFS IAM service do not collide on the same
+ * {@code acs-<uuid>} user and overwrite each other's policy and access
+ * keys.
+ */
+ protected String getUserNameForAccount(Account account, long storeId) {
+ return String.format("%s-%d-%s", ACS_PREFIX, storeId, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach a scoped IAM policy that allows access only to this
+ // account's own buckets (the tenant boundary). Refreshed whenever
+ // buckets are created or deleted.
+ updateAccountIAMPolicy(iamClient, storeId, accountId, null);
+
+ // Reuse the stored access key only if both the access key id and the
+ // secret key are present and the key is still Active in IAM; otherwise
+ // create a replacement.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ String storedSecretKey = details.get(secretKeyDetailKey);
+ if (storedAccessKeyId != null && storedSecretKey != null
+ && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ return true;
+ }
+
+ // The stored key is missing, inactive, or no longer in IAM. Clean up
+ // ALL keys (including the inactive stored one) before creating a
+ // replacement so we do not accumulate keys and hit IAM limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, null);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Persist the credentials in the account details (namespaced by storeId)
+ details.put(accessKeyDetailKey, key.getAccessKeyId());
+ details.put(secretKeyDetailKey, key.getSecretAccessKey());
+ _accountDetailsDao.persist(accountId, details);
+
+ // Update existing bucket records for this account/store with the new
+ // credentials so previously created buckets don't keep handing out
+ // the old (now invalid) key pair.
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ }
+
+ /**
+ * Update the IAM credentials on all BucketVO rows for this store/account
+ * so previously created buckets reflect the new (rotated) key pair.
+ * Mirrors CloudianHyperStoreObjectStoreDriverImpl.updateAccountBucketCredentials.
+ */
+ private void updateAccountBucketCredentials(long storeId, long accountId, AccessKey iamCredential) {
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ for (BucketVO bucketVO : bucketList) {
+ logger.info("Updating accountId={} bucket {} with new IAM credentials", accountId, bucketVO.getName());
+ bucketVO.setAccessKey(iamCredential.getAccessKeyId());
+ bucketVO.setSecretKey(iamCredential.getSecretAccessKey());
+ _bucketDao.update(bucketVO.getId(), bucketVO);
+ }
+ }
+
+ /**
+ * Refresh the per-account IAM user policy so it grants S3 access only to
+ * the account's current buckets (optionally excluding one, e.g. a bucket
+ * being deleted). This is the tenant boundary: each account's IAM
+ * credentials can only operate on that account's own buckets.
+ *
+ * @param iamClient the IAM client
+ * @param storeId the object store
+ * @param accountId the CloudStack account
+ * @param excludeBucket a bucket name to omit (e.g. a bucket being deleted),
+ * or null to include all of the account's buckets
+ */
+ protected void updateAccountIAMPolicy(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ return;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ List<BucketVO> buckets = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ List<String> bucketNames = new ArrayList<>();
+ for (BucketVO bvo : buckets) {
+ if (excludeBucket != null && excludeBucket.equals(bvo.getName())) {
+ continue;
+ }
+ bucketNames.add(bvo.getName());
+ }
+ String policy = SeaweedFSObjectStoreUtil.buildAccountIAMPolicy(bucketNames);
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ SeaweedFSObjectStoreUtil.IAM_USER_POLICY_NAME, policy));
+ }
+
+ /**
+ * Check whether the given access key id is still listed and Active in IAM
+ * for the user. Listing failures are propagated rather than swallowed so
+ * a transient IAM outage does not send createUser into the replacement
+ * path (which would overwrite stored credentials and invalidate bucket
+ * records).
+ */
+ private boolean iamAccessKeyExists(AmazonIdentityManagement iamClient, String userName, String accessKeyId) {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ if (accessKeyId.equals(metadata.getAccessKeyId())) {
+ return "Active".equalsIgnoreCase(metadata.getStatus());
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Delete access keys for the user other than the (optionally) preserved
+ * key id. Used to clean up unmanaged leftover keys before creating a
+ * replacement so repeated calls do not hit IAM access-key limits.
+ */
+ private void deleteUnmanagedAccessKeys(AmazonIdentityManagement iamClient, String userName, String preserveAccessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ String keyId = metadata.getAccessKeyId();
+ if (preserveAccessKeyId != null && preserveAccessKeyId.equals(keyId)) {
+ continue;
+ }
+ DeleteAccessKeyRequest deleteReq =
+ new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(keyId);
+ logger.info("Deleting un-managed IAM access key {} for user {}", keyId, userName);
+ iamClient.deleteAccessKey(deleteReq);
+ }
+ } catch (AmazonClientException e) {
+ logger.warn("Failed to clean up IAM access keys for user {}: {}", userName, e.getMessage());
+ }
+ }
+
+ @Override
+ public Bucket createBucket(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+ long accountId = bucket.getAccountId();
+
+ // Use the store's admin credentials to create the bucket
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Check if the bucket already exists
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket already exists with name " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Create the bucket
+ try {
+ CreateBucketRequest request = new CreateBucketRequest(bucketName);
+ if (objectLock) {
+ request.setObjectLockEnabledForBucket(true);
+ }
+ s3client.createBucket(request);
+ } catch (AmazonClientException e) {
+ logger.error("Create bucket failed", e);
+ throw new CloudRuntimeException(e);
+ }
+
+ // Step 2: update the bucket record with the account's IAM credentials.
+ // If this fails, clean up the remote bucket so a retry does not find
+ // it already existing — mirroring the Cloudian createBucket pattern.
+ try {
+ Map<String, String> accountDetails = _accountDetailsDao.findDetails(accountId);
+ String accessKey = accountDetails.get(SeaweedFSObjectStoreUtil.keyAccessKey(storeId));
+ String secretKey = accountDetails.get(SeaweedFSObjectStoreUtil.keySecretKey(storeId));
+ if (accessKey == null || secretKey == null) {
+ logger.warn("No IAM credentials found for account {}. Bucket will be created without per-account credentials.", accountId);
+ }
+
+ String s3Url = getS3Url(storeId);
+ BucketVO bucketVO = _bucketDao.findById(bucket.getId());
+ bucketVO.setAccessKey(accessKey);
+ bucketVO.setSecretKey(secretKey);
+ bucketVO.setBucketURL(s3Url + "/" + bucketName);
+ _bucketDao.update(bucket.getId(), bucketVO);
+
+ // Refresh the account's IAM policy to include the new bucket
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicy(iamClient, storeId, accountId, null);
+
+ return bucket;
+ } catch (Exception e) {
+ logger.error("Post-create bucket record update failed for {}; cleaning up remote bucket", bucketName, e);
+ try {
+ s3client.deleteBucket(bucketName);
+ logger.info("Cleanup of bucket {} succeeded", bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Cleanup of bucket {} also failed", bucketName, cleanupEx);
+ }
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public List<Bucket> listBuckets(long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ List<Bucket> bucketsList = new ArrayList<>();
+ try {
+ List<com.amazonaws.services.s3.model.Bucket> s3Buckets = s3client.listBuckets();
+ for (com.amazonaws.services.s3.model.Bucket s3Bucket : s3Buckets) {
+ Bucket bucket = new BucketObject();
+ bucket.setName(s3Bucket.getName());
+ bucketsList.add(bucket);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ return bucketsList;
+ }
+
+ @Override
+ public boolean deleteBucket(BucketTO bucket, long storeId) {
+ String bucketName = bucket.getName();
+ long accountId = bucket.getAccountId();
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ if (! s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket doesn't exist: " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ try {
+ s3client.deleteBucket(bucketName);
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ // Best-effort: refresh the account's IAM policy to drop the deleted
+ // bucket. The S3 deletion has already succeeded, so a policy refresh
+ // failure must not cause deleteBucket to throw — that would leave
+ // BucketApiServiceImpl with a BucketVO for a bucket that no longer
+ // exists remotely. The stale policy entry is harmless (it grants
+ // access to a non-existent bucket) and will be corrected on the
+ // next create/delete or manually by an operator.
+ try {
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicy(iamClient, storeId, accountId, bucketName);
+ } catch (Exception e) {
+ logger.warn("Failed to refresh IAM policy after deleting bucket {}: {}", bucketName, e.getMessage());
Review Comment:
This best-effort cleanup can leave the old account's IAM policy granting access to the deleted bucket ARN. Bucket names are reusable, so if another account later creates the same name, the old account's credentials can access the new tenant's bucket; the stale grant is not harmless. Make policy removal a prerequisite (or otherwise retry/block reuse until it succeeds) instead of swallowing this failure.
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/util/SeaweedFSObjectStoreUtil.java:
##########
@@ -0,0 +1,454 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.util;
+
+import org.apache.commons.lang3.StringUtils;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagementClientBuilder;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3ClientBuilder;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * Utility class for the SeaweedFS object storage provider.
+ *
+ * SeaweedFS exposes both an S3-compatible API and an AWS IAM-compatible API,
+ * so this provider needs no proprietary admin client — only the AWS S3 and IAM
+ * SDKs, the same pair Cloudian HyperStore already uses in this tree.
+ */
+public class SeaweedFSObjectStoreUtil {
+
+ /** The name of our Object Store Provider */
+ public static final String OBJECT_STORE_PROVIDER_NAME = "SeaweedFS";
+
+ public static final String STORE_KEY_PROVIDER_NAME = "providerName";
+ public static final String STORE_KEY_URL = "url";
+ public static final String STORE_KEY_NAME = "name";
+ public static final String STORE_KEY_SIZE = "size";
+ public static final String STORE_KEY_DETAILS = "details";
+
+ // Store Details Map key names - managed outside of plugin
+ public static final String STORE_DETAILS_KEY_ACCESS_KEY = "accesskey"; // admin/root access key
+ public static final String STORE_DETAILS_KEY_SECRET_KEY = "secretkey"; // admin/root secret key
+ public static final String STORE_DETAILS_KEY_S3_URL = "s3Url"; // S3 endpoint URL
+ public static final String STORE_DETAILS_KEY_IAM_URL = "iamUrl"; // IAM endpoint URL
+
+ // Account Detail Map key names - credentials created per CloudStack account.
+ // Namespaced by store ID so one account can use multiple SeaweedFS pools
+ // without the second pool overwriting the first pool's credentials.
+ public static final String KEY_ACCESS_KEY_PREFIX = "swfs_AccessKey_";
+ public static final String KEY_SECRET_KEY_PREFIX = "swfs_SecretKey_";
+
+ /**
+ * Build the account-detail key for the IAM access key of a given store.
+ */
+ public static String keyAccessKey(long storeId) {
+ return KEY_ACCESS_KEY_PREFIX + storeId;
+ }
+
+ /**
+ * Build the account-detail key for the IAM secret key of a given store.
+ */
+ public static String keySecretKey(long storeId) {
+ return KEY_SECRET_KEY_PREFIX + storeId;
+ }
+
+ /**
+ * Connect timeout for the S3 extension HTTP client, in seconds.
+ */
+ public static final int S3_EXTENSION_CONNECT_TIMEOUT_SECONDS = 10;
+ /**
+ * Per-request timeout for the S3 extension HTTP request, in seconds.
+ */
+ public static final int S3_EXTENSION_REQUEST_TIMEOUT_SECONDS = 30;
+
+ /**
+ * IAM user policy name applied to each per-account IAM user.
+ */
+ public static final String IAM_USER_POLICY_NAME = "CloudStackPolicy";
+
+ /**
+ * Build an IAM user policy that grants full S3 access only to the given
+ * buckets (both the bucket and its contents), while denying bucket
+ * creation and deletion everywhere so CloudStack retains control of the
+ * bucket lifecycle. When no buckets are provided, all S3 access is denied.
+ *
+ * <p>This is the tenant boundary: each account's IAM credentials can only
+ * operate on that account's own buckets, not on every bucket in the
+ * SeaweedFS pool. The policy is refreshed whenever buckets are created or
+ * deleted (see
+ * {@code SeaweedFSObjectStoreDriverImpl.updateAccountIAMPolicy}).
+ *
+ * @param bucketNames the bucket names the account is allowed to access
+ * @return a JSON IAM policy document
+ */
+ public static String buildAccountIAMPolicy(java.util.List<String> bucketNames) {
+ StringBuilder sb = new StringBuilder();
+ sb.append("{\n");
+ sb.append(" \"Version\": \"2012-10-17\",\n");
+ sb.append(" \"Statement\": [\n");
+ if (bucketNames == null || bucketNames.isEmpty()) {
+ // No buckets: deny all S3 access. A Resource cannot be empty in
+ // an IAM policy, so deny everything explicitly.
+ sb.append(" {\n");
+ sb.append(" \"Sid\": \"DenyAllS3\",\n");
+ sb.append(" \"Effect\": \"Deny\",\n");
+ sb.append(" \"Action\": [\"s3:*\"],\n");
+ sb.append(" \"Resource\": [\"arn:aws:s3:::*\", \"arn:aws:s3:::*/*\"]\n");
+ sb.append(" }\n");
+ } else {
+ sb.append(" {\n");
+ sb.append(" \"Sid\": \"AllowAccountBuckets\",\n");
+ sb.append(" \"Effect\": \"Allow\",\n");
+ sb.append(" \"Action\": [\"s3:*\"],\n");
+ sb.append(" \"Resource\": [\n");
+ for (int i = 0; i < bucketNames.size(); i++) {
+ String name = bucketNames.get(i);
+ sb.append(" \"arn:aws:s3:::").append(name).append("\",\n");
+ sb.append(" \"arn:aws:s3:::").append(name).append("/*\"");
+ if (i < bucketNames.size() - 1) {
+ sb.append(",");
+ }
+ sb.append("\n");
+ }
+ sb.append(" ]\n");
+ sb.append(" }\n");
+ }
+ // Always deny bucket creation/deletion and quota mutation —
+ // CloudStack controls lifecycle and resource accounting. Denying
+ // s3:PutBucketQuota prevents a tenant from using the credentials
+ // returned in BucketResponse to call the SeaweedFS quota extension
+ // directly and bypass CloudStack's resource accounting.
+ sb.append(" ,{\n");
+ sb.append(" \"Sid\": \"DenyBucketLifecycleAndQuota\",\n");
+ sb.append(" \"Effect\": \"Deny\",\n");
+ sb.append(" \"Action\": [\"s3:CreateBucket\", \"s3:DeleteBucket\", \"s3:PutBucketQuota\"],\n");
+ sb.append(" \"Resource\": \"*\"\n");
+ sb.append(" }\n");
+ sb.append(" ]\n");
+ sb.append("}\n");
+ return sb.toString();
+ }
+
+ /**
+ * Returns an S3 connection for the given endpoint and credentials.
+ * Uses path-style access, which SeaweedFS requires.
+ *
+ * @param url the url of the S3 service
+ * @param accessKey the credentials to use for the S3 connection.
+ * @param secretKey the matching secret key.
+ * @return an S3 connection (never null)
+ * @throws CloudRuntimeException on failure.
+ */
+ public static AmazonS3 getS3Client(String url, String accessKey, String secretKey) {
+ AmazonS3 client = AmazonS3ClientBuilder.standard()
+ .enablePathStyleAccess()
+ .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))
+ .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(url, "us-east-1"))
+ .build();
+ if (client == null) {
+ throw new CloudRuntimeException("Error while creating SeaweedFS S3 client");
+ }
+ return client;
+ }
+
+ /**
+ * Returns an IAM connection for the given endpoint and credentials.
+ *
+ * @param url the url of the IAM service
+ * @param accessKey the credentials to use for the iam connection.
+ * @param secretKey the matching secret key.
+ * @return an IAM connection (never null)
+ * @throws CloudRuntimeException on failure.
+ */
+ public static AmazonIdentityManagement getIAMClient(String url, String accessKey, String secretKey) {
+ AmazonIdentityManagement iamClient = AmazonIdentityManagementClientBuilder.standard()
+ .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))
+ .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(url, "us-east-1"))
+ .build();
+ if (iamClient == null) {
+ throw new CloudRuntimeException("Error while creating SeaweedFS IAM client");
+ }
+ return iamClient;
+ }
+
+ /**
+ * Test the S3Url to confirm it behaves like an S3 Service.
+ *
+ * Uses bad credentials and looks for the particular error from S3 that says
+ * InvalidAccessKeyId was used. Quietly returns if we connect and get the
+ * expected error back.
+ *
+ * @param s3Url the url to check
+ * @throws CloudRuntimeException if there is any unexpected issue.
+ */
+ public static void validateS3Url(String s3Url) {
+ try {
+ AmazonS3 s3Client = SeaweedFSObjectStoreUtil.getS3Client(s3Url, "unknown", "unknown");
+ s3Client.listBuckets();
+ } catch (AmazonServiceException e) {
+ if (StringUtils.compareIgnoreCase(e.getErrorCode(), "InvalidAccessKeyId") != 0
+ && StringUtils.compareIgnoreCase(e.getErrorCode(), "SignatureDoesNotMatch") != 0) {
+ throw new CloudRuntimeException("Unexpected response from S3 Endpoint.", e);
+ }
+ }
+ }
+
+ /**
+ * Test the IAMUrl to confirm it behaves like an IAM Service.
+ *
+ * Uses bad credentials and looks for the particular error from IAM that says
+ * InvalidAccessKeyId or InvalidClientTokenId was used. Quietly returns if we
+ * connect and get the expected error back.
+ *
+ * @param iamUrl the url to check
+ * @throws CloudRuntimeException if there is any unexpected issue.
+ */
+ public static void validateIAMUrl(String iamUrl) {
+ try {
+ AmazonIdentityManagement iamClient = SeaweedFSObjectStoreUtil.getIAMClient(iamUrl, "unknown", "unknown");
+ iamClient.listAccessKeys();
+ } catch (AmazonServiceException e) {
+ if (! StringUtils.equalsAnyIgnoreCase(e.getErrorCode(), "InvalidAccessKeyId", "InvalidClientTokenId", "SignatureDoesNotMatch")) {
+ throw new CloudRuntimeException("Unexpected response from IAM Endpoint.", e);
+ }
+ }
+ }
+
+ /**
+ * Set bucket quota via the SeaweedFS S3 extension endpoint.
+ *
+ * SeaweedFS exposes a custom S3 subresource at
+ * PUT /{bucket}?seaweedfs-quota
+ * authenticated via standard S3 SigV4 and authorized via the
+ * s3:PutBucketQuota IAM permission. This avoids the need for a
+ * separate admin API credential.
+ *
+ * The request body is JSON:
+ * {"quota_size": <n>, "quota_unit": "GB", "quota_enabled": true}
+ *
+ * @param s3Url the S3 endpoint URL (e.g. http://host:8333)
+ * @param accessKey the S3 access key (must have s3:PutBucketQuota permission)
+ * @param secretKey the S3 secret key
+ * @param bucketName the bucket name
+ * @param sizeGiB the quota size in GiB (0 to disable quota)
+ * @throws CloudRuntimeException on any failure
+ */
+ public static void setBucketQuotaViaS3Extension(String s3Url, String accessKey, String secretKey, String bucketName, long sizeGiB) {
+ setBucketQuotaViaS3Extension(s3Url, accessKey, secretKey, bucketName, sizeGiB, newS3ExtensionHttpClient());
+ }
+
+ /**
+ * Build a bounded HTTP client for SeaweedFS S3 extension requests with a
+ * connect timeout so a stalled endpoint cannot block the management-server
+ * API thread indefinitely.
+ */
+ public static java.net.http.HttpClient newS3ExtensionHttpClient() {
+ return java.net.http.HttpClient.newBuilder()
+ .connectTimeout(java.time.Duration.ofSeconds(S3_EXTENSION_CONNECT_TIMEOUT_SECONDS))
+ .build();
+ }
+
+ /**
+ * Set bucket quota via the SeaweedFS S3 extension endpoint using the
+ * supplied HTTP client. The client is injected so tests can assert the
+ * signed request without hitting the network.
+ */
+ public static void setBucketQuotaViaS3Extension(String s3Url, String accessKey, String secretKey,
+ String bucketName, long sizeGiB, java.net.http.HttpClient httpClient) {
+ if (sizeGiB < 0) {
+ // Only zero disables a quota; a negative value would corrupt
+ // resource accounting (BucketApiServiceImpl persists the requested
+ // value and computes deltas from it), so reject it outright.
+ throw new CloudRuntimeException("Bucket quota cannot be negative: " + sizeGiB);
Review Comment:
Rejecting the negative value only in the provider is too late: `allocBucket` accepts any Integer quota, persists it, and on async failure calls `decrementResourceCount` with the negative quota. That turns cleanup into an increment of the account's object-storage usage, so repeated negative-quota requests corrupt resource accounting. Validate non-negative quotas before reservation or make rollback apply only positive reservations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r4011933289
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/util/SeaweedFSObjectStoreUtil.java:
##########
@@ -0,0 +1,564 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.util;
+
+import org.apache.commons.lang3.StringUtils;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagementClientBuilder;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3ClientBuilder;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * Utility class for the SeaweedFS object storage provider.
+ *
+ * SeaweedFS exposes both an S3-compatible API and an AWS IAM-compatible API,
+ * so this provider needs no proprietary admin client — only the AWS S3 and IAM
+ * SDKs, the same pair Cloudian HyperStore already uses in this tree.
+ */
+public class SeaweedFSObjectStoreUtil {
+
+ /** The name of our Object Store Provider */
+ public static final String OBJECT_STORE_PROVIDER_NAME = "SeaweedFS";
+
+ public static final String STORE_KEY_PROVIDER_NAME = "providerName";
+ public static final String STORE_KEY_URL = "url";
+ public static final String STORE_KEY_NAME = "name";
+ public static final String STORE_KEY_SIZE = "size";
+ public static final String STORE_KEY_DETAILS = "details";
+
+ // Store Details Map key names - managed outside of plugin
+ public static final String STORE_DETAILS_KEY_ACCESS_KEY = "accesskey"; // admin/root access key
+ public static final String STORE_DETAILS_KEY_SECRET_KEY = "secretkey"; // admin/root secret key
+ public static final String STORE_DETAILS_KEY_S3_URL = "s3Url"; // S3 endpoint URL
+ public static final String STORE_DETAILS_KEY_IAM_URL = "iamUrl"; // IAM endpoint URL
+ public static final String STORE_DETAILS_KEY_METRICS_URL = "metricsUrl"; // Prometheus metrics endpoint URL (optional, for scalable usage reporting)
+
+ // Account Detail Map key names - credentials created per CloudStack account.
+ // Namespaced by store ID so one account can use multiple SeaweedFS pools
+ // without the second pool overwriting the first pool's credentials.
+ public static final String KEY_ACCESS_KEY_PREFIX = "swfs_AccessKey_";
+ public static final String KEY_SECRET_KEY_PREFIX = "swfs_SecretKey_";
+
+ /**
+ * Build the account-detail key for the IAM access key of a given store.
+ */
+ public static String keyAccessKey(long storeId) {
+ return KEY_ACCESS_KEY_PREFIX + storeId;
+ }
+
+ /**
+ * Build the account-detail key for the IAM secret key of a given store.
+ */
+ public static String keySecretKey(long storeId) {
+ return KEY_SECRET_KEY_PREFIX + storeId;
+ }
+
+ /**
+ * Connect timeout for the S3 extension HTTP client, in seconds.
+ */
+ public static final int S3_EXTENSION_CONNECT_TIMEOUT_SECONDS = 10;
+ /**
+ * Per-request timeout for the S3 extension HTTP request, in seconds.
+ */
+ public static final int S3_EXTENSION_REQUEST_TIMEOUT_SECONDS = 30;
+
+ /**
+ * IAM user policy name applied to each per-account IAM user.
+ */
+ public static final String IAM_USER_POLICY_NAME = "CloudStackPolicy";
+
+ /**
+ * Build an IAM user policy that grants full S3 access only to the given
+ * buckets (both the bucket and its contents), while denying bucket
+ * creation and deletion everywhere so CloudStack retains control of the
+ * bucket lifecycle. When no buckets are provided, all S3 access is denied.
+ *
+ * <p>This is the tenant boundary: each account's IAM credentials can only
+ * operate on that account's own buckets, not on every bucket in the
+ * SeaweedFS pool. The policy is refreshed whenever buckets are created or
+ * deleted (see
+ * {@code SeaweedFSObjectStoreDriverImpl.updateAccountIAMPolicy}).
+ *
+ * @param bucketNames the bucket names the account is allowed to access
+ * @return a JSON IAM policy document
+ */
+ public static String buildAccountIAMPolicy(java.util.List<String> bucketNames) {
+ StringBuilder sb = new StringBuilder();
+ sb.append("{\n");
+ sb.append(" \"Version\": \"2012-10-17\",\n");
+ sb.append(" \"Statement\": [\n");
+ if (bucketNames == null || bucketNames.isEmpty()) {
+ // No buckets: deny all S3 access. A Resource cannot be empty in
+ // an IAM policy, so deny everything explicitly.
+ sb.append(" {\n");
+ sb.append(" \"Sid\": \"DenyAllS3\",\n");
+ sb.append(" \"Effect\": \"Deny\",\n");
+ sb.append(" \"Action\": [\"s3:*\"],\n");
+ sb.append(" \"Resource\": [\"arn:aws:s3:::*\", \"arn:aws:s3:::*/*\"]\n");
+ sb.append(" }\n");
+ } else {
+ sb.append(" {\n");
+ sb.append(" \"Sid\": \"AllowAccountBuckets\",\n");
+ sb.append(" \"Effect\": \"Allow\",\n");
+ sb.append(" \"Action\": [\"s3:*\"],\n");
+ sb.append(" \"Resource\": [\n");
+ for (int i = 0; i < bucketNames.size(); i++) {
+ String name = bucketNames.get(i);
+ sb.append(" \"arn:aws:s3:::").append(name).append("\",\n");
+ sb.append(" \"arn:aws:s3:::").append(name).append("/*\"");
+ if (i < bucketNames.size() - 1) {
+ sb.append(",");
+ }
+ sb.append("\n");
+ }
+ sb.append(" ]\n");
+ sb.append(" }\n");
+ }
+ // Always deny bucket creation/deletion and quota mutation —
+ // CloudStack controls lifecycle and resource accounting. Denying
+ // s3:PutBucketQuota prevents a tenant from using the credentials
+ // returned in BucketResponse to call the SeaweedFS quota extension
+ // directly and bypass CloudStack's resource accounting.
+ sb.append(" ,{\n");
+ sb.append(" \"Sid\": \"DenyBucketLifecycleAndQuota\",\n");
+ sb.append(" \"Effect\": \"Deny\",\n");
+ sb.append(" \"Action\": [\"s3:CreateBucket\", \"s3:DeleteBucket\", \"s3:PutBucketQuota\"],\n");
+ sb.append(" \"Resource\": \"*\"\n");
+ sb.append(" }\n");
+ sb.append(" ]\n");
+ sb.append("}\n");
+ return sb.toString();
+ }
+
+ /**
+ * Returns an S3 connection for the given endpoint and credentials.
+ * Uses path-style access, which SeaweedFS requires.
+ *
+ * @param url the url of the S3 service
+ * @param accessKey the credentials to use for the S3 connection.
+ * @param secretKey the matching secret key.
+ * @return an S3 connection (never null)
+ * @throws CloudRuntimeException on failure.
+ */
+ public static AmazonS3 getS3Client(String url, String accessKey, String secretKey) {
+ AmazonS3 client = AmazonS3ClientBuilder.standard()
+ .enablePathStyleAccess()
+ .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))
+ .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(url, "us-east-1"))
+ .build();
+ if (client == null) {
+ throw new CloudRuntimeException("Error while creating SeaweedFS S3 client");
+ }
+ return client;
+ }
+
+ /**
+ * Returns an IAM connection for the given endpoint and credentials.
+ *
+ * @param url the url of the IAM service
+ * @param accessKey the credentials to use for the iam connection.
+ * @param secretKey the matching secret key.
+ * @return an IAM connection (never null)
+ * @throws CloudRuntimeException on failure.
+ */
+ public static AmazonIdentityManagement getIAMClient(String url, String accessKey, String secretKey) {
+ AmazonIdentityManagement iamClient = AmazonIdentityManagementClientBuilder.standard()
+ .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))
+ .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(url, "us-east-1"))
+ .build();
+ if (iamClient == null) {
+ throw new CloudRuntimeException("Error while creating SeaweedFS IAM client");
+ }
+ return iamClient;
+ }
+
+ /**
+ * Test the S3Url to confirm it behaves like an S3 Service.
+ *
+ * Uses bad credentials and looks for the particular error from S3 that says
+ * InvalidAccessKeyId was used. Quietly returns if we connect and get the
+ * expected error back.
+ *
+ * @param s3Url the url to check
+ * @throws CloudRuntimeException if there is any unexpected issue.
+ */
+ public static void validateS3Url(String s3Url) {
+ try {
+ AmazonS3 s3Client = SeaweedFSObjectStoreUtil.getS3Client(s3Url, "unknown", "unknown");
+ s3Client.listBuckets();
+ } catch (AmazonServiceException e) {
+ if (StringUtils.compareIgnoreCase(e.getErrorCode(), "InvalidAccessKeyId") != 0
+ && StringUtils.compareIgnoreCase(e.getErrorCode(), "SignatureDoesNotMatch") != 0) {
+ throw new CloudRuntimeException("Unexpected response from S3 Endpoint.", e);
+ }
+ }
+ }
+
+ /**
+ * Test the IAMUrl to confirm it behaves like an IAM Service.
+ *
+ * Uses bad credentials and looks for the particular error from IAM that says
+ * InvalidAccessKeyId or InvalidClientTokenId was used. Quietly returns if we
+ * connect and get the expected error back.
+ *
+ * @param iamUrl the url to check
+ * @throws CloudRuntimeException if there is any unexpected issue.
+ */
+ public static void validateIAMUrl(String iamUrl) {
+ try {
+ AmazonIdentityManagement iamClient = SeaweedFSObjectStoreUtil.getIAMClient(iamUrl, "unknown", "unknown");
+ iamClient.listAccessKeys();
+ } catch (AmazonServiceException e) {
+ if (! StringUtils.equalsAnyIgnoreCase(e.getErrorCode(), "InvalidAccessKeyId", "InvalidClientTokenId", "SignatureDoesNotMatch")) {
+ throw new CloudRuntimeException("Unexpected response from IAM Endpoint.", e);
+ }
+ }
+ }
+
+ /**
+ * Set bucket quota via the SeaweedFS S3 extension endpoint.
+ *
+ * SeaweedFS exposes a custom S3 subresource at
+ * PUT /{bucket}?seaweedfs-quota
+ * authenticated via standard S3 SigV4 and authorized via the
+ * s3:PutBucketQuota IAM permission. This avoids the need for a
+ * separate admin API credential.
+ *
+ * The request body is JSON:
+ * {"quota_size": <n>, "quota_unit": "GB", "quota_enabled": true}
+ *
+ * @param s3Url the S3 endpoint URL (e.g. http://host:8333)
+ * @param accessKey the S3 access key (must have s3:PutBucketQuota permission)
+ * @param secretKey the S3 secret key
+ * @param bucketName the bucket name
+ * @param sizeGiB the quota size in GiB (0 to disable quota)
+ * @throws CloudRuntimeException on any failure
+ */
+ public static void setBucketQuotaViaS3Extension(String s3Url, String accessKey, String secretKey, String bucketName, long sizeGiB) {
+ setBucketQuotaViaS3Extension(s3Url, accessKey, secretKey, bucketName, sizeGiB, newS3ExtensionHttpClient());
+ }
+
+ /**
+ * Build a bounded HTTP client for SeaweedFS S3 extension requests with a
+ * connect timeout so a stalled endpoint cannot block the management-server
+ * API thread indefinitely.
+ */
+ public static java.net.http.HttpClient newS3ExtensionHttpClient() {
+ return java.net.http.HttpClient.newBuilder()
+ .connectTimeout(java.time.Duration.ofSeconds(S3_EXTENSION_CONNECT_TIMEOUT_SECONDS))
+ .build();
+ }
+
+ /**
+ * Set bucket quota via the SeaweedFS S3 extension endpoint using the
+ * supplied HTTP client. The client is injected so tests can assert the
+ * signed request without hitting the network.
+ */
+ public static void setBucketQuotaViaS3Extension(String s3Url, String accessKey, String secretKey,
+ String bucketName, long sizeGiB, java.net.http.HttpClient httpClient) {
+ if (sizeGiB < 0) {
+ // Only zero disables a quota; a negative value would corrupt
+ // resource accounting (BucketApiServiceImpl persists the requested
+ // value and computes deltas from it), so reject it outright.
+ throw new CloudRuntimeException("Bucket quota cannot be negative: " + sizeGiB);
+ }
+ String body;
+ if (sizeGiB == 0) {
+ body = "{\"quota_size\":0,\"quota_unit\":\"B\",\"quota_enabled\":false}";
+ } else {
+ body = String.format("{\"quota_size\":%d,\"quota_unit\":\"GB\",\"quota_enabled\":true}", sizeGiB);
+ }
+ try {
+ executeSignedS3Request("PUT", s3Url, "/" + bucketName + "?seaweedfs-quota", accessKey, secretKey, body, httpClient);
+ } catch (CloudRuntimeException e) {
+ // A quota of 0 disables the quota, which is the default state for
+ // a newly created bucket. If the SeaweedFS quota extension is not
+ // available (404/405), tolerate the failure for quota 0 so basic
+ // bucket CRUD works on deployments without the extension. A
+ // positive quota still requires the extension and must fail.
+ //
+ // Distinguish "extension not available" from "bucket not found":
+ // SeaweedFS returns a standard S3 NoSuchBucket error (with
+ // <Code>NoSuchBucket</Code> in the body) when the bucket does not
+ // exist, which must NOT be swallowed — it indicates CloudStack and
+ // S3 are out of sync.
+ if (sizeGiB == 0 && e.getMessage() != null
+ && (e.getMessage().contains("status 404") || e.getMessage().contains("status 405"))
+ && !e.getMessage().contains("NoSuchBucket")) {
+ org.apache.logging.log4j.LogManager.getLogger(SeaweedFSObjectStoreUtil.class)
+ .warn("SeaweedFS quota extension not available for bucket {}; skipping quota disable (quota is already off by default)", bucketName);
+ return;
+ }
+ throw e;
+ }
+ }
+
+ /**
+ * Execute a custom S3 request with SigV4 signing.
+ *
+ * Uses the AWS SDK v1 Aws4Signer to sign the request, then sends it via
+ * java.net.http.HttpClient. This allows calling SeaweedFS-specific S3
+ * extensions (like ?seaweedfs-quota) that the AWS SDK doesn't natively
+ * support.
+ *
+ * The query string portion of {@code resourcePath} (e.g.
+ * {@code /bucket?seaweedfs-quota}) is split off and added to the request
+ * via {@code addParameter(...)} before signing, so the signer includes it
+ * in the canonical query string. {@code DefaultRequest.setResourcePath}
+ * does not parse an embedded query string, so passing it verbatim would
+ * leave the subresource unsigned while the outgoing URI would still carry
+ * it, causing a signature mismatch on the server.
+ *
+ * @param method HTTP method (PUT, GET, etc.)
+ * @param s3Url the S3 endpoint base URL
+ * @param resourcePath the path + optional query string (e.g. /bucket?seaweedfs-quota)
+ * @param accessKey S3 access key
+ * @param secretKey S3 secret key
+ * @param body the request body (null for GET)
+ * @param httpClient the HTTP client used to send the request
+ * @return the response body as a string
+ * @throws CloudRuntimeException on any failure
+ */
+ protected static String executeSignedS3Request(String method, String s3Url, String resourcePath,
+ String accessKey, String secretKey, String body,
+ java.net.http.HttpClient httpClient) {
+ try {
+ java.net.URI endpointUri = java.net.URI.create(s3Url);
+
+ // Split the resource path into a path and a query string so the
+ // query parameters are signed as canonical query parameters.
+ String path = resourcePath;
+ String queryString = "";
+ int q = resourcePath.indexOf('?');
+ if (q >= 0) {
+ path = resourcePath.substring(0, q);
+ queryString = resourcePath.substring(q + 1);
+ }
+
+ // The AWS SDK v1 AWS4Signer already combines the endpoint path
+ // (request.getEndpoint().getPath()) with the resource path
+ // (request.getResourcePath()) via SdkHttpUtils.appendUri when
+ // building the canonical URI. Set the resource path to just the
+ // bucket/key path (e.g. /bucket) and let the signer prepend the
+ // endpoint path prefix (e.g. /object-s3). The outgoing URI must
+ // also include the endpoint path so the server sees the same path
+ // the signer canonicalized.
+ String endpointPath = endpointUri.getPath();
+ if (endpointPath == null) {
+ endpointPath = "";
+ }
+ if (endpointPath.endsWith("/")) {
+ endpointPath = endpointPath.substring(0, endpointPath.length() - 1);
+ }
+
+ // Build AWS SDK v1 Request for SigV4 signing
+ com.amazonaws.DefaultRequest<?> request = new com.amazonaws.DefaultRequest<>("s3");
+ request.setEndpoint(endpointUri);
+ request.setHttpMethod(com.amazonaws.http.HttpMethodName.valueOf(method));
+ request.setResourcePath(path);
+ if (! queryString.isEmpty()) {
+ for (String pair : queryString.split("&")) {
+ if (pair.isEmpty()) {
+ continue;
+ }
+ int eq = pair.indexOf('=');
+ if (eq >= 0) {
+ request.addParameter(pair.substring(0, eq), pair.substring(eq + 1));
+ } else {
+ request.addParameter(pair, "");
+ }
+ }
+ }
+ if (body != null) {
+ byte[] bodyBytes = body.getBytes(java.nio.charset.StandardCharsets.UTF_8);
+ request.setContent(new java.io.ByteArrayInputStream(bodyBytes));
+ request.getHeaders().put("Content-Length", String.valueOf(bodyBytes.length));
+ request.getHeaders().put("Content-Type", "application/json");
+ }
+
+ // Sign with SigV4 (AWSS3V4Signer, not the legacy S3Signer which is SigV2)
+ com.amazonaws.auth.AWSCredentials credentials = new com.amazonaws.auth.BasicAWSCredentials(accessKey, secretKey);
+ com.amazonaws.services.s3.internal.AWSS3V4Signer signer = new com.amazonaws.services.s3.internal.AWSS3V4Signer();
+ signer.setServiceName("s3");
+ signer.setRegionName("us-east-1");
+ signer.sign(request, credentials);
+
+ // Build and send the HTTP request with signed headers. The URI
+ // carries the original query string; the signed headers (including
+ // Authorization) are copied from the signed request. Restricted
+ // headers (e.g. Content-Length, Host) are set by the HTTP client /
+ // URI itself and cannot be added via HttpRequest.Builder.header(),
+ // so they are skipped here.
+ // Build the outgoing URI preserving the endpoint path prefix (e.g.
+ // https://host/object-s3) by concatenating it with the resource
+ // path. The signer internally combines the endpoint path with the
+ // resource path to form the same canonical URI, so SigV4 verifies.
+ java.net.URI fullUri = java.net.URI.create(
+ endpointUri.getScheme() + "://" + endpointUri.getRawAuthority()
+ + endpointPath + path);
+ if (! queryString.isEmpty()) {
+ fullUri = java.net.URI.create(fullUri.toString() + "?" + queryString);
+ }
+ java.net.http.HttpRequest.Builder reqBuilder = java.net.http.HttpRequest.newBuilder()
+ .uri(fullUri)
+ .timeout(java.time.Duration.ofSeconds(S3_EXTENSION_REQUEST_TIMEOUT_SECONDS));
+ for (java.util.Map.Entry<String, String> entry : request.getHeaders().entrySet()) {
+ String headerName = entry.getKey();
+ if (headerName == null || entry.getValue() == null) {
+ continue;
+ }
+ if (isRestrictedHttpHeader(headerName)) {
+ continue;
+ }
+ reqBuilder.header(headerName, entry.getValue());
+ }
+ if (body != null) {
+ reqBuilder.method(method, java.net.http.HttpRequest.BodyPublishers.ofString(body));
+ } else {
+ reqBuilder.method(method, java.net.http.HttpRequest.BodyPublishers.noBody());
+ }
+
+ java.net.http.HttpResponse<String> response = httpClient.send(reqBuilder.build(),
+ java.net.http.HttpResponse.BodyHandlers.ofString());
+
+ int statusCode = response.statusCode();
+ if (statusCode < 200 || statusCode >= 300) {
+ throw new CloudRuntimeException(String.format(
+ "S3 extension request %s %s failed with status %d: %s",
+ method, fullUri, statusCode, response.body()));
+ }
+ return response.body();
+ } catch (CloudRuntimeException e) {
+ throw e;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("S3 extension request failed: " + method + " " + resourcePath, e);
+ }
+ }
+
+ /**
+ * Headers that {@code java.net.http.HttpRequest.Builder.header()} rejects
+ * because they are managed by the HTTP client itself (content length is
+ * derived from the body publisher, host from the URI, etc.). They must be
+ * skipped when copying the signed headers onto the outgoing request.
+ */
+ private static boolean isRestrictedHttpHeader(String headerName) {
+ if (headerName == null) {
+ return true;
+ }
+ switch (headerName.toLowerCase(java.util.Locale.ROOT)) {
+ case "content-length":
+ case "host":
+ case "connection":
+ case "expect":
+ case "upgrade":
+ return true;
+ default:
+ return false;
+ }
+ }
+
+ /**
+ * Prometheus metric name for per-bucket logical size. SeaweedFS publishes
+ * this gauge from the S3 API server's bucket-size metrics loop.
+ */
+ public static final String METRIC_BUCKET_SIZE_BYTES = "SeaweedFS_s3_bucket_size_bytes";
+
+ /**
+ * Scrape the SeaweedFS Prometheus {@code /metrics} endpoint and return a
+ * map of bucket name to logical size in bytes.
+ *
+ * <p>This is a single HTTP GET that returns all bucket sizes in O(buckets)
+ * time, replacing the O(total objects) {@code ListObjectsV2} scan used as a
+ * fallback. The operator configures {@code metricsUrl} as a store detail
+ * pointing at the SeaweedFS S3 server's metrics port (or a Prometheus
+ * server that scrapes it).
+ *
+ * @param metricsUrl the base URL of the Prometheus metrics endpoint
+ * @param bucketNames the set of bucket names CloudStack manages (used to
+ * filter the scraped metrics; buckets not in this set
+ * are ignored)
+ * @param httpClient the HTTP client used to send the request
+ * @return a map of bucket name to size in bytes. Every bucket in
+ * {@code bucketNames} is present; buckets not found in the
+ * metrics response are set to 0 so stale sizes from a previous
+ * scan are overwritten.
+ * @throws CloudRuntimeException on any HTTP or parse failure
+ */
+ public static java.util.Map<String, Long> parseBucketUsageFromMetrics(String metricsUrl,
+ java.util.Set<String> bucketNames, java.net.http.HttpClient httpClient) {
+ java.util.Map<String, Long> result = new java.util.HashMap<>();
+ // Initialize all managed buckets to zero so missing samples do not
+ // leave stale sizes from a previous scan in BucketApiServiceImpl.
+ for (String name : bucketNames) {
+ result.put(name, 0L);
+ }
Review Comment:
Initializing every managed bucket to `0` and silently skipping malformed samples makes an incomplete HTTP-200 scrape authoritative. A newly created bucket can be absent until SeaweedFS's periodic metrics refresh, and this value is then persisted by `BucketApiServiceImpl` and included in `usedSize`, under-reporting capacity and quota usage; the existing `getAllBucketsUsage` fallback only runs when an exception is thrown. Treat missing/invalid managed-bucket samples as a scrape failure so the caller falls back to `ListObjectsV2` (or preserves the last known value).
This issue also appears on line 553 of the same file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r4011191337
##########
plugins/storage/object/seaweedfs/src/test/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImplTest.java:
##########
@@ -0,0 +1,746 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.Mockito.anyLong;
Review Comment:
These imports expose the same anyLong() matcher through both ArgumentMatchers and Mockito, so the unqualified anyLong() calls in this test are ambiguous to javac. Keep only one of the imports.
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,730 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ /**
+ * DB-backed global lock name prefix for serializing IAM provisioning and
+ * policy refreshes per store+account. Uses {@link GlobalLock} so the
+ * critical section is serialized across management servers in a
+ * clustered deployment, not just within a single JVM.
+ */
+ private static final String IAM_LOCK_PREFIX = "seaweedfs.iam.";
+
+ private static String getIamLockName(long storeId, long accountId) {
+ return IAM_LOCK_PREFIX + storeId + "." + accountId;
+ }
+
+ /**
+ * Acquire a DB-backed global lock for IAM operations on the given
+ * store+account. Returns a {@link GlobalLock} that the caller must
+ * {@link GlobalLock#unlock()} in a {@code finally} block, or {@code null}
+ * if the lock could not be acquired within the timeout.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ */
+ protected GlobalLock acquireIamLock(long storeId, long accountId) {
+ GlobalLock lock = GlobalLock.getInternLock(getIamLockName(storeId, accountId));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire IAM lock for store {} account {}", storeId, accountId);
+ return null;
+ }
+ return lock;
+ }
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account and
+ * store. The store ID is included so that two CloudStack pools pointing
+ * at the same SeaweedFS IAM service do not collide on the same
+ * {@code acs-<uuid>} user and overwrite each other's policy and access
+ * keys.
+ */
+ protected String getUserNameForAccount(Account account, long storeId) {
+ return String.format("%s-%d-%s", ACS_PREFIX, storeId, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Serialize per store+account across management servers so two
+ // concurrent bucket requests do not both rotate credentials and leave
+ // bucket rows with mismatched key pairs.
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ return false;
+ }
+ try {
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach a scoped IAM policy that allows access only to this
+ // account's own buckets (the tenant boundary). Refreshed whenever
+ // buckets are created or deleted.
+ updateAccountIAMPolicy(iamClient, storeId, accountId, null);
+
+ // Reuse the stored access key only if both the access key id and the
+ // secret key are present and the key is still Active in IAM; otherwise
+ // create a replacement.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ String storedSecretKey = details.get(secretKeyDetailKey);
+ if (storedAccessKeyId != null && storedSecretKey != null
+ && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ return true;
+ }
+
+ // The stored key is missing, inactive, or no longer in IAM. Clean up
+ // ALL keys (including the inactive stored one) before creating a
+ // replacement so we do not accumulate keys and hit IAM limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, null);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Update existing bucket records for this account/store with the new
+ // credentials BEFORE persisting the new key in account details. If a
+ // bucket update fails, the stored key remains the old one and a retry
+ // will re-enter the replacement path; if we persisted first, a retry
+ // would see the new stored key and return without repairing the
+ // remaining buckets.
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ // Persist the credentials in the account details (namespaced by storeId)
+ details.put(accessKeyDetailKey, key.getAccessKeyId());
+ details.put(secretKeyDetailKey, key.getSecretAccessKey());
+ _accountDetailsDao.persist(accountId, details);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ } finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Update the IAM credentials on all BucketVO rows for this store/account
+ * so previously created buckets reflect the new (rotated) key pair.
+ * Mirrors CloudianHyperStoreObjectStoreDriverImpl.updateAccountBucketCredentials.
+ */
+ private void updateAccountBucketCredentials(long storeId, long accountId, AccessKey iamCredential) {
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ for (BucketVO bucketVO : bucketList) {
+ logger.info("Updating accountId={} bucket {} with new IAM credentials", accountId, bucketVO.getName());
+ bucketVO.setAccessKey(iamCredential.getAccessKeyId());
+ bucketVO.setSecretKey(iamCredential.getSecretAccessKey());
+ _bucketDao.update(bucketVO.getId(), bucketVO);
+ }
+ }
+
+ /**
+ * Refresh the per-account IAM user policy so it grants S3 access only to
+ * the account's current buckets (optionally excluding one, e.g. a bucket
+ * being deleted). This is the tenant boundary: each account's IAM
+ * credentials can only operate on that account's own buckets.
+ *
+ * @param iamClient the IAM client
+ * @param storeId the object store
+ * @param accountId the CloudStack account
+ * @param excludeBucket a bucket name to omit (e.g. a bucket being deleted),
+ * or null to include all of the account's buckets
+ */
+ protected void updateAccountIAMPolicy(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ return;
+ }
Review Comment:
`updateAccountIAMPolicy` returns normally when the per-account lock cannot be acquired. Both `createBucket` and `deleteBucket` then continue as if the IAM policy was refreshed: creation leaves the new bucket inaccessible to its account, while deletion can remove the remote bucket/CloudStack row with the old grant still active. Propagate the lock failure (or return a failure that callers honor) instead of treating it as success.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r4011433500
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,789 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.BucketCrossOriginConfiguration;
+import com.amazonaws.services.s3.model.CORSRule;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketCrossOriginConfigurationRequest;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ /**
+ * DB-backed global lock name prefix for serializing IAM provisioning and
+ * policy refreshes per store+account. Uses {@link GlobalLock} so the
+ * critical section is serialized across management servers in a
+ * clustered deployment, not just within a single JVM.
+ */
+ private static final String IAM_LOCK_PREFIX = "seaweedfs.iam.";
+
+ private static String getIamLockName(long storeId, long accountId) {
+ return IAM_LOCK_PREFIX + storeId + "." + accountId;
+ }
+
+ /**
+ * Acquire a DB-backed global lock for IAM operations on the given
+ * store+account. Returns a {@link GlobalLock} that the caller must
+ * {@link GlobalLock#unlock()} in a {@code finally} block, or {@code null}
+ * if the lock could not be acquired within the timeout.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ */
+ protected GlobalLock acquireIamLock(long storeId, long accountId) {
+ GlobalLock lock = GlobalLock.getInternLock(getIamLockName(storeId, accountId));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire IAM lock for store {} account {}", storeId, accountId);
+ lock.releaseRef();
+ return null;
+ }
+ return lock;
+ }
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account and
+ * store. The store ID is included so that two CloudStack pools pointing
+ * at the same SeaweedFS IAM service do not collide on the same
+ * {@code acs-<uuid>} user and overwrite each other's policy and access
+ * keys.
+ */
+ protected String getUserNameForAccount(Account account, long storeId) {
+ return String.format("%s-%d-%s", ACS_PREFIX, storeId, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Serialize per store+account across management servers so two
+ // concurrent bucket requests do not both rotate credentials and leave
+ // bucket rows with mismatched key pairs.
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ return false;
+ }
+ try {
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach a scoped IAM policy that allows access only to this
+ // account's own buckets (the tenant boundary). Refreshed whenever
+ // buckets are created or deleted.
+ updateAccountIAMPolicy(iamClient, storeId, accountId, null);
+
+ // Reuse the stored access key only if both the access key id and the
+ // secret key are present and the key is still Active in IAM; otherwise
+ // create a replacement.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ String storedSecretKey = details.get(secretKeyDetailKey);
+ if (storedAccessKeyId != null && storedSecretKey != null
+ && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ return true;
+ }
+
+ // The stored key is missing, inactive, or no longer in IAM. Clean up
+ // ALL keys (including the inactive stored one) before creating a
+ // replacement so we do not accumulate keys and hit IAM limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, null);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Update existing bucket records for this account/store with the new
+ // credentials BEFORE persisting the new key in account details. If a
+ // bucket update fails, the stored key remains the old one and a retry
+ // will re-enter the replacement path; if we persisted first, a retry
+ // would see the new stored key and return without repairing the
+ // remaining buckets.
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ // Persist the credentials in the account details (namespaced by storeId)
+ details.put(accessKeyDetailKey, key.getAccessKeyId());
+ details.put(secretKeyDetailKey, key.getSecretAccessKey());
+ _accountDetailsDao.persist(accountId, details);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Update the IAM credentials on all BucketVO rows for this store/account
+ * so previously created buckets reflect the new (rotated) key pair.
+ * Mirrors CloudianHyperStoreObjectStoreDriverImpl.updateAccountBucketCredentials.
+ */
+ private void updateAccountBucketCredentials(long storeId, long accountId, AccessKey iamCredential) {
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ for (BucketVO bucketVO : bucketList) {
+ logger.info("Updating accountId={} bucket {} with new IAM credentials", accountId, bucketVO.getName());
+ bucketVO.setAccessKey(iamCredential.getAccessKeyId());
+ bucketVO.setSecretKey(iamCredential.getSecretAccessKey());
+ _bucketDao.update(bucketVO.getId(), bucketVO);
+ }
+ }
+
+ /**
+ * Refresh the per-account IAM user policy so it grants S3 access only to
+ * the account's current buckets (optionally excluding one, e.g. a bucket
+ * being deleted). This is the tenant boundary: each account's IAM
+ * credentials can only operate on that account's own buckets.
+ *
+ * @param iamClient the IAM client
+ * @param storeId the object store
+ * @param accountId the CloudStack account
+ * @param excludeBucket a bucket name to omit (e.g. a bucket being deleted),
+ * or null to include all of the account's buckets
+ */
+ protected void updateAccountIAMPolicy(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ return;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ List<BucketVO> buckets = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ List<String> bucketNames = new ArrayList<>();
+ for (BucketVO bvo : buckets) {
+ if (excludeBucket != null && excludeBucket.equals(bvo.getName())) {
+ continue;
+ }
+ bucketNames.add(bvo.getName());
+ }
+ String policy = SeaweedFSObjectStoreUtil.buildAccountIAMPolicy(bucketNames);
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ SeaweedFSObjectStoreUtil.IAM_USER_POLICY_NAME, policy));
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Check whether the given access key id is still listed and Active in IAM
+ * for the user. Listing failures are propagated rather than swallowed so
+ * a transient IAM outage does not send createUser into the replacement
+ * path (which would overwrite stored credentials and invalidate bucket
+ * records).
+ */
+ private boolean iamAccessKeyExists(AmazonIdentityManagement iamClient, String userName, String accessKeyId) {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ if (accessKeyId.equals(metadata.getAccessKeyId())) {
+ return "Active".equalsIgnoreCase(metadata.getStatus());
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Delete access keys for the user other than the (optionally) preserved
+ * key id. Used to clean up unmanaged leftover keys before creating a
+ * replacement so repeated calls do not hit IAM access-key limits.
+ */
+ private void deleteUnmanagedAccessKeys(AmazonIdentityManagement iamClient, String userName, String preserveAccessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ String keyId = metadata.getAccessKeyId();
+ if (preserveAccessKeyId != null && preserveAccessKeyId.equals(keyId)) {
+ continue;
+ }
+ DeleteAccessKeyRequest deleteReq =
+ new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(keyId);
+ logger.info("Deleting un-managed IAM access key {} for user {}", keyId, userName);
+ iamClient.deleteAccessKey(deleteReq);
+ }
+ } catch (AmazonClientException e) {
+ // Propagate so the caller does not proceed to create a replacement
+ // key while stale unmanaged keys remain (which could hit IAM key
+ // limits or leave orphaned credentials).
+ throw new CloudRuntimeException("Failed to clean up IAM access keys for user " + userName, e);
+ }
+ }
+
+ @Override
+ public Bucket createBucket(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+ long accountId = bucket.getAccountId();
+
+ // Use the store's admin credentials to create the bucket
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Check if the bucket already exists
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket already exists with name " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Create the bucket
+ try {
+ CreateBucketRequest request = new CreateBucketRequest(bucketName);
+ if (objectLock) {
+ request.setObjectLockEnabledForBucket(true);
+ }
+ s3client.createBucket(request);
+ } catch (AmazonClientException e) {
+ logger.error("Create bucket failed", e);
+ throw new CloudRuntimeException(e);
+ }
+
+ // Step 2: update the bucket record with the account's IAM credentials,
+ // configure CORS, and refresh the IAM policy. If any of these fail,
+ // clean up the remote bucket so a retry does not find it already
+ // existing — mirroring the Cloudian createBucket pattern.
+ //
+ // Hold the IAM lock for the account so a concurrent createUser key
+ // rotation does not change the account credentials between reading
+ // them and writing the BucketVO, which would leave the bucket with
+ // a stale key pair.
+ GlobalLock iamLock = acquireIamLock(storeId, accountId);
+ if (iamLock == null) {
+ // The S3 bucket has already been created. Clean it up so a
+ // retry does not find it already existing, then throw.
+ try {
+ s3client.deleteBucket(bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Failed to clean up bucket {} after IAM lock timeout", bucketName, cleanupEx);
+ }
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ // Configure permissive CORS so the CloudStack S3 bucket browser
+ // (which performs list/upload/delete from the browser) can function.
+ // SeaweedFS supports the standard PutBucketCors operation.
+ configureBucketCORS(s3client, bucketName);
+
+ Map<String, String> accountDetails = _accountDetailsDao.findDetails(accountId);
+ String accessKey = accountDetails.get(SeaweedFSObjectStoreUtil.keyAccessKey(storeId));
+ String secretKey = accountDetails.get(SeaweedFSObjectStoreUtil.keySecretKey(storeId));
+ if (accessKey == null || secretKey == null) {
+ throw new CloudRuntimeException("No IAM credentials found for account " + accountId
+ + " on store " + storeId + ". Run createUser before creating a bucket.");
+ }
+
+ String s3Url = getS3Url(storeId);
+ BucketVO bucketVO = _bucketDao.findById(bucket.getId());
+ bucketVO.setAccessKey(accessKey);
+ bucketVO.setSecretKey(secretKey);
+ bucketVO.setBucketURL(s3Url + "/" + bucketName);
+ _bucketDao.update(bucket.getId(), bucketVO);
+
+ // Refresh the account's IAM policy to include the new bucket
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicy(iamClient, storeId, accountId, null);
+
+ return bucket;
+ } catch (Exception e) {
+ logger.error("Post-create bucket record update failed for {}; cleaning up remote bucket", bucketName, e);
+ try {
+ s3client.deleteBucket(bucketName);
+ logger.info("Cleanup of bucket {} succeeded", bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Cleanup of bucket {} also failed", bucketName, cleanupEx);
+ }
+ // Revoke the IAM policy grant for the new bucket so the account's
+ // credentials cannot access a bucket that no longer exists. If the
+ // policy PUT succeeded before the DB update failed, the grant
+ // would otherwise persist and could be reused if another account
+ // later creates the same bucket name.
+ try {
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicy(iamClient, storeId, accountId, bucketName);
+ } catch (Exception policyEx) {
+ logger.warn("Failed to revoke IAM policy for bucket {} after cleanup: {}", bucketName, policyEx.getMessage());
+ }
+ throw new CloudRuntimeException(e);
+ } finally {
+ iamLock.unlock();
+ iamLock.releaseRef();
+ }
+ }
+
+ /**
+ * Configure a permissive CORS policy on the bucket so the CloudStack
+ * S3 bucket browser (which performs list/upload/delete from the
+ * browser) can function. Mirrors the Cloudian configureBucketCORS.
+ */
+ private void configureBucketCORS(AmazonS3 s3client, String bucketName) {
+ logger.debug("Configuring CORS for bucket {}", bucketName);
+ List<CORSRule> corsRules = new ArrayList<>();
+ CORSRule allowAnyRule = new CORSRule().withId("AllowAny");
+ allowAnyRule.setAllowedOrigins("*");
+ allowAnyRule.setAllowedHeaders("*");
+ allowAnyRule.setAllowedMethods(
+ CORSRule.AllowedMethods.HEAD,
+ CORSRule.AllowedMethods.GET,
+ CORSRule.AllowedMethods.PUT,
+ CORSRule.AllowedMethods.POST,
+ CORSRule.AllowedMethods.DELETE);
+ corsRules.add(allowAnyRule);
+ BucketCrossOriginConfiguration corsConfig = new BucketCrossOriginConfiguration();
+ corsConfig.setRules(corsRules);
+ SetBucketCrossOriginConfigurationRequest corsRequest = new SetBucketCrossOriginConfigurationRequest(bucketName, corsConfig);
+ s3client.setBucketCrossOriginConfiguration(corsRequest);
+ logger.info("Successfully configured CORS for bucket {}", bucketName);
+ }
+
+ @Override
+ public List<Bucket> listBuckets(long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ List<Bucket> bucketsList = new ArrayList<>();
+ try {
+ List<com.amazonaws.services.s3.model.Bucket> s3Buckets = s3client.listBuckets();
+ for (com.amazonaws.services.s3.model.Bucket s3Bucket : s3Buckets) {
+ Bucket bucket = new BucketObject();
+ bucket.setName(s3Bucket.getName());
+ bucketsList.add(bucket);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ return bucketsList;
+ }
+
+ @Override
+ public boolean deleteBucket(BucketTO bucket, long storeId) {
+ String bucketName = bucket.getName();
+ long accountId = bucket.getAccountId();
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Delete the S3 bucket first. If this fails (non-empty bucket,
+ // transient error), the IAM policy is still intact so the user
+ // can empty the bucket and retry.
+ //
+ // If the bucket is already gone (e.g. from a previous partial
+ // failure where the S3 delete succeeded but the IAM policy refresh
+ // failed), skip the S3 delete so the retry is idempotent.
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ s3client.deleteBucket(bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Refresh the account's IAM policy to drop the deleted bucket.
+ // Bucket names are reusable, so a stale grant would let the old
+ // account access a new tenant's bucket with the same name. This
+ // must succeed; if it fails, the caller sees the exception and can
+ // retry (the policy refresh is idempotent because the bucket is
+ // already gone from S3 and excludeBucket still applies).
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicy(iamClient, storeId, accountId, bucketName);
Review Comment:
This refresh is not atomic with CloudStack's deletion of the `BucketVO`: `BucketApiServiceImpl` removes that row only after this driver method returns. A concurrent delete/create or `createUser` can rebuild the policy while the row is still present, re-adding the bucket ARN after this exclusion; the final row removal performs no further refresh. If the name is later reused, the old account's credentials can access the new tenant's bucket. Coordinate policy refresh with row removal (or refresh after removal) under the same per-account serialization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "chrislusf (via GitHub)" <gi...@apache.org>.
chrislusf commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r4017612731
##########
server/src/main/java/org/apache/cloudstack/storage/object/BucketApiServiceImpl.java:
##########
@@ -313,22 +332,60 @@ private void updateBucketQuota(UpdateBucketCmd cmd, BucketVO bucket, ObjectStore
return;
}
- int quotaDelta = quota - bucket.getQuota();
- objectStore.setQuota(bucketTO, quota);
- bucket.setQuota(quota);
-
+ int previousQuota = ObjectUtils.defaultIfNull(bucket.getQuota(), 0);
+ int quotaDelta = quota - previousQuota;
long diff = quotaDelta * Resource.ResourceType.bytesToGiB;
if (quotaDelta < 0) {
- resourceLimitManager.decrementResourceCount(bucket.getAccountId(), Resource.ResourceType.object_storage, Math.abs(diff));
- _objectStoreDao.updateAllocatedSize(objectStoreVO, diff);
+ // A decrease needs no reservation. Apply it remotely first, then
+ // compensate the backend if the accounting update fails so the two
+ // cannot diverge.
+ objectStore.setQuota(bucketTO, quota);
+ try {
+ resourceLimitManager.decrementResourceCount(bucket.getAccountId(), Resource.ResourceType.object_storage, Math.abs(diff));
+ _objectStoreDao.updateAllocatedSize(objectStoreVO, diff);
Review Comment:
Fixed in `932cc3c`. Quota updates now check the `updateAllocatedSize` result, compensate the `object_storage` count if that update fails, restore the remote quota, and reset the in-memory quota before rethrowing. The successful path persists the bucket quota inside the quota helper and skips the later duplicate DAO update. Added a regression test for this rollback path.
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,938 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.BucketCrossOriginConfiguration;
+import com.amazonaws.services.s3.model.CORSRule;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketCrossOriginConfigurationRequest;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ /**
+ * DB-backed global lock name prefix for serializing IAM provisioning and
+ * policy refreshes per store+account. Uses {@link GlobalLock} so the
+ * critical section is serialized across management servers in a
+ * clustered deployment, not just within a single JVM.
+ */
+ private static final String IAM_LOCK_PREFIX = "seaweedfs.iam.";
+
+ private static String getIamLockName(long storeId, long accountId) {
+ return IAM_LOCK_PREFIX + storeId + "." + accountId;
+ }
+
+ /**
+ * Acquire a DB-backed global lock for IAM operations on the given
+ * store+account. Returns a {@link GlobalLock} that the caller must
+ * {@link GlobalLock#unlock()} in a {@code finally} block, or {@code null}
+ * if the lock could not be acquired within the timeout.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ */
+ protected GlobalLock acquireIamLock(long storeId, long accountId) {
+ GlobalLock lock = GlobalLock.getInternLock(getIamLockName(storeId, accountId));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire IAM lock for store {} account {}", storeId, accountId);
+ lock.releaseRef();
+ return null;
+ }
+ return lock;
+ }
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account and
+ * store. The store ID is included so that two CloudStack pools pointing
+ * at the same SeaweedFS IAM service do not collide on the same
+ * {@code acs-<uuid>} user and overwrite each other's policy and access
+ * keys.
+ */
+ protected String getUserNameForAccount(Account account, long storeId) {
+ return String.format("%s-%d-%s", ACS_PREFIX, storeId, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Serialize per store+account across management servers so two
+ // concurrent bucket requests do not both rotate credentials and leave
+ // bucket rows with mismatched key pairs.
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ return false;
+ }
+ try {
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach a scoped IAM policy that allows access only to this
+ // account's own buckets (the tenant boundary). Refreshed whenever
+ // buckets are created or deleted. Use the lock-free variant since
+ // createUser already holds the IAM lock.
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, null);
+
+ // Reuse the stored access key only if both the access key id and the
+ // secret key are present and the key is still Active in IAM; otherwise
+ // create a replacement.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ String storedSecretKey = details.get(secretKeyDetailKey);
+ if (storedAccessKeyId != null && storedSecretKey != null
+ && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ return true;
+ }
+
+ // The stored key is missing, inactive, or no longer in IAM. Clean up
+ // ALL keys (including the inactive stored one) before creating a
+ // replacement so we do not accumulate keys and hit IAM limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, null);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Update existing bucket records for this account/store with the new
+ // credentials BEFORE persisting the new key in account details. If a
+ // bucket update fails, the stored key remains the old one and a retry
+ // will re-enter the replacement path; if we persisted first, a retry
+ // would see the new stored key and return without repairing the
+ // remaining buckets.
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ // Persist the credentials in the account details (namespaced by
+ // storeId) with per-key writes. AccountDetailsDao.persist(accountId,
+ // map) expunges every existing detail for the account before inserting
+ // the supplied map, so it would clobber details this snapshot never saw
+ // — including another object store's namespaced credentials, since the
+ // IAM lock is keyed by storeId+accountId and two pools can provision
+ // the same account concurrently. addDetail touches only the named key.
+ details.put(accessKeyDetailKey, key.getAccessKeyId());
+ details.put(secretKeyDetailKey, key.getSecretAccessKey());
+ _accountDetailsDao.addDetail(accountId, accessKeyDetailKey, key.getAccessKeyId(), false);
+ _accountDetailsDao.addDetail(accountId, secretKeyDetailKey, key.getSecretAccessKey(), false);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Update the IAM credentials on all BucketVO rows for this store/account
+ * so previously created buckets reflect the new (rotated) key pair.
+ * Mirrors CloudianHyperStoreObjectStoreDriverImpl.updateAccountBucketCredentials.
+ */
+ private void updateAccountBucketCredentials(long storeId, long accountId, AccessKey iamCredential) {
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ for (BucketVO bucketVO : bucketList) {
+ logger.info("Updating accountId={} bucket {} with new IAM credentials", accountId, bucketVO.getName());
+ bucketVO.setAccessKey(iamCredential.getAccessKeyId());
+ bucketVO.setSecretKey(iamCredential.getSecretAccessKey());
+ _bucketDao.update(bucketVO.getId(), bucketVO);
+ }
+ }
+
+ /**
+ * Refresh the per-account IAM user policy so it grants S3 access only to
+ * the account's current buckets (optionally excluding one, e.g. a bucket
+ * being deleted). This is the tenant boundary: each account's IAM
+ * credentials can only operate on that account's own buckets.
+ *
+ * Acquires the per-store/account IAM lock. Callers that already hold the
+ * lock (e.g. createUser, createBucket post-create) should call
+ * {@link #updateAccountIAMPolicyLocked} instead to avoid re-entrant lock
+ * acquisition warnings from GlobalLock.
+ *
+ * @param iamClient the IAM client
+ * @param storeId the object store
+ * @param accountId the CloudStack account
+ * @param excludeBucket a bucket name to omit (e.g. a bucket being deleted),
+ * or null to include all of the account's buckets
+ */
+ protected void updateAccountIAMPolicy(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, excludeBucket);
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Lock-free variant of {@link #updateAccountIAMPolicy} for callers that
+ * already hold the per-store/account IAM lock. Performs the policy refresh
+ * without reacquiring the lock, avoiding the GlobalLock re-entrant
+ * acquisition warning.
+ */
+ protected void updateAccountIAMPolicyLocked(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ return;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ List<BucketVO> buckets = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ List<String> bucketNames = new ArrayList<>();
+ for (BucketVO bvo : buckets) {
+ if (excludeBucket != null && excludeBucket.equals(bvo.getName())) {
+ continue;
+ }
+ // Skip buckets whose remote counterpart has been deleted but whose
+ // row is still present for resource accounting. Including them
+ // would re-grant a bucket name that is now free for another account
+ // to claim.
+ if (Bucket.State.Destroyed.equals(bvo.getState())) {
+ continue;
+ }
+ bucketNames.add(bvo.getName());
+ }
+ String policy = SeaweedFSObjectStoreUtil.buildAccountIAMPolicy(bucketNames);
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ SeaweedFSObjectStoreUtil.IAM_USER_POLICY_NAME, policy));
+ }
+
+ /**
+ * Check whether the given access key id is still listed and Active in IAM
+ * for the user. Listing failures are propagated rather than swallowed so
+ * a transient IAM outage does not send createUser into the replacement
+ * path (which would overwrite stored credentials and invalidate bucket
+ * records).
+ */
+ private boolean iamAccessKeyExists(AmazonIdentityManagement iamClient, String userName, String accessKeyId) {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ if (accessKeyId.equals(metadata.getAccessKeyId())) {
+ return "Active".equalsIgnoreCase(metadata.getStatus());
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Delete access keys for the user other than the (optionally) preserved
+ * key id. Used to clean up unmanaged leftover keys before creating a
+ * replacement so repeated calls do not hit IAM access-key limits.
+ */
+ private void deleteUnmanagedAccessKeys(AmazonIdentityManagement iamClient, String userName, String preserveAccessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ String keyId = metadata.getAccessKeyId();
+ if (preserveAccessKeyId != null && preserveAccessKeyId.equals(keyId)) {
+ continue;
+ }
+ DeleteAccessKeyRequest deleteReq =
+ new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(keyId);
+ logger.info("Deleting un-managed IAM access key {} for user {}", keyId, userName);
+ iamClient.deleteAccessKey(deleteReq);
+ }
+ } catch (AmazonClientException e) {
+ // Propagate so the caller does not proceed to create a replacement
+ // key while stale unmanaged keys remain (which could hit IAM key
+ // limits or leave orphaned credentials).
+ throw new CloudRuntimeException("Failed to clean up IAM access keys for user " + userName, e);
+ }
+ }
+
+ @Override
+ public Bucket createBucket(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+ long accountId = bucket.getAccountId();
+
+ // Use the store's admin credentials to create the bucket
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Check if the bucket already exists
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket already exists with name " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Create the bucket
+ try {
+ CreateBucketRequest request = new CreateBucketRequest(bucketName);
+ if (objectLock) {
+ request.setObjectLockEnabledForBucket(true);
+ }
+ s3client.createBucket(request);
+ } catch (AmazonClientException e) {
+ logger.error("Create bucket failed", e);
+ throw new CloudRuntimeException(e);
+ }
+
+ // Step 2: update the bucket record with the account's IAM credentials,
+ // configure CORS, and refresh the IAM policy. If any of these fail,
+ // clean up the remote bucket so a retry does not find it already
+ // existing — mirroring the Cloudian createBucket pattern.
+ //
+ // Hold the IAM lock for the account so a concurrent createUser key
+ // rotation does not change the account credentials between reading
+ // them and writing the BucketVO, which would leave the bucket with
+ // a stale key pair.
+ GlobalLock iamLock = acquireIamLock(storeId, accountId);
+ if (iamLock == null) {
+ // The S3 bucket has already been created. Clean it up so a
+ // retry does not find it already existing, then throw.
+ try {
+ s3client.deleteBucket(bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Failed to clean up bucket {} after IAM lock timeout", bucketName, cleanupEx);
+ }
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ // Configure permissive CORS so the CloudStack S3 bucket browser
+ // (which performs list/upload/delete from the browser) can function.
+ // SeaweedFS supports the standard PutBucketCors operation.
+ configureBucketCORS(s3client, bucketName);
+
+ Map<String, String> accountDetails = _accountDetailsDao.findDetails(accountId);
+ String accessKey = accountDetails.get(SeaweedFSObjectStoreUtil.keyAccessKey(storeId));
+ String secretKey = accountDetails.get(SeaweedFSObjectStoreUtil.keySecretKey(storeId));
+ if (accessKey == null || secretKey == null) {
+ throw new CloudRuntimeException("No IAM credentials found for account " + accountId
+ + " on store " + storeId + ". Run createUser before creating a bucket.");
+ }
+
+ String s3Url = getS3Url(storeId);
+ BucketVO bucketVO = _bucketDao.findById(bucket.getId());
+ bucketVO.setAccessKey(accessKey);
+ bucketVO.setSecretKey(secretKey);
+ bucketVO.setBucketURL(s3Url + "/" + bucketName);
+ _bucketDao.update(bucket.getId(), bucketVO);
+
+ // Refresh the account's IAM policy to include the new bucket.
+ // Use the lock-free variant since createBucket already holds the
+ // IAM lock for the post-create section.
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, null);
+
+ // Return the updated BucketVO (not the stale input bucket) so
+ // BucketApiServiceImpl.createBucket does not overwrite the
+ // persisted credentials with the stale values.
+ return bucketVO;
+ } catch (Exception e) {
+ logger.error("Post-create bucket record update failed for {}; cleaning up remote bucket", bucketName, e);
+ CloudRuntimeException primary = new CloudRuntimeException(e);
+ try {
+ s3client.deleteBucket(bucketName);
+ logger.info("Cleanup of bucket {} succeeded", bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Cleanup of bucket {} also failed", bucketName, cleanupEx);
+ primary.addSuppressed(cleanupEx);
+ }
+ // Revoke the IAM policy grant for the new bucket so the account's
+ // credentials cannot access a bucket that no longer exists. If the
+ // policy PUT succeeded before the DB update failed, the grant
+ // would otherwise persist and could be reused if another account
+ // later creates the same bucket name. Use the lock-free variant
+ // since createBucket already holds the IAM lock. Propagate
+ // failures as suppressed exceptions so they are not silently lost.
+ try {
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, bucketName);
+ } catch (Exception policyEx) {
+ logger.warn("Failed to revoke IAM policy for bucket {} after cleanup: {}", bucketName, policyEx.getMessage());
+ primary.addSuppressed(policyEx);
+ }
+ throw primary;
+ } finally {
+ iamLock.unlock();
+ iamLock.releaseRef();
+ }
+ }
+
+ /**
+ * Configure a permissive CORS policy on the bucket so the CloudStack
+ * S3 bucket browser (which performs list/upload/delete from the
+ * browser) can function. Mirrors the Cloudian configureBucketCORS.
+ */
+ private void configureBucketCORS(AmazonS3 s3client, String bucketName) {
+ logger.debug("Configuring CORS for bucket {}", bucketName);
+ List<CORSRule> corsRules = new ArrayList<>();
+ CORSRule allowAnyRule = new CORSRule().withId("AllowAny");
+ allowAnyRule.setAllowedOrigins("*");
+ allowAnyRule.setAllowedHeaders("*");
+ allowAnyRule.setAllowedMethods(
+ CORSRule.AllowedMethods.HEAD,
+ CORSRule.AllowedMethods.GET,
+ CORSRule.AllowedMethods.PUT,
+ CORSRule.AllowedMethods.POST,
+ CORSRule.AllowedMethods.DELETE);
+ corsRules.add(allowAnyRule);
+ BucketCrossOriginConfiguration corsConfig = new BucketCrossOriginConfiguration();
+ corsConfig.setRules(corsRules);
+ SetBucketCrossOriginConfigurationRequest corsRequest = new SetBucketCrossOriginConfigurationRequest(bucketName, corsConfig);
+ s3client.setBucketCrossOriginConfiguration(corsRequest);
+ logger.info("Successfully configured CORS for bucket {}", bucketName);
+ }
+
+ @Override
+ public List<Bucket> listBuckets(long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ List<Bucket> bucketsList = new ArrayList<>();
+ try {
+ List<com.amazonaws.services.s3.model.Bucket> s3Buckets = s3client.listBuckets();
+ for (com.amazonaws.services.s3.model.Bucket s3Bucket : s3Buckets) {
+ Bucket bucket = new BucketObject();
+ bucket.setName(s3Bucket.getName());
+ bucketsList.add(bucket);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ return bucketsList;
+ }
+
+ @Override
+ public boolean deleteBucket(BucketTO bucket, long storeId) {
+ String bucketName = bucket.getName();
+ long accountId = bucket.getAccountId();
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Acquire the per-store/account IAM lock so the S3 delete and the
+ // subsequent policy refresh are atomic with respect to concurrent
+ // createUser/createBucket operations. Without the lock, a concurrent
+ // policy rebuild could re-add the deleted bucket ARN between the
+ // excludeBucket refresh and BucketApiServiceImpl's row removal.
+ GlobalLock iamLock = acquireIamLock(storeId, accountId);
+ if (iamLock == null) {
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ // Delete the S3 bucket first. If this fails (non-empty bucket,
+ // transient error), the IAM policy is still intact so the user
+ // can empty the bucket and retry.
+ //
+ // If the bucket is already gone (e.g. from a previous partial
+ // failure where the S3 delete succeeded but the IAM policy refresh
+ // failed), skip the S3 delete so the retry is idempotent.
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ s3client.deleteBucket(bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Mark the row Destroyed inside the lock. The row itself must
+ // survive so BucketApiServiceImpl.deleteCheckedBucket can still
+ // decrement the resource counts and allocated size, and so a
+ // failure in that cleanup leaves state a retry can reconcile.
+ // Marking it (rather than relying only on excludeBucket) closes the
+ // window where a concurrent createUser/createBucket for this
+ // account rebuilds the policy from the DB and re-adds this bucket's
+ // ARN: buildAccountIAMPolicy skips Destroyed rows, so the grant
+ // cannot come back on a name that is now reusable.
+ markBucketDestroyed(storeId, accountId, bucketName);
Review Comment:
Fixed in `932cc3c`. The driver still marks the bucket row `Destroyed`, but `BucketApiServiceImpl.deleteCheckedBucket` now performs the resource-count updates, allocated-size update, and row removal together in one transaction after the driver returns, checking the DAO results. That keeps retry paths from double-decrementing object-store accounting. Added a regression test for allocated-size failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r4011304271
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,765 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.BucketCrossOriginConfiguration;
+import com.amazonaws.services.s3.model.CORSRule;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketCrossOriginConfigurationRequest;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ /**
+ * DB-backed global lock name prefix for serializing IAM provisioning and
+ * policy refreshes per store+account. Uses {@link GlobalLock} so the
+ * critical section is serialized across management servers in a
+ * clustered deployment, not just within a single JVM.
+ */
+ private static final String IAM_LOCK_PREFIX = "seaweedfs.iam.";
+
+ private static String getIamLockName(long storeId, long accountId) {
+ return IAM_LOCK_PREFIX + storeId + "." + accountId;
+ }
+
+ /**
+ * Acquire a DB-backed global lock for IAM operations on the given
+ * store+account. Returns a {@link GlobalLock} that the caller must
+ * {@link GlobalLock#unlock()} in a {@code finally} block, or {@code null}
+ * if the lock could not be acquired within the timeout.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ */
+ protected GlobalLock acquireIamLock(long storeId, long accountId) {
+ GlobalLock lock = GlobalLock.getInternLock(getIamLockName(storeId, accountId));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire IAM lock for store {} account {}", storeId, accountId);
+ lock.releaseRef();
+ return null;
+ }
+ return lock;
+ }
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account and
+ * store. The store ID is included so that two CloudStack pools pointing
+ * at the same SeaweedFS IAM service do not collide on the same
+ * {@code acs-<uuid>} user and overwrite each other's policy and access
+ * keys.
+ */
+ protected String getUserNameForAccount(Account account, long storeId) {
+ return String.format("%s-%d-%s", ACS_PREFIX, storeId, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Serialize per store+account across management servers so two
+ // concurrent bucket requests do not both rotate credentials and leave
+ // bucket rows with mismatched key pairs.
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ return false;
+ }
+ try {
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach a scoped IAM policy that allows access only to this
+ // account's own buckets (the tenant boundary). Refreshed whenever
+ // buckets are created or deleted.
+ updateAccountIAMPolicy(iamClient, storeId, accountId, null);
+
+ // Reuse the stored access key only if both the access key id and the
+ // secret key are present and the key is still Active in IAM; otherwise
+ // create a replacement.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ String storedSecretKey = details.get(secretKeyDetailKey);
+ if (storedAccessKeyId != null && storedSecretKey != null
+ && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ return true;
+ }
+
+ // The stored key is missing, inactive, or no longer in IAM. Clean up
+ // ALL keys (including the inactive stored one) before creating a
+ // replacement so we do not accumulate keys and hit IAM limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, null);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Update existing bucket records for this account/store with the new
+ // credentials BEFORE persisting the new key in account details. If a
+ // bucket update fails, the stored key remains the old one and a retry
+ // will re-enter the replacement path; if we persisted first, a retry
+ // would see the new stored key and return without repairing the
+ // remaining buckets.
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ // Persist the credentials in the account details (namespaced by storeId)
+ details.put(accessKeyDetailKey, key.getAccessKeyId());
+ details.put(secretKeyDetailKey, key.getSecretAccessKey());
+ _accountDetailsDao.persist(accountId, details);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Update the IAM credentials on all BucketVO rows for this store/account
+ * so previously created buckets reflect the new (rotated) key pair.
+ * Mirrors CloudianHyperStoreObjectStoreDriverImpl.updateAccountBucketCredentials.
+ */
+ private void updateAccountBucketCredentials(long storeId, long accountId, AccessKey iamCredential) {
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ for (BucketVO bucketVO : bucketList) {
+ logger.info("Updating accountId={} bucket {} with new IAM credentials", accountId, bucketVO.getName());
+ bucketVO.setAccessKey(iamCredential.getAccessKeyId());
+ bucketVO.setSecretKey(iamCredential.getSecretAccessKey());
+ _bucketDao.update(bucketVO.getId(), bucketVO);
+ }
+ }
+
+ /**
+ * Refresh the per-account IAM user policy so it grants S3 access only to
+ * the account's current buckets (optionally excluding one, e.g. a bucket
+ * being deleted). This is the tenant boundary: each account's IAM
+ * credentials can only operate on that account's own buckets.
+ *
+ * @param iamClient the IAM client
+ * @param storeId the object store
+ * @param accountId the CloudStack account
+ * @param excludeBucket a bucket name to omit (e.g. a bucket being deleted),
+ * or null to include all of the account's buckets
+ */
+ protected void updateAccountIAMPolicy(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ return;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ List<BucketVO> buckets = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ List<String> bucketNames = new ArrayList<>();
+ for (BucketVO bvo : buckets) {
+ if (excludeBucket != null && excludeBucket.equals(bvo.getName())) {
+ continue;
+ }
+ bucketNames.add(bvo.getName());
+ }
+ String policy = SeaweedFSObjectStoreUtil.buildAccountIAMPolicy(bucketNames);
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ SeaweedFSObjectStoreUtil.IAM_USER_POLICY_NAME, policy));
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Check whether the given access key id is still listed and Active in IAM
+ * for the user. Listing failures are propagated rather than swallowed so
+ * a transient IAM outage does not send createUser into the replacement
+ * path (which would overwrite stored credentials and invalidate bucket
+ * records).
+ */
+ private boolean iamAccessKeyExists(AmazonIdentityManagement iamClient, String userName, String accessKeyId) {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ if (accessKeyId.equals(metadata.getAccessKeyId())) {
+ return "Active".equalsIgnoreCase(metadata.getStatus());
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Delete access keys for the user other than the (optionally) preserved
+ * key id. Used to clean up unmanaged leftover keys before creating a
+ * replacement so repeated calls do not hit IAM access-key limits.
+ */
+ private void deleteUnmanagedAccessKeys(AmazonIdentityManagement iamClient, String userName, String preserveAccessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ String keyId = metadata.getAccessKeyId();
+ if (preserveAccessKeyId != null && preserveAccessKeyId.equals(keyId)) {
+ continue;
+ }
+ DeleteAccessKeyRequest deleteReq =
+ new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(keyId);
+ logger.info("Deleting un-managed IAM access key {} for user {}", keyId, userName);
+ iamClient.deleteAccessKey(deleteReq);
+ }
+ } catch (AmazonClientException e) {
+ // Propagate so the caller does not proceed to create a replacement
+ // key while stale unmanaged keys remain (which could hit IAM key
+ // limits or leave orphaned credentials).
+ throw new CloudRuntimeException("Failed to clean up IAM access keys for user " + userName, e);
+ }
+ }
+
+ @Override
+ public Bucket createBucket(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+ long accountId = bucket.getAccountId();
+
+ // Use the store's admin credentials to create the bucket
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Check if the bucket already exists
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket already exists with name " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Create the bucket
+ try {
+ CreateBucketRequest request = new CreateBucketRequest(bucketName);
+ if (objectLock) {
+ request.setObjectLockEnabledForBucket(true);
+ }
+ s3client.createBucket(request);
+ } catch (AmazonClientException e) {
+ logger.error("Create bucket failed", e);
+ throw new CloudRuntimeException(e);
+ }
+
+ // Configure permissive CORS so the CloudStack S3 bucket browser
+ // (which performs list/upload/delete from the browser) can function.
+ // SeaweedFS supports the standard PutBucketCors operation.
+ configureBucketCORS(s3client, bucketName);
+
+ // Step 2: update the bucket record with the account's IAM credentials.
+ // If this fails, clean up the remote bucket so a retry does not find
+ // it already existing — mirroring the Cloudian createBucket pattern.
+ try {
+ Map<String, String> accountDetails = _accountDetailsDao.findDetails(accountId);
+ String accessKey = accountDetails.get(SeaweedFSObjectStoreUtil.keyAccessKey(storeId));
+ String secretKey = accountDetails.get(SeaweedFSObjectStoreUtil.keySecretKey(storeId));
+ if (accessKey == null || secretKey == null) {
+ throw new CloudRuntimeException("No IAM credentials found for account " + accountId
+ + " on store " + storeId + ". Run createUser before creating a bucket.");
+ }
+
+ String s3Url = getS3Url(storeId);
+ BucketVO bucketVO = _bucketDao.findById(bucket.getId());
+ bucketVO.setAccessKey(accessKey);
+ bucketVO.setSecretKey(secretKey);
+ bucketVO.setBucketURL(s3Url + "/" + bucketName);
+ _bucketDao.update(bucket.getId(), bucketVO);
+
+ // Refresh the account's IAM policy to include the new bucket
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicy(iamClient, storeId, accountId, null);
Review Comment:
`createUser` rotates/deletes IAM keys while holding the per-store/account lock and updates all existing bucket rows, but this path reads the account credentials and writes the new `BucketVO` without that lock. A rotation between the read and `_bucketDao.update` can therefore leave this bucket with the old key pair after account details have moved to the new key, making the bucket credentials unusable. Acquire the same lock around the credential read/update and policy refresh.
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,765 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.BucketCrossOriginConfiguration;
+import com.amazonaws.services.s3.model.CORSRule;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketCrossOriginConfigurationRequest;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ /**
+ * DB-backed global lock name prefix for serializing IAM provisioning and
+ * policy refreshes per store+account. Uses {@link GlobalLock} so the
+ * critical section is serialized across management servers in a
+ * clustered deployment, not just within a single JVM.
+ */
+ private static final String IAM_LOCK_PREFIX = "seaweedfs.iam.";
+
+ private static String getIamLockName(long storeId, long accountId) {
+ return IAM_LOCK_PREFIX + storeId + "." + accountId;
+ }
+
+ /**
+ * Acquire a DB-backed global lock for IAM operations on the given
+ * store+account. Returns a {@link GlobalLock} that the caller must
+ * {@link GlobalLock#unlock()} in a {@code finally} block, or {@code null}
+ * if the lock could not be acquired within the timeout.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ */
+ protected GlobalLock acquireIamLock(long storeId, long accountId) {
+ GlobalLock lock = GlobalLock.getInternLock(getIamLockName(storeId, accountId));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire IAM lock for store {} account {}", storeId, accountId);
+ lock.releaseRef();
+ return null;
+ }
+ return lock;
+ }
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account and
+ * store. The store ID is included so that two CloudStack pools pointing
+ * at the same SeaweedFS IAM service do not collide on the same
+ * {@code acs-<uuid>} user and overwrite each other's policy and access
+ * keys.
+ */
+ protected String getUserNameForAccount(Account account, long storeId) {
+ return String.format("%s-%d-%s", ACS_PREFIX, storeId, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Serialize per store+account across management servers so two
+ // concurrent bucket requests do not both rotate credentials and leave
+ // bucket rows with mismatched key pairs.
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ return false;
+ }
+ try {
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach a scoped IAM policy that allows access only to this
+ // account's own buckets (the tenant boundary). Refreshed whenever
+ // buckets are created or deleted.
+ updateAccountIAMPolicy(iamClient, storeId, accountId, null);
+
+ // Reuse the stored access key only if both the access key id and the
+ // secret key are present and the key is still Active in IAM; otherwise
+ // create a replacement.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ String storedSecretKey = details.get(secretKeyDetailKey);
+ if (storedAccessKeyId != null && storedSecretKey != null
+ && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ return true;
+ }
+
+ // The stored key is missing, inactive, or no longer in IAM. Clean up
+ // ALL keys (including the inactive stored one) before creating a
+ // replacement so we do not accumulate keys and hit IAM limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, null);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Update existing bucket records for this account/store with the new
+ // credentials BEFORE persisting the new key in account details. If a
+ // bucket update fails, the stored key remains the old one and a retry
+ // will re-enter the replacement path; if we persisted first, a retry
+ // would see the new stored key and return without repairing the
+ // remaining buckets.
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ // Persist the credentials in the account details (namespaced by storeId)
+ details.put(accessKeyDetailKey, key.getAccessKeyId());
+ details.put(secretKeyDetailKey, key.getSecretAccessKey());
+ _accountDetailsDao.persist(accountId, details);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Update the IAM credentials on all BucketVO rows for this store/account
+ * so previously created buckets reflect the new (rotated) key pair.
+ * Mirrors CloudianHyperStoreObjectStoreDriverImpl.updateAccountBucketCredentials.
+ */
+ private void updateAccountBucketCredentials(long storeId, long accountId, AccessKey iamCredential) {
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ for (BucketVO bucketVO : bucketList) {
+ logger.info("Updating accountId={} bucket {} with new IAM credentials", accountId, bucketVO.getName());
+ bucketVO.setAccessKey(iamCredential.getAccessKeyId());
+ bucketVO.setSecretKey(iamCredential.getSecretAccessKey());
+ _bucketDao.update(bucketVO.getId(), bucketVO);
+ }
+ }
+
+ /**
+ * Refresh the per-account IAM user policy so it grants S3 access only to
+ * the account's current buckets (optionally excluding one, e.g. a bucket
+ * being deleted). This is the tenant boundary: each account's IAM
+ * credentials can only operate on that account's own buckets.
+ *
+ * @param iamClient the IAM client
+ * @param storeId the object store
+ * @param accountId the CloudStack account
+ * @param excludeBucket a bucket name to omit (e.g. a bucket being deleted),
+ * or null to include all of the account's buckets
+ */
+ protected void updateAccountIAMPolicy(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ return;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ List<BucketVO> buckets = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ List<String> bucketNames = new ArrayList<>();
+ for (BucketVO bvo : buckets) {
+ if (excludeBucket != null && excludeBucket.equals(bvo.getName())) {
+ continue;
+ }
+ bucketNames.add(bvo.getName());
+ }
+ String policy = SeaweedFSObjectStoreUtil.buildAccountIAMPolicy(bucketNames);
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ SeaweedFSObjectStoreUtil.IAM_USER_POLICY_NAME, policy));
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Check whether the given access key id is still listed and Active in IAM
+ * for the user. Listing failures are propagated rather than swallowed so
+ * a transient IAM outage does not send createUser into the replacement
+ * path (which would overwrite stored credentials and invalidate bucket
+ * records).
+ */
+ private boolean iamAccessKeyExists(AmazonIdentityManagement iamClient, String userName, String accessKeyId) {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ if (accessKeyId.equals(metadata.getAccessKeyId())) {
+ return "Active".equalsIgnoreCase(metadata.getStatus());
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Delete access keys for the user other than the (optionally) preserved
+ * key id. Used to clean up unmanaged leftover keys before creating a
+ * replacement so repeated calls do not hit IAM access-key limits.
+ */
+ private void deleteUnmanagedAccessKeys(AmazonIdentityManagement iamClient, String userName, String preserveAccessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ String keyId = metadata.getAccessKeyId();
+ if (preserveAccessKeyId != null && preserveAccessKeyId.equals(keyId)) {
+ continue;
+ }
+ DeleteAccessKeyRequest deleteReq =
+ new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(keyId);
+ logger.info("Deleting un-managed IAM access key {} for user {}", keyId, userName);
+ iamClient.deleteAccessKey(deleteReq);
+ }
+ } catch (AmazonClientException e) {
+ // Propagate so the caller does not proceed to create a replacement
+ // key while stale unmanaged keys remain (which could hit IAM key
+ // limits or leave orphaned credentials).
+ throw new CloudRuntimeException("Failed to clean up IAM access keys for user " + userName, e);
+ }
+ }
+
+ @Override
+ public Bucket createBucket(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+ long accountId = bucket.getAccountId();
+
+ // Use the store's admin credentials to create the bucket
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Check if the bucket already exists
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket already exists with name " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Create the bucket
+ try {
+ CreateBucketRequest request = new CreateBucketRequest(bucketName);
+ if (objectLock) {
+ request.setObjectLockEnabledForBucket(true);
+ }
+ s3client.createBucket(request);
+ } catch (AmazonClientException e) {
+ logger.error("Create bucket failed", e);
+ throw new CloudRuntimeException(e);
+ }
+
+ // Configure permissive CORS so the CloudStack S3 bucket browser
+ // (which performs list/upload/delete from the browser) can function.
+ // SeaweedFS supports the standard PutBucketCors operation.
+ configureBucketCORS(s3client, bucketName);
+
+ // Step 2: update the bucket record with the account's IAM credentials.
+ // If this fails, clean up the remote bucket so a retry does not find
+ // it already existing — mirroring the Cloudian createBucket pattern.
+ try {
Review Comment:
`configureBucketCORS` runs after the remote bucket is created but before the `try` block that performs cleanup. If SeaweedFS rejects `PutBucketCors` (or a transient request fails), this exception escapes without deleting the newly created bucket; the API then removes only the CloudStack row and retries will find an orphaned bucket. Move this call into the post-create `try` so the existing cleanup path handles CORS failures too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#issuecomment-5643804609
Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
Here are some useful points:
- In case of a new feature add useful documentation (raise doc PR at https://github.com/apache/cloudstack-documentation)
- Be patient and persistent. It might take some time to get a review or get the final approval from the committers.
- Pay attention to the quality of your code, ensure tests are passing and your PR doesn't have conflicts.
- Please follow [ASF Code of Conduct](https://github.com/apache/.github/blob/main/.github/CODE_OF_CONDUCT.md) for all communication including (but not limited to) comments on Pull Requests, Issues, Mailing list and Slack.
- Be sure to read the [CloudStack Coding Conventions](https://cwiki.apache.org/confluence/display/CLOUDSTACK/Coding+conventions).
Apache CloudStack is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@cloudstack.apache.org (https://cloudstack.apache.org/mailing-lists.html)
Slack: https://apachecloudstack.slack.com/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r4010843459
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,554 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account.
+ * Uses the account UUID prefixed with "acs-" for namespacing.
+ */
+ protected String getUserNameForAccount(Account account) {
+ return String.format("%s-%s", ACS_PREFIX, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach the restricted S3 policy (idempotent — overwrites if present)
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ "CloudStackPolicy", SeaweedFSObjectStoreUtil.IAM_USER_POLICY));
+
+ // Reuse the stored access key if it is still present in IAM; only
+ // create a new one when no usable key exists.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String storedAccessKeyId = details.get(SeaweedFSObjectStoreUtil.KEY_ACCESS_KEY);
+ if (storedAccessKeyId != null && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ return true;
+ }
+
+ // The stored key is missing or no longer in IAM. Clean up any
+ // unmanaged leftover keys before creating a replacement so we do not
+ // accumulate keys and hit IAM access-key limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, storedAccessKeyId);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Persist the credentials in the account details
+ details.put(SeaweedFSObjectStoreUtil.KEY_ACCESS_KEY, key.getAccessKeyId());
+ details.put(SeaweedFSObjectStoreUtil.KEY_SECRET_KEY, key.getSecretAccessKey());
+ _accountDetailsDao.persist(accountId, details);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ }
+
+ /**
+ * Check whether the given access key id is still listed in IAM for the user.
+ */
+ private boolean iamAccessKeyExists(AmazonIdentityManagement iamClient, String userName, String accessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ if (accessKeyId.equals(metadata.getAccessKeyId())) {
+ return true;
+ }
+ }
+ } catch (AmazonClientException e) {
+ logger.warn("Failed to list IAM access keys for user {}: {}", userName, e.getMessage());
+ }
+ return false;
+ }
+
+ /**
+ * Delete access keys for the user other than the (optionally) preserved
+ * key id. Used to clean up unmanaged leftover keys before creating a
+ * replacement so repeated calls do not hit IAM access-key limits.
+ */
+ private void deleteUnmanagedAccessKeys(AmazonIdentityManagement iamClient, String userName, String preserveAccessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ String keyId = metadata.getAccessKeyId();
+ if (preserveAccessKeyId != null && preserveAccessKeyId.equals(keyId)) {
+ continue;
+ }
+ DeleteAccessKeyRequest deleteReq =
+ new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(keyId);
+ logger.info("Deleting un-managed IAM access key {} for user {}", keyId, userName);
+ iamClient.deleteAccessKey(deleteReq);
+ }
+ } catch (AmazonClientException e) {
+ logger.warn("Failed to clean up IAM access keys for user {}: {}", userName, e.getMessage());
+ }
+ }
+
+ @Override
+ public Bucket createBucket(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+ long accountId = bucket.getAccountId();
+
+ // Use the store's admin credentials to create the bucket
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Check if the bucket already exists
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket already exists with name " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Create the bucket
+ try {
+ CreateBucketRequest request = new CreateBucketRequest(bucketName);
+ if (objectLock) {
+ request.setObjectLockEnabledForBucket(true);
+ }
+ s3client.createBucket(request);
+ } catch (AmazonClientException e) {
+ logger.error("Create bucket failed", e);
+ throw new CloudRuntimeException(e);
+ }
+
+ // Update the bucket record with the account's IAM credentials
+ Map<String, String> accountDetails = _accountDetailsDao.findDetails(accountId);
+ String accessKey = accountDetails.get(SeaweedFSObjectStoreUtil.KEY_ACCESS_KEY);
+ String secretKey = accountDetails.get(SeaweedFSObjectStoreUtil.KEY_SECRET_KEY);
+ if (accessKey == null || secretKey == null) {
+ logger.warn("No IAM credentials found for account {}. Bucket will be created without per-account credentials.", accountId);
+ }
+
+ ObjectStoreVO store = _storeDao.findById(storeId);
+ String s3Url = getS3Url(storeId);
+ BucketVO bucketVO = _bucketDao.findById(bucket.getId());
+ bucketVO.setAccessKey(accessKey);
+ bucketVO.setSecretKey(secretKey);
+ bucketVO.setBucketURL(s3Url + "/" + bucketName);
+ _bucketDao.update(bucket.getId(), bucketVO);
+ return bucket;
+ }
+
+ @Override
+ public List<Bucket> listBuckets(long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ List<Bucket> bucketsList = new ArrayList<>();
+ try {
+ List<com.amazonaws.services.s3.model.Bucket> s3Buckets = s3client.listBuckets();
+ for (com.amazonaws.services.s3.model.Bucket s3Bucket : s3Buckets) {
+ Bucket bucket = new BucketObject();
+ bucket.setName(s3Bucket.getName());
+ bucketsList.add(bucket);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ return bucketsList;
+ }
+
+ @Override
+ public boolean deleteBucket(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ if (! s3client.doesBucketExistV2(bucket.getName())) {
+ throw new CloudRuntimeException("Bucket doesn't exist: " + bucket.getName());
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ try {
+ s3client.deleteBucket(bucket.getName());
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ return true;
+ }
+
+ @Override
+ public AccessControlList getBucketAcl(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ return s3client.getBucketAcl(bucket.getName());
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public void setBucketAcl(BucketTO bucket, AccessControlList acl, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ s3client.setBucketAcl(bucket.getName(), acl);
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public void setBucketPolicy(BucketTO bucket, String policy, long storeId) {
+ if ("private".equalsIgnoreCase(policy)) {
+ deleteBucketPolicy(bucket, storeId);
+ return;
+ }
+
+ StringBuilder sb = new StringBuilder();
+ sb.append("{\n");
+ sb.append(" \"Version\": \"2012-10-17\",\n");
+ sb.append(" \"Statement\": [\n");
+ sb.append(" {\n");
+ sb.append(" \"Sid\": \"PublicReadForObjects\",\n");
+ sb.append(" \"Effect\": \"Allow\",\n");
+ sb.append(" \"Principal\": \"*\",\n");
+ sb.append(" \"Action\": \"s3:GetObject\",\n");
+ sb.append(" \"Resource\": \"arn:aws:s3:::%s/*\"\n");
+ sb.append(" }\n");
+ sb.append(" ]\n");
+ sb.append("}\n");
+
+ String jsonPolicy = String.format(sb.toString(), bucket.getName());
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ s3client.setBucketPolicy(bucket.getName(), jsonPolicy);
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public BucketPolicy getBucketPolicy(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ return s3client.getBucketPolicy(new GetBucketPolicyRequest(bucket.getName()));
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public void deleteBucketPolicy(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ s3client.deleteBucketPolicy(new DeleteBucketPolicyRequest(bucket.getName()));
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public boolean setBucketEncryption(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ SetBucketEncryptionRequest eRequest = new SetBucketEncryptionRequest();
+ eRequest.setBucketName(bucket.getName());
+
+ ServerSideEncryptionByDefault sseByDefault = new ServerSideEncryptionByDefault();
+ sseByDefault.setSSEAlgorithm(SSEAlgorithm.AES256.toString());
+
+ ServerSideEncryptionRule sseRule = new ServerSideEncryptionRule();
+ sseRule.setApplyServerSideEncryptionByDefault(sseByDefault);
+
+ List<ServerSideEncryptionRule> sseRules = new ArrayList<>();
+ sseRules.add(sseRule);
+
+ ServerSideEncryptionConfiguration sseConf = new ServerSideEncryptionConfiguration();
+ sseConf.setRules(sseRules);
+
+ eRequest.setServerSideEncryptionConfiguration(sseConf);
+ s3client.setBucketEncryption(eRequest);
+ return true;
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public boolean deleteBucketEncryption(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ s3client.deleteBucketEncryption(bucket.getName());
+ return true;
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public boolean setBucketVersioning(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ BucketVersioningConfiguration vConf = new BucketVersioningConfiguration(BucketVersioningConfiguration.ENABLED);
+ s3client.setBucketVersioningConfiguration(
+ new SetBucketVersioningConfigurationRequest(bucket.getName(), vConf));
+ return true;
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public boolean deleteBucketVersioning(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ BucketVersioningConfiguration vConf = new BucketVersioningConfiguration(BucketVersioningConfiguration.SUSPENDED);
+ s3client.setBucketVersioningConfiguration(
+ new SetBucketVersioningConfigurationRequest(bucket.getName(), vConf));
+ return true;
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ /**
+ * Set the bucket quota via the SeaweedFS admin REST API.
+ *
+ * SeaweedFS enforces bucket quota server-side by setting a read-only flag
+ * when usage exceeds the configured limit. The quota is configured via the
+ * SeaweedFS S3 extension endpoint PUT /{bucket}?seaweedfs-quota,
+ * authenticated via standard S3 SigV4 and authorized via the
+ * s3:PutBucketQuota IAM permission.
+ *
+ * @param size the GiB size to set the quota to. 0 disables quota.
+ * @throws CloudRuntimeException if the S3 endpoint or credentials are missing or the request fails.
+ */
+ @Override
+ public void setBucketQuota(BucketTO bucket, long storeId, long size) {
+ String s3Url = getS3Url(storeId);
+ String accessKey = getAccessKey(storeId);
+ String secretKey = getSecretKey(storeId);
+ if (s3Url == null || s3Url.isEmpty() || accessKey == null || accessKey.isEmpty() || secretKey == null || secretKey.isEmpty()) {
+ throw new CloudRuntimeException("SeaweedFS S3 URL and credentials are required to set bucket quota. " +
+ "Configure 's3Url', 'accesskey', and 'secretkey' in the object store details.");
+ }
+ SeaweedFSObjectStoreUtil.setBucketQuotaViaS3Extension(s3Url, accessKey, secretKey, bucket.getName(), size, getS3ExtensionHttpClient());
+ }
+
+ /**
+ * Returns the HTTP client used to send SeaweedFS S3 extension requests
+ * (e.g. PUT /{bucket}?seaweedfs-quota). Exposed as a protected seam so
+ * tests can inject a mock client and assert the signed request without
+ * touching the network.
+ */
+ protected java.net.http.HttpClient getS3ExtensionHttpClient() {
+ return java.net.http.HttpClient.newHttpClient();
Review Comment:
`HttpClient.newHttpClient()` has no connect timeout, and `executeSignedS3Request` builds the request without a per-request timeout. A stalled or unreachable SeaweedFS endpoint can therefore block the synchronous bucket create/update API indefinitely. Configure bounded timeouts (preferably through the provider's configured client) before sending the quota request.
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/util/SeaweedFSObjectStoreUtil.java:
##########
@@ -0,0 +1,362 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.util;
+
+import org.apache.commons.lang3.StringUtils;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagementClientBuilder;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3ClientBuilder;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * Utility class for the SeaweedFS object storage provider.
+ *
+ * SeaweedFS exposes both an S3-compatible API and an AWS IAM-compatible API,
+ * so this provider needs no proprietary admin client — only the AWS S3 and IAM
+ * SDKs, the same pair Cloudian HyperStore already uses in this tree.
+ */
+public class SeaweedFSObjectStoreUtil {
+
+ /** The name of our Object Store Provider */
+ public static final String OBJECT_STORE_PROVIDER_NAME = "SeaweedFS";
+
+ public static final String STORE_KEY_PROVIDER_NAME = "providerName";
+ public static final String STORE_KEY_URL = "url";
+ public static final String STORE_KEY_NAME = "name";
+ public static final String STORE_KEY_SIZE = "size";
+ public static final String STORE_KEY_DETAILS = "details";
+
+ // Store Details Map key names - managed outside of plugin
+ public static final String STORE_DETAILS_KEY_ACCESS_KEY = "accesskey"; // admin/root access key
+ public static final String STORE_DETAILS_KEY_SECRET_KEY = "secretkey"; // admin/root secret key
+ public static final String STORE_DETAILS_KEY_S3_URL = "s3Url"; // S3 endpoint URL
+ public static final String STORE_DETAILS_KEY_IAM_URL = "iamUrl"; // IAM endpoint URL
+
+ // Account Detail Map key names - credentials created per CloudStack account
+ public static final String KEY_ACCESS_KEY = "swfs_AccessKey";
+ public static final String KEY_SECRET_KEY = "swfs_SecretKey";
+
+ /**
+ * IAM user policy applied to each per-account IAM user. Grants full S3
+ * access except bucket creation/deletion, so CloudStack retains control of
+ * bucket lifecycle while the account's IAM credentials can manage objects.
+ */
+ public static final String IAM_USER_POLICY = "{\n" +
+ " \"Version\": \"2012-10-17\",\n" +
+ " \"Statement\": [\n" +
+ " {\n" +
+ " \"Sid\": \"AllowFullS3Access\",\n" +
+ " \"Effect\": \"Allow\",\n" +
+ " \"Action\": [\n" +
+ " \"s3:*\"\n" +
+ " ],\n" +
+ " \"Resource\": \"*\"\n" +
+ " },\n" +
+ " {\n" +
+ " \"Sid\": \"ExceptBucketCreationOrDeletion\",\n" +
+ " \"Effect\": \"Deny\",\n" +
+ " \"Action\": [\n" +
+ " \"s3:CreateBucket\",\n" +
+ " \"s3:DeleteBucket\"\n" +
+ " ],\n" +
+ " \"Resource\": \"*\"\n" +
+ " }\n" +
+ " ]\n" +
+ "}\n";
+
+ // The CloudStack service credential (the accesskey/secretkey configured on
+ // the object store) is the admin credential used for ALL driver operations:
+ // - AmazonS3 client: bucket CRUD, policy, versioning, encryption, listing
+ // - AmazonIdentityManagement client: per-account IAM user provisioning
+ // - setBucketQuotaViaS3Extension: PUT /{bucket}?seaweedfs-quota
+ // It must therefore have broad S3 and IAM permissions. It is NOT scoped
+ // down to only s3:PutBucketQuota/s3:GetBucketQuota — that was an earlier
+ // design idea that does not match the implementation. The per-account IAM
+ // users (created by createUser) are the ones with restricted permissions
+ // (see IAM_USER_POLICY above).
+
+ /**
+ * Returns an S3 connection for the given endpoint and credentials.
+ * Uses path-style access, which SeaweedFS requires.
+ *
+ * @param url the url of the S3 service
+ * @param accessKey the credentials to use for the S3 connection.
+ * @param secretKey the matching secret key.
+ * @return an S3 connection (never null)
+ * @throws CloudRuntimeException on failure.
+ */
+ public static AmazonS3 getS3Client(String url, String accessKey, String secretKey) {
+ AmazonS3 client = AmazonS3ClientBuilder.standard()
+ .enablePathStyleAccess()
+ .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))
+ .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(url, "us-east-1"))
+ .build();
+ if (client == null) {
+ throw new CloudRuntimeException("Error while creating SeaweedFS S3 client");
+ }
+ return client;
+ }
+
+ /**
+ * Returns an IAM connection for the given endpoint and credentials.
+ *
+ * @param url the url of the IAM service
+ * @param accessKey the credentials to use for the iam connection.
+ * @param secretKey the matching secret key.
+ * @return an IAM connection (never null)
+ * @throws CloudRuntimeException on failure.
+ */
+ public static AmazonIdentityManagement getIAMClient(String url, String accessKey, String secretKey) {
+ AmazonIdentityManagement iamClient = AmazonIdentityManagementClientBuilder.standard()
+ .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))
+ .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(url, "us-east-1"))
+ .build();
+ if (iamClient == null) {
+ throw new CloudRuntimeException("Error while creating SeaweedFS IAM client");
+ }
+ return iamClient;
+ }
+
+ /**
+ * Test the S3Url to confirm it behaves like an S3 Service.
+ *
+ * Uses bad credentials and looks for the particular error from S3 that says
+ * InvalidAccessKeyId was used. Quietly returns if we connect and get the
+ * expected error back.
+ *
+ * @param s3Url the url to check
+ * @throws CloudRuntimeException if there is any unexpected issue.
+ */
+ public static void validateS3Url(String s3Url) {
+ try {
+ AmazonS3 s3Client = SeaweedFSObjectStoreUtil.getS3Client(s3Url, "unknown", "unknown");
+ s3Client.listBuckets();
+ } catch (AmazonServiceException e) {
+ if (StringUtils.compareIgnoreCase(e.getErrorCode(), "InvalidAccessKeyId") != 0
+ && StringUtils.compareIgnoreCase(e.getErrorCode(), "SignatureDoesNotMatch") != 0) {
+ throw new CloudRuntimeException("Unexpected response from S3 Endpoint.", e);
+ }
+ }
+ }
+
+ /**
+ * Test the IAMUrl to confirm it behaves like an IAM Service.
+ *
+ * Uses bad credentials and looks for the particular error from IAM that says
+ * InvalidAccessKeyId or InvalidClientTokenId was used. Quietly returns if we
+ * connect and get the expected error back.
+ *
+ * @param iamUrl the url to check
+ * @throws CloudRuntimeException if there is any unexpected issue.
+ */
+ public static void validateIAMUrl(String iamUrl) {
+ try {
+ AmazonIdentityManagement iamClient = SeaweedFSObjectStoreUtil.getIAMClient(iamUrl, "unknown", "unknown");
+ iamClient.listAccessKeys();
+ } catch (AmazonServiceException e) {
+ if (! StringUtils.equalsAnyIgnoreCase(e.getErrorCode(), "InvalidAccessKeyId", "InvalidClientTokenId", "SignatureDoesNotMatch")) {
+ throw new CloudRuntimeException("Unexpected response from IAM Endpoint.", e);
+ }
+ }
+ }
+
+ /**
+ * Set bucket quota via the SeaweedFS S3 extension endpoint.
+ *
+ * SeaweedFS exposes a custom S3 subresource at
+ * PUT /{bucket}?seaweedfs-quota
+ * authenticated via standard S3 SigV4 and authorized via the
+ * s3:PutBucketQuota IAM permission. This avoids the need for a
+ * separate admin API credential.
+ *
+ * The request body is JSON:
+ * {"quota_size": <n>, "quota_unit": "GB", "quota_enabled": true}
+ *
+ * @param s3Url the S3 endpoint URL (e.g. http://host:8333)
+ * @param accessKey the S3 access key (must have s3:PutBucketQuota permission)
+ * @param secretKey the S3 secret key
+ * @param bucketName the bucket name
+ * @param sizeGiB the quota size in GiB (0 to disable quota)
+ * @throws CloudRuntimeException on any failure
+ */
+ public static void setBucketQuotaViaS3Extension(String s3Url, String accessKey, String secretKey, String bucketName, long sizeGiB) {
+ setBucketQuotaViaS3Extension(s3Url, accessKey, secretKey, bucketName, sizeGiB, java.net.http.HttpClient.newHttpClient());
Review Comment:
This quota request is executed synchronously from bucket create/update, but the default `HttpClient` and `HttpRequest` have neither a connect timeout nor a request timeout. If the SeaweedFS endpoint becomes unreachable, a management-server API thread can block indefinitely instead of failing promptly. Configure bounded connect and per-request timeouts (preferably using CloudStack's existing timeout configuration).
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,554 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account.
+ * Uses the account UUID prefixed with "acs-" for namespacing.
+ */
+ protected String getUserNameForAccount(Account account) {
+ return String.format("%s-%s", ACS_PREFIX, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach the restricted S3 policy (idempotent — overwrites if present)
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ "CloudStackPolicy", SeaweedFSObjectStoreUtil.IAM_USER_POLICY));
+
+ // Reuse the stored access key if it is still present in IAM; only
+ // create a new one when no usable key exists.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String storedAccessKeyId = details.get(SeaweedFSObjectStoreUtil.KEY_ACCESS_KEY);
+ if (storedAccessKeyId != null && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ return true;
+ }
+
+ // The stored key is missing or no longer in IAM. Clean up any
+ // unmanaged leftover keys before creating a replacement so we do not
+ // accumulate keys and hit IAM access-key limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, storedAccessKeyId);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Persist the credentials in the account details
+ details.put(SeaweedFSObjectStoreUtil.KEY_ACCESS_KEY, key.getAccessKeyId());
+ details.put(SeaweedFSObjectStoreUtil.KEY_SECRET_KEY, key.getSecretAccessKey());
+ _accountDetailsDao.persist(accountId, details);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ }
+
+ /**
+ * Check whether the given access key id is still listed in IAM for the user.
+ */
+ private boolean iamAccessKeyExists(AmazonIdentityManagement iamClient, String userName, String accessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ if (accessKeyId.equals(metadata.getAccessKeyId())) {
+ return true;
+ }
+ }
+ } catch (AmazonClientException e) {
+ logger.warn("Failed to list IAM access keys for user {}: {}", userName, e.getMessage());
+ }
+ return false;
+ }
+
+ /**
+ * Delete access keys for the user other than the (optionally) preserved
+ * key id. Used to clean up unmanaged leftover keys before creating a
+ * replacement so repeated calls do not hit IAM access-key limits.
+ */
+ private void deleteUnmanagedAccessKeys(AmazonIdentityManagement iamClient, String userName, String preserveAccessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ String keyId = metadata.getAccessKeyId();
+ if (preserveAccessKeyId != null && preserveAccessKeyId.equals(keyId)) {
+ continue;
+ }
+ DeleteAccessKeyRequest deleteReq =
+ new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(keyId);
+ logger.info("Deleting un-managed IAM access key {} for user {}", keyId, userName);
+ iamClient.deleteAccessKey(deleteReq);
+ }
+ } catch (AmazonClientException e) {
+ logger.warn("Failed to clean up IAM access keys for user {}: {}", userName, e.getMessage());
+ }
+ }
+
+ @Override
+ public Bucket createBucket(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+ long accountId = bucket.getAccountId();
+
+ // Use the store's admin credentials to create the bucket
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Check if the bucket already exists
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket already exists with name " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Create the bucket
+ try {
+ CreateBucketRequest request = new CreateBucketRequest(bucketName);
+ if (objectLock) {
+ request.setObjectLockEnabledForBucket(true);
+ }
+ s3client.createBucket(request);
+ } catch (AmazonClientException e) {
+ logger.error("Create bucket failed", e);
+ throw new CloudRuntimeException(e);
+ }
+
+ // Update the bucket record with the account's IAM credentials
+ Map<String, String> accountDetails = _accountDetailsDao.findDetails(accountId);
+ String accessKey = accountDetails.get(SeaweedFSObjectStoreUtil.KEY_ACCESS_KEY);
+ String secretKey = accountDetails.get(SeaweedFSObjectStoreUtil.KEY_SECRET_KEY);
+ if (accessKey == null || secretKey == null) {
+ logger.warn("No IAM credentials found for account {}. Bucket will be created without per-account credentials.", accountId);
+ }
+
+ ObjectStoreVO store = _storeDao.findById(storeId);
+ String s3Url = getS3Url(storeId);
+ BucketVO bucketVO = _bucketDao.findById(bucket.getId());
+ bucketVO.setAccessKey(accessKey);
+ bucketVO.setSecretKey(secretKey);
+ bucketVO.setBucketURL(s3Url + "/" + bucketName);
+ _bucketDao.update(bucket.getId(), bucketVO);
+ return bucket;
+ }
+
+ @Override
+ public List<Bucket> listBuckets(long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ List<Bucket> bucketsList = new ArrayList<>();
+ try {
+ List<com.amazonaws.services.s3.model.Bucket> s3Buckets = s3client.listBuckets();
+ for (com.amazonaws.services.s3.model.Bucket s3Bucket : s3Buckets) {
+ Bucket bucket = new BucketObject();
+ bucket.setName(s3Bucket.getName());
+ bucketsList.add(bucket);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ return bucketsList;
+ }
+
+ @Override
+ public boolean deleteBucket(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ if (! s3client.doesBucketExistV2(bucket.getName())) {
+ throw new CloudRuntimeException("Bucket doesn't exist: " + bucket.getName());
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ try {
+ s3client.deleteBucket(bucket.getName());
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ return true;
+ }
+
+ @Override
+ public AccessControlList getBucketAcl(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ return s3client.getBucketAcl(bucket.getName());
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public void setBucketAcl(BucketTO bucket, AccessControlList acl, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ s3client.setBucketAcl(bucket.getName(), acl);
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public void setBucketPolicy(BucketTO bucket, String policy, long storeId) {
+ if ("private".equalsIgnoreCase(policy)) {
+ deleteBucketPolicy(bucket, storeId);
+ return;
+ }
+
+ StringBuilder sb = new StringBuilder();
+ sb.append("{\n");
+ sb.append(" \"Version\": \"2012-10-17\",\n");
+ sb.append(" \"Statement\": [\n");
+ sb.append(" {\n");
+ sb.append(" \"Sid\": \"PublicReadForObjects\",\n");
+ sb.append(" \"Effect\": \"Allow\",\n");
+ sb.append(" \"Principal\": \"*\",\n");
+ sb.append(" \"Action\": \"s3:GetObject\",\n");
+ sb.append(" \"Resource\": \"arn:aws:s3:::%s/*\"\n");
+ sb.append(" }\n");
+ sb.append(" ]\n");
+ sb.append("}\n");
+
+ String jsonPolicy = String.format(sb.toString(), bucket.getName());
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ s3client.setBucketPolicy(bucket.getName(), jsonPolicy);
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public BucketPolicy getBucketPolicy(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ return s3client.getBucketPolicy(new GetBucketPolicyRequest(bucket.getName()));
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public void deleteBucketPolicy(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ s3client.deleteBucketPolicy(new DeleteBucketPolicyRequest(bucket.getName()));
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public boolean setBucketEncryption(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ SetBucketEncryptionRequest eRequest = new SetBucketEncryptionRequest();
+ eRequest.setBucketName(bucket.getName());
+
+ ServerSideEncryptionByDefault sseByDefault = new ServerSideEncryptionByDefault();
+ sseByDefault.setSSEAlgorithm(SSEAlgorithm.AES256.toString());
+
+ ServerSideEncryptionRule sseRule = new ServerSideEncryptionRule();
+ sseRule.setApplyServerSideEncryptionByDefault(sseByDefault);
+
+ List<ServerSideEncryptionRule> sseRules = new ArrayList<>();
+ sseRules.add(sseRule);
+
+ ServerSideEncryptionConfiguration sseConf = new ServerSideEncryptionConfiguration();
+ sseConf.setRules(sseRules);
+
+ eRequest.setServerSideEncryptionConfiguration(sseConf);
+ s3client.setBucketEncryption(eRequest);
+ return true;
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public boolean deleteBucketEncryption(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ s3client.deleteBucketEncryption(bucket.getName());
+ return true;
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public boolean setBucketVersioning(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ BucketVersioningConfiguration vConf = new BucketVersioningConfiguration(BucketVersioningConfiguration.ENABLED);
+ s3client.setBucketVersioningConfiguration(
+ new SetBucketVersioningConfigurationRequest(bucket.getName(), vConf));
+ return true;
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public boolean deleteBucketVersioning(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ BucketVersioningConfiguration vConf = new BucketVersioningConfiguration(BucketVersioningConfiguration.SUSPENDED);
+ s3client.setBucketVersioningConfiguration(
+ new SetBucketVersioningConfigurationRequest(bucket.getName(), vConf));
+ return true;
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ /**
+ * Set the bucket quota via the SeaweedFS admin REST API.
+ *
+ * SeaweedFS enforces bucket quota server-side by setting a read-only flag
+ * when usage exceeds the configured limit. The quota is configured via the
+ * SeaweedFS S3 extension endpoint PUT /{bucket}?seaweedfs-quota,
+ * authenticated via standard S3 SigV4 and authorized via the
+ * s3:PutBucketQuota IAM permission.
+ *
+ * @param size the GiB size to set the quota to. 0 disables quota.
+ * @throws CloudRuntimeException if the S3 endpoint or credentials are missing or the request fails.
+ */
+ @Override
+ public void setBucketQuota(BucketTO bucket, long storeId, long size) {
+ String s3Url = getS3Url(storeId);
+ String accessKey = getAccessKey(storeId);
+ String secretKey = getSecretKey(storeId);
+ if (s3Url == null || s3Url.isEmpty() || accessKey == null || accessKey.isEmpty() || secretKey == null || secretKey.isEmpty()) {
+ throw new CloudRuntimeException("SeaweedFS S3 URL and credentials are required to set bucket quota. " +
+ "Configure 's3Url', 'accesskey', and 'secretkey' in the object store details.");
+ }
+ SeaweedFSObjectStoreUtil.setBucketQuotaViaS3Extension(s3Url, accessKey, secretKey, bucket.getName(), size, getS3ExtensionHttpClient());
+ }
+
+ /**
+ * Returns the HTTP client used to send SeaweedFS S3 extension requests
+ * (e.g. PUT /{bucket}?seaweedfs-quota). Exposed as a protected seam so
+ * tests can inject a mock client and assert the signed request without
+ * touching the network.
+ */
+ protected java.net.http.HttpClient getS3ExtensionHttpClient() {
+ return java.net.http.HttpClient.newHttpClient();
+ }
+
+ @Override
+ public Map<String, Long> getAllBucketsUsage(long storeId) {
+ Map<String, Long> bucketUsage = new HashMap<>();
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreId(storeId);
+ if (bucketList.isEmpty()) {
+ return bucketUsage;
+ }
+
+ // List objects per bucket via S3 (no admin API needed).
+ // SeaweedFS also publishes per-bucket Prometheus metrics and an SOSAPI
+ // capacity.xml response; operators who need scalable usage reporting
+ // should consume those instead of S3 list-based aggregation.
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ for (BucketVO bucket : bucketList) {
+ try {
+ long size = 0L;
+ com.amazonaws.services.s3.model.ListObjectsV2Result result;
+ String continuationToken = null;
+ do {
+ com.amazonaws.services.s3.model.ListObjectsV2Request req =
+ new com.amazonaws.services.s3.model.ListObjectsV2Request()
+ .withBucketName(bucket.getName())
+ .withMaxKeys(1000);
+ if (continuationToken != null) {
+ req.setContinuationToken(continuationToken);
+ }
+ result = s3client.listObjectsV2(req);
+ for (com.amazonaws.services.s3.model.S3ObjectSummary summary : result.getObjectSummaries()) {
+ size += summary.getSize();
+ }
+ continuationToken = result.getNextContinuationToken();
+ } while (result.isTruncated());
+ bucketUsage.put(bucket.getName(), size);
+ } catch (AmazonClientException e) {
+ logger.warn("Failed to get usage for bucket {}: {}", bucket.getName(), e.getMessage());
+ bucketUsage.put(bucket.getName(), 0L);
+ }
Review Comment:
Returning 0 for an S3 listing failure converts "unknown" into a valid usage value. `BucketApiServiceImpl` writes every returned size into `BucketVO.size`, so a transient endpoint or permission failure will erase the stored usage and under-report capacity. Omit the bucket or propagate the failure so the caller retains or skips the previous value.
##########
plugins/storage/object/seaweedfs/src/test/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImplTest.java:
##########
@@ -0,0 +1,549 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.lenient;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Flow;
+
+import java.io.ByteArrayOutputStream;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.Bucket;
+
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysResult;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.ListObjectsV2Request;
+import com.amazonaws.services.s3.model.ListObjectsV2Result;
+import com.amazonaws.services.s3.model.S3ObjectSummary;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.AccountVO;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.ArgumentCaptor;
+import org.mockito.ArgumentMatchers;
+import org.mockito.Mock;
+import org.mockito.MockitoAnnotations;
+import org.mockito.Spy;
+import org.mockito.junit.MockitoJUnitRunner;
+
+@RunWith(MockitoJUnitRunner.Silent.class)
+public class SeaweedFSObjectStoreDriverImplTest {
+
+ @Spy
+ SeaweedFSObjectStoreDriverImpl driver = new SeaweedFSObjectStoreDriverImpl();
+
+ @Mock
+ AmazonS3 s3Client;
+ @Mock
+ AmazonIdentityManagement iamClient;
+ @Mock
+ ObjectStoreDao objectStoreDao;
+ @Mock
+ ObjectStoreVO objectStoreVO;
+ @Mock
+ ObjectStoreDetailsDao objectStoreDetailsDao;
+ @Mock
+ AccountDao accountDao;
+ @Mock
+ BucketDao bucketDao;
+ @Mock
+ AccountDetailsDao accountDetailsDao;
+ @Mock
+ AccountVO account;
+
+ BucketVO bucketVo;
+ Map<String, String> storeDetailsMap;
+ Map<String, String> accountDetailsMap;
+
+ static long TEST_STORE_ID = 1010L;
+ static long TEST_ACCOUNT_ID = 2010L;
+ static long TEST_DOMAIN_ID = 3010L;
+ static String TEST_ACCESS_KEY = "test_access_key";
+ static String TEST_SECRET_KEY = "test_secret_key";
+ static String TEST_BUCKET_NAME = "testbucketname";
+ static String TEST_S3_URL = "http://s3-endpoint";
+ static String TEST_IAM_URL = "http://iam-endpoint";
+ static String TEST_AK = "user_access_key";
+ static String TEST_SK = "user_secret_key";
+ static String TEST_BUCKET_URL = TEST_S3_URL + "/" + TEST_BUCKET_NAME;
+ static String TEST_ACCOUNT_UUID = "account-uuid-1234";
+
+ private AutoCloseable closeable;
+
+ @Before
+ public void setUp() {
+ closeable = MockitoAnnotations.openMocks(this);
+ driver._storeDao = objectStoreDao;
+ driver._storeDetailsDao = objectStoreDetailsDao;
+ driver._accountDao = accountDao;
+ driver._bucketDao = bucketDao;
+ driver._accountDetailsDao = accountDetailsDao;
+
+ lenient().when(objectStoreDao.findById(TEST_STORE_ID)).thenReturn(objectStoreVO);
+ lenient().when(objectStoreVO.getUrl()).thenReturn(TEST_S3_URL);
+
+ storeDetailsMap = new HashMap<>();
+ storeDetailsMap.put(SeaweedFSObjectStoreUtil.STORE_DETAILS_KEY_ACCESS_KEY, TEST_ACCESS_KEY);
+ storeDetailsMap.put(SeaweedFSObjectStoreUtil.STORE_DETAILS_KEY_SECRET_KEY, TEST_SECRET_KEY);
+ storeDetailsMap.put(SeaweedFSObjectStoreUtil.STORE_DETAILS_KEY_S3_URL, TEST_S3_URL);
+ storeDetailsMap.put(SeaweedFSObjectStoreUtil.STORE_DETAILS_KEY_IAM_URL, TEST_IAM_URL);
+ lenient().when(objectStoreDetailsDao.getDetails(TEST_STORE_ID)).thenReturn(storeDetailsMap);
+
+ accountDetailsMap = new HashMap<>();
+ accountDetailsMap.put(SeaweedFSObjectStoreUtil.KEY_ACCESS_KEY, TEST_AK);
+ accountDetailsMap.put(SeaweedFSObjectStoreUtil.KEY_SECRET_KEY, TEST_SK);
+ lenient().when(accountDetailsDao.findDetails(TEST_ACCOUNT_ID)).thenReturn(accountDetailsMap);
+
+ bucketVo = new BucketVO(TEST_ACCOUNT_ID, TEST_DOMAIN_ID, TEST_STORE_ID, TEST_BUCKET_NAME, null, false, false, false, null);
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ closeable.close();
+ }
+
+ @Test
+ public void testGetStoreTO() {
+ assertNull(driver.getStoreTO(null));
+ }
+
+ @Test
+ public void testCreateBucket() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ when(s3Client.doesBucketExistV2(TEST_BUCKET_NAME)).thenReturn(false);
+ when(bucketDao.findById(anyLong())).thenReturn(bucketVo);
+
+ Bucket result = driver.createBucket(bucketVo, false);
+
+ assertEquals(TEST_BUCKET_NAME, result.getName());
+
+ ArgumentCaptor<BucketVO> captor = ArgumentCaptor.forClass(BucketVO.class);
+ verify(bucketDao, times(1)).update(any(), captor.capture());
+ BucketVO updated = captor.getValue();
+ assertEquals(TEST_AK, updated.getAccessKey());
+ assertEquals(TEST_SK, updated.getSecretKey());
+ assertEquals(TEST_BUCKET_URL, updated.getBucketURL());
+
+ verify(s3Client, times(1)).createBucket(any(CreateBucketRequest.class));
+ }
+
+ @Test
+ public void testCreateBucketAlreadyExists() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ when(s3Client.doesBucketExistV2(TEST_BUCKET_NAME)).thenReturn(true);
+
+ assertThrows(CloudRuntimeException.class, () -> driver.createBucket(bucketVo, false));
+ verify(s3Client, never()).createBucket(any(CreateBucketRequest.class));
+ }
+
+ @Test
+ public void testListBuckets() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ List<com.amazonaws.services.s3.model.Bucket> s3Buckets = new ArrayList<>();
+ s3Buckets.add(new com.amazonaws.services.s3.model.Bucket("bucket1"));
+ s3Buckets.add(new com.amazonaws.services.s3.model.Bucket("bucket2"));
+ when(s3Client.listBuckets()).thenReturn(s3Buckets);
+
+ List<Bucket> result = driver.listBuckets(TEST_STORE_ID);
+
+ assertEquals(2, result.size());
+ assertEquals("bucket1", result.get(0).getName());
+ assertEquals("bucket2", result.get(1).getName());
+ }
+
+ @Test
+ public void testDeleteBucket() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ BucketTO bucketTO = mock(BucketTO.class);
+ when(bucketTO.getName()).thenReturn(TEST_BUCKET_NAME);
+ when(s3Client.doesBucketExistV2(TEST_BUCKET_NAME)).thenReturn(true);
+
+ assertTrue(driver.deleteBucket(bucketTO, TEST_STORE_ID));
+ verify(s3Client, times(1)).deleteBucket(TEST_BUCKET_NAME);
+ }
+
+ @Test
+ public void testDeleteBucketNotFound() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ BucketTO bucketTO = mock(BucketTO.class);
+ when(bucketTO.getName()).thenReturn(TEST_BUCKET_NAME);
+ when(s3Client.doesBucketExistV2(TEST_BUCKET_NAME)).thenReturn(false);
+
+ assertThrows(CloudRuntimeException.class, () -> driver.deleteBucket(bucketTO, TEST_STORE_ID));
+ }
+
+ @Test
+ public void testSetBucketVersioning() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ BucketTO bucketTO = mock(BucketTO.class);
+ when(bucketTO.getName()).thenReturn(TEST_BUCKET_NAME);
+
+ assertTrue(driver.setBucketVersioning(bucketTO, TEST_STORE_ID));
+ verify(s3Client, times(1)).setBucketVersioningConfiguration(any(SetBucketVersioningConfigurationRequest.class));
+ }
+
+ @Test
+ public void testDeleteBucketVersioning() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ BucketTO bucketTO = mock(BucketTO.class);
+ when(bucketTO.getName()).thenReturn(TEST_BUCKET_NAME);
+
+ assertTrue(driver.deleteBucketVersioning(bucketTO, TEST_STORE_ID));
+ ArgumentCaptor<SetBucketVersioningConfigurationRequest> captor =
+ ArgumentCaptor.forClass(SetBucketVersioningConfigurationRequest.class);
+ verify(s3Client, times(1)).setBucketVersioningConfiguration(captor.capture());
+ assertEquals(BucketVersioningConfiguration.SUSPENDED, captor.getValue().getVersioningConfiguration().getStatus());
+ }
+
+ @Test
+ public void testSetBucketQuotaZero() throws Exception {
+ BucketTO bucketTO = mock(BucketTO.class);
+ when(bucketTO.getName()).thenReturn(TEST_BUCKET_NAME);
+ doReturn(TEST_S3_URL).when(driver).getS3Url(TEST_STORE_ID);
+ doReturn("access-key").when(driver).getAccessKey(TEST_STORE_ID);
+ doReturn("secret-key").when(driver).getSecretKey(TEST_STORE_ID);
+
+ HttpClient mockHttpClient = mock(HttpClient.class);
+ HttpResponse<String> mockResponse = mock(HttpResponse.class);
+ when(mockResponse.statusCode()).thenReturn(200);
+ when(mockResponse.body()).thenReturn("");
+ when(mockHttpClient.send(ArgumentMatchers.<HttpRequest>any(),
+ ArgumentMatchers.<HttpResponse.BodyHandler<String>>any())).thenReturn(mockResponse);
+ doReturn(mockHttpClient).when(driver).getS3ExtensionHttpClient();
+
+ driver.setBucketQuota(bucketTO, TEST_STORE_ID, 0);
+
+ ArgumentCaptor<HttpRequest> reqCaptor = ArgumentCaptor.forClass(HttpRequest.class);
+ verify(mockHttpClient, times(1)).send(reqCaptor.capture(),
+ ArgumentMatchers.<HttpResponse.BodyHandler<String>>any());
+ HttpRequest sent = reqCaptor.getValue();
+ assertEquals("PUT", sent.method());
+ assertEquals("/" + TEST_BUCKET_NAME, sent.uri().getPath());
+ assertTrue("query must carry the seaweedfs-quota subresource",
+ sent.uri().getQuery().contains("seaweedfs-quota"));
+ assertNotNull("request must be SigV4-signed", sent.headers().firstValue("Authorization"));
+ assertEquals("{\"quota_size\":0,\"quota_unit\":\"B\",\"quota_enabled\":false}", extractBody(sent));
Review Comment:
These tests only check that an `Authorization` header exists; they do not verify the SigV4 canonical query, payload hash, or signed headers against a known signature or S3-compatible verifier. A signing mismatch would therefore pass the suite and make every quota operation fail against SeaweedFS. Add a deterministic signature-verification test for the actual request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r4011645205
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,838 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.BucketCrossOriginConfiguration;
+import com.amazonaws.services.s3.model.CORSRule;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketCrossOriginConfigurationRequest;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ /**
+ * DB-backed global lock name prefix for serializing IAM provisioning and
+ * policy refreshes per store+account. Uses {@link GlobalLock} so the
+ * critical section is serialized across management servers in a
+ * clustered deployment, not just within a single JVM.
+ */
+ private static final String IAM_LOCK_PREFIX = "seaweedfs.iam.";
+
+ private static String getIamLockName(long storeId, long accountId) {
+ return IAM_LOCK_PREFIX + storeId + "." + accountId;
+ }
+
+ /**
+ * Acquire a DB-backed global lock for IAM operations on the given
+ * store+account. Returns a {@link GlobalLock} that the caller must
+ * {@link GlobalLock#unlock()} in a {@code finally} block, or {@code null}
+ * if the lock could not be acquired within the timeout.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ */
+ protected GlobalLock acquireIamLock(long storeId, long accountId) {
+ GlobalLock lock = GlobalLock.getInternLock(getIamLockName(storeId, accountId));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire IAM lock for store {} account {}", storeId, accountId);
+ lock.releaseRef();
+ return null;
+ }
+ return lock;
+ }
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account and
+ * store. The store ID is included so that two CloudStack pools pointing
+ * at the same SeaweedFS IAM service do not collide on the same
+ * {@code acs-<uuid>} user and overwrite each other's policy and access
+ * keys.
+ */
+ protected String getUserNameForAccount(Account account, long storeId) {
+ return String.format("%s-%d-%s", ACS_PREFIX, storeId, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Serialize per store+account across management servers so two
+ // concurrent bucket requests do not both rotate credentials and leave
+ // bucket rows with mismatched key pairs.
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ return false;
+ }
+ try {
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach a scoped IAM policy that allows access only to this
+ // account's own buckets (the tenant boundary). Refreshed whenever
+ // buckets are created or deleted.
+ updateAccountIAMPolicy(iamClient, storeId, accountId, null);
+
+ // Reuse the stored access key only if both the access key id and the
+ // secret key are present and the key is still Active in IAM; otherwise
+ // create a replacement.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ String storedSecretKey = details.get(secretKeyDetailKey);
+ if (storedAccessKeyId != null && storedSecretKey != null
+ && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ return true;
+ }
+
+ // The stored key is missing, inactive, or no longer in IAM. Clean up
+ // ALL keys (including the inactive stored one) before creating a
+ // replacement so we do not accumulate keys and hit IAM limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, null);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Update existing bucket records for this account/store with the new
+ // credentials BEFORE persisting the new key in account details. If a
+ // bucket update fails, the stored key remains the old one and a retry
+ // will re-enter the replacement path; if we persisted first, a retry
+ // would see the new stored key and return without repairing the
+ // remaining buckets.
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ // Persist the credentials in the account details (namespaced by storeId)
+ details.put(accessKeyDetailKey, key.getAccessKeyId());
+ details.put(secretKeyDetailKey, key.getSecretAccessKey());
+ _accountDetailsDao.persist(accountId, details);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Update the IAM credentials on all BucketVO rows for this store/account
+ * so previously created buckets reflect the new (rotated) key pair.
+ * Mirrors CloudianHyperStoreObjectStoreDriverImpl.updateAccountBucketCredentials.
+ */
+ private void updateAccountBucketCredentials(long storeId, long accountId, AccessKey iamCredential) {
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ for (BucketVO bucketVO : bucketList) {
+ logger.info("Updating accountId={} bucket {} with new IAM credentials", accountId, bucketVO.getName());
+ bucketVO.setAccessKey(iamCredential.getAccessKeyId());
+ bucketVO.setSecretKey(iamCredential.getSecretAccessKey());
+ _bucketDao.update(bucketVO.getId(), bucketVO);
+ }
+ }
+
+ /**
+ * Refresh the per-account IAM user policy so it grants S3 access only to
+ * the account's current buckets (optionally excluding one, e.g. a bucket
+ * being deleted). This is the tenant boundary: each account's IAM
+ * credentials can only operate on that account's own buckets.
+ *
+ * @param iamClient the IAM client
+ * @param storeId the object store
+ * @param accountId the CloudStack account
+ * @param excludeBucket a bucket name to omit (e.g. a bucket being deleted),
+ * or null to include all of the account's buckets
+ */
+ protected void updateAccountIAMPolicy(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ return;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ List<BucketVO> buckets = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ List<String> bucketNames = new ArrayList<>();
+ for (BucketVO bvo : buckets) {
+ if (excludeBucket != null && excludeBucket.equals(bvo.getName())) {
+ continue;
+ }
+ bucketNames.add(bvo.getName());
+ }
+ String policy = SeaweedFSObjectStoreUtil.buildAccountIAMPolicy(bucketNames);
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ SeaweedFSObjectStoreUtil.IAM_USER_POLICY_NAME, policy));
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Check whether the given access key id is still listed and Active in IAM
+ * for the user. Listing failures are propagated rather than swallowed so
+ * a transient IAM outage does not send createUser into the replacement
+ * path (which would overwrite stored credentials and invalidate bucket
+ * records).
+ */
+ private boolean iamAccessKeyExists(AmazonIdentityManagement iamClient, String userName, String accessKeyId) {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ if (accessKeyId.equals(metadata.getAccessKeyId())) {
+ return "Active".equalsIgnoreCase(metadata.getStatus());
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Delete access keys for the user other than the (optionally) preserved
+ * key id. Used to clean up unmanaged leftover keys before creating a
+ * replacement so repeated calls do not hit IAM access-key limits.
+ */
+ private void deleteUnmanagedAccessKeys(AmazonIdentityManagement iamClient, String userName, String preserveAccessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ String keyId = metadata.getAccessKeyId();
+ if (preserveAccessKeyId != null && preserveAccessKeyId.equals(keyId)) {
+ continue;
+ }
+ DeleteAccessKeyRequest deleteReq =
+ new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(keyId);
+ logger.info("Deleting un-managed IAM access key {} for user {}", keyId, userName);
+ iamClient.deleteAccessKey(deleteReq);
+ }
+ } catch (AmazonClientException e) {
+ // Propagate so the caller does not proceed to create a replacement
+ // key while stale unmanaged keys remain (which could hit IAM key
+ // limits or leave orphaned credentials).
+ throw new CloudRuntimeException("Failed to clean up IAM access keys for user " + userName, e);
+ }
+ }
+
+ @Override
+ public Bucket createBucket(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+ long accountId = bucket.getAccountId();
+
+ // Use the store's admin credentials to create the bucket
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Check if the bucket already exists
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket already exists with name " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Create the bucket
+ try {
+ CreateBucketRequest request = new CreateBucketRequest(bucketName);
+ if (objectLock) {
+ request.setObjectLockEnabledForBucket(true);
+ }
+ s3client.createBucket(request);
+ } catch (AmazonClientException e) {
+ logger.error("Create bucket failed", e);
+ throw new CloudRuntimeException(e);
+ }
+
+ // Step 2: update the bucket record with the account's IAM credentials,
+ // configure CORS, and refresh the IAM policy. If any of these fail,
+ // clean up the remote bucket so a retry does not find it already
+ // existing — mirroring the Cloudian createBucket pattern.
+ //
+ // Hold the IAM lock for the account so a concurrent createUser key
+ // rotation does not change the account credentials between reading
+ // them and writing the BucketVO, which would leave the bucket with
+ // a stale key pair.
+ GlobalLock iamLock = acquireIamLock(storeId, accountId);
+ if (iamLock == null) {
+ // The S3 bucket has already been created. Clean it up so a
+ // retry does not find it already existing, then throw.
+ try {
+ s3client.deleteBucket(bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Failed to clean up bucket {} after IAM lock timeout", bucketName, cleanupEx);
+ }
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ // Configure permissive CORS so the CloudStack S3 bucket browser
+ // (which performs list/upload/delete from the browser) can function.
+ // SeaweedFS supports the standard PutBucketCors operation.
+ configureBucketCORS(s3client, bucketName);
+
+ Map<String, String> accountDetails = _accountDetailsDao.findDetails(accountId);
+ String accessKey = accountDetails.get(SeaweedFSObjectStoreUtil.keyAccessKey(storeId));
+ String secretKey = accountDetails.get(SeaweedFSObjectStoreUtil.keySecretKey(storeId));
+ if (accessKey == null || secretKey == null) {
+ throw new CloudRuntimeException("No IAM credentials found for account " + accountId
+ + " on store " + storeId + ". Run createUser before creating a bucket.");
+ }
+
+ String s3Url = getS3Url(storeId);
+ BucketVO bucketVO = _bucketDao.findById(bucket.getId());
+ bucketVO.setAccessKey(accessKey);
+ bucketVO.setSecretKey(secretKey);
+ bucketVO.setBucketURL(s3Url + "/" + bucketName);
+ _bucketDao.update(bucket.getId(), bucketVO);
+
+ // Refresh the account's IAM policy to include the new bucket
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicy(iamClient, storeId, accountId, null);
+
+ // Return the updated BucketVO (not the stale input bucket) so
+ // BucketApiServiceImpl.createBucket does not overwrite the
+ // persisted credentials with the stale values.
+ return bucketVO;
+ } catch (Exception e) {
+ logger.error("Post-create bucket record update failed for {}; cleaning up remote bucket", bucketName, e);
+ try {
+ s3client.deleteBucket(bucketName);
+ logger.info("Cleanup of bucket {} succeeded", bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Cleanup of bucket {} also failed", bucketName, cleanupEx);
+ }
+ // Revoke the IAM policy grant for the new bucket so the account's
+ // credentials cannot access a bucket that no longer exists. If the
+ // policy PUT succeeded before the DB update failed, the grant
+ // would otherwise persist and could be reused if another account
+ // later creates the same bucket name.
+ try {
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicy(iamClient, storeId, accountId, bucketName);
+ } catch (Exception policyEx) {
+ logger.warn("Failed to revoke IAM policy for bucket {} after cleanup: {}", bucketName, policyEx.getMessage());
+ }
+ throw new CloudRuntimeException(e);
+ } finally {
+ iamLock.unlock();
+ iamLock.releaseRef();
+ }
+ }
+
+ /**
+ * Configure a permissive CORS policy on the bucket so the CloudStack
+ * S3 bucket browser (which performs list/upload/delete from the
+ * browser) can function. Mirrors the Cloudian configureBucketCORS.
+ */
+ private void configureBucketCORS(AmazonS3 s3client, String bucketName) {
+ logger.debug("Configuring CORS for bucket {}", bucketName);
+ List<CORSRule> corsRules = new ArrayList<>();
+ CORSRule allowAnyRule = new CORSRule().withId("AllowAny");
+ allowAnyRule.setAllowedOrigins("*");
+ allowAnyRule.setAllowedHeaders("*");
+ allowAnyRule.setAllowedMethods(
+ CORSRule.AllowedMethods.HEAD,
+ CORSRule.AllowedMethods.GET,
+ CORSRule.AllowedMethods.PUT,
+ CORSRule.AllowedMethods.POST,
+ CORSRule.AllowedMethods.DELETE);
+ corsRules.add(allowAnyRule);
+ BucketCrossOriginConfiguration corsConfig = new BucketCrossOriginConfiguration();
+ corsConfig.setRules(corsRules);
+ SetBucketCrossOriginConfigurationRequest corsRequest = new SetBucketCrossOriginConfigurationRequest(bucketName, corsConfig);
+ s3client.setBucketCrossOriginConfiguration(corsRequest);
+ logger.info("Successfully configured CORS for bucket {}", bucketName);
+ }
+
+ @Override
+ public List<Bucket> listBuckets(long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ List<Bucket> bucketsList = new ArrayList<>();
+ try {
+ List<com.amazonaws.services.s3.model.Bucket> s3Buckets = s3client.listBuckets();
+ for (com.amazonaws.services.s3.model.Bucket s3Bucket : s3Buckets) {
+ Bucket bucket = new BucketObject();
+ bucket.setName(s3Bucket.getName());
+ bucketsList.add(bucket);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ return bucketsList;
+ }
+
+ @Override
+ public boolean deleteBucket(BucketTO bucket, long storeId) {
+ String bucketName = bucket.getName();
+ long accountId = bucket.getAccountId();
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Delete the S3 bucket first. If this fails (non-empty bucket,
+ // transient error), the IAM policy is still intact so the user
+ // can empty the bucket and retry.
+ //
+ // If the bucket is already gone (e.g. from a previous partial
+ // failure where the S3 delete succeeded but the IAM policy refresh
+ // failed), skip the S3 delete so the retry is idempotent.
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ s3client.deleteBucket(bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Remove the BucketVO row before refreshing the IAM policy so a
+ // concurrent createUser/createBucket policy rebuild (which reads the
+ // bucket list from the DB) cannot re-add the deleted bucket ARN
+ // between this exclusion and BucketApiServiceImpl's row removal.
+ // BucketApiServiceImpl's subsequent _bucketDao.remove is idempotent.
+ for (BucketVO bvo : _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId)) {
+ if (bucketName.equals(bvo.getName())) {
+ _bucketDao.remove(bvo.getId());
+ break;
+ }
+ }
+
+ // Refresh the account's IAM policy to drop the deleted bucket.
+ // Bucket names are reusable, so a stale grant would let the old
+ // account access a new tenant's bucket with the same name. This
+ // must succeed; if it fails, the caller sees the exception and can
+ // retry (the policy refresh is idempotent because the bucket is
+ // already gone from S3 and the DB, so excludeBucket is a no-op).
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicy(iamClient, storeId, accountId, bucketName);
+ return true;
+ }
+
+ @Override
+ public AccessControlList getBucketAcl(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ return s3client.getBucketAcl(bucket.getName());
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public void setBucketAcl(BucketTO bucket, AccessControlList acl, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ s3client.setBucketAcl(bucket.getName(), acl);
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public void setBucketPolicy(BucketTO bucket, String policy, long storeId) {
+ if ("private".equalsIgnoreCase(policy)) {
+ deleteBucketPolicy(bucket, storeId);
+ return;
+ }
+
+ StringBuilder sb = new StringBuilder();
+ sb.append("{\n");
+ sb.append(" \"Version\": \"2012-10-17\",\n");
+ sb.append(" \"Statement\": [\n");
+ sb.append(" {\n");
+ sb.append(" \"Sid\": \"PublicReadForObjects\",\n");
+ sb.append(" \"Effect\": \"Allow\",\n");
+ sb.append(" \"Principal\": \"*\",\n");
+ sb.append(" \"Action\": \"s3:GetObject\",\n");
+ sb.append(" \"Resource\": \"arn:aws:s3:::%s/*\"\n");
+ sb.append(" }\n");
+ sb.append(" ]\n");
+ sb.append("}\n");
+
+ String jsonPolicy = String.format(sb.toString(), bucket.getName());
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ s3client.setBucketPolicy(bucket.getName(), jsonPolicy);
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public BucketPolicy getBucketPolicy(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ return s3client.getBucketPolicy(new GetBucketPolicyRequest(bucket.getName()));
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public void deleteBucketPolicy(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ s3client.deleteBucketPolicy(new DeleteBucketPolicyRequest(bucket.getName()));
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public boolean setBucketEncryption(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ SetBucketEncryptionRequest eRequest = new SetBucketEncryptionRequest();
+ eRequest.setBucketName(bucket.getName());
+
+ ServerSideEncryptionByDefault sseByDefault = new ServerSideEncryptionByDefault();
+ sseByDefault.setSSEAlgorithm(SSEAlgorithm.AES256.toString());
+
+ ServerSideEncryptionRule sseRule = new ServerSideEncryptionRule();
+ sseRule.setApplyServerSideEncryptionByDefault(sseByDefault);
+
+ List<ServerSideEncryptionRule> sseRules = new ArrayList<>();
+ sseRules.add(sseRule);
+
+ ServerSideEncryptionConfiguration sseConf = new ServerSideEncryptionConfiguration();
+ sseConf.setRules(sseRules);
+
+ eRequest.setServerSideEncryptionConfiguration(sseConf);
+ s3client.setBucketEncryption(eRequest);
+ return true;
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public boolean deleteBucketEncryption(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ s3client.deleteBucketEncryption(bucket.getName());
+ return true;
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public boolean setBucketVersioning(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ BucketVersioningConfiguration vConf = new BucketVersioningConfiguration(BucketVersioningConfiguration.ENABLED);
+ s3client.setBucketVersioningConfiguration(
+ new SetBucketVersioningConfigurationRequest(bucket.getName(), vConf));
+ return true;
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public boolean deleteBucketVersioning(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ BucketVersioningConfiguration vConf = new BucketVersioningConfiguration(BucketVersioningConfiguration.SUSPENDED);
+ s3client.setBucketVersioningConfiguration(
+ new SetBucketVersioningConfigurationRequest(bucket.getName(), vConf));
+ return true;
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ /**
+ * Set the bucket quota via the SeaweedFS S3 extension
+ * ({@code PUT /{bucket}?seaweedfs-quota}), signed with the store admin
+ * S3 credentials.
+ *
+ * SeaweedFS enforces bucket quota server-side by setting a read-only flag
+ * when usage exceeds the configured limit. The quota is configured via the
+ * SeaweedFS S3 extension endpoint PUT /{bucket}?seaweedfs-quota,
+ * authenticated via standard S3 SigV4 and authorized via the
+ * s3:PutBucketQuota IAM permission.
+ *
+ * @param size the GiB size to set the quota to. 0 disables quota.
+ * @throws CloudRuntimeException if the S3 endpoint or credentials are missing or the request fails.
+ */
+ @Override
+ public void setBucketQuota(BucketTO bucket, long storeId, long size) {
+ String s3Url = getS3Url(storeId);
+ String accessKey = getAccessKey(storeId);
+ String secretKey = getSecretKey(storeId);
+ if (s3Url == null || s3Url.isEmpty() || accessKey == null || accessKey.isEmpty() || secretKey == null || secretKey.isEmpty()) {
+ throw new CloudRuntimeException("SeaweedFS S3 URL and credentials are required to set bucket quota. " +
+ "Configure 's3Url', 'accesskey', and 'secretkey' in the object store details.");
+ }
+ SeaweedFSObjectStoreUtil.setBucketQuotaViaS3Extension(s3Url, accessKey, secretKey, bucket.getName(), size, getS3ExtensionHttpClient());
+ }
+
+ /**
+ * Returns the HTTP client used to send SeaweedFS S3 extension requests
+ * (e.g. PUT /{bucket}?seaweedfs-quota). Exposed as a protected seam so
+ * tests can inject a mock client and assert the signed request without
+ * touching the network.
+ */
+ protected java.net.http.HttpClient getS3ExtensionHttpClient() {
+ return SeaweedFSObjectStoreUtil.newS3ExtensionHttpClient();
+ }
+
+ @Override
+ public Map<String, Long> getAllBucketsUsage(long storeId) {
+ Map<String, Long> bucketUsage = new HashMap<>();
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreId(storeId);
+ if (bucketList.isEmpty()) {
+ return bucketUsage;
+ }
+
+ // If the operator has configured a Prometheus metricsUrl, scrape
+ // per-bucket sizes from the /metrics endpoint in a single HTTP GET.
+ // This is O(buckets) and avoids the O(total objects) ListObjectsV2
+ // scan that doesn't scale to large deployments. Falls back to
+ // ListObjectsV2 when metricsUrl is not configured.
+ String metricsUrl = getMetricsUrl(storeId);
+ if (metricsUrl != null) {
+ java.util.Set<String> bucketNames = new java.util.HashSet<>();
+ for (BucketVO bucket : bucketList) {
+ bucketNames.add(bucket.getName());
+ }
+ try {
+ return SeaweedFSObjectStoreUtil.parseBucketUsageFromMetrics(
+ metricsUrl, bucketNames, getS3ExtensionHttpClient());
Review Comment:
The metrics parser intentionally omits managed buckets that are absent or unparseable, but `BucketApiServiceImpl.getBucketUsage()` only updates and sums entries when `size != null`. Returning this partial map therefore leaves stale bucket sizes and under-reports object-store usage instead of applying the documented zero value. Populate every managed bucket with zero before returning (or treat missing samples as a scrape failure so the S3 fallback runs).
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/util/SeaweedFSObjectStoreUtil.java:
##########
@@ -0,0 +1,555 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.util;
+
+import org.apache.commons.lang3.StringUtils;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagementClientBuilder;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3ClientBuilder;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * Utility class for the SeaweedFS object storage provider.
+ *
+ * SeaweedFS exposes both an S3-compatible API and an AWS IAM-compatible API,
+ * so this provider needs no proprietary admin client — only the AWS S3 and IAM
+ * SDKs, the same pair Cloudian HyperStore already uses in this tree.
+ */
+public class SeaweedFSObjectStoreUtil {
+
+ /** The name of our Object Store Provider */
+ public static final String OBJECT_STORE_PROVIDER_NAME = "SeaweedFS";
+
+ public static final String STORE_KEY_PROVIDER_NAME = "providerName";
+ public static final String STORE_KEY_URL = "url";
+ public static final String STORE_KEY_NAME = "name";
+ public static final String STORE_KEY_SIZE = "size";
+ public static final String STORE_KEY_DETAILS = "details";
+
+ // Store Details Map key names - managed outside of plugin
+ public static final String STORE_DETAILS_KEY_ACCESS_KEY = "accesskey"; // admin/root access key
+ public static final String STORE_DETAILS_KEY_SECRET_KEY = "secretkey"; // admin/root secret key
+ public static final String STORE_DETAILS_KEY_S3_URL = "s3Url"; // S3 endpoint URL
+ public static final String STORE_DETAILS_KEY_IAM_URL = "iamUrl"; // IAM endpoint URL
+ public static final String STORE_DETAILS_KEY_METRICS_URL = "metricsUrl"; // Prometheus metrics endpoint URL (optional, for scalable usage reporting)
+
+ // Account Detail Map key names - credentials created per CloudStack account.
+ // Namespaced by store ID so one account can use multiple SeaweedFS pools
+ // without the second pool overwriting the first pool's credentials.
+ public static final String KEY_ACCESS_KEY_PREFIX = "swfs_AccessKey_";
+ public static final String KEY_SECRET_KEY_PREFIX = "swfs_SecretKey_";
+
+ /**
+ * Build the account-detail key for the IAM access key of a given store.
+ */
+ public static String keyAccessKey(long storeId) {
+ return KEY_ACCESS_KEY_PREFIX + storeId;
+ }
+
+ /**
+ * Build the account-detail key for the IAM secret key of a given store.
+ */
+ public static String keySecretKey(long storeId) {
+ return KEY_SECRET_KEY_PREFIX + storeId;
+ }
+
+ /**
+ * Connect timeout for the S3 extension HTTP client, in seconds.
+ */
+ public static final int S3_EXTENSION_CONNECT_TIMEOUT_SECONDS = 10;
+ /**
+ * Per-request timeout for the S3 extension HTTP request, in seconds.
+ */
+ public static final int S3_EXTENSION_REQUEST_TIMEOUT_SECONDS = 30;
+
+ /**
+ * IAM user policy name applied to each per-account IAM user.
+ */
+ public static final String IAM_USER_POLICY_NAME = "CloudStackPolicy";
+
+ /**
+ * Build an IAM user policy that grants full S3 access only to the given
+ * buckets (both the bucket and its contents), while denying bucket
+ * creation and deletion everywhere so CloudStack retains control of the
+ * bucket lifecycle. When no buckets are provided, all S3 access is denied.
+ *
+ * <p>This is the tenant boundary: each account's IAM credentials can only
+ * operate on that account's own buckets, not on every bucket in the
+ * SeaweedFS pool. The policy is refreshed whenever buckets are created or
+ * deleted (see
+ * {@code SeaweedFSObjectStoreDriverImpl.updateAccountIAMPolicy}).
+ *
+ * @param bucketNames the bucket names the account is allowed to access
+ * @return a JSON IAM policy document
+ */
+ public static String buildAccountIAMPolicy(java.util.List<String> bucketNames) {
+ StringBuilder sb = new StringBuilder();
+ sb.append("{\n");
+ sb.append(" \"Version\": \"2012-10-17\",\n");
+ sb.append(" \"Statement\": [\n");
+ if (bucketNames == null || bucketNames.isEmpty()) {
+ // No buckets: deny all S3 access. A Resource cannot be empty in
+ // an IAM policy, so deny everything explicitly.
+ sb.append(" {\n");
+ sb.append(" \"Sid\": \"DenyAllS3\",\n");
+ sb.append(" \"Effect\": \"Deny\",\n");
+ sb.append(" \"Action\": [\"s3:*\"],\n");
+ sb.append(" \"Resource\": [\"arn:aws:s3:::*\", \"arn:aws:s3:::*/*\"]\n");
+ sb.append(" }\n");
+ } else {
+ sb.append(" {\n");
+ sb.append(" \"Sid\": \"AllowAccountBuckets\",\n");
+ sb.append(" \"Effect\": \"Allow\",\n");
+ sb.append(" \"Action\": [\"s3:*\"],\n");
+ sb.append(" \"Resource\": [\n");
+ for (int i = 0; i < bucketNames.size(); i++) {
+ String name = bucketNames.get(i);
+ sb.append(" \"arn:aws:s3:::").append(name).append("\",\n");
+ sb.append(" \"arn:aws:s3:::").append(name).append("/*\"");
+ if (i < bucketNames.size() - 1) {
+ sb.append(",");
+ }
+ sb.append("\n");
+ }
+ sb.append(" ]\n");
+ sb.append(" }\n");
+ }
+ // Always deny bucket creation/deletion and quota mutation —
+ // CloudStack controls lifecycle and resource accounting. Denying
+ // s3:PutBucketQuota prevents a tenant from using the credentials
+ // returned in BucketResponse to call the SeaweedFS quota extension
+ // directly and bypass CloudStack's resource accounting.
+ sb.append(" ,{\n");
+ sb.append(" \"Sid\": \"DenyBucketLifecycleAndQuota\",\n");
+ sb.append(" \"Effect\": \"Deny\",\n");
+ sb.append(" \"Action\": [\"s3:CreateBucket\", \"s3:DeleteBucket\", \"s3:PutBucketQuota\"],\n");
+ sb.append(" \"Resource\": \"*\"\n");
+ sb.append(" }\n");
+ sb.append(" ]\n");
+ sb.append("}\n");
+ return sb.toString();
+ }
+
+ /**
+ * Returns an S3 connection for the given endpoint and credentials.
+ * Uses path-style access, which SeaweedFS requires.
+ *
+ * @param url the url of the S3 service
+ * @param accessKey the credentials to use for the S3 connection.
+ * @param secretKey the matching secret key.
+ * @return an S3 connection (never null)
+ * @throws CloudRuntimeException on failure.
+ */
+ public static AmazonS3 getS3Client(String url, String accessKey, String secretKey) {
+ AmazonS3 client = AmazonS3ClientBuilder.standard()
+ .enablePathStyleAccess()
+ .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))
+ .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(url, "us-east-1"))
+ .build();
+ if (client == null) {
+ throw new CloudRuntimeException("Error while creating SeaweedFS S3 client");
+ }
+ return client;
+ }
+
+ /**
+ * Returns an IAM connection for the given endpoint and credentials.
+ *
+ * @param url the url of the IAM service
+ * @param accessKey the credentials to use for the iam connection.
+ * @param secretKey the matching secret key.
+ * @return an IAM connection (never null)
+ * @throws CloudRuntimeException on failure.
+ */
+ public static AmazonIdentityManagement getIAMClient(String url, String accessKey, String secretKey) {
+ AmazonIdentityManagement iamClient = AmazonIdentityManagementClientBuilder.standard()
+ .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))
+ .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(url, "us-east-1"))
+ .build();
+ if (iamClient == null) {
+ throw new CloudRuntimeException("Error while creating SeaweedFS IAM client");
+ }
+ return iamClient;
+ }
+
+ /**
+ * Test the S3Url to confirm it behaves like an S3 Service.
+ *
+ * Uses bad credentials and looks for the particular error from S3 that says
+ * InvalidAccessKeyId was used. Quietly returns if we connect and get the
+ * expected error back.
+ *
+ * @param s3Url the url to check
+ * @throws CloudRuntimeException if there is any unexpected issue.
+ */
+ public static void validateS3Url(String s3Url) {
+ try {
+ AmazonS3 s3Client = SeaweedFSObjectStoreUtil.getS3Client(s3Url, "unknown", "unknown");
+ s3Client.listBuckets();
+ } catch (AmazonServiceException e) {
+ if (StringUtils.compareIgnoreCase(e.getErrorCode(), "InvalidAccessKeyId") != 0
+ && StringUtils.compareIgnoreCase(e.getErrorCode(), "SignatureDoesNotMatch") != 0) {
+ throw new CloudRuntimeException("Unexpected response from S3 Endpoint.", e);
+ }
+ }
+ }
+
+ /**
+ * Test the IAMUrl to confirm it behaves like an IAM Service.
+ *
+ * Uses bad credentials and looks for the particular error from IAM that says
+ * InvalidAccessKeyId or InvalidClientTokenId was used. Quietly returns if we
+ * connect and get the expected error back.
+ *
+ * @param iamUrl the url to check
+ * @throws CloudRuntimeException if there is any unexpected issue.
+ */
+ public static void validateIAMUrl(String iamUrl) {
+ try {
+ AmazonIdentityManagement iamClient = SeaweedFSObjectStoreUtil.getIAMClient(iamUrl, "unknown", "unknown");
+ iamClient.listAccessKeys();
+ } catch (AmazonServiceException e) {
+ if (! StringUtils.equalsAnyIgnoreCase(e.getErrorCode(), "InvalidAccessKeyId", "InvalidClientTokenId", "SignatureDoesNotMatch")) {
+ throw new CloudRuntimeException("Unexpected response from IAM Endpoint.", e);
+ }
+ }
+ }
+
+ /**
+ * Set bucket quota via the SeaweedFS S3 extension endpoint.
+ *
+ * SeaweedFS exposes a custom S3 subresource at
+ * PUT /{bucket}?seaweedfs-quota
+ * authenticated via standard S3 SigV4 and authorized via the
+ * s3:PutBucketQuota IAM permission. This avoids the need for a
+ * separate admin API credential.
+ *
+ * The request body is JSON:
+ * {"quota_size": <n>, "quota_unit": "GB", "quota_enabled": true}
+ *
+ * @param s3Url the S3 endpoint URL (e.g. http://host:8333)
+ * @param accessKey the S3 access key (must have s3:PutBucketQuota permission)
+ * @param secretKey the S3 secret key
+ * @param bucketName the bucket name
+ * @param sizeGiB the quota size in GiB (0 to disable quota)
+ * @throws CloudRuntimeException on any failure
+ */
+ public static void setBucketQuotaViaS3Extension(String s3Url, String accessKey, String secretKey, String bucketName, long sizeGiB) {
+ setBucketQuotaViaS3Extension(s3Url, accessKey, secretKey, bucketName, sizeGiB, newS3ExtensionHttpClient());
+ }
+
+ /**
+ * Build a bounded HTTP client for SeaweedFS S3 extension requests with a
+ * connect timeout so a stalled endpoint cannot block the management-server
+ * API thread indefinitely.
+ */
+ public static java.net.http.HttpClient newS3ExtensionHttpClient() {
+ return java.net.http.HttpClient.newBuilder()
+ .connectTimeout(java.time.Duration.ofSeconds(S3_EXTENSION_CONNECT_TIMEOUT_SECONDS))
+ .build();
+ }
+
+ /**
+ * Set bucket quota via the SeaweedFS S3 extension endpoint using the
+ * supplied HTTP client. The client is injected so tests can assert the
+ * signed request without hitting the network.
+ */
+ public static void setBucketQuotaViaS3Extension(String s3Url, String accessKey, String secretKey,
+ String bucketName, long sizeGiB, java.net.http.HttpClient httpClient) {
+ if (sizeGiB < 0) {
+ // Only zero disables a quota; a negative value would corrupt
+ // resource accounting (BucketApiServiceImpl persists the requested
+ // value and computes deltas from it), so reject it outright.
+ throw new CloudRuntimeException("Bucket quota cannot be negative: " + sizeGiB);
+ }
+ String body;
+ if (sizeGiB == 0) {
+ body = "{\"quota_size\":0,\"quota_unit\":\"B\",\"quota_enabled\":false}";
+ } else {
+ body = String.format("{\"quota_size\":%d,\"quota_unit\":\"GB\",\"quota_enabled\":true}", sizeGiB);
+ }
+ try {
+ executeSignedS3Request("PUT", s3Url, "/" + bucketName + "?seaweedfs-quota", accessKey, secretKey, body, httpClient);
+ } catch (CloudRuntimeException e) {
+ // A quota of 0 disables the quota, which is the default state for
+ // a newly created bucket. If the SeaweedFS quota extension is not
+ // available (404/405), tolerate the failure for quota 0 so basic
+ // bucket CRUD works on deployments without the extension. A
+ // positive quota still requires the extension and must fail.
+ //
+ // Distinguish "extension not available" from "bucket not found":
+ // SeaweedFS returns a standard S3 NoSuchBucket error (with
+ // <Code>NoSuchBucket</Code> in the body) when the bucket does not
+ // exist, which must NOT be swallowed — it indicates CloudStack and
+ // S3 are out of sync.
+ if (sizeGiB == 0 && e.getMessage() != null
+ && (e.getMessage().contains("status 404") || e.getMessage().contains("status 405"))
+ && !e.getMessage().contains("NoSuchBucket")) {
+ org.apache.logging.log4j.LogManager.getLogger(SeaweedFSObjectStoreUtil.class)
+ .warn("SeaweedFS quota extension not available for bucket {}; skipping quota disable (quota is already off by default)", bucketName);
+ return;
+ }
+ throw e;
+ }
+ }
+
+ /**
+ * Execute a custom S3 request with SigV4 signing.
+ *
+ * Uses the AWS SDK v1 Aws4Signer to sign the request, then sends it via
+ * java.net.http.HttpClient. This allows calling SeaweedFS-specific S3
+ * extensions (like ?seaweedfs-quota) that the AWS SDK doesn't natively
+ * support.
+ *
+ * The query string portion of {@code resourcePath} (e.g.
+ * {@code /bucket?seaweedfs-quota}) is split off and added to the request
+ * via {@code addParameter(...)} before signing, so the signer includes it
+ * in the canonical query string. {@code DefaultRequest.setResourcePath}
+ * does not parse an embedded query string, so passing it verbatim would
+ * leave the subresource unsigned while the outgoing URI would still carry
+ * it, causing a signature mismatch on the server.
+ *
+ * @param method HTTP method (PUT, GET, etc.)
+ * @param s3Url the S3 endpoint base URL
+ * @param resourcePath the path + optional query string (e.g. /bucket?seaweedfs-quota)
+ * @param accessKey S3 access key
+ * @param secretKey S3 secret key
+ * @param body the request body (null for GET)
+ * @param httpClient the HTTP client used to send the request
+ * @return the response body as a string
+ * @throws CloudRuntimeException on any failure
+ */
+ protected static String executeSignedS3Request(String method, String s3Url, String resourcePath,
+ String accessKey, String secretKey, String body,
+ java.net.http.HttpClient httpClient) {
+ try {
+ java.net.URI endpointUri = java.net.URI.create(s3Url);
+
+ // Split the resource path into a path and a query string so the
+ // query parameters are signed as canonical query parameters.
+ String path = resourcePath;
+ String queryString = "";
+ int q = resourcePath.indexOf('?');
+ if (q >= 0) {
+ path = resourcePath.substring(0, q);
+ queryString = resourcePath.substring(q + 1);
+ }
+
+ // Prepend the endpoint's path prefix (e.g. /object-s3) to the
+ // resource path so the SigV4 canonical URI matches the outgoing
+ // request URI. Without this, a path-prefixed endpoint behind a
+ // reverse proxy would sign /bucket but send /object-s3/bucket,
+ // causing SignatureDoesNotMatch.
+ String endpointPath = endpointUri.getPath();
+ if (endpointPath == null) {
+ endpointPath = "";
+ }
+ if (endpointPath.endsWith("/")) {
+ endpointPath = endpointPath.substring(0, endpointPath.length() - 1);
+ }
+ String signedResourcePath = endpointPath + path;
+
+ // Build AWS SDK v1 Request for SigV4 signing
+ com.amazonaws.DefaultRequest<?> request = new com.amazonaws.DefaultRequest<>("s3");
+ request.setEndpoint(endpointUri);
+ request.setHttpMethod(com.amazonaws.http.HttpMethodName.valueOf(method));
+ request.setResourcePath(signedResourcePath);
+ if (! queryString.isEmpty()) {
+ for (String pair : queryString.split("&")) {
+ if (pair.isEmpty()) {
+ continue;
+ }
+ int eq = pair.indexOf('=');
+ if (eq >= 0) {
+ request.addParameter(pair.substring(0, eq), pair.substring(eq + 1));
+ } else {
+ request.addParameter(pair, "");
+ }
+ }
+ }
+ if (body != null) {
+ byte[] bodyBytes = body.getBytes(java.nio.charset.StandardCharsets.UTF_8);
+ request.setContent(new java.io.ByteArrayInputStream(bodyBytes));
+ request.getHeaders().put("Content-Length", String.valueOf(bodyBytes.length));
+ request.getHeaders().put("Content-Type", "application/json");
+ }
+
+ // Sign with SigV4 (AWSS3V4Signer, not the legacy S3Signer which is SigV2)
+ com.amazonaws.auth.AWSCredentials credentials = new com.amazonaws.auth.BasicAWSCredentials(accessKey, secretKey);
+ com.amazonaws.services.s3.internal.AWSS3V4Signer signer = new com.amazonaws.services.s3.internal.AWSS3V4Signer();
+ signer.setServiceName("s3");
+ signer.setRegionName("us-east-1");
+ signer.sign(request, credentials);
+
+ // Build and send the HTTP request with signed headers. The URI
+ // carries the original query string; the signed headers (including
+ // Authorization) are copied from the signed request. Restricted
+ // headers (e.g. Content-Length, Host) are set by the HTTP client /
+ // URI itself and cannot be added via HttpRequest.Builder.header(),
+ // so they are skipped here.
+ // Build the outgoing URI preserving the endpoint path prefix (e.g.
+ // https://host/object-s3) by concatenating it with the resource
+ // path. This matches the signed resource path so SigV4 verifies.
+ java.net.URI fullUri = java.net.URI.create(
+ endpointUri.getScheme() + "://" + endpointUri.getRawAuthority()
+ + endpointPath + path);
+ if (! queryString.isEmpty()) {
+ fullUri = java.net.URI.create(fullUri.toString() + "?" + queryString);
+ }
+ java.net.http.HttpRequest.Builder reqBuilder = java.net.http.HttpRequest.newBuilder()
+ .uri(fullUri)
+ .timeout(java.time.Duration.ofSeconds(S3_EXTENSION_REQUEST_TIMEOUT_SECONDS));
+ for (java.util.Map.Entry<String, String> entry : request.getHeaders().entrySet()) {
+ String headerName = entry.getKey();
+ if (headerName == null || entry.getValue() == null) {
+ continue;
+ }
+ if (isRestrictedHttpHeader(headerName)) {
+ continue;
+ }
+ reqBuilder.header(headerName, entry.getValue());
+ }
+ if (body != null) {
+ reqBuilder.method(method, java.net.http.HttpRequest.BodyPublishers.ofString(body));
+ } else {
+ reqBuilder.method(method, java.net.http.HttpRequest.BodyPublishers.noBody());
+ }
+
+ java.net.http.HttpResponse<String> response = httpClient.send(reqBuilder.build(),
+ java.net.http.HttpResponse.BodyHandlers.ofString());
+
+ int statusCode = response.statusCode();
+ if (statusCode < 200 || statusCode >= 300) {
+ throw new CloudRuntimeException(String.format(
+ "S3 extension request %s %s failed with status %d: %s",
+ method, fullUri, statusCode, response.body()));
+ }
+ return response.body();
+ } catch (CloudRuntimeException e) {
+ throw e;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("S3 extension request failed: " + method + " " + resourcePath, e);
+ }
+ }
+
+ /**
+ * Headers that {@code java.net.http.HttpRequest.Builder.header()} rejects
+ * because they are managed by the HTTP client itself (content length is
+ * derived from the body publisher, host from the URI, etc.). They must be
+ * skipped when copying the signed headers onto the outgoing request.
+ */
+ private static boolean isRestrictedHttpHeader(String headerName) {
+ if (headerName == null) {
+ return true;
+ }
+ switch (headerName.toLowerCase(java.util.Locale.ROOT)) {
+ case "content-length":
+ case "host":
+ case "connection":
+ case "expect":
+ case "upgrade":
+ return true;
+ default:
+ return false;
+ }
+ }
+
+ /**
+ * Prometheus metric name for per-bucket logical size. SeaweedFS publishes
+ * this gauge from the S3 API server's bucket-size metrics loop.
+ */
+ public static final String METRIC_BUCKET_SIZE_BYTES = "seaweed_s3_bucket_size_bytes";
+
+ /**
+ * Scrape the SeaweedFS Prometheus {@code /metrics} endpoint and return a
+ * map of bucket name to logical size in bytes.
+ *
+ * <p>This is a single HTTP GET that returns all bucket sizes in O(buckets)
+ * time, replacing the O(total objects) {@code ListObjectsV2} scan used as a
+ * fallback. The operator configures {@code metricsUrl} as a store detail
+ * pointing at the SeaweedFS S3 server's metrics port (or a Prometheus
+ * server that scrapes it).
Review Comment:
`metricsUrl` is documented as accepting a Prometheus server, but the implementation appends `/metrics` and parses the response as SeaweedFS exporter text. A Prometheus server's `/metrics` endpoint exposes Prometheus's own metrics, not the scraped `seaweed_s3_bucket_size_bytes` series, so this configuration produces an empty/partial map and suppresses the S3 fallback. Query the Prometheus HTTP API instead, or restrict the contract to a SeaweedFS exporter endpoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r4012630288
##########
server/src/main/java/com/cloud/storage/StorageManagerImpl.java:
##########
@@ -4812,6 +4813,30 @@ public ObjectStore updateObjectStore(Long id, UpdateObjectStoragePoolCmd cmd) {
return objectStoreVO;
}
+ /**
+ * Rewrite the stored bucketURL of every bucket on an object store after the
+ * store URL changes.
+ *
+ * BucketVO.bucketURL is written only at bucket creation. BucketResponse
+ * exposes it and the object-store browser builds its S3 client from it, so
+ * without this the browser keeps targeting the old endpoint after a URL
+ * change while the management server uses the new one.
+ */
+ private void updateBucketUrls(Long storeId, String oldUrl, String newUrl) {
+ if (oldUrl == null || oldUrl.equals(newUrl)) {
+ return;
+ }
+ for (BucketVO bucket : _bucketDao.listByObjectStoreId(storeId)) {
+ String bucketUrl = bucket.getBucketURL();
+ if (bucketUrl == null || !bucketUrl.startsWith(oldUrl)) {
Review Comment:
`startsWith(oldUrl)` also matches unrelated explicit endpoints whose text merely begins with the old URL. For example, with a store URL of `http://host` and an explicit SeaweedFS S3 endpoint of `http://host:8333`, changing the store URL rewrites that override and corrupts the bucket URL. Require an endpoint boundary (normalized `oldUrl + "/"`) before replacing.
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,931 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.BucketCrossOriginConfiguration;
+import com.amazonaws.services.s3.model.CORSRule;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketCrossOriginConfigurationRequest;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ /**
+ * DB-backed global lock name prefix for serializing IAM provisioning and
+ * policy refreshes per store+account. Uses {@link GlobalLock} so the
+ * critical section is serialized across management servers in a
+ * clustered deployment, not just within a single JVM.
+ */
+ private static final String IAM_LOCK_PREFIX = "seaweedfs.iam.";
+
+ private static String getIamLockName(long storeId, long accountId) {
+ return IAM_LOCK_PREFIX + storeId + "." + accountId;
+ }
+
+ /**
+ * Acquire a DB-backed global lock for IAM operations on the given
+ * store+account. Returns a {@link GlobalLock} that the caller must
+ * {@link GlobalLock#unlock()} in a {@code finally} block, or {@code null}
+ * if the lock could not be acquired within the timeout.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ */
+ protected GlobalLock acquireIamLock(long storeId, long accountId) {
+ GlobalLock lock = GlobalLock.getInternLock(getIamLockName(storeId, accountId));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire IAM lock for store {} account {}", storeId, accountId);
+ lock.releaseRef();
+ return null;
+ }
+ return lock;
+ }
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account and
+ * store. The store ID is included so that two CloudStack pools pointing
+ * at the same SeaweedFS IAM service do not collide on the same
+ * {@code acs-<uuid>} user and overwrite each other's policy and access
+ * keys.
+ */
+ protected String getUserNameForAccount(Account account, long storeId) {
+ return String.format("%s-%d-%s", ACS_PREFIX, storeId, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Serialize per store+account across management servers so two
+ // concurrent bucket requests do not both rotate credentials and leave
+ // bucket rows with mismatched key pairs.
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ return false;
+ }
+ try {
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach a scoped IAM policy that allows access only to this
+ // account's own buckets (the tenant boundary). Refreshed whenever
+ // buckets are created or deleted. Use the lock-free variant since
+ // createUser already holds the IAM lock.
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, null);
+
+ // Reuse the stored access key only if both the access key id and the
+ // secret key are present and the key is still Active in IAM; otherwise
+ // create a replacement.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ String storedSecretKey = details.get(secretKeyDetailKey);
+ if (storedAccessKeyId != null && storedSecretKey != null
+ && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ return true;
+ }
+
+ // The stored key is missing, inactive, or no longer in IAM. Clean up
+ // ALL keys (including the inactive stored one) before creating a
+ // replacement so we do not accumulate keys and hit IAM limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, null);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Update existing bucket records for this account/store with the new
+ // credentials BEFORE persisting the new key in account details. If a
+ // bucket update fails, the stored key remains the old one and a retry
+ // will re-enter the replacement path; if we persisted first, a retry
+ // would see the new stored key and return without repairing the
+ // remaining buckets.
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ // Persist the credentials in the account details (namespaced by storeId)
+ details.put(accessKeyDetailKey, key.getAccessKeyId());
+ details.put(secretKeyDetailKey, key.getSecretAccessKey());
+ _accountDetailsDao.persist(accountId, details);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Update the IAM credentials on all BucketVO rows for this store/account
+ * so previously created buckets reflect the new (rotated) key pair.
+ * Mirrors CloudianHyperStoreObjectStoreDriverImpl.updateAccountBucketCredentials.
+ */
+ private void updateAccountBucketCredentials(long storeId, long accountId, AccessKey iamCredential) {
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ for (BucketVO bucketVO : bucketList) {
+ logger.info("Updating accountId={} bucket {} with new IAM credentials", accountId, bucketVO.getName());
+ bucketVO.setAccessKey(iamCredential.getAccessKeyId());
+ bucketVO.setSecretKey(iamCredential.getSecretAccessKey());
+ _bucketDao.update(bucketVO.getId(), bucketVO);
+ }
+ }
+
+ /**
+ * Refresh the per-account IAM user policy so it grants S3 access only to
+ * the account's current buckets (optionally excluding one, e.g. a bucket
+ * being deleted). This is the tenant boundary: each account's IAM
+ * credentials can only operate on that account's own buckets.
+ *
+ * Acquires the per-store/account IAM lock. Callers that already hold the
+ * lock (e.g. createUser, createBucket post-create) should call
+ * {@link #updateAccountIAMPolicyLocked} instead to avoid re-entrant lock
+ * acquisition warnings from GlobalLock.
+ *
+ * @param iamClient the IAM client
+ * @param storeId the object store
+ * @param accountId the CloudStack account
+ * @param excludeBucket a bucket name to omit (e.g. a bucket being deleted),
+ * or null to include all of the account's buckets
+ */
+ protected void updateAccountIAMPolicy(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, excludeBucket);
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Lock-free variant of {@link #updateAccountIAMPolicy} for callers that
+ * already hold the per-store/account IAM lock. Performs the policy refresh
+ * without reacquiring the lock, avoiding the GlobalLock re-entrant
+ * acquisition warning.
+ */
+ protected void updateAccountIAMPolicyLocked(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ return;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ List<BucketVO> buckets = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ List<String> bucketNames = new ArrayList<>();
+ for (BucketVO bvo : buckets) {
+ if (excludeBucket != null && excludeBucket.equals(bvo.getName())) {
+ continue;
+ }
+ // Skip buckets whose remote counterpart has been deleted but whose
+ // row is still present for resource accounting. Including them
+ // would re-grant a bucket name that is now free for another account
+ // to claim.
+ if (Bucket.State.Destroyed.equals(bvo.getState())) {
+ continue;
+ }
+ bucketNames.add(bvo.getName());
+ }
+ String policy = SeaweedFSObjectStoreUtil.buildAccountIAMPolicy(bucketNames);
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ SeaweedFSObjectStoreUtil.IAM_USER_POLICY_NAME, policy));
+ }
+
+ /**
+ * Check whether the given access key id is still listed and Active in IAM
+ * for the user. Listing failures are propagated rather than swallowed so
+ * a transient IAM outage does not send createUser into the replacement
+ * path (which would overwrite stored credentials and invalidate bucket
+ * records).
+ */
+ private boolean iamAccessKeyExists(AmazonIdentityManagement iamClient, String userName, String accessKeyId) {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ if (accessKeyId.equals(metadata.getAccessKeyId())) {
+ return "Active".equalsIgnoreCase(metadata.getStatus());
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Delete access keys for the user other than the (optionally) preserved
+ * key id. Used to clean up unmanaged leftover keys before creating a
+ * replacement so repeated calls do not hit IAM access-key limits.
+ */
+ private void deleteUnmanagedAccessKeys(AmazonIdentityManagement iamClient, String userName, String preserveAccessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ String keyId = metadata.getAccessKeyId();
+ if (preserveAccessKeyId != null && preserveAccessKeyId.equals(keyId)) {
+ continue;
+ }
+ DeleteAccessKeyRequest deleteReq =
+ new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(keyId);
+ logger.info("Deleting un-managed IAM access key {} for user {}", keyId, userName);
+ iamClient.deleteAccessKey(deleteReq);
+ }
+ } catch (AmazonClientException e) {
+ // Propagate so the caller does not proceed to create a replacement
+ // key while stale unmanaged keys remain (which could hit IAM key
+ // limits or leave orphaned credentials).
+ throw new CloudRuntimeException("Failed to clean up IAM access keys for user " + userName, e);
+ }
+ }
+
+ @Override
+ public Bucket createBucket(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+ long accountId = bucket.getAccountId();
+
+ // Use the store's admin credentials to create the bucket
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Check if the bucket already exists
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket already exists with name " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Create the bucket
+ try {
+ CreateBucketRequest request = new CreateBucketRequest(bucketName);
+ if (objectLock) {
+ request.setObjectLockEnabledForBucket(true);
+ }
+ s3client.createBucket(request);
+ } catch (AmazonClientException e) {
+ logger.error("Create bucket failed", e);
+ throw new CloudRuntimeException(e);
+ }
+
+ // Step 2: update the bucket record with the account's IAM credentials,
+ // configure CORS, and refresh the IAM policy. If any of these fail,
+ // clean up the remote bucket so a retry does not find it already
+ // existing — mirroring the Cloudian createBucket pattern.
+ //
+ // Hold the IAM lock for the account so a concurrent createUser key
+ // rotation does not change the account credentials between reading
+ // them and writing the BucketVO, which would leave the bucket with
+ // a stale key pair.
+ GlobalLock iamLock = acquireIamLock(storeId, accountId);
+ if (iamLock == null) {
+ // The S3 bucket has already been created. Clean it up so a
+ // retry does not find it already existing, then throw.
+ try {
+ s3client.deleteBucket(bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Failed to clean up bucket {} after IAM lock timeout", bucketName, cleanupEx);
+ }
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ // Configure permissive CORS so the CloudStack S3 bucket browser
+ // (which performs list/upload/delete from the browser) can function.
+ // SeaweedFS supports the standard PutBucketCors operation.
+ configureBucketCORS(s3client, bucketName);
+
+ Map<String, String> accountDetails = _accountDetailsDao.findDetails(accountId);
+ String accessKey = accountDetails.get(SeaweedFSObjectStoreUtil.keyAccessKey(storeId));
+ String secretKey = accountDetails.get(SeaweedFSObjectStoreUtil.keySecretKey(storeId));
+ if (accessKey == null || secretKey == null) {
+ throw new CloudRuntimeException("No IAM credentials found for account " + accountId
+ + " on store " + storeId + ". Run createUser before creating a bucket.");
+ }
+
+ String s3Url = getS3Url(storeId);
+ BucketVO bucketVO = _bucketDao.findById(bucket.getId());
+ bucketVO.setAccessKey(accessKey);
+ bucketVO.setSecretKey(secretKey);
+ bucketVO.setBucketURL(s3Url + "/" + bucketName);
+ _bucketDao.update(bucket.getId(), bucketVO);
+
+ // Refresh the account's IAM policy to include the new bucket.
+ // Use the lock-free variant since createBucket already holds the
+ // IAM lock for the post-create section.
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, null);
+
+ // Return the updated BucketVO (not the stale input bucket) so
+ // BucketApiServiceImpl.createBucket does not overwrite the
+ // persisted credentials with the stale values.
+ return bucketVO;
+ } catch (Exception e) {
+ logger.error("Post-create bucket record update failed for {}; cleaning up remote bucket", bucketName, e);
+ CloudRuntimeException primary = new CloudRuntimeException(e);
+ try {
+ s3client.deleteBucket(bucketName);
+ logger.info("Cleanup of bucket {} succeeded", bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Cleanup of bucket {} also failed", bucketName, cleanupEx);
+ primary.addSuppressed(cleanupEx);
+ }
+ // Revoke the IAM policy grant for the new bucket so the account's
+ // credentials cannot access a bucket that no longer exists. If the
+ // policy PUT succeeded before the DB update failed, the grant
+ // would otherwise persist and could be reused if another account
+ // later creates the same bucket name. Use the lock-free variant
+ // since createBucket already holds the IAM lock. Propagate
+ // failures as suppressed exceptions so they are not silently lost.
+ try {
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, bucketName);
+ } catch (Exception policyEx) {
+ logger.warn("Failed to revoke IAM policy for bucket {} after cleanup: {}", bucketName, policyEx.getMessage());
+ primary.addSuppressed(policyEx);
+ }
+ throw primary;
+ } finally {
+ iamLock.unlock();
+ iamLock.releaseRef();
+ }
+ }
+
+ /**
+ * Configure a permissive CORS policy on the bucket so the CloudStack
+ * S3 bucket browser (which performs list/upload/delete from the
+ * browser) can function. Mirrors the Cloudian configureBucketCORS.
+ */
+ private void configureBucketCORS(AmazonS3 s3client, String bucketName) {
+ logger.debug("Configuring CORS for bucket {}", bucketName);
+ List<CORSRule> corsRules = new ArrayList<>();
+ CORSRule allowAnyRule = new CORSRule().withId("AllowAny");
+ allowAnyRule.setAllowedOrigins("*");
+ allowAnyRule.setAllowedHeaders("*");
+ allowAnyRule.setAllowedMethods(
+ CORSRule.AllowedMethods.HEAD,
+ CORSRule.AllowedMethods.GET,
+ CORSRule.AllowedMethods.PUT,
+ CORSRule.AllowedMethods.POST,
+ CORSRule.AllowedMethods.DELETE);
+ corsRules.add(allowAnyRule);
+ BucketCrossOriginConfiguration corsConfig = new BucketCrossOriginConfiguration();
+ corsConfig.setRules(corsRules);
+ SetBucketCrossOriginConfigurationRequest corsRequest = new SetBucketCrossOriginConfigurationRequest(bucketName, corsConfig);
+ s3client.setBucketCrossOriginConfiguration(corsRequest);
+ logger.info("Successfully configured CORS for bucket {}", bucketName);
+ }
+
+ @Override
+ public List<Bucket> listBuckets(long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ List<Bucket> bucketsList = new ArrayList<>();
+ try {
+ List<com.amazonaws.services.s3.model.Bucket> s3Buckets = s3client.listBuckets();
+ for (com.amazonaws.services.s3.model.Bucket s3Bucket : s3Buckets) {
+ Bucket bucket = new BucketObject();
+ bucket.setName(s3Bucket.getName());
+ bucketsList.add(bucket);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ return bucketsList;
+ }
+
+ @Override
+ public boolean deleteBucket(BucketTO bucket, long storeId) {
+ String bucketName = bucket.getName();
+ long accountId = bucket.getAccountId();
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Acquire the per-store/account IAM lock so the S3 delete and the
+ // subsequent policy refresh are atomic with respect to concurrent
+ // createUser/createBucket operations. Without the lock, a concurrent
+ // policy rebuild could re-add the deleted bucket ARN between the
+ // excludeBucket refresh and BucketApiServiceImpl's row removal.
Review Comment:
Using a lock keyed by `(storeId, accountId)` does not make this sequence atomic with another account. After this code deletes the globally named S3 bucket, account B can pass `doesBucketExistV2` and create the same name while account A's policy still contains that ARN; until the later policy refresh completes, A's credentials can access B's bucket. Serialize bucket-name reuse and policy refresh with a store-wide (or bucket-name) lock, including the create path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#issuecomment-5735459517
## [Codecov](https://app.codecov.io/gh/apache/cloudstack/pull/14160?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 3.70%. Comparing base ([`dc0ba1c`](https://app.codecov.io/gh/apache/cloudstack/commit/dc0ba1cba7475f2c5e0d6346a0e974549c35f00e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`932cc3c`](https://app.codecov.io/gh/apache/cloudstack/commit/932cc3c95a8015306e89015834544c518274fec1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 13 commits behind head on main.
> :exclamation: There is a different number of reports uploaded between BASE (dc0ba1c) and HEAD (932cc3c). Click for more details.
>
> <details><summary>HEAD has 1 upload less than BASE</summary>
>
>| Flag | BASE (dc0ba1c) | HEAD (932cc3c) |
>|------|------|------|
>|unittests|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #14160 +/- ##
=============================================
- Coverage 19.89% 3.70% -16.19%
=============================================
Files 6371 487 -5884
Lines 576829 42006 -534823
Branches 70627 7946 -62681
=============================================
- Hits 114778 1558 -113220
+ Misses 449507 40222 -409285
+ Partials 12544 226 -12318
```
| [Flag](https://app.codecov.io/gh/apache/cloudstack/pull/14160/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [uitests](https://app.codecov.io/gh/apache/cloudstack/pull/14160/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `3.70% <ø> (-0.01%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/cloudstack/pull/14160/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/cloudstack/pull/14160?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "chrislusf (via GitHub)" <gi...@apache.org>.
chrislusf commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r4017611100
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,938 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.BucketCrossOriginConfiguration;
+import com.amazonaws.services.s3.model.CORSRule;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketCrossOriginConfigurationRequest;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ /**
+ * DB-backed global lock name prefix for serializing IAM provisioning and
+ * policy refreshes per store+account. Uses {@link GlobalLock} so the
+ * critical section is serialized across management servers in a
+ * clustered deployment, not just within a single JVM.
+ */
+ private static final String IAM_LOCK_PREFIX = "seaweedfs.iam.";
+
+ private static String getIamLockName(long storeId, long accountId) {
+ return IAM_LOCK_PREFIX + storeId + "." + accountId;
+ }
+
+ /**
+ * Acquire a DB-backed global lock for IAM operations on the given
+ * store+account. Returns a {@link GlobalLock} that the caller must
+ * {@link GlobalLock#unlock()} in a {@code finally} block, or {@code null}
+ * if the lock could not be acquired within the timeout.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ */
+ protected GlobalLock acquireIamLock(long storeId, long accountId) {
+ GlobalLock lock = GlobalLock.getInternLock(getIamLockName(storeId, accountId));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire IAM lock for store {} account {}", storeId, accountId);
+ lock.releaseRef();
+ return null;
+ }
+ return lock;
+ }
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account and
+ * store. The store ID is included so that two CloudStack pools pointing
+ * at the same SeaweedFS IAM service do not collide on the same
+ * {@code acs-<uuid>} user and overwrite each other's policy and access
+ * keys.
+ */
+ protected String getUserNameForAccount(Account account, long storeId) {
+ return String.format("%s-%d-%s", ACS_PREFIX, storeId, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Serialize per store+account across management servers so two
+ // concurrent bucket requests do not both rotate credentials and leave
+ // bucket rows with mismatched key pairs.
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ return false;
+ }
+ try {
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach a scoped IAM policy that allows access only to this
+ // account's own buckets (the tenant boundary). Refreshed whenever
+ // buckets are created or deleted. Use the lock-free variant since
+ // createUser already holds the IAM lock.
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, null);
+
+ // Reuse the stored access key only if both the access key id and the
+ // secret key are present and the key is still Active in IAM; otherwise
+ // create a replacement.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ String storedSecretKey = details.get(secretKeyDetailKey);
+ if (storedAccessKeyId != null && storedSecretKey != null
+ && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ return true;
+ }
+
+ // The stored key is missing, inactive, or no longer in IAM. Clean up
+ // ALL keys (including the inactive stored one) before creating a
+ // replacement so we do not accumulate keys and hit IAM limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, null);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Update existing bucket records for this account/store with the new
+ // credentials BEFORE persisting the new key in account details. If a
+ // bucket update fails, the stored key remains the old one and a retry
+ // will re-enter the replacement path; if we persisted first, a retry
+ // would see the new stored key and return without repairing the
+ // remaining buckets.
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ // Persist the credentials in the account details (namespaced by
+ // storeId) with per-key writes. AccountDetailsDao.persist(accountId,
+ // map) expunges every existing detail for the account before inserting
+ // the supplied map, so it would clobber details this snapshot never saw
+ // — including another object store's namespaced credentials, since the
+ // IAM lock is keyed by storeId+accountId and two pools can provision
+ // the same account concurrently. addDetail touches only the named key.
+ details.put(accessKeyDetailKey, key.getAccessKeyId());
+ details.put(secretKeyDetailKey, key.getSecretAccessKey());
+ _accountDetailsDao.addDetail(accountId, accessKeyDetailKey, key.getAccessKeyId(), false);
+ _accountDetailsDao.addDetail(accountId, secretKeyDetailKey, key.getSecretAccessKey(), false);
Review Comment:
Fixed in `932cc3c`. `createUser` now persists the access/secret pair via `persistAccountCredentialsOrRollback`; if either DAO write fails it restores the prior details and deletes the newly-created IAM key, so a mixed credential pair is not left behind. I also added coverage for the failed-secret-write path and for reconciling bucket rows when reusing a stored key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r4013302489
##########
server/src/main/java/com/cloud/storage/StorageManagerImpl.java:
##########
@@ -4812,6 +4813,57 @@ public ObjectStore updateObjectStore(Long id, UpdateObjectStoragePoolCmd cmd) {
return objectStoreVO;
}
+ /**
+ * Object store detail key holding an explicit S3 endpoint override. Providers
+ * that support it (SeaweedFS, Cloudian HyperStore) prefer it over the generic
+ * ObjectStoreVO.url.
+ */
+ private static final String OBJECT_STORE_DETAIL_S3_URL = "s3Url";
+
+ /**
+ * Rewrite the stored bucketURL of every bucket on an object store after the
+ * generic store URL changes.
+ *
+ * BucketVO.bucketURL is written only at bucket creation. BucketResponse
+ * exposes it and the object-store browser builds its S3 client from it, so
+ * without this the browser keeps targeting the old endpoint after a URL
+ * change while the management server uses the new one.
+ *
+ * Skipped when the pool has an explicit s3Url detail: providers that support
+ * that override (SeaweedFS, Cloudian HyperStore) keep using it regardless of
+ * ObjectStoreVO.url, so rewriting the bucket URLs off the generic URL would
+ * point the browser at an endpoint the driver never uses.
+ */
+ private void updateBucketUrls(Long storeId, String oldUrl, String newUrl) {
+ if (oldUrl == null || newUrl == null || oldUrl.equals(newUrl)) {
+ return;
+ }
+ Map<String, String> storeDetails = _objectStoreDetailsDao.getDetails(storeId);
+ if (storeDetails != null && StringUtils.isNotBlank(storeDetails.get(OBJECT_STORE_DETAIL_S3_URL))) {
+ logger.debug("Object store {} has an explicit s3Url; leaving stored bucket URLs unchanged", storeId);
+ return;
+ }
+ // Both URLs are operator-supplied and may carry a trailing slash. Strip
+ // them so the retained suffix (which starts with '/') is not appended to
+ // a base that already ends in one, which would rewrite every bucket URL
+ // to '...//bucket' and break browser access.
+ String oldBase = StringUtils.stripEnd(oldUrl, "/");
+ String newBase = StringUtils.stripEnd(newUrl, "/");
+ if (oldBase.equals(newBase)) {
+ return;
+ }
+ for (BucketVO bucket : _bucketDao.listByObjectStoreId(storeId)) {
+ String bucketUrl = bucket.getBucketURL();
+ if (bucketUrl == null || !bucketUrl.startsWith(oldBase)) {
+ continue;
+ }
+ String suffix = bucketUrl.substring(oldBase.length());
+ bucket.setBucketURL(newBase + (suffix.startsWith("/") ? suffix : "/" + suffix));
+ _bucketDao.update(bucket.getId(), bucket);
Review Comment:
The return value of `_bucketDao.update` is ignored, so `updateObjectStore` can return success after changing the store endpoint while some bucket rows still contain the old URL. The management server will use the new endpoint but the browser will continue using those stale `bucketURL` values. Check the update result and fail/rollback the object-store URL change when a bucket rewrite cannot be persisted.
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,1072 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.BucketCrossOriginConfiguration;
+import com.amazonaws.services.s3.model.CORSRule;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketCrossOriginConfigurationRequest;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ /**
+ * DB-backed global lock name prefix for serializing IAM provisioning and
+ * policy refreshes per store+account. Uses {@link GlobalLock} so the
+ * critical section is serialized across management servers in a
+ * clustered deployment, not just within a single JVM.
+ */
+ private static final String IAM_LOCK_PREFIX = "seaweedfs.iam.";
+ private static final String BUCKET_NAME_LOCK_PREFIX = "seaweedfs.bucket.";
+
+ private static String getIamLockName(long storeId, long accountId) {
+ return IAM_LOCK_PREFIX + storeId + "." + accountId;
+ }
+
+ /**
+ * Acquire a DB-backed global lock for IAM operations on the given
+ * store+account. Returns a {@link GlobalLock} that the caller must
+ * {@link GlobalLock#unlock()} in a {@code finally} block, or {@code null}
+ * if the lock could not be acquired within the timeout.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ */
+ protected GlobalLock acquireIamLock(long storeId, long accountId) {
+ GlobalLock lock = GlobalLock.getInternLock(getIamLockName(storeId, accountId));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire IAM lock for store {} account {}", storeId, accountId);
+ lock.releaseRef();
+ return null;
+ }
+ return lock;
+ }
+
+ private static String getBucketNameLockName(long storeId, String bucketName) {
+ return BUCKET_NAME_LOCK_PREFIX + storeId + "." + bucketName;
+ }
+
+ /**
+ * Acquire a DB-backed global lock covering a bucket <em>name</em> on a store,
+ * independent of the owning account.
+ *
+ * S3 bucket names are globally unique within a store and are reusable after
+ * deletion, while the IAM lock is scoped to (store, account). Without this
+ * lock, once a delete removes the remote bucket but before the old owner's
+ * IAM policy is refreshed, a different account can recreate the same name and
+ * the old owner's credentials would still grant that ARN. Both createBucket
+ * and deleteBucket take this lock so the two never interleave for a name.
+ *
+ * <p>Protected so tests can override with a no-op lock (the DB-backed
+ * {@link GlobalLock} requires a real transaction context).
+ *
+ * @return the held lock, which the caller must {@link GlobalLock#unlock()}
+ * and {@link GlobalLock#releaseRef()}, or {@code null} on timeout
+ */
+ protected GlobalLock acquireBucketNameLock(long storeId, String bucketName) {
+ GlobalLock lock = GlobalLock.getInternLock(getBucketNameLockName(storeId, bucketName));
+ if (!lock.lock(300)) {
+ logger.warn("Failed to acquire bucket name lock for store {} bucket {}", storeId, bucketName);
+ lock.releaseRef();
+ return null;
+ }
+ return lock;
+ }
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account and
+ * store. The store ID is included so that two CloudStack pools pointing
+ * at the same SeaweedFS IAM service do not collide on the same
+ * {@code acs-<uuid>} user and overwrite each other's policy and access
+ * keys.
+ */
+ protected String getUserNameForAccount(Account account, long storeId) {
+ return String.format("%s-%d-%s", ACS_PREFIX, storeId, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Serialize per store+account across management servers so two
+ // concurrent bucket requests do not both rotate credentials and leave
+ // bucket rows with mismatched key pairs.
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ return false;
+ }
+ try {
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach a scoped IAM policy that allows access only to this
+ // account's own buckets (the tenant boundary). Refreshed whenever
+ // buckets are created or deleted. Use the lock-free variant since
+ // createUser already holds the IAM lock.
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, null);
+
+ // Reuse the stored access key only if both the access key id and the
+ // secret key are present and the key is still Active in IAM; otherwise
+ // create a replacement.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ String storedSecretKey = details.get(secretKeyDetailKey);
+ if (storedAccessKeyId != null && storedSecretKey != null
+ && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ updateAccountBucketCredentials(storeId, accountId, storedAccessKeyId, storedSecretKey);
+ return true;
+ }
+
+ // The stored key is missing, inactive, or no longer in IAM. Clean up
+ // ALL keys (including the inactive stored one) before creating a
+ // replacement so we do not accumulate keys and hit IAM limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, null);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Persist the credential pair in account details (namespaced by storeId)
+ // before updating BucketVO rows. The reuse path above reconciles bucket
+ // rows every time, so a later bucket update failure is repairable on
+ // retry. AccountDetailsDao.persist(accountId, map) is deliberately not
+ // used: it expunges every detail for the account and can clobber another
+ // store's namespaced credentials.
+ try {
+ persistAccountCredentialsOrRollback(accountId, accessKeyDetailKey, secretKeyDetailKey,
+ storedAccessKeyId, storedSecretKey, key);
+ } catch (RuntimeException e) {
+ deleteAccessKeyAfterCredentialPersistenceFailure(iamClient, userName, key.getAccessKeyId(), e);
+ throw e;
+ }
+
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Persist a SeaweedFS IAM credential pair without using
+ * AccountDetailsDao.persist(accountId, map), and restore the previous pair if
+ * either single-key write fails so callers never observe a half-new pair.
+ */
+ private void persistAccountCredentialsOrRollback(long accountId, String accessKeyDetailKey, String secretKeyDetailKey,
+ String previousAccessKey, String previousSecretKey, AccessKey key) {
+ try {
+ _accountDetailsDao.addDetail(accountId, accessKeyDetailKey, key.getAccessKeyId(), false);
+ _accountDetailsDao.addDetail(accountId, secretKeyDetailKey, key.getSecretAccessKey(), false);
+ } catch (RuntimeException e) {
+ CloudRuntimeException wrapped = new CloudRuntimeException("Failed to persist SeaweedFS IAM credential pair for account " + accountId, e);
+ restoreAccountCredentialDetail(accountId, accessKeyDetailKey, previousAccessKey, wrapped);
+ restoreAccountCredentialDetail(accountId, secretKeyDetailKey, previousSecretKey, wrapped);
+ throw wrapped;
+ }
+ }
+
+ private void restoreAccountCredentialDetail(long accountId, String detailKey, String previousValue, RuntimeException cause) {
+ try {
+ if (previousValue == null) {
+ _accountDetailsDao.removeDetail(accountId, detailKey);
+ } else {
+ _accountDetailsDao.addDetail(accountId, detailKey, previousValue, false);
+ }
+ } catch (RuntimeException rollbackEx) {
+ logger.error("Failed to restore account detail {} for account {} after SeaweedFS credential persistence failed",
+ detailKey, accountId, rollbackEx);
+ cause.addSuppressed(rollbackEx);
+ }
+ }
+
+ private void deleteAccessKeyAfterCredentialPersistenceFailure(AmazonIdentityManagement iamClient, String userName,
+ String accessKeyId, RuntimeException cause) {
+ try {
+ iamClient.deleteAccessKey(new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(accessKeyId));
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Failed to delete IAM access key {} for user {} after account credential persistence failed",
+ accessKeyId, userName, cleanupEx);
+ cause.addSuppressed(cleanupEx);
+ }
+ }
+
+ private void updateAccountBucketCredentials(long storeId, long accountId, AccessKey iamCredential) {
+ updateAccountBucketCredentials(storeId, accountId, iamCredential.getAccessKeyId(), iamCredential.getSecretAccessKey());
+ }
+
+ /**
+ * Update the IAM credentials on all BucketVO rows for this store/account so
+ * previously created buckets reflect the current key pair.
+ */
+ private void updateAccountBucketCredentials(long storeId, long accountId, String accessKeyId, String secretAccessKey) {
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ for (BucketVO bucketVO : bucketList) {
+ if (accessKeyId.equals(bucketVO.getAccessKey()) && secretAccessKey.equals(bucketVO.getSecretKey())) {
+ continue;
+ }
+ logger.info("Updating accountId={} bucket {} with new IAM credentials", accountId, bucketVO.getName());
+ bucketVO.setAccessKey(accessKeyId);
+ bucketVO.setSecretKey(secretAccessKey);
+ if (!_bucketDao.update(bucketVO.getId(), bucketVO)) {
+ throw new CloudRuntimeException("Failed to update IAM credentials on bucket " + bucketVO.getName());
+ }
+ }
+ }
+
+ /**
+ * Refresh the per-account IAM user policy so it grants S3 access only to
+ * the account's current buckets (optionally excluding one, e.g. a bucket
+ * being deleted). This is the tenant boundary: each account's IAM
+ * credentials can only operate on that account's own buckets.
+ *
+ * Acquires the per-store/account IAM lock. Callers that already hold the
+ * lock (e.g. createUser, createBucket post-create) should call
+ * {@link #updateAccountIAMPolicyLocked} instead to avoid re-entrant lock
+ * acquisition warnings from GlobalLock.
+ *
+ * @param iamClient the IAM client
+ * @param storeId the object store
+ * @param accountId the CloudStack account
+ * @param excludeBucket a bucket name to omit (e.g. a bucket being deleted),
+ * or null to include all of the account's buckets
+ */
+ protected void updateAccountIAMPolicy(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ GlobalLock lock = acquireIamLock(storeId, accountId);
+ if (lock == null) {
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ updateAccountIAMPolicyLocked(iamClient, storeId, accountId, excludeBucket);
+ } finally {
+ lock.unlock();
+ lock.releaseRef();
+ }
+ }
+
+ /**
+ * Lock-free variant of {@link #updateAccountIAMPolicy} for callers that
+ * already hold the per-store/account IAM lock. Performs the policy refresh
+ * without reacquiring the lock, avoiding the GlobalLock re-entrant
+ * acquisition warning.
+ */
+ protected void updateAccountIAMPolicyLocked(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ return;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ List<BucketVO> buckets = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ List<String> bucketNames = new ArrayList<>();
+ for (BucketVO bvo : buckets) {
+ if (excludeBucket != null && excludeBucket.equals(bvo.getName())) {
+ continue;
+ }
+ // Skip buckets whose remote counterpart has been deleted but whose
+ // row is still present for resource accounting. Including them
+ // would re-grant a bucket name that is now free for another account
+ // to claim.
+ if (Bucket.State.Destroyed.equals(bvo.getState())) {
+ continue;
+ }
+ bucketNames.add(bvo.getName());
+ }
+ String policy = SeaweedFSObjectStoreUtil.buildAccountIAMPolicy(bucketNames);
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ SeaweedFSObjectStoreUtil.IAM_USER_POLICY_NAME, policy));
+ }
+
+ /**
+ * Check whether the given access key id is still listed and Active in IAM
+ * for the user. Listing failures are propagated rather than swallowed so
+ * a transient IAM outage does not send createUser into the replacement
+ * path (which would overwrite stored credentials and invalidate bucket
+ * records).
+ */
+ private boolean iamAccessKeyExists(AmazonIdentityManagement iamClient, String userName, String accessKeyId) {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ if (accessKeyId.equals(metadata.getAccessKeyId())) {
+ return "Active".equalsIgnoreCase(metadata.getStatus());
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Delete access keys for the user other than the (optionally) preserved
+ * key id. Used to clean up unmanaged leftover keys before creating a
+ * replacement so repeated calls do not hit IAM access-key limits.
+ */
+ private void deleteUnmanagedAccessKeys(AmazonIdentityManagement iamClient, String userName, String preserveAccessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ String keyId = metadata.getAccessKeyId();
+ if (preserveAccessKeyId != null && preserveAccessKeyId.equals(keyId)) {
+ continue;
+ }
+ DeleteAccessKeyRequest deleteReq =
+ new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(keyId);
+ logger.info("Deleting un-managed IAM access key {} for user {}", keyId, userName);
+ iamClient.deleteAccessKey(deleteReq);
+ }
+ } catch (AmazonClientException e) {
+ // Propagate so the caller does not proceed to create a replacement
+ // key while stale unmanaged keys remain (which could hit IAM key
+ // limits or leave orphaned credentials).
+ throw new CloudRuntimeException("Failed to clean up IAM access keys for user " + userName, e);
+ }
+ }
+
+ @Override
+ public Bucket createBucket(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+
+ // Serialize against a concurrent deleteBucket of the same name by any
+ // account on this store. Bucket names are globally unique per store and
+ // reusable, so without this an account could claim a name while the
+ // previous owner's IAM policy still granted that ARN.
+ GlobalLock nameLock = acquireBucketNameLock(storeId, bucketName);
+ if (nameLock == null) {
+ throw new CloudRuntimeException("Failed to acquire bucket name lock for store " + storeId + " bucket " + bucketName);
+ }
+ try {
+ return createBucketLocked(bucket, objectLock);
+ } finally {
+ nameLock.unlock();
+ nameLock.releaseRef();
+ }
+ }
+
+ private Bucket createBucketLocked(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+ long accountId = bucket.getAccountId();
+
+ // Use the store's admin credentials to create the bucket
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Check if the bucket already exists
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket already exists with name " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Create the bucket
+ try {
+ CreateBucketRequest request = new CreateBucketRequest(bucketName);
+ if (objectLock) {
+ request.setObjectLockEnabledForBucket(true);
+ }
+ s3client.createBucket(request);
+ } catch (AmazonClientException e) {
+ logger.error("Create bucket failed", e);
+ throw new CloudRuntimeException(e);
+ }
+
+ // Step 2: update the bucket record with the account's IAM credentials,
+ // configure CORS, and refresh the IAM policy. If any of these fail,
+ // clean up the remote bucket so a retry does not find it already
+ // existing — mirroring the Cloudian createBucket pattern.
+ //
+ // Hold the IAM lock for the account so a concurrent createUser key
+ // rotation does not change the account credentials between reading
+ // them and writing the BucketVO, which would leave the bucket with
+ // a stale key pair.
+ GlobalLock iamLock = acquireIamLock(storeId, accountId);
+ if (iamLock == null) {
+ // The S3 bucket has already been created. Clean it up so a
+ // retry does not find it already existing, then throw.
+ try {
+ s3client.deleteBucket(bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Failed to clean up bucket {} after IAM lock timeout", bucketName, cleanupEx);
+ }
+ throw new CloudRuntimeException("Failed to acquire IAM lock for store " + storeId + " account " + accountId);
+ }
+ try {
+ // Configure permissive CORS so the CloudStack S3 bucket browser
+ // (which performs list/upload/delete from the browser) can function.
+ // SeaweedFS supports the standard PutBucketCors operation.
+ configureBucketCORS(s3client, bucketName);
+
+ Map<String, String> accountDetails = _accountDetailsDao.findDetails(accountId);
+ String accessKey = accountDetails.get(SeaweedFSObjectStoreUtil.keyAccessKey(storeId));
+ String secretKey = accountDetails.get(SeaweedFSObjectStoreUtil.keySecretKey(storeId));
+ if (accessKey == null || secretKey == null) {
+ throw new CloudRuntimeException("No IAM credentials found for account " + accountId
+ + " on store " + storeId + ". Run createUser before creating a bucket.");
+ }
+
+ String s3Url = getS3Url(storeId);
+ BucketVO bucketVO = _bucketDao.findById(bucket.getId());
+ bucketVO.setAccessKey(accessKey);
+ bucketVO.setSecretKey(secretKey);
+ // Normalize the endpoint: s3Url is operator-supplied and may carry a
+ // trailing slash, which would persist a broken "...//bucket" URL
+ // that BucketResponse and the object store browser both use.
+ bucketVO.setBucketURL(SeaweedFSObjectStoreUtil.stripTrailingSlashes(s3Url) + "/" + bucketName);
+ _bucketDao.update(bucket.getId(), bucketVO);
Review Comment:
This ignores the DAO's boolean result. If the row update affects no rows, the method still refreshes IAM and returns `bucketVO`; `BucketApiServiceImpl` can then promote a stale re-read and report success without persisting the credentials or bucket URL, leaving the bucket unusable through the returned credentials. Treat a `false` result as a failure so the existing cleanup path removes the remote bucket and revokes the policy grant.
This issue also appears on line 897 of the same file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r4011123894
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,682 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ /**
+ * Lock object map for serializing IAM provisioning and policy refreshes
+ * per store+account. Prevents concurrent createUser calls from both
+ * rotating credentials and concurrent policy refreshes from building
+ * different snapshots.
+ */
+ private static final java.util.Map<String, Object> IAM_LOCKS = new java.util.concurrent.ConcurrentHashMap<>();
+
+ private static Object getIamLock(long storeId, long accountId) {
+ return IAM_LOCKS.computeIfAbsent(storeId + ":" + accountId, k -> new Object());
+ }
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account and
+ * store. The store ID is included so that two CloudStack pools pointing
+ * at the same SeaweedFS IAM service do not collide on the same
+ * {@code acs-<uuid>} user and overwrite each other's policy and access
+ * keys.
+ */
+ protected String getUserNameForAccount(Account account, long storeId) {
+ return String.format("%s-%d-%s", ACS_PREFIX, storeId, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Serialize per store+account so two concurrent bucket requests do
+ // not both rotate credentials and leave bucket rows with mismatched
+ // key pairs.
+ synchronized (getIamLock(storeId, accountId)) {
Review Comment:
This lock only serializes threads inside one management-server JVM. In a multi-management-server deployment, two nodes can both observe a missing/inactive key, rotate IAM credentials, and race the shared AccountDetails/BucketVO updates, leaving the persisted key pair inconsistent with bucket rows. Use a DB-backed `GlobalLock` (with timeout and release) for this critical section.
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,682 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ /**
+ * Lock object map for serializing IAM provisioning and policy refreshes
+ * per store+account. Prevents concurrent createUser calls from both
+ * rotating credentials and concurrent policy refreshes from building
+ * different snapshots.
+ */
+ private static final java.util.Map<String, Object> IAM_LOCKS = new java.util.concurrent.ConcurrentHashMap<>();
+
+ private static Object getIamLock(long storeId, long accountId) {
+ return IAM_LOCKS.computeIfAbsent(storeId + ":" + accountId, k -> new Object());
+ }
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account and
+ * store. The store ID is included so that two CloudStack pools pointing
+ * at the same SeaweedFS IAM service do not collide on the same
+ * {@code acs-<uuid>} user and overwrite each other's policy and access
+ * keys.
+ */
+ protected String getUserNameForAccount(Account account, long storeId) {
+ return String.format("%s-%d-%s", ACS_PREFIX, storeId, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Serialize per store+account so two concurrent bucket requests do
+ // not both rotate credentials and leave bucket rows with mismatched
+ // key pairs.
+ synchronized (getIamLock(storeId, accountId)) {
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach a scoped IAM policy that allows access only to this
+ // account's own buckets (the tenant boundary). Refreshed whenever
+ // buckets are created or deleted.
+ updateAccountIAMPolicy(iamClient, storeId, accountId, null);
+
+ // Reuse the stored access key only if both the access key id and the
+ // secret key are present and the key is still Active in IAM; otherwise
+ // create a replacement.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ String storedSecretKey = details.get(secretKeyDetailKey);
+ if (storedAccessKeyId != null && storedSecretKey != null
+ && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ return true;
+ }
+
+ // The stored key is missing, inactive, or no longer in IAM. Clean up
+ // ALL keys (including the inactive stored one) before creating a
+ // replacement so we do not accumulate keys and hit IAM limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, null);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Update existing bucket records for this account/store with the new
+ // credentials BEFORE persisting the new key in account details. If a
+ // bucket update fails, the stored key remains the old one and a retry
+ // will re-enter the replacement path; if we persisted first, a retry
+ // would see the new stored key and return without repairing the
+ // remaining buckets.
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ // Persist the credentials in the account details (namespaced by storeId)
+ details.put(accessKeyDetailKey, key.getAccessKeyId());
+ details.put(secretKeyDetailKey, key.getSecretAccessKey());
+ _accountDetailsDao.persist(accountId, details);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ } // synchronized
+ }
+
+ /**
+ * Update the IAM credentials on all BucketVO rows for this store/account
+ * so previously created buckets reflect the new (rotated) key pair.
+ * Mirrors CloudianHyperStoreObjectStoreDriverImpl.updateAccountBucketCredentials.
+ */
+ private void updateAccountBucketCredentials(long storeId, long accountId, AccessKey iamCredential) {
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ for (BucketVO bucketVO : bucketList) {
+ logger.info("Updating accountId={} bucket {} with new IAM credentials", accountId, bucketVO.getName());
+ bucketVO.setAccessKey(iamCredential.getAccessKeyId());
+ bucketVO.setSecretKey(iamCredential.getSecretAccessKey());
+ _bucketDao.update(bucketVO.getId(), bucketVO);
+ }
+ }
+
+ /**
+ * Refresh the per-account IAM user policy so it grants S3 access only to
+ * the account's current buckets (optionally excluding one, e.g. a bucket
+ * being deleted). This is the tenant boundary: each account's IAM
+ * credentials can only operate on that account's own buckets.
+ *
+ * @param iamClient the IAM client
+ * @param storeId the object store
+ * @param accountId the CloudStack account
+ * @param excludeBucket a bucket name to omit (e.g. a bucket being deleted),
+ * or null to include all of the account's buckets
+ */
+ protected void updateAccountIAMPolicy(AmazonIdentityManagement iamClient, long storeId, long accountId, String excludeBucket) {
+ synchronized (getIamLock(storeId, accountId)) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ return;
+ }
+ String userName = getUserNameForAccount(account, storeId);
+ List<BucketVO> buckets = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ List<String> bucketNames = new ArrayList<>();
+ for (BucketVO bvo : buckets) {
+ if (excludeBucket != null && excludeBucket.equals(bvo.getName())) {
+ continue;
+ }
+ bucketNames.add(bvo.getName());
+ }
+ String policy = SeaweedFSObjectStoreUtil.buildAccountIAMPolicy(bucketNames);
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ SeaweedFSObjectStoreUtil.IAM_USER_POLICY_NAME, policy));
+ }
+ }
+
+ /**
+ * Check whether the given access key id is still listed and Active in IAM
+ * for the user. Listing failures are propagated rather than swallowed so
+ * a transient IAM outage does not send createUser into the replacement
+ * path (which would overwrite stored credentials and invalidate bucket
+ * records).
+ */
+ private boolean iamAccessKeyExists(AmazonIdentityManagement iamClient, String userName, String accessKeyId) {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ if (accessKeyId.equals(metadata.getAccessKeyId())) {
+ return "Active".equalsIgnoreCase(metadata.getStatus());
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Delete access keys for the user other than the (optionally) preserved
+ * key id. Used to clean up unmanaged leftover keys before creating a
+ * replacement so repeated calls do not hit IAM access-key limits.
+ */
+ private void deleteUnmanagedAccessKeys(AmazonIdentityManagement iamClient, String userName, String preserveAccessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ String keyId = metadata.getAccessKeyId();
+ if (preserveAccessKeyId != null && preserveAccessKeyId.equals(keyId)) {
+ continue;
+ }
+ DeleteAccessKeyRequest deleteReq =
+ new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(keyId);
+ logger.info("Deleting un-managed IAM access key {} for user {}", keyId, userName);
+ iamClient.deleteAccessKey(deleteReq);
+ }
+ } catch (AmazonClientException e) {
+ logger.warn("Failed to clean up IAM access keys for user {}: {}", userName, e.getMessage());
+ }
+ }
+
+ @Override
+ public Bucket createBucket(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+ long accountId = bucket.getAccountId();
+
+ // Use the store's admin credentials to create the bucket
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Check if the bucket already exists
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket already exists with name " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Create the bucket
+ try {
+ CreateBucketRequest request = new CreateBucketRequest(bucketName);
+ if (objectLock) {
+ request.setObjectLockEnabledForBucket(true);
+ }
+ s3client.createBucket(request);
+ } catch (AmazonClientException e) {
+ logger.error("Create bucket failed", e);
+ throw new CloudRuntimeException(e);
+ }
+
+ // Step 2: update the bucket record with the account's IAM credentials.
+ // If this fails, clean up the remote bucket so a retry does not find
+ // it already existing — mirroring the Cloudian createBucket pattern.
+ try {
+ Map<String, String> accountDetails = _accountDetailsDao.findDetails(accountId);
+ String accessKey = accountDetails.get(SeaweedFSObjectStoreUtil.keyAccessKey(storeId));
+ String secretKey = accountDetails.get(SeaweedFSObjectStoreUtil.keySecretKey(storeId));
+ if (accessKey == null || secretKey == null) {
+ throw new CloudRuntimeException("No IAM credentials found for account " + accountId
+ + " on store " + storeId + ". Run createUser before creating a bucket.");
+ }
+
+ String s3Url = getS3Url(storeId);
+ BucketVO bucketVO = _bucketDao.findById(bucket.getId());
+ bucketVO.setAccessKey(accessKey);
+ bucketVO.setSecretKey(secretKey);
+ bucketVO.setBucketURL(s3Url + "/" + bucketName);
+ _bucketDao.update(bucket.getId(), bucketVO);
+
+ // Refresh the account's IAM policy to include the new bucket
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+ updateAccountIAMPolicy(iamClient, storeId, accountId, null);
+
+ return bucket;
+ } catch (Exception e) {
+ logger.error("Post-create bucket record update failed for {}; cleaning up remote bucket", bucketName, e);
+ try {
+ s3client.deleteBucket(bucketName);
+ logger.info("Cleanup of bucket {} succeeded", bucketName);
+ } catch (AmazonClientException cleanupEx) {
+ logger.error("Cleanup of bucket {} also failed", bucketName, cleanupEx);
Review Comment:
If the IAM policy PUT has reached SeaweedFS but its response fails, this compensation deletes the bucket without revoking the newly added bucket ARN. The API then removes the CloudStack bucket row after this method throws, so the old account can retain a grant for the deleted name and access a future bucket that reuses it. Recompute/revoke the account policy as part of the rollback and reconcile if that cleanup also fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(seaweedfs): add SeaweedFS object storage provider [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14160:
URL: https://github.com/apache/cloudstack/pull/14160#discussion_r4010913333
##########
plugins/storage/object/seaweedfs/src/main/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImpl.java:
##########
@@ -0,0 +1,582 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.DeleteBucketPolicyRequest;
+import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
+import com.amazonaws.services.s3.model.SSEAlgorithm;
+import com.amazonaws.services.s3.model.ServerSideEncryptionByDefault;
+import com.amazonaws.services.s3.model.ServerSideEncryptionConfiguration;
+import com.amazonaws.services.s3.model.ServerSideEncryptionRule;
+import com.amazonaws.services.s3.model.SetBucketEncryptionRequest;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+/**
+ * SeaweedFS object store driver.
+ *
+ * Bucket operations use the AWS S3 SDK v1 (path-style access, endpoint-pinned).
+ * User/credential management uses the AWS IAM SDK v1, since SeaweedFS exposes a
+ * standard AWS IAM-compatible API. No proprietary admin client is needed.
+ *
+ * Modeled on CloudianHyperStoreObjectStoreDriverImpl, which uses the same
+ * S3 + IAM SDK pair.
+ */
+public class SeaweedFSObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+ @Inject
+ AccountDao _accountDao;
+
+ @Inject
+ AccountDetailsDao _accountDetailsDao;
+
+ @Inject
+ ObjectStoreDao _storeDao;
+
+ @Inject
+ BucketDao _bucketDao;
+
+ @Inject
+ ObjectStoreDetailsDao _storeDetailsDao;
+
+ private static final String ACS_PREFIX = "acs";
+
+ @Override
+ public DataStoreTO getStoreTO(DataStore store) {
+ return null;
+ }
+
+ /**
+ * Get the SeaweedFS IAM user name for the given CloudStack account.
+ * Uses the account UUID prefixed with "acs-" for namespacing.
+ */
+ protected String getUserNameForAccount(Account account) {
+ return String.format("%s-%s", ACS_PREFIX, account.getUuid());
+ }
+
+ /**
+ * Create the IAM user for the CloudStack account if it doesn't exist,
+ * attach the restricted S3 policy, and ensure the account has a usable
+ * IAM access key persisted in its account details.
+ *
+ * <p>If a previously stored access key is still present in IAM, it is
+ * reused rather than rotated. A new key is only created when no stored
+ * key exists or the stored key is no longer found in IAM; in the latter
+ * case any unmanaged (leftover) keys for the user are deleted first to
+ * avoid hitting IAM access-key limits. This keeps bucket records that
+ * reference the stored credentials valid across repeated calls.
+ *
+ * @return true if the user exists or was created, false on failure.
+ */
+ @Override
+ public boolean createUser(long accountId, long storeId) {
+ Account account = _accountDao.findById(accountId);
+ if (account == null) {
+ logger.error("Account {} not found", accountId);
+ return false;
+ }
+ String userName = getUserNameForAccount(account);
+ AmazonIdentityManagement iamClient = getIAMClient(storeId);
+
+ // Create the IAM user if it doesn't already exist
+ try {
+ iamClient.createUser(new CreateUserRequest(userName));
+ logger.info("Created IAM user {} for account {}", userName, account.getAccountName());
+ } catch (EntityAlreadyExistsException e) {
+ logger.debug("IAM user {} already exists", userName);
+ }
+
+ // Attach the restricted S3 policy (idempotent — overwrites if present)
+ iamClient.putUserPolicy(new PutUserPolicyRequest(userName,
+ "CloudStackPolicy", SeaweedFSObjectStoreUtil.IAM_USER_POLICY));
+
+ // Reuse the stored access key if it is still present in IAM; only
+ // create a new one when no usable key exists.
+ Map<String, String> details = _accountDetailsDao.findDetails(accountId);
+ String accessKeyDetailKey = SeaweedFSObjectStoreUtil.keyAccessKey(storeId);
+ String secretKeyDetailKey = SeaweedFSObjectStoreUtil.keySecretKey(storeId);
+ String storedAccessKeyId = details.get(accessKeyDetailKey);
+ if (storedAccessKeyId != null && iamAccessKeyExists(iamClient, userName, storedAccessKeyId)) {
+ logger.debug("Reusing existing IAM access key {} for user {}", storedAccessKeyId, userName);
+ return true;
+ }
+
+ // The stored key is missing or no longer in IAM. Clean up any
+ // unmanaged leftover keys before creating a replacement so we do not
+ // accumulate keys and hit IAM access-key limits.
+ deleteUnmanagedAccessKeys(iamClient, userName, storedAccessKeyId);
+
+ CreateAccessKeyResult result = iamClient.createAccessKey(
+ new CreateAccessKeyRequest().withUserName(userName));
+ AccessKey key = result.getAccessKey();
+
+ // Persist the credentials in the account details (namespaced by storeId)
+ details.put(accessKeyDetailKey, key.getAccessKeyId());
+ details.put(secretKeyDetailKey, key.getSecretAccessKey());
+ _accountDetailsDao.persist(accountId, details);
+
+ // Update existing bucket records for this account/store with the new
+ // credentials so previously created buckets don't keep handing out
+ // the old (now invalid) key pair.
+ updateAccountBucketCredentials(storeId, accountId, key);
+
+ logger.info("Created IAM credentials {} for user {}", key.getAccessKeyId(), userName);
+ return true;
+ }
+
+ /**
+ * Update the IAM credentials on all BucketVO rows for this store/account
+ * so previously created buckets reflect the new (rotated) key pair.
+ * Mirrors CloudianHyperStoreObjectStoreDriverImpl.updateAccountBucketCredentials.
+ */
+ private void updateAccountBucketCredentials(long storeId, long accountId, AccessKey iamCredential) {
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreIdAndAccountId(storeId, accountId);
+ for (BucketVO bucketVO : bucketList) {
+ logger.info("Updating accountId={} bucket {} with new IAM credentials", accountId, bucketVO.getName());
+ bucketVO.setAccessKey(iamCredential.getAccessKeyId());
+ bucketVO.setSecretKey(iamCredential.getSecretAccessKey());
+ _bucketDao.update(bucketVO.getId(), bucketVO);
+ }
+ }
+
+ /**
+ * Check whether the given access key id is still listed in IAM for the user.
+ */
+ private boolean iamAccessKeyExists(AmazonIdentityManagement iamClient, String userName, String accessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ if (accessKeyId.equals(metadata.getAccessKeyId())) {
+ return true;
+ }
+ }
+ } catch (AmazonClientException e) {
+ logger.warn("Failed to list IAM access keys for user {}: {}", userName, e.getMessage());
+ }
+ return false;
+ }
+
+ /**
+ * Delete access keys for the user other than the (optionally) preserved
+ * key id. Used to clean up unmanaged leftover keys before creating a
+ * replacement so repeated calls do not hit IAM access-key limits.
+ */
+ private void deleteUnmanagedAccessKeys(AmazonIdentityManagement iamClient, String userName, String preserveAccessKeyId) {
+ try {
+ for (AccessKeyMetadata metadata :
+ iamClient.listAccessKeys(new ListAccessKeysRequest()
+ .withUserName(userName)).getAccessKeyMetadata()) {
+ String keyId = metadata.getAccessKeyId();
+ if (preserveAccessKeyId != null && preserveAccessKeyId.equals(keyId)) {
+ continue;
+ }
+ DeleteAccessKeyRequest deleteReq =
+ new DeleteAccessKeyRequest()
+ .withUserName(userName)
+ .withAccessKeyId(keyId);
+ logger.info("Deleting un-managed IAM access key {} for user {}", keyId, userName);
+ iamClient.deleteAccessKey(deleteReq);
+ }
+ } catch (AmazonClientException e) {
+ logger.warn("Failed to clean up IAM access keys for user {}: {}", userName, e.getMessage());
+ }
+ }
+
+ @Override
+ public Bucket createBucket(Bucket bucket, boolean objectLock) {
+ String bucketName = bucket.getName();
+ long storeId = bucket.getObjectStoreId();
+ long accountId = bucket.getAccountId();
+
+ // Use the store's admin credentials to create the bucket
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+
+ // Check if the bucket already exists
+ try {
+ if (s3client.doesBucketExistV2(bucketName)) {
+ throw new CloudRuntimeException("Bucket already exists with name " + bucketName);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+
+ // Create the bucket
+ try {
+ CreateBucketRequest request = new CreateBucketRequest(bucketName);
+ if (objectLock) {
+ request.setObjectLockEnabledForBucket(true);
+ }
+ s3client.createBucket(request);
+ } catch (AmazonClientException e) {
+ logger.error("Create bucket failed", e);
+ throw new CloudRuntimeException(e);
+ }
+
+ // Update the bucket record with the account's IAM credentials
+ Map<String, String> accountDetails = _accountDetailsDao.findDetails(accountId);
+ String accessKey = accountDetails.get(SeaweedFSObjectStoreUtil.keyAccessKey(storeId));
+ String secretKey = accountDetails.get(SeaweedFSObjectStoreUtil.keySecretKey(storeId));
+ if (accessKey == null || secretKey == null) {
+ logger.warn("No IAM credentials found for account {}. Bucket will be created without per-account credentials.", accountId);
+ }
+
+ ObjectStoreVO store = _storeDao.findById(storeId);
+ String s3Url = getS3Url(storeId);
+ BucketVO bucketVO = _bucketDao.findById(bucket.getId());
+ bucketVO.setAccessKey(accessKey);
+ bucketVO.setSecretKey(secretKey);
+ bucketVO.setBucketURL(s3Url + "/" + bucketName);
+ _bucketDao.update(bucket.getId(), bucketVO);
+ return bucket;
+ }
+
+ @Override
+ public List<Bucket> listBuckets(long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ List<Bucket> bucketsList = new ArrayList<>();
+ try {
+ List<com.amazonaws.services.s3.model.Bucket> s3Buckets = s3client.listBuckets();
+ for (com.amazonaws.services.s3.model.Bucket s3Bucket : s3Buckets) {
+ Bucket bucket = new BucketObject();
+ bucket.setName(s3Bucket.getName());
+ bucketsList.add(bucket);
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ return bucketsList;
+ }
+
+ @Override
+ public boolean deleteBucket(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ if (! s3client.doesBucketExistV2(bucket.getName())) {
+ throw new CloudRuntimeException("Bucket doesn't exist: " + bucket.getName());
+ }
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ try {
+ s3client.deleteBucket(bucket.getName());
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ return true;
+ }
+
+ @Override
+ public AccessControlList getBucketAcl(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ return s3client.getBucketAcl(bucket.getName());
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public void setBucketAcl(BucketTO bucket, AccessControlList acl, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ s3client.setBucketAcl(bucket.getName(), acl);
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public void setBucketPolicy(BucketTO bucket, String policy, long storeId) {
+ if ("private".equalsIgnoreCase(policy)) {
+ deleteBucketPolicy(bucket, storeId);
+ return;
+ }
+
+ StringBuilder sb = new StringBuilder();
+ sb.append("{\n");
+ sb.append(" \"Version\": \"2012-10-17\",\n");
+ sb.append(" \"Statement\": [\n");
+ sb.append(" {\n");
+ sb.append(" \"Sid\": \"PublicReadForObjects\",\n");
+ sb.append(" \"Effect\": \"Allow\",\n");
+ sb.append(" \"Principal\": \"*\",\n");
+ sb.append(" \"Action\": \"s3:GetObject\",\n");
+ sb.append(" \"Resource\": \"arn:aws:s3:::%s/*\"\n");
+ sb.append(" }\n");
+ sb.append(" ]\n");
+ sb.append("}\n");
+
+ String jsonPolicy = String.format(sb.toString(), bucket.getName());
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ s3client.setBucketPolicy(bucket.getName(), jsonPolicy);
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public BucketPolicy getBucketPolicy(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ return s3client.getBucketPolicy(new GetBucketPolicyRequest(bucket.getName()));
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public void deleteBucketPolicy(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ s3client.deleteBucketPolicy(new DeleteBucketPolicyRequest(bucket.getName()));
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public boolean setBucketEncryption(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ SetBucketEncryptionRequest eRequest = new SetBucketEncryptionRequest();
+ eRequest.setBucketName(bucket.getName());
+
+ ServerSideEncryptionByDefault sseByDefault = new ServerSideEncryptionByDefault();
+ sseByDefault.setSSEAlgorithm(SSEAlgorithm.AES256.toString());
+
+ ServerSideEncryptionRule sseRule = new ServerSideEncryptionRule();
+ sseRule.setApplyServerSideEncryptionByDefault(sseByDefault);
+
+ List<ServerSideEncryptionRule> sseRules = new ArrayList<>();
+ sseRules.add(sseRule);
+
+ ServerSideEncryptionConfiguration sseConf = new ServerSideEncryptionConfiguration();
+ sseConf.setRules(sseRules);
+
+ eRequest.setServerSideEncryptionConfiguration(sseConf);
+ s3client.setBucketEncryption(eRequest);
+ return true;
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public boolean deleteBucketEncryption(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ s3client.deleteBucketEncryption(bucket.getName());
+ return true;
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public boolean setBucketVersioning(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ BucketVersioningConfiguration vConf = new BucketVersioningConfiguration(BucketVersioningConfiguration.ENABLED);
+ s3client.setBucketVersioningConfiguration(
+ new SetBucketVersioningConfigurationRequest(bucket.getName(), vConf));
+ return true;
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ @Override
+ public boolean deleteBucketVersioning(BucketTO bucket, long storeId) {
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ try {
+ BucketVersioningConfiguration vConf = new BucketVersioningConfiguration(BucketVersioningConfiguration.SUSPENDED);
+ s3client.setBucketVersioningConfiguration(
+ new SetBucketVersioningConfigurationRequest(bucket.getName(), vConf));
+ return true;
+ } catch (AmazonClientException e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ /**
+ * Set the bucket quota via the SeaweedFS admin REST API.
+ *
+ * SeaweedFS enforces bucket quota server-side by setting a read-only flag
+ * when usage exceeds the configured limit. The quota is configured via the
+ * SeaweedFS S3 extension endpoint PUT /{bucket}?seaweedfs-quota,
+ * authenticated via standard S3 SigV4 and authorized via the
+ * s3:PutBucketQuota IAM permission.
+ *
+ * @param size the GiB size to set the quota to. 0 disables quota.
+ * @throws CloudRuntimeException if the S3 endpoint or credentials are missing or the request fails.
+ */
+ @Override
+ public void setBucketQuota(BucketTO bucket, long storeId, long size) {
+ String s3Url = getS3Url(storeId);
+ String accessKey = getAccessKey(storeId);
+ String secretKey = getSecretKey(storeId);
+ if (s3Url == null || s3Url.isEmpty() || accessKey == null || accessKey.isEmpty() || secretKey == null || secretKey.isEmpty()) {
+ throw new CloudRuntimeException("SeaweedFS S3 URL and credentials are required to set bucket quota. " +
+ "Configure 's3Url', 'accesskey', and 'secretkey' in the object store details.");
+ }
+ SeaweedFSObjectStoreUtil.setBucketQuotaViaS3Extension(s3Url, accessKey, secretKey, bucket.getName(), size, getS3ExtensionHttpClient());
+ }
+
+ /**
+ * Returns the HTTP client used to send SeaweedFS S3 extension requests
+ * (e.g. PUT /{bucket}?seaweedfs-quota). Exposed as a protected seam so
+ * tests can inject a mock client and assert the signed request without
+ * touching the network.
+ */
+ protected java.net.http.HttpClient getS3ExtensionHttpClient() {
+ return SeaweedFSObjectStoreUtil.newS3ExtensionHttpClient();
+ }
+
+ @Override
+ public Map<String, Long> getAllBucketsUsage(long storeId) {
+ Map<String, Long> bucketUsage = new HashMap<>();
+ List<BucketVO> bucketList = _bucketDao.listByObjectStoreId(storeId);
+ if (bucketList.isEmpty()) {
+ return bucketUsage;
+ }
+
+ // List objects per bucket via S3 (no admin API needed).
+ // SeaweedFS also publishes per-bucket Prometheus metrics and an SOSAPI
+ // capacity.xml response; operators who need scalable usage reporting
+ // should consume those instead of S3 list-based aggregation.
+ AmazonS3 s3client = getS3ClientByStoreId(storeId);
+ for (BucketVO bucket : bucketList) {
+ try {
+ long size = 0L;
+ com.amazonaws.services.s3.model.ListObjectsV2Result result;
+ String continuationToken = null;
+ do {
+ com.amazonaws.services.s3.model.ListObjectsV2Request req =
+ new com.amazonaws.services.s3.model.ListObjectsV2Request()
+ .withBucketName(bucket.getName())
+ .withMaxKeys(1000);
+ if (continuationToken != null) {
+ req.setContinuationToken(continuationToken);
+ }
+ result = s3client.listObjectsV2(req);
+ for (com.amazonaws.services.s3.model.S3ObjectSummary summary : result.getObjectSummaries()) {
+ size += summary.getSize();
+ }
+ continuationToken = result.getNextContinuationToken();
+ } while (result.isTruncated());
+ bucketUsage.put(bucket.getName(), size);
+ } catch (AmazonClientException e) {
+ // Omit the bucket rather than reporting 0 — returning 0 would
+ // cause BucketApiServiceImpl to overwrite the stored size with
+ // a false zero, erasing known usage on a transient failure.
+ logger.warn("Failed to get usage for bucket {} (omitting from result): {}", bucket.getName(), e.getMessage());
+ }
+ }
+ return bucketUsage;
+ }
+
+ // ---- Client builders ----
+
+ protected String getS3Url(long storeId) {
+ // Prefer the current store URL (ObjectStoreVO.url) over the persisted
+ // s3Url detail. initialize() persists a resolved s3Url detail, but if
+ // an administrator later updates the store URL via updateObjectStore,
+ // the detail becomes stale. Using the current store URL keeps bucket
+ // operations pointed at the live endpoint.
+ ObjectStoreVO store = _storeDao.findById(storeId);
+ if (store != null && store.getUrl() != null && ! store.getUrl().isEmpty()) {
+ return store.getUrl();
Review Comment:
When `details` contains a distinct `s3Url` (which `initialize` explicitly supports and persists), `ObjectStoreVO.url` is always non-empty and this branch returns it, so the configured S3 endpoint is ignored for bucket operations and quota requests. Either read the persisted `s3Url` first or make the resolved S3 endpoint the stored object-store URL; currently the separate endpoint option is not functional.
##########
plugins/storage/object/seaweedfs/src/test/java/org/apache/cloudstack/storage/datastore/driver/SeaweedFSObjectStoreDriverImplTest.java:
##########
@@ -0,0 +1,654 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// SPDX-License-Identifier: Apache-2.0
+package org.apache.cloudstack.storage.datastore.driver;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.lenient;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Flow;
+
+import java.io.ByteArrayOutputStream;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.datastore.util.SeaweedFSObjectStoreUtil;
+import org.apache.cloudstack.storage.object.Bucket;
+
+import com.amazonaws.services.identitymanagement.AmazonIdentityManagement;
+import com.amazonaws.services.identitymanagement.model.AccessKey;
+import com.amazonaws.services.identitymanagement.model.AccessKeyMetadata;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.CreateAccessKeyResult;
+import com.amazonaws.services.identitymanagement.model.CreateUserRequest;
+import com.amazonaws.services.identitymanagement.model.DeleteAccessKeyRequest;
+import com.amazonaws.services.identitymanagement.model.EntityAlreadyExistsException;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysRequest;
+import com.amazonaws.services.identitymanagement.model.ListAccessKeysResult;
+import com.amazonaws.services.identitymanagement.model.PutUserPolicyRequest;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
+import com.amazonaws.services.s3.model.CreateBucketRequest;
+import com.amazonaws.services.s3.model.ListObjectsV2Request;
+import com.amazonaws.services.s3.model.ListObjectsV2Result;
+import com.amazonaws.services.s3.model.S3ObjectSummary;
+import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.AccountVO;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.ArgumentCaptor;
+import org.mockito.ArgumentMatchers;
+import org.mockito.Mock;
+import org.mockito.MockitoAnnotations;
+import org.mockito.Spy;
+import org.mockito.junit.MockitoJUnitRunner;
+
+@RunWith(MockitoJUnitRunner.Silent.class)
+public class SeaweedFSObjectStoreDriverImplTest {
+
+ @Spy
+ SeaweedFSObjectStoreDriverImpl driver = new SeaweedFSObjectStoreDriverImpl();
+
+ @Mock
+ AmazonS3 s3Client;
+ @Mock
+ AmazonIdentityManagement iamClient;
+ @Mock
+ ObjectStoreDao objectStoreDao;
+ @Mock
+ ObjectStoreVO objectStoreVO;
+ @Mock
+ ObjectStoreDetailsDao objectStoreDetailsDao;
+ @Mock
+ AccountDao accountDao;
+ @Mock
+ BucketDao bucketDao;
+ @Mock
+ AccountDetailsDao accountDetailsDao;
+ @Mock
+ AccountVO account;
+
+ BucketVO bucketVo;
+ Map<String, String> storeDetailsMap;
+ Map<String, String> accountDetailsMap;
+
+ static long TEST_STORE_ID = 1010L;
+ static long TEST_ACCOUNT_ID = 2010L;
+ static long TEST_DOMAIN_ID = 3010L;
+ static String TEST_ACCESS_KEY = "test_access_key";
+ static String TEST_SECRET_KEY = "test_secret_key";
+ static String TEST_BUCKET_NAME = "testbucketname";
+ static String TEST_S3_URL = "http://s3-endpoint";
+ static String TEST_IAM_URL = "http://iam-endpoint";
+ static String TEST_AK = "user_access_key";
+ static String TEST_SK = "user_secret_key";
+ static String TEST_BUCKET_URL = TEST_S3_URL + "/" + TEST_BUCKET_NAME;
+ static String TEST_ACCOUNT_UUID = "account-uuid-1234";
+
+ private AutoCloseable closeable;
+
+ @Before
+ public void setUp() {
+ closeable = MockitoAnnotations.openMocks(this);
+ driver._storeDao = objectStoreDao;
+ driver._storeDetailsDao = objectStoreDetailsDao;
+ driver._accountDao = accountDao;
+ driver._bucketDao = bucketDao;
+ driver._accountDetailsDao = accountDetailsDao;
+
+ lenient().when(objectStoreDao.findById(TEST_STORE_ID)).thenReturn(objectStoreVO);
+ lenient().when(objectStoreVO.getUrl()).thenReturn(TEST_S3_URL);
+
+ storeDetailsMap = new HashMap<>();
+ storeDetailsMap.put(SeaweedFSObjectStoreUtil.STORE_DETAILS_KEY_ACCESS_KEY, TEST_ACCESS_KEY);
+ storeDetailsMap.put(SeaweedFSObjectStoreUtil.STORE_DETAILS_KEY_SECRET_KEY, TEST_SECRET_KEY);
+ storeDetailsMap.put(SeaweedFSObjectStoreUtil.STORE_DETAILS_KEY_S3_URL, TEST_S3_URL);
+ storeDetailsMap.put(SeaweedFSObjectStoreUtil.STORE_DETAILS_KEY_IAM_URL, TEST_IAM_URL);
+ lenient().when(objectStoreDetailsDao.getDetails(TEST_STORE_ID)).thenReturn(storeDetailsMap);
+
+ accountDetailsMap = new HashMap<>();
+ accountDetailsMap.put(SeaweedFSObjectStoreUtil.keyAccessKey(TEST_STORE_ID), TEST_AK);
+ accountDetailsMap.put(SeaweedFSObjectStoreUtil.keySecretKey(TEST_STORE_ID), TEST_SK);
+ lenient().when(accountDetailsDao.findDetails(TEST_ACCOUNT_ID)).thenReturn(accountDetailsMap);
+
+ bucketVo = new BucketVO(TEST_ACCOUNT_ID, TEST_DOMAIN_ID, TEST_STORE_ID, TEST_BUCKET_NAME, null, false, false, false, null);
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ closeable.close();
+ }
+
+ @Test
+ public void testGetStoreTO() {
+ assertNull(driver.getStoreTO(null));
+ }
+
+ @Test
+ public void testCreateBucket() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ when(s3Client.doesBucketExistV2(TEST_BUCKET_NAME)).thenReturn(false);
+ when(bucketDao.findById(anyLong())).thenReturn(bucketVo);
+
+ Bucket result = driver.createBucket(bucketVo, false);
+
+ assertEquals(TEST_BUCKET_NAME, result.getName());
+
+ ArgumentCaptor<BucketVO> captor = ArgumentCaptor.forClass(BucketVO.class);
+ verify(bucketDao, times(1)).update(any(), captor.capture());
+ BucketVO updated = captor.getValue();
+ assertEquals(TEST_AK, updated.getAccessKey());
+ assertEquals(TEST_SK, updated.getSecretKey());
+ assertEquals(TEST_BUCKET_URL, updated.getBucketURL());
+
+ verify(s3Client, times(1)).createBucket(any(CreateBucketRequest.class));
+ }
+
+ @Test
+ public void testCreateBucketAlreadyExists() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ when(s3Client.doesBucketExistV2(TEST_BUCKET_NAME)).thenReturn(true);
+
+ assertThrows(CloudRuntimeException.class, () -> driver.createBucket(bucketVo, false));
+ verify(s3Client, never()).createBucket(any(CreateBucketRequest.class));
+ }
+
+ @Test
+ public void testListBuckets() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ List<com.amazonaws.services.s3.model.Bucket> s3Buckets = new ArrayList<>();
+ s3Buckets.add(new com.amazonaws.services.s3.model.Bucket("bucket1"));
+ s3Buckets.add(new com.amazonaws.services.s3.model.Bucket("bucket2"));
+ when(s3Client.listBuckets()).thenReturn(s3Buckets);
+
+ List<Bucket> result = driver.listBuckets(TEST_STORE_ID);
+
+ assertEquals(2, result.size());
+ assertEquals("bucket1", result.get(0).getName());
+ assertEquals("bucket2", result.get(1).getName());
+ }
+
+ @Test
+ public void testDeleteBucket() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ BucketTO bucketTO = mock(BucketTO.class);
+ when(bucketTO.getName()).thenReturn(TEST_BUCKET_NAME);
+ when(s3Client.doesBucketExistV2(TEST_BUCKET_NAME)).thenReturn(true);
+
+ assertTrue(driver.deleteBucket(bucketTO, TEST_STORE_ID));
+ verify(s3Client, times(1)).deleteBucket(TEST_BUCKET_NAME);
+ }
+
+ @Test
+ public void testDeleteBucketNotFound() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ BucketTO bucketTO = mock(BucketTO.class);
+ when(bucketTO.getName()).thenReturn(TEST_BUCKET_NAME);
+ when(s3Client.doesBucketExistV2(TEST_BUCKET_NAME)).thenReturn(false);
+
+ assertThrows(CloudRuntimeException.class, () -> driver.deleteBucket(bucketTO, TEST_STORE_ID));
+ }
+
+ @Test
+ public void testSetBucketVersioning() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ BucketTO bucketTO = mock(BucketTO.class);
+ when(bucketTO.getName()).thenReturn(TEST_BUCKET_NAME);
+
+ assertTrue(driver.setBucketVersioning(bucketTO, TEST_STORE_ID));
+ verify(s3Client, times(1)).setBucketVersioningConfiguration(any(SetBucketVersioningConfigurationRequest.class));
+ }
+
+ @Test
+ public void testDeleteBucketVersioning() throws Exception {
+ doReturn(s3Client).when(driver).getS3ClientByStoreId(TEST_STORE_ID);
+ BucketTO bucketTO = mock(BucketTO.class);
+ when(bucketTO.getName()).thenReturn(TEST_BUCKET_NAME);
+
+ assertTrue(driver.deleteBucketVersioning(bucketTO, TEST_STORE_ID));
+ ArgumentCaptor<SetBucketVersioningConfigurationRequest> captor =
+ ArgumentCaptor.forClass(SetBucketVersioningConfigurationRequest.class);
+ verify(s3Client, times(1)).setBucketVersioningConfiguration(captor.capture());
+ assertEquals(BucketVersioningConfiguration.SUSPENDED, captor.getValue().getVersioningConfiguration().getStatus());
+ }
+
+ @Test
+ public void testSetBucketQuotaZero() throws Exception {
+ BucketTO bucketTO = mock(BucketTO.class);
+ when(bucketTO.getName()).thenReturn(TEST_BUCKET_NAME);
+ doReturn(TEST_S3_URL).when(driver).getS3Url(TEST_STORE_ID);
+ doReturn("access-key").when(driver).getAccessKey(TEST_STORE_ID);
+ doReturn("secret-key").when(driver).getSecretKey(TEST_STORE_ID);
+
+ HttpClient mockHttpClient = mock(HttpClient.class);
+ HttpResponse<String> mockResponse = mock(HttpResponse.class);
+ when(mockResponse.statusCode()).thenReturn(200);
+ when(mockResponse.body()).thenReturn("");
+ when(mockHttpClient.send(ArgumentMatchers.<HttpRequest>any(),
+ ArgumentMatchers.<HttpResponse.BodyHandler<String>>any())).thenReturn(mockResponse);
+ doReturn(mockHttpClient).when(driver).getS3ExtensionHttpClient();
+
+ driver.setBucketQuota(bucketTO, TEST_STORE_ID, 0);
+
+ ArgumentCaptor<HttpRequest> reqCaptor = ArgumentCaptor.forClass(HttpRequest.class);
+ verify(mockHttpClient, times(1)).send(reqCaptor.capture(),
+ ArgumentMatchers.<HttpResponse.BodyHandler<String>>any());
+ HttpRequest sent = reqCaptor.getValue();
+ assertEquals("PUT", sent.method());
+ assertEquals("/" + TEST_BUCKET_NAME, sent.uri().getPath());
+ assertTrue("query must carry the seaweedfs-quota subresource",
+ sent.uri().getQuery().contains("seaweedfs-quota"));
+ assertNotNull("request must be SigV4-signed", sent.headers().firstValue("Authorization"));
+ assertEquals("{\"quota_size\":0,\"quota_unit\":\"B\",\"quota_enabled\":false}", extractBody(sent));
+ }
+
+ @Test
+ public void testSetBucketQuotaNonZero() throws Exception {
+ BucketTO bucketTO = mock(BucketTO.class);
+ when(bucketTO.getName()).thenReturn(TEST_BUCKET_NAME);
+ doReturn(TEST_S3_URL).when(driver).getS3Url(TEST_STORE_ID);
+ doReturn("access-key").when(driver).getAccessKey(TEST_STORE_ID);
+ doReturn("secret-key").when(driver).getSecretKey(TEST_STORE_ID);
+
+ HttpClient mockHttpClient = mock(HttpClient.class);
+ HttpResponse<String> mockResponse = mock(HttpResponse.class);
+ when(mockResponse.statusCode()).thenReturn(200);
+ when(mockResponse.body()).thenReturn("");
+ when(mockHttpClient.send(ArgumentMatchers.<HttpRequest>any(),
+ ArgumentMatchers.<HttpResponse.BodyHandler<String>>any())).thenReturn(mockResponse);
+ doReturn(mockHttpClient).when(driver).getS3ExtensionHttpClient();
+
+ driver.setBucketQuota(bucketTO, TEST_STORE_ID, 10);
+
+ ArgumentCaptor<HttpRequest> reqCaptor = ArgumentCaptor.forClass(HttpRequest.class);
+ verify(mockHttpClient, times(1)).send(reqCaptor.capture(),
+ ArgumentMatchers.<HttpResponse.BodyHandler<String>>any());
+ HttpRequest sent = reqCaptor.getValue();
+ assertEquals("PUT", sent.method());
+ assertEquals("/" + TEST_BUCKET_NAME, sent.uri().getPath());
+ assertTrue(sent.uri().getQuery().contains("seaweedfs-quota"));
+ assertNotNull(sent.headers().firstValue("Authorization"));
+ assertEquals("{\"quota_size\":10,\"quota_unit\":\"GB\",\"quota_enabled\":true}", extractBody(sent));
+ }
+
+ @Test
+ public void testSetBucketQuotaPropagatesFailure() throws Exception {
+ BucketTO bucketTO = mock(BucketTO.class);
+ when(bucketTO.getName()).thenReturn(TEST_BUCKET_NAME);
+ doReturn(TEST_S3_URL).when(driver).getS3Url(TEST_STORE_ID);
+ doReturn("access-key").when(driver).getAccessKey(TEST_STORE_ID);
+ doReturn("secret-key").when(driver).getSecretKey(TEST_STORE_ID);
+
+ HttpClient mockHttpClient = mock(HttpClient.class);
+ HttpResponse<String> mockResponse = mock(HttpResponse.class);
+ when(mockResponse.statusCode()).thenReturn(403);
+ when(mockResponse.body()).thenReturn("forbidden");
+ when(mockHttpClient.send(ArgumentMatchers.<HttpRequest>any(),
+ ArgumentMatchers.<HttpResponse.BodyHandler<String>>any())).thenReturn(mockResponse);
+ doReturn(mockHttpClient).when(driver).getS3ExtensionHttpClient();
+
+ assertThrows(CloudRuntimeException.class, () -> driver.setBucketQuota(bucketTO, TEST_STORE_ID, 10));
+ }
+
+ @Test
+ public void testSetBucketQuotaRejects3xx() throws Exception {
+ BucketTO bucketTO = mock(BucketTO.class);
+ when(bucketTO.getName()).thenReturn(TEST_BUCKET_NAME);
+ doReturn(TEST_S3_URL).when(driver).getS3Url(TEST_STORE_ID);
+ doReturn("access-key").when(driver).getAccessKey(TEST_STORE_ID);
+ doReturn("secret-key").when(driver).getSecretKey(TEST_STORE_ID);
+
+ HttpClient mockHttpClient = mock(HttpClient.class);
+ HttpResponse<String> mockResponse = mock(HttpResponse.class);
+ // 3xx must NOT be treated as success — the mutation was not applied
+ when(mockResponse.statusCode()).thenReturn(302);
+ when(mockResponse.body()).thenReturn("redirect");
+ when(mockHttpClient.send(ArgumentMatchers.<HttpRequest>any(),
+ ArgumentMatchers.<HttpResponse.BodyHandler<String>>any())).thenReturn(mockResponse);
+ doReturn(mockHttpClient).when(driver).getS3ExtensionHttpClient();
+
+ assertThrows(CloudRuntimeException.class, () -> driver.setBucketQuota(bucketTO, TEST_STORE_ID, 10));
+ }
+
+ /**
+ * Deterministic SigV4 signature-verification test.
+ *
+ * Signs the same request through the AWS SDK v1 AWSS3V4Signer (the same
+ * signer the production code uses) and asserts that the Authorization
+ * header, signed headers, x-amz-content-sha256, and x-amz-date produced
+ * by the driver's request match. This catches signing regressions (e.g.
+ * the query parameter not being in the canonical query string) that a
+ * mere "header exists" check would miss.
+ */
+ @Test
+ public void testSetBucketQuotaSigV4SignatureVerification() throws Exception {
+ String accessKey = "AKIAIOSFODNN7EXAMPLE";
+ String secretKey = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY";
+ String bucketName = "quota-sig-test";
+ String s3Url = "http://s3.example.com:8333";
+ long quotaGiB = 5;
+
+ BucketTO bucketTO = mock(BucketTO.class);
+ when(bucketTO.getName()).thenReturn(bucketName);
+ doReturn(s3Url).when(driver).getS3Url(TEST_STORE_ID);
+ doReturn(accessKey).when(driver).getAccessKey(TEST_STORE_ID);
+ doReturn(secretKey).when(driver).getSecretKey(TEST_STORE_ID);
+
+ HttpClient mockHttpClient = mock(HttpClient.class);
+ HttpResponse<String> mockResponse = mock(HttpResponse.class);
+ when(mockResponse.statusCode()).thenReturn(200);
+ when(mockResponse.body()).thenReturn("");
+ when(mockHttpClient.send(ArgumentMatchers.<HttpRequest>any(),
+ ArgumentMatchers.<HttpResponse.BodyHandler<String>>any())).thenReturn(mockResponse);
+ doReturn(mockHttpClient).when(driver).getS3ExtensionHttpClient();
+
+ driver.setBucketQuota(bucketTO, TEST_STORE_ID, quotaGiB);
+
+ ArgumentCaptor<HttpRequest> reqCaptor = ArgumentCaptor.forClass(HttpRequest.class);
+ verify(mockHttpClient, times(1)).send(reqCaptor.capture(),
+ ArgumentMatchers.<HttpResponse.BodyHandler<String>>any());
+ HttpRequest sent = reqCaptor.getValue();
+
+ // Build the expected signed request the same way the production code does
+ String expectedBody = String.format("{\"quota_size\":%d,\"quota_unit\":\"GB\",\"quota_enabled\":true}", quotaGiB);
+ byte[] bodyBytes = expectedBody.getBytes(StandardCharsets.UTF_8);
+
+ com.amazonaws.DefaultRequest<?> expectedRequest = new com.amazonaws.DefaultRequest<>("s3");
+ expectedRequest.setEndpoint(java.net.URI.create(s3Url));
+ expectedRequest.setHttpMethod(com.amazonaws.http.HttpMethodName.PUT);
+ expectedRequest.setResourcePath("/" + bucketName);
+ expectedRequest.addParameter("seaweedfs-quota", "");
+ expectedRequest.setContent(new java.io.ByteArrayInputStream(bodyBytes));
+ expectedRequest.getHeaders().put("Content-Length", String.valueOf(bodyBytes.length));
+ expectedRequest.getHeaders().put("Content-Type", "application/json");
+
+ com.amazonaws.auth.AWSCredentials credentials = new com.amazonaws.auth.BasicAWSCredentials(accessKey, secretKey);
+ com.amazonaws.services.s3.internal.AWSS3V4Signer signer = new com.amazonaws.services.s3.internal.AWSS3V4Signer();
+ signer.setServiceName("s3");
+ signer.setRegionName("us-east-1");
+ signer.sign(expectedRequest, credentials);
Review Comment:
The "deterministic" reference is signed with a second `AWSS3V4Signer` call, which derives `x-amz-date` from the current clock. If production signing and this call straddle a second boundary, the exact Authorization, date, and signature assertions fail intermittently even when the implementation is correct. Fix the signing timestamp via an injectable clock/fixed signer input, or avoid exact time-dependent header comparison.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org