You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@dolphinscheduler.apache.org by "wcmolin (via GitHub)" <gi...@apache.org> on 2026/05/15 01:39:31 UTC

[I] [Bug] [Remote Logging] S3-compatible storage with custom endpoint should allow custom region [dolphinscheduler]

wcmolin opened a new issue, #18267:
URL: https://github.com/apache/dolphinscheduler/issues/18267

### Search before asking

- [x] I had searched in the issues and found no similar issues.

### What happened

We use a self-hosted MinIO service in our internal network as S3-compatible storage for remote logging.

Example configuration:

remote.logging:
target: S3
base.dir: task-logs
thread.pool.size: 10

aws:
s3:
credentials.provider.type: AWSStaticCredentialsProvider
access.key.id: xxx
access.key.secret: xxx
region: internal-region
bucket.name: dolphinscheduler
endpoint: http://10.10.20.30:9000

After a task finishes, DolphinScheduler tries to upload the task log to MinIO, but creating the S3 client fails:

java.lang.IllegalArgumentException: Cannot create enum from internal-region value!
at com.amazonaws.regions.Regions.fromName(Regions.java:101)
at org.apache.dolphinscheduler.authentication.aws.AmazonS3ClientFactory.createAmazonS3Client(AmazonS3ClientFactory.java:35)

The problem is that AmazonS3ClientFactory still calls Regions.fromName(region) even when a custom S3 endpoint is configured.

For self-hosted MinIO or other S3-compatible storage, the region can be a custom value and may not be an official AWS region.

### What you expected to happen

When aws.s3.endpoint is configured, DolphinScheduler should allow custom region values for S3-compatible storage.

It should only require official AWS regions when no custom endpoint is configured and the client is connecting to AWS S3 directly.

### How to reproduce

1. Deploy a self-hosted MinIO service in an internal network, for example http://10.10.20.30:9000.
2. Configure MinIO with a custom region, for example internal-region.
3. Enable DolphinScheduler remote logging.
4. Configure remote.logging.target as S3.
5. Configure aws.yaml with the MinIO endpoint and custom region.
6. Run a task and wait for it to finish.
7. The remote log upload fails when creating the S3 client.

### Anything else

A possible fix is to use the raw configured region string when a custom endpoint is present, and only call Regions.fromName(region) when no endpoint is configured.

### Version

dev

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project Code of Conduct

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Improvement] [Remote Logging] S3-compatible storage with custom endpoint should allow custom region [dolphinscheduler]

Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.

ruanwenjun commented on issue #18267:
URL: https://github.com/apache/dolphinscheduler/issues/18267#issuecomment-4456392917

   I would perfer to add a new storage plugin; I'm not sure if there will be more compatibility change like this in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Improvement] [Remote Logging] S3-compatible storage with custom endpoint should allow custom region [dolphinscheduler]

Posted by "wcmolin (via GitHub)" <gi...@apache.org>.

wcmolin commented on issue #18267:
URL: https://github.com/apache/dolphinscheduler/issues/18267#issuecomment-4473393535

I agree with the analysis from @leocook and lean toward not introducing a separate plugin for now.

In my view, this is not a new storage type, but an endpoint/region compatibility issue within the S3 protocol implementation. AWS SDK's endpoint configuration is the documented mechanism for overriding the S3 endpoint, and the region is used as the signing region string. When a custom endpoint is configured, DolphinScheduler can pass the configured region string directly to `EndpointConfiguration`; validating it with `Regions.fromName()` is stricter than what the SDK requires and breaks S3-compatible storage such as MinIO.

From the current code structure, both S3 resource storage and remote logging go through the same `AmazonS3ClientFactory`, so handling this in the factory keeps the change small and centralized: use the raw region string when `endpoint` is configured, and only use `Regions.fromName()` when no custom endpoint is configured. `S3StorageOperator` already reuses the same factory, so this avoids near-duplicate implementations, two configuration entries, and duplicated future bug fixes.

I understand @ruanwenjun's concern that more compatibility differences may appear in the future. Maybe a reasonable boundary is: as long as the behavior is still S3 API based and can be handled by AWS SDK configuration, we keep it in the existing S3 plugin; if we later find substantial differences in authentication model, object semantics, listing/multipart behavior, or other SDK-level behavior that cannot be unified cleanly, then introducing a separate plugin would be justified.

This keeps the current fix minimal while still leaving room to split it later if real divergence appears.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Improvement] [Remote Logging] S3-compatible storage with custom endpoint should allow custom region [dolphinscheduler]

Posted by "wcmolin (via GitHub)" <gi...@apache.org>.

wcmolin commented on issue #18267:
URL: https://github.com/apache/dolphinscheduler/issues/18267#issuecomment-4483923633

Thanks, that makes sense. I agree we should not assume MinIO supports all AWS S3 features. I also agree that compatibility `if` branches should have a clear boundary; otherwise they can make the code harder to maintain.

To clarify, this change does not rely on MinIO supporting the full S3 feature set. MinIO’s own S3 compatibility document lists both supported and unsupported APIs, so I agree we should not treat it as a complete AWS S3 replacement: [MinIO S3 API Compatibility](https://minio.community/community/minio-object-store/reference/s3-api-compatibility.html).

But DolphinScheduler currently uses only a small subset of basic S3 object/bucket APIs in resource storage and remote logging, such as `PutObject`, `GetObject`, `CopyObject`, `DeleteObject`, `ListObjectsV2`, `HeadObject` / `doesObjectExist`, and `HeadBucket` / `doesBucketExistV2`. These APIs are within MinIO’s documented compatibility scope. So the current change does not require MinIO to include all S3 features; it only depends on the subset that DolphinScheduler actually uses.

Also, to avoid making this look like a random compatibility `if`, we can make the boundary explicit in the code/comment. This is not a MinIO-specific branch. It is the SDK endpoint configuration path:

- custom endpoint: use `EndpointConfiguration(endpoint, region)` and pass the configured region as the signing region string;
- no custom endpoint: keep the existing AWS S3 path and continue validating the region with `Regions.fromName()`.

AWS SDK documents `EndpointConfiguration` as the configuration for service endpoint and signing region, and the region selection doc also describes using explicit endpoint configuration when choosing a specific endpoint: [AWS SDK v1 EndpointConfiguration Javadoc](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/client/builder/AwsClientBuilder.EndpointConfiguration.html), [AWS SDK for Java 1.x Region Selection](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-region-selection.html).

If DolphinScheduler later needs APIs or behavior that MinIO does not support, then splitting a dedicated plugin would be a better option.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Improvement] [Remote Logging] S3-compatible storage with custom endpoint should allow custom region [dolphinscheduler]

Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.

SbloodyS closed issue #18267: [Improvement] [Remote Logging] S3-compatible storage with custom endpoint should allow custom region
URL: https://github.com/apache/dolphinscheduler/issues/18267


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] [Remote Logging] S3-compatible storage with custom endpoint should allow custom region [dolphinscheduler]

Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.

SbloodyS commented on issue #18267:
URL: https://github.com/apache/dolphinscheduler/issues/18267#issuecomment-4456149038

   This is not a bug. I'll mark it as an `Improvement`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Improvement] [Remote Logging] S3-compatible storage with custom endpoint should allow custom region [dolphinscheduler]

Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.

SbloodyS commented on issue #18267:
URL: https://github.com/apache/dolphinscheduler/issues/18267#issuecomment-4456595847

   > I would perfer to add a new storage plugin; I'm not sure if there will be more compatibility change like this in the future.
   
   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Improvement] [Remote Logging] S3-compatible storage with custom endpoint should allow custom region [dolphinscheduler]

Posted by "leocook (via GitHub)" <gi...@apache.org>.

leocook commented on issue #18267:
URL: https://github.com/apache/dolphinscheduler/issues/18267#issuecomment-4465310612

   Hi, I did some quick research on how other popular projects handle S3-compatible storage (like MinIO), hope it helps:
   
   - **Hadoop S3A** — same connector, custom endpoint + any region string, no enum validation
   - **Trino** — same connector, `s3.endpoint` + `s3.region` accepts any value
   - **Flink / Spark / Airflow / GitLab** — all same approach, just add endpoint config
   
   **None of them create a separate plugin for this.**
   Because MinIO is basically S3 protocol with a different endpoint, not really a different storage type.
   
   I understand the concern about future compatibility issues, but personally I feel creating a new plugin may introduce near-duplicate code and user confusion (should I choose S3 or S3-Compatible?). The fix itself is quite small — just skip `Regions.fromName()` when custom endpoint is configured, probably less than 10 lines.
   
   Of course this is just my thought, happy to hear other opinions. And happy to help if needed :)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Improvement] [Remote Logging] S3-compatible storage with custom endpoint should allow custom region [dolphinscheduler]

Posted by "wcmolin (via GitHub)" <gi...@apache.org>.

wcmolin commented on issue #18267:
URL: https://github.com/apache/dolphinscheduler/issues/18267#issuecomment-4473395784

I agree with @leocook's analysis and lean toward not introducing a separate plugin for now.

This keeps the current fix minimal while still leaving room to split it later if real divergence appears.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Improvement] [Remote Logging] S3-compatible storage with custom endpoint should allow custom region [dolphinscheduler]

Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.

ruanwenjun commented on issue #18267:
URL: https://github.com/apache/dolphinscheduler/issues/18267#issuecomment-4477544059

   As for the issue of duplicate code, I don’t think we need to worry about it, because Mino can reduce duplicate code by importing the current S3 plugin module. Of course, this raises a question of scope: specifically, whether Mino includes all of S3’s features.
   If you’re certain that this is the only difference in Mino’s use of S3, I agree that we should handle it by adding an `if` statement, but we need to clearly explain this in the comments. Many projects end up with messy code precisely because of this kind of haphazard addition of `if` statements for compatibility purposes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org