[I] [File system] Add Baidu Cloud BOS integration [fluss]

["falsity (via GitHub)" <gi...@apache.org>]

falsity opened a new issue, #4789:
URL: https://github.com/apache/fluss/issues/4789

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
   
   
   ### Motivation
   
   Add support for Baidu Cloud BOS (Baidu Object Storage) file system integration to Fluss.
   
   Fluss already supports remote filesystems such as OSS, OBS and COS, but BOS is not supported yet. Adding BOS support would allow users to use Baidu Object Storage as remote storage for snapshots and tiered log segments.
   
   ### Solution
   
   Add a new plugin module `fluss-fs-bos` under `fluss-filesystems/`, following the same architecture as OSS / OBS / COS:
   
   - Register scheme `bos://` via `FileSystemPlugin` SPI
   - Propagate `fs.bos.*` configs to the underlying filesystem/SDK as-is
   - Implement security token provider/receiver so Fluss clients can access BOS data (same requirement as OBS/COS)
   
   ### Proposed approach: direct Baidu BCE Java SDK
   
   Integrate via `com.baidubce:bce-java-sdk`, and implement the Fluss filesystem APIs on top of it.
   
   COS and OBS wrap a published Hadoop connector (`org.apache.hadoop:hadoop-cos`, `org.apache.hadoop:hadoop-huaweicloud`). BOS does not have an equivalent Maven dependency today:
   
   - The Baidu BOS HDFS tool (https://github.com/baidubce/bos-hdfs) is distributed as a downloadable jar to put on the Hadoop classpath.
   - Apache Hadoop `hadoop-bos` (https://github.com/apache/hadoop/pull/8347, HDFS-11161) was merged into `trunk` on Jul 16, 2026, on the Hadoop 3.6.0-SNAPSHOT line. It is not in the released 3.4.3 or 3.5.0 line, and `org.apache.hadoop:hadoop-bos` is not published on Maven Central. Fluss currently shades Hadoop 3.4.3, so this module cannot be used as a dependency yet.
   
   `bce-java-sdk` is available on Maven Central. A local POC already uses it, including security-token behavior tests. The plugin still registers `bos://`, propagates `fs.bos.*`, and implements the Fluss security token provider/receiver, same as OBS/COS.
   
   If a released `hadoop-bos` later matches Fluss's Hadoop version, the plugin can be switched to `HadoopFileSystem` without changing the `bos://` scheme or the user-facing configuration.
   
   ### Configuration Example
   
   ```yaml
   remote.data.dir: bos://bucket/fluss
   bos.endpoint: http://hkg.bcebos.com
   bos.region: hkg
   bos.access.key: your-access-key
   bos.secret.key: your-secret-key
   # bos.sts.endpoint is optional, default https://sts.bj.baidubce.com
   ```
   
   `fs.bos.*` is accepted as well. `bos.region` is required on the server to issue security tokens.
   
   ### Testing
   
   Local testing with environment variables. Required:
   
   - `BOS_ENDPOINT`
   - `BOS_BUCKET`
   - `BOS_REGION`
   - `BOS_ACCESS_KEY`
   - `BOS_SECRET_KEY`
   
   Optional:
   
   - `BOS_STS_ENDPOINT` (only when the default BCE STS endpoint `https://sts.bj.baidubce.com` is not the one to use)
   
   IT cases are skipped when the required credentials are unavailable, same as OSS / OBS / COS.
   
   ### References
   
   - Baidu BOS docs: https://cloud.baidu.com/doc/BOS/index.html
   - BCE Java SDK: https://central.sonatype.com/artifact/com.baidubce/bce-java-sdk
   - BOS HDFS connector: https://github.com/baidubce/bos-hdfs
   - Hadoop BOS work: https://github.com/apache/hadoop/pull/8347
   - Similar Fluss work:
     - https://github.com/apache/fluss/issues/1227
     - https://github.com/apache/fluss/issues/2686
   
   ### Anything else?
   
   _No response_
   
   ### Willingness to contribute
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org