You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by ma...@apache.org on 2026/09/25 15:42:58 UTC
(flink) branch release-1.20 updated (01a3822f16e -> 5998abe4a9a)
This is an automated email from the ASF dual-hosted git repository.
MartijnVisser pushed a change to branch release-1.20
in repository https://gitbox.apache.org/repos/asf/flink.git
from 01a3822f16e [FLINK-40786][build] Skip japicmp for flink-s3-fs-presto
new 191270a65f3 [FLINK-39499][tests] Replace MinIO with SeaweedFS
new 0d8fcb2ecfd [FLINK-40083][tests] Launch the SeaweedFS S3 test backend via 'weed mini'
new 5998abe4a9a [FLINK-40083][tests] Bump SeaweedFS test image to 4.47
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
flink-end-to-end-tests/test-scripts/common_s3.sh | 2 +-
.../{common_s3_minio.sh => common_s3_seaweedfs.sh} | 59 +++++++++++++---------
.../test-scripts/test_batch_wordcount.sh | 17 ++++---
.../test-scripts/test_file_sink.sh | 7 +--
flink-filesystems/flink-s3-fs-base/pom.xml | 2 +-
....java => HAJobRunOnSeaweedFsS3StoreITCase.java} | 29 ++++++-----
...tContainer.java => SeaweedFsTestContainer.java} | 53 +++++++++----------
...erTest.java => SeaweedFsTestContainerTest.java} | 16 +++---
.../HAJobRunOnHadoopS3FileSystemITCase.java | 6 +--
.../HAJobRunOnPrestoS3FileSystemITCase.java | 6 +--
.../org/apache/flink/util/DockerImageVersions.java | 2 +-
11 files changed, 101 insertions(+), 98 deletions(-)
rename flink-end-to-end-tests/test-scripts/{common_s3_minio.sh => common_s3_seaweedfs.sh} (62%)
rename flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/{HAJobRunOnMinioS3StoreITCase.java => HAJobRunOnSeaweedFsS3StoreITCase.java} (84%)
rename flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/{MinioTestContainer.java => SeaweedFsTestContainer.java} (74%)
rename flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/{MinioTestContainerTest.java => SeaweedFsTestContainerTest.java} (89%)
(flink) 02/03: [FLINK-40083][tests] Launch the SeaweedFS S3 test backend via 'weed mini'
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
MartijnVisser pushed a commit to branch release-1.20
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 0d8fcb2ecfd528086d6665b7d9bb63ca0ff38edb
Author: Chris Lu <ch...@gmail.com>
AuthorDate: Wed Sep 16 15:58:13 2026 -0700
[FLINK-40083][tests] Launch the SeaweedFS S3 test backend via 'weed mini'
weed mini pre-creates the bucket (-bucket) and emits its readiness
banner after the enabled components are up. Both the Java test container
and the S3 end-to-end script now use mini instead of creating the bucket
client-side. The Java container also verifies that the requested bucket
exists before tests proceed.
This drops the startup-503 retry workaround, the explicit aws s3 mb
step, and the /healthz poll in favour of the mini readiness banner.
Backport for FLINK-40802.
Generated-by: Claude Code (claude-opus-4-8)
Generated-by: GitHub Copilot CLI 1.0.83
Generated-by: Codex (GPT-5)
(cherry picked from commit 97ad29b8ed436c72098afa2f1d01c8b336e4d32e)
---
.../test-scripts/common_s3_seaweedfs.sh | 12 +++++------
.../flink/fs/s3/common/SeaweedFsTestContainer.java | 24 +++++++++-------------
2 files changed, 15 insertions(+), 21 deletions(-)
diff --git a/flink-end-to-end-tests/test-scripts/common_s3_seaweedfs.sh b/flink-end-to-end-tests/test-scripts/common_s3_seaweedfs.sh
index a8060aa8d14..9da0fce0e49 100644
--- a/flink-end-to-end-tests/test-scripts/common_s3_seaweedfs.sh
+++ b/flink-end-to-end-tests/test-scripts/common_s3_seaweedfs.sh
@@ -48,10 +48,10 @@ function s3_start {
-P \
-e "AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID" -e "AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY" \
chrislusf/seaweedfs:4.34 \
- server \
- -s3 \
+ mini \
-s3.port=8333 \
- -dir=/data)
+ -dir=/data \
+ -bucket="$IT_CASE_S3_BUCKET")
while [[ "$(docker inspect -f {{.State.Running}} "$SEAWEEDFS_CONTAINER_ID")" -ne "true" ]]; do
sleep 0.1
done
@@ -59,7 +59,8 @@ function s3_start {
echo "Started seaweedfs @ $S3_ENDPOINT"
on_exit s3_stop
- while [[ "$(curl -s -o /dev/null -w '%{http_code}' "$S3_ENDPOINT/healthz")" != "200" ]]; do
+ # mini pre-creates the bucket and reports readiness only once all components are up.
+ while ! docker logs "$SEAWEEDFS_CONTAINER_ID" 2>&1 | grep "All enabled components are running and ready to use"; do
sleep 0.1
done
echo "Seaweedfs S3 gateway is up @ $S3_ENDPOINT"
@@ -123,7 +124,4 @@ function s3_setup_with_provider {
source "$(dirname "$0")"/common_s3_operations.sh
-# SeaweedFS does not expose on-disk directories as S3 buckets,
-# so the bucket must be created explicitly
-aws_cli s3 mb "s3://$IT_CASE_S3_BUCKET"
aws_cli s3 cp "/hostdir/test-data/words" "$S3_TEST_DATA_WORDS_URI"
diff --git a/flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/SeaweedFsTestContainer.java b/flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/SeaweedFsTestContainer.java
index d6f8b5c25d0..679140a33fd 100644
--- a/flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/SeaweedFsTestContainer.java
+++ b/flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/SeaweedFsTestContainer.java
@@ -30,7 +30,7 @@ import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.github.dockerjava.api.command.InspectContainerResponse;
import org.testcontainers.containers.GenericContainer;
-import org.testcontainers.containers.wait.strategy.HttpWaitStrategy;
+import org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy;
import org.testcontainers.utility.Base58;
import java.time.Duration;
@@ -47,7 +47,6 @@ public class SeaweedFsTestContainer extends GenericContainer<SeaweedFsTestContai
private static final String AWS_SECRET_ACCESS_KEY = "AWS_SECRET_ACCESS_KEY";
private static final String DEFAULT_STORAGE_DIRECTORY = "/data";
- private static final String HEALTH_ENDPOINT = "/healthz";
private final String accessKey;
private final String secretKey;
@@ -70,21 +69,22 @@ public class SeaweedFsTestContainer extends GenericContainer<SeaweedFsTestContai
withEnv(AWS_ACCESS_KEY_ID, this.accessKey);
withEnv(AWS_SECRET_ACCESS_KEY, this.secretKey);
withCommand(
- "server", "-s3", "-s3.port=" + DEFAULT_PORT, "-dir=" + DEFAULT_STORAGE_DIRECTORY);
+ "mini",
+ "-s3.port=" + DEFAULT_PORT,
+ "-dir=" + DEFAULT_STORAGE_DIRECTORY,
+ "-bucket=" + defaultBucketName);
setWaitStrategy(
- new HttpWaitStrategy()
- .forPort(DEFAULT_PORT)
- .forPath(HEALTH_ENDPOINT)
+ new LogMessageWaitStrategy()
+ .withRegEx("(?s).*All enabled components are running and ready to use.*")
.withStartupTimeout(Duration.ofMinutes(2)));
- // Very rarely, a 503 status will be returned continuously while the container is
- // starting up, slipping past the AmazonS3 client's default retry strategy.
- withStartupAttempts(3);
}
@Override
protected void containerIsStarted(InspectContainerResponse containerInfo) {
super.containerIsStarted(containerInfo);
- createDefaultBucket();
+ Preconditions.checkState(
+ getClient().doesBucketExistV2(defaultBucketName),
+ "SeaweedFS did not create the requested default bucket.");
}
private static String randomString(String prefix, int length) {
@@ -128,10 +128,6 @@ public class SeaweedFsTestContainer extends GenericContainer<SeaweedFsTestContai
FileSystem.initialize(config, null);
}
- private void createDefaultBucket() {
- getClient().createBucket(defaultBucketName);
- }
-
/** Returns the internally used default bucket. */
public String getDefaultBucketName() {
return defaultBucketName;
(flink) 01/03: [FLINK-39499][tests] Replace MinIO with SeaweedFS
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
MartijnVisser pushed a commit to branch release-1.20
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 191270a65f3f02ce135d276463f30cb4f17e48b4
Author: Efrat Levitan <41...@users.noreply.github.com>
AuthorDate: Thu Jun 25 09:31:53 2026 +0300
[FLINK-39499][tests] Replace MinIO with SeaweedFS
Backport for FLINK-40802, adapted to this branch. The new
run-nightly-tests.sh entries are left out.
Generated-by: Claude Code (Claude Opus 5.5)
(cherry picked from commit adc9c1d1f61e00d9ff7f98c64c379786afef6e43)
---
flink-end-to-end-tests/test-scripts/common_s3.sh | 2 +-
.../{common_s3_minio.sh => common_s3_seaweedfs.sh} | 59 +++++++++++++---------
.../test-scripts/test_batch_wordcount.sh | 17 ++++---
.../test-scripts/test_file_sink.sh | 7 +--
flink-filesystems/flink-s3-fs-base/pom.xml | 2 +-
....java => HAJobRunOnSeaweedFsS3StoreITCase.java} | 29 ++++++-----
...tContainer.java => SeaweedFsTestContainer.java} | 33 ++++++------
...erTest.java => SeaweedFsTestContainerTest.java} | 16 +++---
.../HAJobRunOnHadoopS3FileSystemITCase.java | 6 +--
.../HAJobRunOnPrestoS3FileSystemITCase.java | 6 +--
.../org/apache/flink/util/DockerImageVersions.java | 2 +-
11 files changed, 94 insertions(+), 85 deletions(-)
diff --git a/flink-end-to-end-tests/test-scripts/common_s3.sh b/flink-end-to-end-tests/test-scripts/common_s3.sh
index 903fb6a3ff8..6de1604a61c 100644
--- a/flink-end-to-end-tests/test-scripts/common_s3.sh
+++ b/flink-end-to-end-tests/test-scripts/common_s3.sh
@@ -18,7 +18,7 @@
################################################################################
if [[ $S3_SOURCED ]]; then
- echo "Only source common_s3.sh or common_s3_minio.sh in the same test, previously sourced $S3_SOURCED" && exit 1
+ echo "Only source common_s3.sh or common_s3_seaweedfs.sh in the same test, previously sourced $S3_SOURCED" && exit 1
fi
export S3_SOURCED="common_s3.sh"
diff --git a/flink-end-to-end-tests/test-scripts/common_s3_minio.sh b/flink-end-to-end-tests/test-scripts/common_s3_seaweedfs.sh
similarity index 63%
rename from flink-end-to-end-tests/test-scripts/common_s3_minio.sh
rename to flink-end-to-end-tests/test-scripts/common_s3_seaweedfs.sh
index 319a46478d0..a8060aa8d14 100644
--- a/flink-end-to-end-tests/test-scripts/common_s3_minio.sh
+++ b/flink-end-to-end-tests/test-scripts/common_s3_seaweedfs.sh
@@ -18,9 +18,9 @@
################################################################################
if [[ $S3_SOURCED ]]; then
- echo "Only source common_s3.sh or common_s3_minio.sh in the same test, previously sourced $S3_SOURCED" && exit 1
+ echo "Only source common_s3.sh or common_s3_seaweedfs.sh in the same test, previously sourced $S3_SOURCED" && exit 1
fi
-export S3_SOURCED="common_s3_minio.sh"
+export S3_SOURCED="common_s3_seaweedfs.sh"
# export credentials into environment variables for AWS client
export AWS_REGION=us-east-1
@@ -32,48 +32,54 @@ IT_CASE_S3_BUCKET=test-data
S3_TEST_DATA_WORDS_URI="s3://$IT_CASE_S3_BUCKET/words"
###################################
-# Starts a docker container for s3 minio.
+# Starts a docker container for s3 seaweedfs.
#
# Globals:
# TEST_INFRA_DIR
# AWS_ACCESS_KEY_ID
# AWS_SECRET_ACCESS_KEY
# Exports:
-# MINIO_CONTAINER_ID
+# SEAWEEDFS_CONTAINER_ID
# S3_ENDPOINT
###################################
function s3_start {
- echo "Spawning minio for s3 tests"
- export MINIO_CONTAINER_ID=$(docker run -d \
+ echo "Spawning seaweedfs for s3 tests"
+ export SEAWEEDFS_CONTAINER_ID=$(docker run -d \
-P \
- --mount type=bind,source="$TEST_INFRA_DIR",target=/data \
- -e "MINIO_ACCESS_KEY=$AWS_ACCESS_KEY_ID" -e "MINIO_SECRET_KEY=$AWS_SECRET_ACCESS_KEY" -e "MINIO_DOMAIN=localhost" \
- quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z \
+ -e "AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID" -e "AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY" \
+ chrislusf/seaweedfs:4.34 \
server \
- /data)
- while [[ "$(docker inspect -f {{.State.Running}} "$MINIO_CONTAINER_ID")" -ne "true" ]]; do
+ -s3 \
+ -s3.port=8333 \
+ -dir=/data)
+ while [[ "$(docker inspect -f {{.State.Running}} "$SEAWEEDFS_CONTAINER_ID")" -ne "true" ]]; do
sleep 0.1
done
- export S3_ENDPOINT="http://$(docker port "$MINIO_CONTAINER_ID" 9000 | grep -F '0.0.0.0' | sed s'/0\.0\.0\.0/localhost/')"
- echo "Started minio @ $S3_ENDPOINT"
+ export S3_ENDPOINT="http://$(docker port "$SEAWEEDFS_CONTAINER_ID" 8333 | grep -F '0.0.0.0' | sed s'/0\.0\.0\.0/localhost/')"
+ echo "Started seaweedfs @ $S3_ENDPOINT"
on_exit s3_stop
+
+ while [[ "$(curl -s -o /dev/null -w '%{http_code}' "$S3_ENDPOINT/healthz")" != "200" ]]; do
+ sleep 0.1
+ done
+ echo "Seaweedfs S3 gateway is up @ $S3_ENDPOINT"
}
###################################
-# Stops the docker container of minio.
+# Stops the docker container of seaweedfs.
#
# Globals:
-# MINIO_CONTAINER_ID
+# SEAWEEDFS_CONTAINER_ID
###################################
function s3_stop {
- docker kill "$MINIO_CONTAINER_ID"
- docker rm "$MINIO_CONTAINER_ID"
+ docker kill "$SEAWEEDFS_CONTAINER_ID"
+ docker rm "$SEAWEEDFS_CONTAINER_ID"
export S3_ENDPOINT=
- export MINIO_CONTAINER_ID=
+ export SEAWEEDFS_CONTAINER_ID=
}
-# always start it while sourcing, so that MINIO_CONTAINER_ID is available from parent script
-if [[ $MINIO_CONTAINER_ID ]]; then
+# always start it while sourcing, so that SEAWEEDFS_CONTAINER_ID is available from parent script
+if [[ $SEAWEEDFS_CONTAINER_ID ]]; then
s3_stop
fi
s3_start
@@ -95,9 +101,9 @@ function s3_setup {
set_config_key "s3.access-key" "$AWS_ACCESS_KEY_ID"
set_config_key "s3.secret-key" "$AWS_SECRET_ACCESS_KEY"
- # change endpoint to minio's location
+ # change endpoint to seaweedfs's location
set_config_key "s3.endpoint" "$S3_ENDPOINT"
- # If the test is using virtual host style (default), then it tries to reach minio on <bucket>.localhost:<port>,
+ # If the test is using virtual host style (default), then it tries to reach seaweedfs on <bucket>.localhost:<port>,
# which docker does not properly forward.
set_config_key "s3.path.style.access" "true"
set_config_key "s3.path-style-access" "true"
@@ -107,12 +113,17 @@ function s3_setup_with_provider {
add_optional_plugin "s3-fs-$1"
# reads (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
set_config_key "$2" "com.amazonaws.auth.EnvironmentVariableCredentialsProvider"
- # change endpoint to minio's location
+ # change endpoint to seaweedfs's location
set_config_key "s3.endpoint" "$S3_ENDPOINT"
- # If the test is using virtual host style (default), then it tries to reach minio on <bucket>.localhost:<port>,
+ # If the test is using virtual host style (default), then it tries to reach seaweedfs on <bucket>.localhost:<port>,
# which docker does not properly forward.
set_config_key "s3.path.style.access" "true"
set_config_key "s3.path-style-access" "true"
}
source "$(dirname "$0")"/common_s3_operations.sh
+
+# SeaweedFS does not expose on-disk directories as S3 buckets,
+# so the bucket must be created explicitly
+aws_cli s3 mb "s3://$IT_CASE_S3_BUCKET"
+aws_cli s3 cp "/hostdir/test-data/words" "$S3_TEST_DATA_WORDS_URI"
diff --git a/flink-end-to-end-tests/test-scripts/test_batch_wordcount.sh b/flink-end-to-end-tests/test-scripts/test_batch_wordcount.sh
index 59870f7ee67..cd039cb2959 100755
--- a/flink-end-to-end-tests/test-scripts/test_batch_wordcount.sh
+++ b/flink-end-to-end-tests/test-scripts/test_batch_wordcount.sh
@@ -38,11 +38,13 @@ case $INPUT_TYPE in
on_exit "s3_delete_by_full_path_prefix '$S3_PREFIX'"
fetch_complete_result=(s3_get_by_full_path_and_filename_prefix "$OUTPUT_PATH" "${S3_PREFIX}" "" false)
;;
- (hadoop_minio)
- source "$(dirname "$0")"/common_s3_minio.sh
+ (hadoop_seaweedfs)
+ source "$(dirname "$0")"/common_s3_seaweedfs.sh
s3_setup hadoop
ARGS="--input ${S3_TEST_DATA_WORDS_URI} --output s3://$IT_CASE_S3_BUCKET/$S3_PREFIX"
- OUTPUT_PATH="$TEST_INFRA_DIR/$IT_CASE_S3_BUCKET/$S3_PREFIX"
+ OUTPUT_PATH="$TEST_DATA_DIR/$S3_PREFIX"
+ on_exit "s3_delete_by_full_path_prefix '$S3_PREFIX'"
+ fetch_complete_result=(s3_get_by_full_path_and_filename_prefix "$OUTPUT_PATH" "${S3_PREFIX}" "" false)
;;
(hadoop_with_provider)
source "$(dirname "$0")"/common_s3.sh
@@ -60,11 +62,10 @@ case $INPUT_TYPE in
on_exit "s3_delete_by_full_path_prefix '$S3_PREFIX'"
fetch_complete_result=(s3_get_by_full_path_and_filename_prefix "$OUTPUT_PATH" "${S3_PREFIX}" "" false)
;;
- (presto_minio)
- source "$(dirname "$0")"/common_s3_minio.sh
+ (presto_seaweedfs_read)
+ source "$(dirname "$0")"/common_s3_seaweedfs.sh
s3_setup presto
- ARGS="--input ${S3_TEST_DATA_WORDS_URI} --output s3://$IT_CASE_S3_BUCKET/$S3_PREFIX"
- OUTPUT_PATH="$TEST_INFRA_DIR/$IT_CASE_S3_BUCKET/$S3_PREFIX"
+ ARGS="--input ${S3_TEST_DATA_WORDS_URI} --output ${OUTPUT_PATH}"
;;
(dummy-fs)
source "$(dirname "$0")"/common_dummy_fs.sh
@@ -84,7 +85,7 @@ start_cluster
# The test may run against different source types.
# But the sources should provide the same test data, so the checksum stays the same for all tests.
${FLINK_DIR}/bin/flink run -p 1 ${FLINK_DIR}/examples/batch/WordCount.jar ${ARGS}
-# Fetches result from AWS s3 to the OUTPUT_PATH, no-op for other filesystems and minio-based tests
+# Fetches result from AWS s3 to the OUTPUT_PATH, no-op for other filesystems and the presto_seaweedfs_read test
# it seems we need a function for retry_times
function fetch_it() {
diff --git a/flink-end-to-end-tests/test-scripts/test_file_sink.sh b/flink-end-to-end-tests/test-scripts/test_file_sink.sh
index 0acc9425bcb..8aeda7ac5d5 100755
--- a/flink-end-to-end-tests/test-scripts/test_file_sink.sh
+++ b/flink-end-to-end-tests/test-scripts/test_file_sink.sh
@@ -62,7 +62,7 @@ function get_total_number_of_valid_lines {
if [ "${OUT_TYPE}" == "local" ]; then
echo "[INFO] Test run in local environment: No S3 environment is loaded."
elif [ "${OUT_TYPE}" == "s3" ]; then
- source "$(dirname "$0")"/common_s3_minio.sh
+ source "$(dirname "$0")"/common_s3_seaweedfs.sh
s3_setup hadoop
# overwrites JOB_OUTPUT_PATH to point to S3
@@ -80,11 +80,6 @@ elif [ "${OUT_TYPE}" == "s3" ]; then
find "${LOCAL_JOB_OUTPUT_PATH}" -type f \( -iname "part-*" \) -exec cat {} + | sort -g
}
- # overwrites implementation for local runs
- function get_total_number_of_valid_lines {
- s3_get_number_of_lines_by_prefix "${S3_DATA_PREFIX}" "part-"
- }
-
# make sure we delete the file at the end
function out_cleanup {
s3_delete_by_full_path_prefix "${S3_DATA_PREFIX}"
diff --git a/flink-filesystems/flink-s3-fs-base/pom.xml b/flink-filesystems/flink-s3-fs-base/pom.xml
index a078be6d73e..5a437d5aeb9 100644
--- a/flink-filesystems/flink-s3-fs-base/pom.xml
+++ b/flink-filesystems/flink-s3-fs-base/pom.xml
@@ -241,7 +241,7 @@ under the License.
</exclusions>
</dependency>
- <!-- flink-runtime and curator dependencies are needed for the HAJobRunOnMinioS3StoreITCase -->
+ <!-- flink-runtime and curator dependencies are needed for the HAJobRunOnSeaweedFsS3StoreITCase -->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-runtime</artifactId>
diff --git a/flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/HAJobRunOnMinioS3StoreITCase.java b/flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/HAJobRunOnSeaweedFsS3StoreITCase.java
similarity index 84%
rename from flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/HAJobRunOnMinioS3StoreITCase.java
rename to flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/HAJobRunOnSeaweedFsS3StoreITCase.java
index d6fa0f7c2ea..2bf13e85907 100644
--- a/flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/HAJobRunOnMinioS3StoreITCase.java
+++ b/flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/HAJobRunOnSeaweedFsS3StoreITCase.java
@@ -44,20 +44,21 @@ import static org.apache.flink.shaded.guava31.com.google.common.base.Predicates.
import static org.assertj.core.api.Assertions.assertThat;
/**
- * {@code HAJobRunOnMinioS3StoreITCase} covers a job run where the HA data is stored in Minio. The
- * implementation verifies whether the {@code JobResult} was written into the FileSystem-backed
- * {@code JobResultStore}.
+ * {@code HAJobRunOnSeaweedFsS3StoreITCase} covers a job run where the HA data is stored in
+ * SeaweedFs. The implementation verifies whether the {@code JobResult} was written into the
+ * FileSystem-backed {@code JobResultStore}.
*/
-public abstract class HAJobRunOnMinioS3StoreITCase extends AbstractHAJobRunITCase {
+public abstract class HAJobRunOnSeaweedFsS3StoreITCase extends AbstractHAJobRunITCase {
private static final String CLUSTER_ID = "test-cluster";
private static final String JOB_RESULT_STORE_FOLDER = "jrs";
@RegisterExtension
@Order(2)
- private static final AllCallbackWrapper<TestContainerExtension<MinioTestContainer>>
- MINIO_EXTENSION =
- new AllCallbackWrapper<>(new TestContainerExtension<>(MinioTestContainer::new));
+ private static final AllCallbackWrapper<TestContainerExtension<SeaweedFsTestContainer>>
+ SEAWEEDFS_EXTENSION =
+ new AllCallbackWrapper<>(
+ new TestContainerExtension<>(SeaweedFsTestContainer::new));
@RegisterExtension
@Order(3)
@@ -71,19 +72,19 @@ public abstract class HAJobRunOnMinioS3StoreITCase extends AbstractHAJobRunITCas
.build();
});
- private static MinioTestContainer getMinioContainer() {
- return MINIO_EXTENSION.getCustomExtension().getTestContainer();
+ private static SeaweedFsTestContainer getSeaweedFsContainer() {
+ return SEAWEEDFS_EXTENSION.getCustomExtension().getTestContainer();
}
private static String createS3URIWithSubPath(String... subfolders) {
- return getMinioContainer().getS3UriForDefaultBucket() + createSubPath(subfolders);
+ return getSeaweedFsContainer().getS3UriForDefaultBucket() + createSubPath(subfolders);
}
private static List<S3ObjectSummary> getObjectsFromJobResultStore() {
- return getMinioContainer()
+ return getSeaweedFsContainer()
.getClient()
.listObjects(
- getMinioContainer().getDefaultBucketName(),
+ getSeaweedFsContainer().getDefaultBucketName(),
createSubPath(CLUSTER_ID, JOB_RESULT_STORE_FOLDER))
.getObjectSummaries();
}
@@ -96,7 +97,7 @@ public abstract class HAJobRunOnMinioS3StoreITCase extends AbstractHAJobRunITCas
private static Configuration createConfiguration() {
final Configuration config = new Configuration();
- getMinioContainer().setS3ConfigOptions(config);
+ getSeaweedFsContainer().setS3ConfigOptions(config);
// JobResultStore configuration
config.set(JobResultStoreOptions.DELETE_ON_COMMIT, Boolean.FALSE);
@@ -136,7 +137,7 @@ public abstract class HAJobRunOnMinioS3StoreITCase extends AbstractHAJobRunITCas
.matches(not(FileSystemJobResultStore::hasValidDirtyJobResultStoreEntryExtension));
final String objContent =
- getMinioContainer()
+ getSeaweedFsContainer()
.getClient()
.getObjectAsString(objRef.getBucketName(), objRef.getKey());
assertThat(objContent).contains(ApplicationStatus.SUCCEEDED.name());
diff --git a/flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/MinioTestContainer.java b/flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/SeaweedFsTestContainer.java
similarity index 82%
rename from flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/MinioTestContainer.java
rename to flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/SeaweedFsTestContainer.java
index 354be949aed..d6f8b5c25d0 100644
--- a/flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/MinioTestContainer.java
+++ b/flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/SeaweedFsTestContainer.java
@@ -36,40 +36,41 @@ import org.testcontainers.utility.Base58;
import java.time.Duration;
import java.util.Locale;
-/** {@code MinioTestContainer} provides a {@code Minio} test instance. */
-public class MinioTestContainer extends GenericContainer<MinioTestContainer> {
+/** {@code SeaweedFsTestContainer} provides a {@code SeaweedFs} test instance. */
+public class SeaweedFsTestContainer extends GenericContainer<SeaweedFsTestContainer> {
private static final String FLINK_CONFIG_S3_ENDPOINT = "s3.endpoint";
- private static final int DEFAULT_PORT = 9000;
+ private static final int DEFAULT_PORT = 8333;
- private static final String MINIO_ACCESS_KEY = "MINIO_ROOT_USER";
- private static final String MINIO_SECRET_KEY = "MINIO_ROOT_PASSWORD";
+ private static final String AWS_ACCESS_KEY_ID = "AWS_ACCESS_KEY_ID";
+ private static final String AWS_SECRET_ACCESS_KEY = "AWS_SECRET_ACCESS_KEY";
private static final String DEFAULT_STORAGE_DIRECTORY = "/data";
- private static final String HEALTH_ENDPOINT = "/minio/health/ready";
+ private static final String HEALTH_ENDPOINT = "/healthz";
private final String accessKey;
private final String secretKey;
private final String defaultBucketName;
- public MinioTestContainer() {
+ public SeaweedFsTestContainer() {
this(randomString("bucket", 6));
}
- public MinioTestContainer(String defaultBucketName) {
- super(DockerImageVersions.MINIO);
+ public SeaweedFsTestContainer(String defaultBucketName) {
+ super(DockerImageVersions.SEAWEEDFS);
this.accessKey = randomString("accessKey", 10);
// secrets must have at least 8 characters
this.secretKey = randomString("secret", 10);
this.defaultBucketName = Preconditions.checkNotNull(defaultBucketName);
- withNetworkAliases(randomString("minio", 6));
+ withNetworkAliases(randomString("seaweedfs", 6));
addExposedPort(DEFAULT_PORT);
- withEnv(MINIO_ACCESS_KEY, this.accessKey);
- withEnv(MINIO_SECRET_KEY, this.secretKey);
- withCommand("server", DEFAULT_STORAGE_DIRECTORY);
+ withEnv(AWS_ACCESS_KEY_ID, this.accessKey);
+ withEnv(AWS_SECRET_ACCESS_KEY, this.secretKey);
+ withCommand(
+ "server", "-s3", "-s3.port=" + DEFAULT_PORT, "-dir=" + DEFAULT_STORAGE_DIRECTORY);
setWaitStrategy(
new HttpWaitStrategy()
.forPort(DEFAULT_PORT)
@@ -90,7 +91,7 @@ public class MinioTestContainer extends GenericContainer<MinioTestContainer> {
return String.format("%s-%s", prefix, Base58.randomString(length).toLowerCase(Locale.ROOT));
}
- /** Creates {@link AmazonS3} client for accessing the {@code Minio} instance. */
+ /** Creates {@link AmazonS3} client for accessing the {@code SeaweedFs} instance. */
public AmazonS3 getClient() {
return AmazonS3Client.builder()
.withCredentials(
@@ -108,9 +109,9 @@ public class MinioTestContainer extends GenericContainer<MinioTestContainer> {
}
/**
- * Initializes the Minio instance (i.e. creating the default bucket and initializing Flink's
+ * Initializes the SeaweedFs instance (i.e. creating the default bucket and initializing Flink's
* FileSystems). Additionally, the passed Flink {@link Configuration} is extended by all
- * relevant parameter to access the {@code Minio} instance.
+ * relevant parameter to access the {@code SeaweedFs} instance.
*/
public void setS3ConfigOptions(Configuration config) {
config.setString(FLINK_CONFIG_S3_ENDPOINT, getHttpEndpoint());
diff --git a/flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/MinioTestContainerTest.java b/flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/SeaweedFsTestContainerTest.java
similarity index 89%
rename from flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/MinioTestContainerTest.java
rename to flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/SeaweedFsTestContainerTest.java
index 988df5104fc..f692153264a 100644
--- a/flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/MinioTestContainerTest.java
+++ b/flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/SeaweedFsTestContainerTest.java
@@ -35,22 +35,22 @@ import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/**
- * {@code MinioTestContainerTest} tests some basic functionality provided by {@link
- * MinioTestContainer}.
+ * {@code SeaweedFsTestContainerTest} tests some basic functionality provided by {@link
+ * SeaweedFsTestContainer}.
*/
-public class MinioTestContainerTest {
+public class SeaweedFsTestContainerTest {
private static final String DEFAULT_BUCKET_NAME = "test-bucket";
@RegisterExtension
- private static final EachCallbackWrapper<TestContainerExtension<MinioTestContainer>>
- MINIO_EXTENSION =
+ private static final EachCallbackWrapper<TestContainerExtension<SeaweedFsTestContainer>>
+ SEAWEEDFS_EXTENSION =
new EachCallbackWrapper<>(
new TestContainerExtension<>(
- () -> new MinioTestContainer(DEFAULT_BUCKET_NAME)));
+ () -> new SeaweedFsTestContainer(DEFAULT_BUCKET_NAME)));
- private static MinioTestContainer getTestContainer() {
- return MINIO_EXTENSION.getCustomExtension().getTestContainer();
+ private static SeaweedFsTestContainer getTestContainer() {
+ return SEAWEEDFS_EXTENSION.getCustomExtension().getTestContainer();
}
private static AmazonS3 getClient() {
diff --git a/flink-filesystems/flink-s3-fs-hadoop/src/test/java/org/apache/flink/fs/s3hadoop/HAJobRunOnHadoopS3FileSystemITCase.java b/flink-filesystems/flink-s3-fs-hadoop/src/test/java/org/apache/flink/fs/s3hadoop/HAJobRunOnHadoopS3FileSystemITCase.java
index ebe5f713e79..318f29769e6 100644
--- a/flink-filesystems/flink-s3-fs-hadoop/src/test/java/org/apache/flink/fs/s3hadoop/HAJobRunOnHadoopS3FileSystemITCase.java
+++ b/flink-filesystems/flink-s3-fs-hadoop/src/test/java/org/apache/flink/fs/s3hadoop/HAJobRunOnHadoopS3FileSystemITCase.java
@@ -18,7 +18,7 @@
package org.apache.flink.fs.s3hadoop;
-import org.apache.flink.fs.s3.common.HAJobRunOnMinioS3StoreITCase;
+import org.apache.flink.fs.s3.common.HAJobRunOnSeaweedFsS3StoreITCase;
-/** Runs the {@link HAJobRunOnMinioS3StoreITCase} on the Hadoop S3 file system. */
-public class HAJobRunOnHadoopS3FileSystemITCase extends HAJobRunOnMinioS3StoreITCase {}
+/** Runs the {@link HAJobRunOnSeaweedFsS3StoreITCase} on the Hadoop S3 file system. */
+public class HAJobRunOnHadoopS3FileSystemITCase extends HAJobRunOnSeaweedFsS3StoreITCase {}
diff --git a/flink-filesystems/flink-s3-fs-presto/src/test/java/org/apache/flink/fs/s3presto/HAJobRunOnPrestoS3FileSystemITCase.java b/flink-filesystems/flink-s3-fs-presto/src/test/java/org/apache/flink/fs/s3presto/HAJobRunOnPrestoS3FileSystemITCase.java
index 2bd9d82fc8d..fa4179d8cf9 100644
--- a/flink-filesystems/flink-s3-fs-presto/src/test/java/org/apache/flink/fs/s3presto/HAJobRunOnPrestoS3FileSystemITCase.java
+++ b/flink-filesystems/flink-s3-fs-presto/src/test/java/org/apache/flink/fs/s3presto/HAJobRunOnPrestoS3FileSystemITCase.java
@@ -18,7 +18,7 @@
package org.apache.flink.fs.s3presto;
-import org.apache.flink.fs.s3.common.HAJobRunOnMinioS3StoreITCase;
+import org.apache.flink.fs.s3.common.HAJobRunOnSeaweedFsS3StoreITCase;
-/** Runs the {@link HAJobRunOnMinioS3StoreITCase} on the Presto S3 file system. */
-public class HAJobRunOnPrestoS3FileSystemITCase extends HAJobRunOnMinioS3StoreITCase {}
+/** Runs the {@link HAJobRunOnSeaweedFsS3StoreITCase} on the Presto S3 file system. */
+public class HAJobRunOnPrestoS3FileSystemITCase extends HAJobRunOnSeaweedFsS3StoreITCase {}
diff --git a/flink-test-utils-parent/flink-test-utils-junit/src/main/java/org/apache/flink/util/DockerImageVersions.java b/flink-test-utils-parent/flink-test-utils-junit/src/main/java/org/apache/flink/util/DockerImageVersions.java
index ee597b08264..655f4e3b466 100644
--- a/flink-test-utils-parent/flink-test-utils-junit/src/main/java/org/apache/flink/util/DockerImageVersions.java
+++ b/flink-test-utils-parent/flink-test-utils-junit/src/main/java/org/apache/flink/util/DockerImageVersions.java
@@ -36,7 +36,7 @@ public class DockerImageVersions {
public static final String LOCALSTACK = "localstack/localstack:0.13.3";
- public static final String MINIO = "quay.io/minio/minio:RELEASE.2022-02-07T08-17-33Z";
+ public static final String SEAWEEDFS = "chrislusf/seaweedfs:4.34";
public static final String ZOOKEEPER = "zookeeper:3.4.14";
(flink) 03/03: [FLINK-40083][tests] Bump SeaweedFS test image to 4.47
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
MartijnVisser pushed a commit to branch release-1.20
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 5998abe4a9af954703cbcfbc0ca27da40a53f402
Author: Chris Lu <ch...@gmail.com>
AuthorDate: Wed Sep 16 15:58:13 2026 -0700
[FLINK-40083][tests] Bump SeaweedFS test image to 4.47
4.47 answers S3 directory-path probes the way AWS does, so Flink
savepoints/checkpoints restore correctly against SeaweedFS; it also
carries multipart and truncated-PUT robustness fixes exercised by the
recoverable writer.
Backport for FLINK-40802.
Generated-by: Claude Code (claude-opus-4-8)
Generated-by: GitHub Copilot CLI 1.0.83
(cherry picked from commit 2440f65c9fdfa1aef8441b04a6846e42d8304142)
---
flink-end-to-end-tests/test-scripts/common_s3_seaweedfs.sh | 2 +-
.../src/main/java/org/apache/flink/util/DockerImageVersions.java | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/flink-end-to-end-tests/test-scripts/common_s3_seaweedfs.sh b/flink-end-to-end-tests/test-scripts/common_s3_seaweedfs.sh
index 9da0fce0e49..8bd0836779a 100644
--- a/flink-end-to-end-tests/test-scripts/common_s3_seaweedfs.sh
+++ b/flink-end-to-end-tests/test-scripts/common_s3_seaweedfs.sh
@@ -47,7 +47,7 @@ function s3_start {
export SEAWEEDFS_CONTAINER_ID=$(docker run -d \
-P \
-e "AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID" -e "AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY" \
- chrislusf/seaweedfs:4.34 \
+ chrislusf/seaweedfs:4.47 \
mini \
-s3.port=8333 \
-dir=/data \
diff --git a/flink-test-utils-parent/flink-test-utils-junit/src/main/java/org/apache/flink/util/DockerImageVersions.java b/flink-test-utils-parent/flink-test-utils-junit/src/main/java/org/apache/flink/util/DockerImageVersions.java
index 655f4e3b466..850dabe7f56 100644
--- a/flink-test-utils-parent/flink-test-utils-junit/src/main/java/org/apache/flink/util/DockerImageVersions.java
+++ b/flink-test-utils-parent/flink-test-utils-junit/src/main/java/org/apache/flink/util/DockerImageVersions.java
@@ -36,7 +36,7 @@ public class DockerImageVersions {
public static final String LOCALSTACK = "localstack/localstack:0.13.3";
- public static final String SEAWEEDFS = "chrislusf/seaweedfs:4.34";
+ public static final String SEAWEEDFS = "chrislusf/seaweedfs:4.47";
public static final String ZOOKEEPER = "zookeeper:3.4.14";