You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "Gezi-lzq (via GitHub)" <gi...@apache.org> on 2026/08/05 02:32:58 UTC
[PR] [build] Add a local development kit for contributors [fluss]
Gezi-lzq opened a new pull request, #3851:
URL: https://github.com/apache/fluss/pull/3851
<!-- Generated-by: OpenAI Codex (GPT-5) following https://github.com/apache/fluss/blob/main/AGENTS.md -->
### Purpose
Closes #3850.
This draft provides a concrete implementation of a contributor-oriented DevKit for running and validating locally built Fluss artifacts. It is opened as a draft to collect feedback on its scope, dependencies, and maintenance boundary.
- [ ] No generative AI tools used
- [x] Yes, OpenAI Codex was used to assist design, implementation, testing, and review
### Brief change log
- Add a Docker Compose and `just` based local development environment
- Mount the local `build-target` without building a Fluss image
- Support one or three TabletServers
- Add Core, Iceberg, Paimon, Lance, and Hudi profiles
- Add composable commands for startup, logs, SQL execution, tiering validation, and cleanup
- Propagate configured Hadoop settings to the Hudi write client so the S3A-backed profile retains its endpoint and credentials
### Tests
- `just --list`
- Docker Compose rendering for all profiles
- Core single-node and three-node startup
- Iceberg: create table -> write -> tier -> query
- Paimon: create table -> write -> tier -> query
- Hudi: create table -> write -> tier -> query
- Lance: create table -> write -> tiering job
- Host SQL file execution through `just run-sql`
- Maven validate, Checkstyle, Spotless, and RAT
Full `mvn verify` has not yet completed in the current local environment, so this PR remains a draft.
### API and Format
No public API or storage format changes.
### Documentation
Add `devkit/README.md` covering setup, profiles, operations, validation, and profile extension.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4232602079
##########
devkit/docker-compose.yml:
##########
@@ -0,0 +1,179 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: fluss-devkit
+
+x-fluss-common: &fluss-common
+ image: ${FLUSS_DEVKIT_IMAGE:-eclipse-temurin:17-jre-noble}
+ init: true
+ restart: unless-stopped
+ stop_grace_period: 2m
+ volumes:
+ - ../build-target:/opt/fluss:ro
+ - ${FLUSS_DEVKIT_CONFIG:-./profiles/core/server.yaml}:/opt/fluss/conf/server.yaml:ro
+ - shared-data:/tmp/fluss
+ logging:
+ driver: json-file
+ options:
+ max-size: 10m
+ max-file: "3"
+
+x-tablet-common: &tablet-common
+ <<: *fluss-common
+ entrypoint: ["/opt/fluss/bin/tablet-server.sh", "start-foreground"]
+ depends_on:
+ coordinator-server:
+ condition: service_healthy
+ environment:
Review Comment:
Sorry, I forgot to remove this earlier. I also checked the other shared Compose configurations and did not find any similar ineffective blocks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub) via issues" <is...@fluss.apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4483179175
##########
.github/workflows/devkit-smoke.yaml:
##########
@@ -0,0 +1,175 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+name: DevKit Smoke
+
+permissions:
+ contents: read
+
+on:
+ pull_request:
+ paths:
+ - 'devkit/**'
+ - 'pom.xml'
+ - '.github/workflows/devkit-smoke.yaml'
+ push:
+ branches:
+ - main
+ - release-**
+ paths:
+ - 'devkit/**'
+ - 'pom.xml'
+ - '.github/workflows/devkit-smoke.yaml'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }}
+ cancel-in-progress: true
+
+jobs:
+ smoke:
+ runs-on: ubuntu-latest
+ timeout-minutes: 45
+ steps:
+ - uses: actions/checkout@v6
+
+ - name: Set up JDK 11
+ uses: actions/setup-java@v5
+ with:
+ java-version: '11'
+ distribution: temurin
+
+ - name: Install just
+ uses: taiki-e/install-action@v2
+ with:
+ tool: just@1.58.0
+
+ - name: Validate managed profile dependency versions
+ run: bash devkit/validate-managed-dependency-versions.sh
+
+ - name: Core smoke
+ working-directory: devkit
+ run: |
+ set -euo pipefail
+ just build core
+ just up core
+ just run-sql examples/core/setup.sql
+ core_output=$(just run-sql examples/core/batch-scan.sql)
+ printf '%s\n' "$core_output"
+ grep -Fq 'Alice' <<<"$core_output"
+ grep -Fq 'Bob' <<<"$core_output"
+ grep -Fq 'alpha' <<<"$core_output"
+ grep -Fq 'beta' <<<"$core_output"
+ just clean
+
+ - name: Gateway smoke
+ working-directory: devkit
+ run: |
+ set -euo pipefail
+ just build gateway
+ just up gateway
+ curl -fsS -X POST \
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on PR #3851:
URL: https://github.com/apache/fluss/pull/3851#issuecomment-5660052586
Hi @beryllw, I noticed the recent work on the Gateway lakehouse quickstart. #3851 also includes a contributor-focused Gateway scenario that builds and runs Gateway from the current checkout against a real Fluss cluster, covering readiness, metadata access, REST writes, and configuration or integration changes.
From a Gateway development perspective, could you take a look and let me know whether this workflow would be useful and what it might still be missing for day-to-day development? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4232741260
##########
devkit/justfile:
##########
@@ -0,0 +1,514 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set dotenv-load := false
+
+# --- Paths --------------------------------------------------------------------
+
+DEVKIT_DIR := justfile_directory()
+FLUSS_DEV_HOME := DEVKIT_DIR + "/.."
+PROFILES_DIR := DEVKIT_DIR + "/profiles"
+DEPS_DIR := DEVKIT_DIR + "/.deps"
+CACHE_DIR := DEPS_DIR + "/cache"
+ACTIVE_PROFILE_FILE := DEPS_DIR + "/active-profile"
+
+# --- Docker Compose commands --------------------------------------------------
+
+COMPOSE := "docker compose -f " + DEVKIT_DIR + "/docker-compose.yml"
+ALL_COMPOSE := COMPOSE + " -f " + DEVKIT_DIR + "/compose/tiering.yml -f " + DEVKIT_DIR + "/compose/gateway.yml -f " + DEVKIT_DIR + "/compose/lake-common.yml -f " + DEVKIT_DIR + "/compose/iceberg.yml"
+ALL_PROFILES := "--profile single --profile cluster --profile flink --profile tiering --profile gateway"
+
+_default:
+ @just --list
+
+[doc("Prepare every artifact required by a profile")]
+build profile="core":
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ just validate-profile "$PROFILE"
+ TARGETS=$(just profile-runtime-targets "$PROFILE")
+ just deps "$PROFILE"
+ clean_dist() {
+ cd "{{ FLUSS_DEV_HOME }}"
+ ./mvnw -pl fluss-dist clean
+ }
+ build_core() {
+ clean_dist
+ cd "{{ FLUSS_DEV_HOME }}"
+ ./mvnw -DskipTests -pl fluss-dist,fluss-flink/fluss-flink-1.20 -am package
+ }
+ build_tiering() {
+ clean_dist
+ cd "{{ FLUSS_DEV_HOME }}"
+ ./mvnw -DskipTests -pl fluss-dist,fluss-flink/fluss-flink-1.20,fluss-flink/fluss-flink-tiering -am package
+ }
+ build_gateway() {
+ mkdir -p "{{ DEPS_DIR }}/cargo" "{{ DEPS_DIR }}/gateway"
+ docker run --rm \
+ --user "$(id -u):$(id -g)" \
+ -e CARGO_HOME=/cache \
+ -e CARGO_TARGET_DIR=/output \
+ -e HOME=/tmp \
+ -e RUSTUP_TOOLCHAIN=1.88.0 \
+ -v "{{ FLUSS_DEV_HOME }}:/workspace:ro" \
+ -v "{{ DEPS_DIR }}/cargo:/cache" \
+ -v "{{ DEPS_DIR }}/gateway:/output" \
+ -w /workspace/fluss-gateway \
+ rust:1.88-bookworm \
+ cargo build --locked --bin fluss-gateway
+ }
+ if grep -Fxq tiering <<<"$TARGETS"; then
+ build_tiering
+ elif grep -Eq '^(core|flink)$' <<<"$TARGETS"; then
+ build_core
+ fi
+ if grep -Fxq gateway <<<"$TARGETS"; then
+ build_gateway
+ fi
+
+[private]
+ensure-dist:
+ #!/usr/bin/env bash
+ set -e
+ DIST_HOME="{{ FLUSS_DEV_HOME }}/build-target"
+ if [ ! -x "$DIST_HOME/bin/coordinator-server.sh" ] || \
+ [ ! -x "$DIST_HOME/bin/tablet-server.sh" ] || \
+ [ ! -f "$DIST_HOME/conf/server.yaml" ] || \
+ [ ! -d "$DIST_HOME/lib" ]; then
+ echo "Complete local distribution not found. Run: just build <profile>"
+ exit 1
+ fi
+
+[private]
+ensure-gateway:
+ #!/usr/bin/env bash
+ if [ ! -x "{{ DEPS_DIR }}/gateway/debug/fluss-gateway" ]; then
+ echo "Local Gateway executable not found. Run: just build gateway"
+ exit 1
+ fi
+
+[private]
+validate-profile profile:
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ if [[ ! "$PROFILE" =~ ^[a-z0-9][a-z0-9-]*$ ]]; then
+ echo "profile must match [a-z0-9][a-z0-9-]*"
+ exit 1
+ fi
+ if [ ! -f "{{ PROFILES_DIR }}/$PROFILE/server.yaml" ]; then
+ echo "Unknown profile: $PROFILE"
+ exit 1
+ fi
+ if [ ! -f "{{ PROFILES_DIR }}/$PROFILE/runtime.targets" ]; then
+ echo "Profile $PROFILE does not declare runtime.targets"
+ exit 1
+ fi
+
+[private]
+profile-runtime-targets profile:
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ just validate-profile "$PROFILE"
+ TARGETS=$(sed 's/#.*//; /^[[:space:]]*$/d' "{{ PROFILES_DIR }}/$PROFILE/runtime.targets")
+ SEEN=""
+ for TARGET in $TARGETS; do
+ case "$TARGET" in
+ core|flink|gateway|tiering) ;;
+ *) echo "Unknown runtime target in profile $PROFILE: $TARGET" >&2; exit 1 ;;
+ esac
+ case " $SEEN " in
+ *" $TARGET "*)
+ echo "Duplicate runtime target in profile $PROFILE: $TARGET" >&2
+ exit 1
+ ;;
+ esac
+ SEEN="$SEEN $TARGET"
+ done
+ has_target() {
+ [[ " $SEEN " == *" $1 "* ]]
+ }
+ require_target() {
+ if has_target "$1" && ! has_target "$2"; then
+ echo "Profile $PROFILE requires runtime target: $2 for $1" >&2
+ exit 1
+ fi
+ }
+ require_target flink core
+ require_target tiering flink
+ require_target gateway core
+ if ! has_target core; then
+ echo "Profile $PROFILE must include runtime target: core" >&2
+ exit 1
+ fi
+ for TARGET in $SEEN; do
+ echo "$TARGET"
+ done
+
+[private]
+profile-format profile:
+ #!/usr/bin/env bash
+ PROFILE={{ quote(profile) }}
+ sed -n 's/^[[:space:]]*datalake\.format[[:space:]]*: \([^#]*\).*$/\1/p' \
+ "{{ PROFILES_DIR }}/$PROFILE/server.yaml" | sed 's/[[:space:]]*$//' | tail -n 1
+
+[private]
+compose profile topology *ARGS:
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ TOPOLOGY={{ quote(topology) }}
+ ARGS=( {{ ARGS }} )
+ just validate-profile "$PROFILE"
+ case "$TOPOLOGY" in
+ single) REPLICATION_FACTOR=1 ;;
+ cluster) REPLICATION_FACTOR=3 ;;
+ *) echo "Unknown topology: $TOPOLOGY"; exit 1 ;;
+ esac
+ PROFILE_DIR="{{ PROFILES_DIR }}/$PROFILE"
+ COMPOSE_ARGS=(-f "{{ DEVKIT_DIR }}/docker-compose.yml")
+ PROFILES=(--profile "$TOPOLOGY")
+ for TARGET in $(just profile-runtime-targets "$PROFILE"); do
+ case "$TARGET" in
+ core) ;;
+ flink)
+ COMPOSE_ARGS+=(-f "{{ DEVKIT_DIR }}/compose/tiering.yml")
+ PROFILES+=(--profile flink)
+ ;;
+ tiering) PROFILES+=(--profile tiering) ;;
+ gateway)
+ COMPOSE_ARGS+=(-f "{{ DEVKIT_DIR }}/compose/gateway.yml")
+ PROFILES+=(--profile gateway)
+ ;;
+ esac
+ done
+ if [ -f "$PROFILE_DIR/compose.files" ]; then
+ while IFS= read -r FILE; do
+ FILE="${FILE%%#*}"
+ FILE="${FILE//[[:space:]]/}"
+ [ -n "$FILE" ] || continue
+ if [ ! -f "{{ DEVKIT_DIR }}/$FILE" ]; then
+ echo "Compose file not found: $FILE"
+ exit 1
+ fi
+ COMPOSE_ARGS+=(-f "{{ DEVKIT_DIR }}/$FILE")
+ done < "$PROFILE_DIR/compose.files"
+ fi
+ export FLUSS_DEVKIT_CONFIG="$PROFILE_DIR/server.yaml"
+ export FLUSS_REPLICATION_FACTOR="$REPLICATION_FACTOR"
+ docker compose "${COMPOSE_ARGS[@]}" "${PROFILES[@]}" "${ARGS[@]}"
+
+[private]
+compose-active *ARGS:
+ #!/usr/bin/env bash
+ set -e
+ ARGS=( {{ ARGS }} )
+ if [ -f "{{ ACTIVE_PROFILE_FILE }}" ]; then
+ read -r PROFILE TOPOLOGY < "{{ ACTIVE_PROFILE_FILE }}"
+ if just validate-profile "$PROFILE" >/dev/null 2>&1; then
+ just compose "$PROFILE" "$TOPOLOGY" "${ARGS[@]}"
+ exit 0
+ fi
+ fi
+ {{ ALL_COMPOSE }} {{ ALL_PROFILES }} "${ARGS[@]}"
+
+[private]
+deps profile:
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ just validate-profile "$PROFILE"
+ PROFILE_DIR="{{ PROFILES_DIR }}/$PROFILE"
+ mkdir -p "{{ CACHE_DIR }}"
+ for FILE in "$PROFILE_DIR"/{jars,server,flink}.urls; do
+ [ -f "$FILE" ] || continue
+ while IFS= read -r URL; do
+ NAME="$(basename "${URL%%\?*}")"
+ TARGET="{{ CACHE_DIR }}/$NAME"
+ [ -s "$TARGET" ] && continue
+ echo "Downloading $(basename "$TARGET")..."
+ curl -fL --retry 3 -o "${TARGET}.tmp" "$URL"
+ mv "${TARGET}.tmp" "$TARGET"
+ done < <(sed 's/#.*//; s/[[:space:]]//g; /^$/d' "$FILE")
+ done
+
+[private]
+ensure-flink:
+ #!/usr/bin/env bash
+ set -e
+ ACTIVE_DIR="{{ DEPS_DIR }}/flink/active"
+ MATCHES=( {{ FLUSS_DEV_HOME }}/fluss-flink/fluss-flink-1.20/target/fluss-flink-1.20-*.jar )
+ if [ "${#MATCHES[@]}" -ne 1 ] || [ ! -f "${MATCHES[0]}" ]; then
+ echo "Expected one Flink connector artifact"
+ echo "Run: just build <profile>"
+ exit 1
+ fi
+ mkdir -p "$ACTIVE_DIR/lib" "$ACTIVE_DIR/jobs"
+ rm -f "$ACTIVE_DIR/lib/"*.jar "$ACTIVE_DIR/jobs/"*.jar
+ cp "${MATCHES[0]}" "$ACTIVE_DIR/lib/fluss-flink-1.20.jar"
+ chmod -R a+rX "$ACTIVE_DIR"
+
+[private]
+stage-profile-jars profile:
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ PROFILE_DIR="{{ PROFILES_DIR }}/$PROFILE"
+ FORMAT=$(just profile-format "$PROFILE")
+ SERVER_PLUGIN_DIR="{{ FLUSS_DEV_HOME }}/build-target/plugins/${FORMAT:-$PROFILE}"
+ rm -f "$SERVER_PLUGIN_DIR"/devkit-*.jar
+ copy_urls() {
+ local FILE="$1"
+ local TARGET="$2"
+ local PREFIX="${3:-}"
+ local URL NAME SOURCE
+ [ -f "$FILE" ] || return 0
+ while IFS= read -r URL; do
+ NAME="$(basename "${URL%%\?*}")"
+ SOURCE="{{ CACHE_DIR }}/$NAME"
+ if [ ! -s "$SOURCE" ]; then
+ echo "Profile dependency not found: $NAME"
+ echo "Run: just build $PROFILE"
+ exit 1
+ fi
+ mkdir -p "$TARGET"
+ cp "$SOURCE" "$TARGET/$PREFIX$NAME"
+ done < <(sed 's/#.*//; s/[[:space:]]//g; /^$/d' "$FILE")
+ }
+ ACTIVE_DIR="{{ DEPS_DIR }}/flink/active"
+ mkdir -p "$ACTIVE_DIR/lib" "$ACTIVE_DIR/jobs"
+ copy_urls "$PROFILE_DIR/jars.urls" "$SERVER_PLUGIN_DIR" "devkit-"
+ copy_urls "$PROFILE_DIR/jars.urls" "$ACTIVE_DIR/lib"
+ copy_urls "$PROFILE_DIR/server.urls" "$SERVER_PLUGIN_DIR" "devkit-"
+ copy_urls "$PROFILE_DIR/flink.urls" "$ACTIVE_DIR/lib"
+ chmod -R a+rX "$ACTIVE_DIR"
+
+[private]
+ensure-tiering profile:
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ just validate-profile "$PROFILE"
+ PROFILE_DIR="{{ PROFILES_DIR }}/$PROFILE"
+ FORMAT=$(just profile-format "$PROFILE")
+ if [ -z "$FORMAT" ]; then
+ echo "Profile $PROFILE does not configure datalake.format"
+ exit 1
+ fi
+ stage() {
+ local PATTERN="$1"
+ local TARGET="$2"
+ MATCHES=( $PATTERN )
+ if [ "${#MATCHES[@]}" -ne 1 ] || [ ! -f "${MATCHES[0]}" ]; then
+ echo "Expected one build artifact: $PATTERN"
+ echo "Run: just build $PROFILE"
+ exit 1
+ fi
+ cp "${MATCHES[0]}" "$TARGET"
+ }
+ ACTIVE_DIR="{{ DEPS_DIR }}/flink/active"
+ stage "{{ FLUSS_DEV_HOME }}/fluss-lake/fluss-lake-$FORMAT/target/fluss-lake-$FORMAT-*.jar" "$ACTIVE_DIR/lib/fluss-lake-$FORMAT.jar"
+ stage "{{ FLUSS_DEV_HOME }}/fluss-flink/fluss-flink-tiering/target/fluss-flink-tiering-*.jar" "$ACTIVE_DIR/jobs/fluss-flink-tiering.jar"
+
+ SERVER_PLUGIN_DIR="{{ FLUSS_DEV_HOME }}/build-target/plugins/$FORMAT"
+ mkdir -p "$SERVER_PLUGIN_DIR"
+ # Filesystem plugins must stay isolated from lake plugin classpaths.
+ rm -f "$SERVER_PLUGIN_DIR"/fluss-fs-*.jar
+ REMOTE_DATA_DIR=$(sed -n 's/^[[:space:]]*remote\.data\.dir[[:space:]]*: \([^#]*\).*$/\1/p' "$PROFILE_DIR/server.yaml" | sed 's/[[:space:]]*$//' | tail -n 1)
+ SCHEME="${REMOTE_DATA_DIR%%:*}"
+ if [ "$SCHEME" != "$REMOTE_DATA_DIR" ] && [ -d "{{ FLUSS_DEV_HOME }}/build-target/plugins/$SCHEME" ]; then
+ cp "{{ FLUSS_DEV_HOME }}/build-target/plugins/$SCHEME/"*.jar "$ACTIVE_DIR/lib/"
+ fi
+
+ chmod -R a+rX "$ACTIVE_DIR"
+
+[doc("Start a built profile from the profiles directory")]
+up profile="core" nodes="1":
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ just validate-profile "$PROFILE"
+ case "{{ nodes }}" in
+ 1) TOPOLOGY=single ;;
+ 3) TOPOLOGY=cluster ;;
+ *) echo "nodes must be 1 or 3"; exit 1 ;;
+ esac
+ ENABLE_TIERING=false
+ for TARGET in $(just profile-runtime-targets "$PROFILE"); do
+ case "$TARGET" in
+ core) just ensure-dist ;;
+ flink)
+ just ensure-flink
+ ;;
+ tiering) ENABLE_TIERING=true ;;
+ gateway)
+ just ensure-gateway
+ ;;
+ esac
+ done
+ just stage-profile-jars "$PROFILE"
+ if [ "$ENABLE_TIERING" = "true" ]; then
+ just ensure-tiering "$PROFILE"
+ fi
+ just compose-active down --remove-orphans >/dev/null 2>&1 || true
Review Comment:
We had missed the case of switching profiles without cleaning up the previous data. Different profiles must not reuse the same data, so `just up` now asks the user to run `just clean` before switching, while restarting the same profile still preserves data for debugging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub) via issues" <is...@fluss.apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4482518783
##########
devkit/profiles/core/server.yaml:
##########
@@ -0,0 +1,19 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+remote.data.dir: file:///tmp/fluss/remote-data
Review Comment:
We originally kept the Core and Gateway profiles on a shared file:// directory to minimize their startup dependencies. The lake profiles require object storage for tiering, so they already start RustFS. On reflection, using RustFS by default for Core and Gateway also makes sense: it would give the profiles a consistent remote-storage configuration and cover the S3 configuration and plugin setup. I’ll update those two profiles accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r3846361496
##########
devkit/profiles/hudi/server.yaml:
##########
@@ -0,0 +1,41 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+remote.data.dir: s3://fluss/remote-data
+s3.endpoint: http://rustfs:9000
+s3.access-key: rustfsadmin
+s3.secret-key: rustfsadmin
+s3.region: us-east-1
+s3.path-style-access: true
+s3.assumed.role.arn: arn:aws:iam::000000000000:role/rustfsadmin
+s3.assumed.role.sts.endpoint: http://rustfs:9000
+
+datalake.enabled: true
+datalake.format: hudi
+datalake.hudi.mode: dfs
+datalake.hudi.catalog.path: s3a://fluss/hudi
+datalake.hudi.hadoop.fs.s3a.endpoint: http://rustfs:9000
+datalake.hudi.hadoop.fs.s3a.access.key: rustfsadmin
+datalake.hudi.hadoop.fs.s3a.secret.key: rustfsadmin
+datalake.hudi.hadoop.fs.s3a.path.style.access: true
+datalake.hudi.hadoop.fs.s3a.connection.ssl.enabled: false
+datalake.hudi.hadoop.fs.s3a.impl.disable.cache: true
+datalake.hudi.hadoop.fs.s3a.aws.credentials.provider: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
+datalake.hudi.hoodie.write.lock.provider: org.apache.hudi.client.transaction.lock.InProcessLockProvider
+
+fluss.tiering.poll.table.interval: 1s
Review Comment:
Good catch — this was an oversight, not intentional. The Iceberg, Paimon, and Lance profiles now all use fluss.tiering.poll.table.interval: 1s. The Hudi profile has been removed for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r3846369662
##########
devkit/docker-compose.yml:
##########
@@ -0,0 +1,181 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: fluss-devkit
+
+x-fluss-common: &fluss-common
+ image: ${FLUSS_DEVKIT_IMAGE:-eclipse-temurin:17-jre-noble}
+ init: true
+ restart: unless-stopped
+ stop_grace_period: 2m
+ volumes:
+ - ../build-target:/opt/fluss:ro
+ - ${FLUSS_DEVKIT_CONFIG:-./profiles/core/server.yaml}:/opt/fluss/conf/server.yaml:ro
+ - shared-data:/tmp/fluss
+ environment:
+ FLUSS_LOG_DIR: /tmp/fluss-logs
Review Comment:
Good catch. In Compose, a service-level environment mapping replaces the inherited mapping from the anchor. I removed the ineffective shared environment block and now define the required variables explicitly for the CoordinatorServer and TabletServers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on PR #3851:
URL: https://github.com/apache/fluss/pull/3851#issuecomment-5399640454
Sorry for the delayed update—I only got a chance to update the PR today. Thanks for the detailed feedback and support.
I’ve added the DevKit Smoke workflow and updated the CI path filters so DevKit-only changes do not trigger the full Java matrix.
I prefer keeping it under devkit/ because it is intended for day-to-day contributor development and validation, as well as documentation reuse. It also covers process-level checks such as Docker init, JDWP, and Prometheus endpoints. I’ve updated AGENTS.md accordingly.
HudiWriteTableInfo will be handled in a follow-up PR. The Hudi DevKit profile and examples have been removed for now.
cc: @fresh-borzoni
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on PR #3851:
URL: https://github.com/apache/fluss/pull/3851#issuecomment-5338507872
Hi @wuchong @luoyuxia @swuferhong, this PR adds a local DevKit for Fluss contributors. It is a development tooling contribution rather than a user-facing feature, so I’d appreciate your thoughts on whether it fits the project and any feedback when you have time. If it is not a good fit for the repository or the current scope, please feel free to let me know. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4233132320
##########
devkit/compose/tiering.yml:
##########
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+x-flink-common: &flink-common
+ image: ${FLUSS_DEVKIT_FLINK_IMAGE:-flink:1.20.3-scala_2.12-java17}
+ entrypoint:
+ - /bin/bash
+ - -c
+ - |
+ cp /opt/devkit-jars/lib/*.jar /opt/flink/lib/
+ chmod a+r /opt/flink/lib/*.jar
+ exec /docker-entrypoint.sh "$$@"
+ - --
+ volumes:
+ - ./.deps/flink/active:/opt/devkit-jars:ro
+ - ${FLUSS_DEVKIT_CONFIG:-./profiles/core/server.yaml}:/opt/devkit-config/server.yaml:ro
+ - shared-data:/tmp/fluss
+
+services:
+ jobmanager:
+ <<: *flink-common
+ profiles: [flink]
+ command: ["jobmanager"]
+ depends_on:
+ coordinator-server:
+ condition: service_healthy
+ environment:
+ FLINK_PROPERTIES: |
+ jobmanager.rpc.address: jobmanager
+ jobmanager.memory.process.size: 1024m
+ parallelism.default: 1
+ rest.address: jobmanager
+ ports:
+ - "${FLINK_REST_PORT:-8083}:8081"
+ healthcheck:
+ test: ["CMD-SHELL", "curl -fsS http://localhost:8081/overview >/dev/null"]
+ interval: 5s
+ timeout: 5s
+ retries: 24
+ start_period: 10s
+
+ taskmanager:
+ <<: *flink-common
+ profiles: [flink]
+ command: ["taskmanager"]
+ depends_on:
+ jobmanager:
+ condition: service_healthy
+ environment:
+ FLINK_PROPERTIES: |
+ jobmanager.rpc.address: jobmanager
+ taskmanager.memory.process.size: 1536m
+ taskmanager.numberOfTaskSlots: 4
+ healthcheck:
+ test:
+ - CMD-SHELL
+ - curl -fsS http://jobmanager:8081/taskmanagers | grep -q '"id"'
+ interval: 5s
+ timeout: 5s
+ retries: 24
+ start_period: 10s
+
+ tiering-submit:
+ <<: *flink-common
+ profiles: [tiering]
+ restart: "no"
Review Comment:
Good point. All published ports, including JDWP, Gateway, RustFS, Flink, metrics, and Fluss server ports, are now bound to 127.0.0.1 by default. The host port numbers remain configurable through the existing environment variables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "wuchong (via GitHub) via issues" <is...@fluss.apache.org>.
wuchong merged PR #3851:
URL: https://github.com/apache/fluss/pull/3851
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4232745615
##########
AGENTS.md:
##########
@@ -196,6 +196,8 @@ Apache Fluss follows a layered Maven module architecture: `fluss-common` (founda
**`fluss-dist`** - Binary distribution with `bin/` scripts and `conf/` templates
+**`devkit/`** - Local development toolkit for running and testing Fluss across core and lakehouse scenarios
Review Comment:
Addressed. DevKit has been moved out of the Maven module list into Development Tooling, with a short note on using it for local end-to-end validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4233132320
##########
devkit/compose/tiering.yml:
##########
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+x-flink-common: &flink-common
+ image: ${FLUSS_DEVKIT_FLINK_IMAGE:-flink:1.20.3-scala_2.12-java17}
+ entrypoint:
+ - /bin/bash
+ - -c
+ - |
+ cp /opt/devkit-jars/lib/*.jar /opt/flink/lib/
+ chmod a+r /opt/flink/lib/*.jar
+ exec /docker-entrypoint.sh "$$@"
+ - --
+ volumes:
+ - ./.deps/flink/active:/opt/devkit-jars:ro
+ - ${FLUSS_DEVKIT_CONFIG:-./profiles/core/server.yaml}:/opt/devkit-config/server.yaml:ro
+ - shared-data:/tmp/fluss
+
+services:
+ jobmanager:
+ <<: *flink-common
+ profiles: [flink]
+ command: ["jobmanager"]
+ depends_on:
+ coordinator-server:
+ condition: service_healthy
+ environment:
+ FLINK_PROPERTIES: |
+ jobmanager.rpc.address: jobmanager
+ jobmanager.memory.process.size: 1024m
+ parallelism.default: 1
+ rest.address: jobmanager
+ ports:
+ - "${FLINK_REST_PORT:-8083}:8081"
+ healthcheck:
+ test: ["CMD-SHELL", "curl -fsS http://localhost:8081/overview >/dev/null"]
+ interval: 5s
+ timeout: 5s
+ retries: 24
+ start_period: 10s
+
+ taskmanager:
+ <<: *flink-common
+ profiles: [flink]
+ command: ["taskmanager"]
+ depends_on:
+ jobmanager:
+ condition: service_healthy
+ environment:
+ FLINK_PROPERTIES: |
+ jobmanager.rpc.address: jobmanager
+ taskmanager.memory.process.size: 1536m
+ taskmanager.numberOfTaskSlots: 4
+ healthcheck:
+ test:
+ - CMD-SHELL
+ - curl -fsS http://jobmanager:8081/taskmanagers | grep -q '"id"'
+ interval: 5s
+ timeout: 5s
+ retries: 24
+ start_period: 10s
+
+ tiering-submit:
+ <<: *flink-common
+ profiles: [tiering]
+ restart: "no"
Review Comment:
Good point. All published ports, including JDWP, Gateway, RustFS, Flink, metrics, and Fluss server ports, are now bound to 127.0.0.1 by default. The host port numbers remain configurable through the existing environment variables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on PR #3851:
URL: https://github.com/apache/fluss/pull/3851#issuecomment-5250493278
I recommend trying the DevKit introduced by this PR locally. Compared with starting tests from IntelliJ or building Docker images and assembling a custom Docker Compose setup, it provides a more convenient workflow and makes it easier for both developers and coding agents to launch Fluss for end-to-end testing and troubleshooting.
I have frequently used a similar approach in the [AutoMQ DevKit](https://github.com/AutoMQ/automq/tree/main/devkit), and found it helpful for local development. I believe it could provide a similar improvement to the Fluss development experience.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r3846372293
##########
devkit/profiles/hudi/server.urls:
##########
@@ -0,0 +1,21 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+https://repo.maven.apache.org/maven2/io/trino/hadoop/hadoop-apache/3.3.5-2/hadoop-apache-3.3.5-2.jar
+https://repo.maven.apache.org/maven2/org/apache/flink/flink-core/1.20.1/flink-core-1.20.1.jar
Review Comment:
Thanks for looking into this. The Hudi profile and examples have been removed for now, and HudiWriteTableInfo will be addressed in a follow-up PR. This keeps the current PR focused on the DevKit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r3846363615
##########
devkit/tests/paimon-create-table.sql:
##########
@@ -0,0 +1,33 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+CREATE CATALOG fluss_catalog WITH (
+ 'type' = 'fluss',
+ 'bootstrap.servers' = 'coordinator-server:19123',
+ 'paimon.s3.access-key' = 'rustfsadmin',
+ 'paimon.s3.secret-key' = 'rustfsadmin'
+);
+
+USE CATALOG fluss_catalog;
+
+CREATE TABLE ${DEVKIT_TABLE} (
Review Comment:
Agreed. The lake examples now use Primary Key Tables. The Paimon end-to-end example includes both a Log Table and a Primary Key Table, and covers Union Read and Lake-only Read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4232688741
##########
devkit/justfile:
##########
@@ -0,0 +1,514 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set dotenv-load := false
+
+# --- Paths --------------------------------------------------------------------
+
+DEVKIT_DIR := justfile_directory()
+FLUSS_DEV_HOME := DEVKIT_DIR + "/.."
+PROFILES_DIR := DEVKIT_DIR + "/profiles"
+DEPS_DIR := DEVKIT_DIR + "/.deps"
+CACHE_DIR := DEPS_DIR + "/cache"
+ACTIVE_PROFILE_FILE := DEPS_DIR + "/active-profile"
+
+# --- Docker Compose commands --------------------------------------------------
+
+COMPOSE := "docker compose -f " + DEVKIT_DIR + "/docker-compose.yml"
+ALL_COMPOSE := COMPOSE + " -f " + DEVKIT_DIR + "/compose/tiering.yml -f " + DEVKIT_DIR + "/compose/gateway.yml -f " + DEVKIT_DIR + "/compose/lake-common.yml -f " + DEVKIT_DIR + "/compose/iceberg.yml"
+ALL_PROFILES := "--profile single --profile cluster --profile flink --profile tiering --profile gateway"
+
+_default:
+ @just --list
+
+[doc("Prepare every artifact required by a profile")]
+build profile="core":
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ just validate-profile "$PROFILE"
+ TARGETS=$(just profile-runtime-targets "$PROFILE")
+ just deps "$PROFILE"
+ clean_dist() {
+ cd "{{ FLUSS_DEV_HOME }}"
+ ./mvnw -pl fluss-dist clean
+ }
+ build_core() {
+ clean_dist
+ cd "{{ FLUSS_DEV_HOME }}"
+ ./mvnw -DskipTests -pl fluss-dist,fluss-flink/fluss-flink-1.20 -am package
+ }
+ build_tiering() {
+ clean_dist
+ cd "{{ FLUSS_DEV_HOME }}"
+ ./mvnw -DskipTests -pl fluss-dist,fluss-flink/fluss-flink-1.20,fluss-flink/fluss-flink-tiering -am package
+ }
+ build_gateway() {
+ mkdir -p "{{ DEPS_DIR }}/cargo" "{{ DEPS_DIR }}/gateway"
+ docker run --rm \
+ --user "$(id -u):$(id -g)" \
+ -e CARGO_HOME=/cache \
+ -e CARGO_TARGET_DIR=/output \
+ -e HOME=/tmp \
+ -e RUSTUP_TOOLCHAIN=1.88.0 \
+ -v "{{ FLUSS_DEV_HOME }}:/workspace:ro" \
+ -v "{{ DEPS_DIR }}/cargo:/cache" \
+ -v "{{ DEPS_DIR }}/gateway:/output" \
+ -w /workspace/fluss-gateway \
+ rust:1.88-bookworm \
+ cargo build --locked --bin fluss-gateway
+ }
+ if grep -Fxq tiering <<<"$TARGETS"; then
+ build_tiering
+ elif grep -Eq '^(core|flink)$' <<<"$TARGETS"; then
+ build_core
+ fi
+ if grep -Fxq gateway <<<"$TARGETS"; then
+ build_gateway
+ fi
+
+[private]
+ensure-dist:
+ #!/usr/bin/env bash
+ set -e
+ DIST_HOME="{{ FLUSS_DEV_HOME }}/build-target"
+ if [ ! -x "$DIST_HOME/bin/coordinator-server.sh" ] || \
+ [ ! -x "$DIST_HOME/bin/tablet-server.sh" ] || \
+ [ ! -f "$DIST_HOME/conf/server.yaml" ] || \
+ [ ! -d "$DIST_HOME/lib" ]; then
+ echo "Complete local distribution not found. Run: just build <profile>"
+ exit 1
+ fi
+
+[private]
+ensure-gateway:
+ #!/usr/bin/env bash
+ if [ ! -x "{{ DEPS_DIR }}/gateway/debug/fluss-gateway" ]; then
+ echo "Local Gateway executable not found. Run: just build gateway"
+ exit 1
+ fi
+
+[private]
+validate-profile profile:
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ if [[ ! "$PROFILE" =~ ^[a-z0-9][a-z0-9-]*$ ]]; then
+ echo "profile must match [a-z0-9][a-z0-9-]*"
+ exit 1
+ fi
+ if [ ! -f "{{ PROFILES_DIR }}/$PROFILE/server.yaml" ]; then
+ echo "Unknown profile: $PROFILE"
+ exit 1
+ fi
+ if [ ! -f "{{ PROFILES_DIR }}/$PROFILE/runtime.targets" ]; then
+ echo "Profile $PROFILE does not declare runtime.targets"
+ exit 1
+ fi
+
+[private]
+profile-runtime-targets profile:
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ just validate-profile "$PROFILE"
+ TARGETS=$(sed 's/#.*//; /^[[:space:]]*$/d' "{{ PROFILES_DIR }}/$PROFILE/runtime.targets")
+ SEEN=""
+ for TARGET in $TARGETS; do
+ case "$TARGET" in
+ core|flink|gateway|tiering) ;;
+ *) echo "Unknown runtime target in profile $PROFILE: $TARGET" >&2; exit 1 ;;
+ esac
+ case " $SEEN " in
+ *" $TARGET "*)
+ echo "Duplicate runtime target in profile $PROFILE: $TARGET" >&2
+ exit 1
+ ;;
+ esac
+ SEEN="$SEEN $TARGET"
+ done
+ has_target() {
+ [[ " $SEEN " == *" $1 "* ]]
+ }
+ require_target() {
+ if has_target "$1" && ! has_target "$2"; then
+ echo "Profile $PROFILE requires runtime target: $2 for $1" >&2
+ exit 1
+ fi
+ }
+ require_target flink core
+ require_target tiering flink
+ require_target gateway core
+ if ! has_target core; then
+ echo "Profile $PROFILE must include runtime target: core" >&2
+ exit 1
+ fi
+ for TARGET in $SEEN; do
+ echo "$TARGET"
+ done
+
+[private]
+profile-format profile:
+ #!/usr/bin/env bash
+ PROFILE={{ quote(profile) }}
+ sed -n 's/^[[:space:]]*datalake\.format[[:space:]]*: \([^#]*\).*$/\1/p' \
+ "{{ PROFILES_DIR }}/$PROFILE/server.yaml" | sed 's/[[:space:]]*$//' | tail -n 1
+
+[private]
+compose profile topology *ARGS:
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ TOPOLOGY={{ quote(topology) }}
+ ARGS=( {{ ARGS }} )
+ just validate-profile "$PROFILE"
+ case "$TOPOLOGY" in
+ single) REPLICATION_FACTOR=1 ;;
+ cluster) REPLICATION_FACTOR=3 ;;
+ *) echo "Unknown topology: $TOPOLOGY"; exit 1 ;;
+ esac
+ PROFILE_DIR="{{ PROFILES_DIR }}/$PROFILE"
+ COMPOSE_ARGS=(-f "{{ DEVKIT_DIR }}/docker-compose.yml")
+ PROFILES=(--profile "$TOPOLOGY")
+ for TARGET in $(just profile-runtime-targets "$PROFILE"); do
+ case "$TARGET" in
+ core) ;;
+ flink)
+ COMPOSE_ARGS+=(-f "{{ DEVKIT_DIR }}/compose/tiering.yml")
+ PROFILES+=(--profile flink)
+ ;;
+ tiering) PROFILES+=(--profile tiering) ;;
+ gateway)
+ COMPOSE_ARGS+=(-f "{{ DEVKIT_DIR }}/compose/gateway.yml")
+ PROFILES+=(--profile gateway)
+ ;;
+ esac
+ done
+ if [ -f "$PROFILE_DIR/compose.files" ]; then
+ while IFS= read -r FILE; do
+ FILE="${FILE%%#*}"
+ FILE="${FILE//[[:space:]]/}"
+ [ -n "$FILE" ] || continue
+ if [ ! -f "{{ DEVKIT_DIR }}/$FILE" ]; then
+ echo "Compose file not found: $FILE"
+ exit 1
+ fi
+ COMPOSE_ARGS+=(-f "{{ DEVKIT_DIR }}/$FILE")
+ done < "$PROFILE_DIR/compose.files"
+ fi
+ export FLUSS_DEVKIT_CONFIG="$PROFILE_DIR/server.yaml"
+ export FLUSS_REPLICATION_FACTOR="$REPLICATION_FACTOR"
+ docker compose "${COMPOSE_ARGS[@]}" "${PROFILES[@]}" "${ARGS[@]}"
+
+[private]
+compose-active *ARGS:
+ #!/usr/bin/env bash
+ set -e
+ ARGS=( {{ ARGS }} )
+ if [ -f "{{ ACTIVE_PROFILE_FILE }}" ]; then
+ read -r PROFILE TOPOLOGY < "{{ ACTIVE_PROFILE_FILE }}"
+ if just validate-profile "$PROFILE" >/dev/null 2>&1; then
+ just compose "$PROFILE" "$TOPOLOGY" "${ARGS[@]}"
+ exit 0
+ fi
+ fi
+ {{ ALL_COMPOSE }} {{ ALL_PROFILES }} "${ARGS[@]}"
+
+[private]
+deps profile:
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ just validate-profile "$PROFILE"
+ PROFILE_DIR="{{ PROFILES_DIR }}/$PROFILE"
+ mkdir -p "{{ CACHE_DIR }}"
+ for FILE in "$PROFILE_DIR"/{jars,server,flink}.urls; do
+ [ -f "$FILE" ] || continue
+ while IFS= read -r URL; do
+ NAME="$(basename "${URL%%\?*}")"
+ TARGET="{{ CACHE_DIR }}/$NAME"
+ [ -s "$TARGET" ] && continue
Review Comment:
Added checksum verification. For example, Paimon Flink 2.0.0 does not publish a `.sha512` file, but it does provide `.sha1`, so `.sha1` is used consistently to verify cached and downloaded artifacts.
> - [paimon-flink-1.20-2.0.0.jar.sha512](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.20/2.0.0/paimon-flink-1.20-2.0.0.jar.sha512)：HTTP 404
> - [paimon-flink-1.20-2.0.0.jar.sha1](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.20/2.0.0/paimon-flink-1.20-2.0.0.jar.sha1)：HTTP 200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub) via issues" <is...@fluss.apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4483174583
##########
.github/workflows/ci.yaml:
##########
@@ -49,6 +51,8 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v6
+ - name: Validate DevKit managed dependency versions
+ run: bash devkit/validate-managed-dependency-versions.sh
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub) via issues" <is...@fluss.apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4482873252
##########
.github/workflows/devkit-smoke.yaml:
##########
@@ -0,0 +1,175 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+name: DevKit Smoke
+
+permissions:
+ contents: read
+
+on:
+ pull_request:
+ paths:
+ - 'devkit/**'
+ - 'pom.xml'
+ - '.github/workflows/devkit-smoke.yaml'
+ push:
+ branches:
+ - main
+ - release-**
+ paths:
+ - 'devkit/**'
+ - 'pom.xml'
+ - '.github/workflows/devkit-smoke.yaml'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }}
+ cancel-in-progress: true
+
+jobs:
+ smoke:
+ runs-on: ubuntu-latest
+ timeout-minutes: 45
+ steps:
+ - uses: actions/checkout@v6
+
+ - name: Set up JDK 11
+ uses: actions/setup-java@v5
+ with:
+ java-version: '11'
+ distribution: temurin
+
+ - name: Install just
+ uses: taiki-e/install-action@v2
+ with:
+ tool: just@1.58.0
+
+ - name: Validate managed profile dependency versions
+ run: bash devkit/validate-managed-dependency-versions.sh
+
+ - name: Core smoke
+ working-directory: devkit
+ run: |
+ set -euo pipefail
+ just build core
+ just up core
+ just run-sql examples/core/setup.sql
+ core_output=$(just run-sql examples/core/batch-scan.sql)
+ printf '%s\n' "$core_output"
+ grep -Fq 'Alice' <<<"$core_output"
+ grep -Fq 'Bob' <<<"$core_output"
+ grep -Fq 'alpha' <<<"$core_output"
+ grep -Fq 'beta' <<<"$core_output"
+ just clean
+
+ - name: Gateway smoke
+ working-directory: devkit
+ run: |
+ set -euo pipefail
+ just build gateway
+ just up gateway
+ curl -fsS -X POST \
+ -H 'Content-Type: application/json' \
+ http://localhost:8080/v1/clusters/default/databases/fluss/tables \
+ -d '{
+ "table_name": "gateway_events",
+ "columns": [
+ {"name": "event_id", "data_type": {"type": "BIGINT"}, "nullable": false},
+ {"name": "message", "data_type": {"type": "STRING"}, "nullable": false}
+ ],
+ "distribution": {"bucket_count": 1, "bucket_keys": []}
+ }'
+ curl -fsS \
+ http://localhost:8080/v1/clusters/default/databases/fluss/tables/gateway_events
+ curl -fsS -X POST \
+ -H 'Content-Type: application/json' \
+ http://localhost:8080/v1/clusters/default/databases/fluss/tables/gateway_events/records \
+ -d '{
+ "entries": [
+ {"id": "event-1", "append": {"event_id": "1", "message": "hello from Gateway"}}
+ ]
+ }'
+ just clean
+
+ - name: Paimon smoke
+ working-directory: devkit
+ run: |
+ set -euo pipefail
+ just build paimon
+ just up paimon
+ just run-sql examples/lake/paimon/setup.sql
+ just run-sql examples/lake/paimon/union-read.sql
+ for attempt in {1..30}; do
+ lake_output=$(just run-sql examples/lake/paimon/lake-only-read.sql || true)
+ printf '%s\n' "$lake_output"
+ if grep -Fq 'Alice' <<<"$lake_output" && grep -Fq 'Bob' <<<"$lake_output"; then
+ break
+ fi
+ if [ "$attempt" -eq 30 ]; then
+ echo 'Paimon lake-only query did not return Alice and Bob' >&2
+ just logs tiering-submit 200
+ exit 1
+ fi
+ sleep 2
+ done
+ just clean
+
+ - name: Iceberg smoke
+ working-directory: devkit
+ run: |
+ set -euo pipefail
+ just build iceberg
+ just up iceberg
+ just run-sql examples/lake/iceberg/create-table.sql
+ just run-sql examples/lake/iceberg/write-data.sql
+ iceberg_output=''
+ for attempt in {1..30}; do
+ iceberg_output=$(just run-sql examples/lake/iceberg/query-lake.sql || true)
+ printf '%s\n' "$iceberg_output"
+ if grep -Eq '3.*6.*3' <<<"$iceberg_output"; then
Review Comment:
Get, I originally added these checks to catch DevKit profiles that stop working as the project changes. I’ve moved the scenarios into a small unittest script with shared setup and cleanup, so we can add focused checks when needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4232552916
##########
.github/workflows/devkit-smoke.yaml:
##########
@@ -0,0 +1,82 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+name: DevKit Smoke
+
+permissions:
+ contents: read
+
+on:
+ pull_request:
+ paths:
+ - 'devkit/**'
+ push:
+ branches:
+ - main
+ - release-**
+ paths:
+ - 'devkit/**'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }}
+ cancel-in-progress: true
+
+jobs:
+ smoke:
+ runs-on: ubuntu-latest
+ timeout-minutes: 45
+ steps:
+ - uses: actions/checkout@v6
+
+ - name: Set up JDK 11
+ uses: actions/setup-java@v5
+ with:
+ java-version: '11'
+ distribution: temurin
+
+ - name: Install just
+ uses: taiki-e/install-action@v2
+ with:
+ tool: just@1.40.0
+
+ - name: Validate managed profile dependency versions
+ working-directory: devkit
+ run: |
+ set -euo pipefail
+ paimon_version=$(sed -n 's:.*<paimon.version>\([^<]*\)</paimon.version>.*:\1:p' ../pom.xml)
+ iceberg_version=$(sed -n 's:.*<iceberg.version>\([^<]*\)</iceberg.version>.*:\1:p' ../pom.xml)
+ grep -Fq "/paimon-s3/${paimon_version}/paimon-s3-${paimon_version}.jar" profiles/paimon/jars.urls
+ grep -Fq "/paimon-flink-1.20/${paimon_version}/paimon-flink-1.20-${paimon_version}.jar" profiles/paimon/flink.urls
+ grep -Fq "/iceberg-aws/${iceberg_version}/iceberg-aws-${iceberg_version}.jar" profiles/iceberg/jars.urls
+ grep -Fq "/iceberg-aws-bundle/${iceberg_version}/iceberg-aws-bundle-${iceberg_version}.jar" profiles/iceberg/jars.urls
+ grep -Fq "/iceberg-flink-runtime-1.20/${iceberg_version}/iceberg-flink-runtime-1.20-${iceberg_version}.jar" profiles/iceberg/flink.urls
+
+ - name: Run Gateway and Paimon smoke tests
+ working-directory: devkit
+ run: |
+ set -euo pipefail
+ trap 'just clean >/dev/null 2>&1 || true' EXIT
+ just build gateway
+ just up gateway
+ curl --fail --silent --show-error http://localhost:8080/v1/clusters/default/databases
+ just down
+ just build paimon
+ just up paimon
+ just run-sql examples/lake/paimon/setup.sql
+ just run-sql examples/lake/paimon/union-read.sql
+ just run-sql examples/lake/paimon/lake-only-read.sql
Review Comment:
An empty Lake result could previously pass. The smoke now waits for Alice and Bob and fails with Tiering logs if they do not appear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4232565355
##########
.github/workflows/devkit-smoke.yaml:
##########
@@ -0,0 +1,82 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+name: DevKit Smoke
+
+permissions:
+ contents: read
+
+on:
+ pull_request:
+ paths:
+ - 'devkit/**'
Review Comment:
Added pom.xml to the workflow triggers and added Iceberg and Lance smoke tests covering Lake queries and Lance client read-back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4232745615
##########
AGENTS.md:
##########
@@ -196,6 +196,8 @@ Apache Fluss follows a layered Maven module architecture: `fluss-common` (founda
**`fluss-dist`** - Binary distribution with `bin/` scripts and `conf/` templates
+**`devkit/`** - Local development toolkit for running and testing Fluss across core and lakehouse scenarios
Review Comment:
Addressed. DevKit has been moved out of the Maven module list into Development Tooling, with a short note on using it for local validation workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on PR #3851:
URL: https://github.com/apache/fluss/pull/3851#issuecomment-5250450667
This CI failure appears to be caused by a timing issue in the existing test rather than a Coordinator failure. The Coordinator restarted successfully, but the test retried against the stale endpoint before all TabletServers had refreshed their metadata.
The test already calls `waitUntilAllGatewayHasSameMetadata()` after the third restart, but not after the first two. I currently do not have permission to rerun the CI, so this cannot be confirmed by a rerun. It may be worth adding the same metadata synchronization after each restart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r3824381846
##########
devkit/compose/tiering.yml:
##########
@@ -0,0 +1,114 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+x-flink-common: &flink-common
+ image: flink:1.20.0-scala_2.12-java17
Review Comment:
We compile `fluss-flink-1.20` against `1.20.3`, and `hudi/server.urls` pulls `flink-core` `1.20.1`.
Three patch versions with nothing tying them together.
Can we pin to the pom property?
##########
devkit/tests/paimon-create-table.sql:
##########
@@ -0,0 +1,33 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+CREATE CATALOG fluss_catalog WITH (
+ 'type' = 'fluss',
+ 'bootstrap.servers' = 'coordinator-server:19123',
+ 'paimon.s3.access-key' = 'rustfsadmin',
+ 'paimon.s3.secret-key' = 'rustfsadmin'
+);
+
+USE CATALOG fluss_catalog;
+
+CREATE TABLE ${DEVKIT_TABLE} (
Review Comment:
All four fixtures are append-only.
Shall we add a PK table?
##########
devkit/profiles/hudi/server.yaml:
##########
@@ -0,0 +1,41 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+remote.data.dir: s3://fluss/remote-data
+s3.endpoint: http://rustfs:9000
+s3.access-key: rustfsadmin
+s3.secret-key: rustfsadmin
+s3.region: us-east-1
+s3.path-style-access: true
+s3.assumed.role.arn: arn:aws:iam::000000000000:role/rustfsadmin
+s3.assumed.role.sts.endpoint: http://rustfs:9000
+
+datalake.enabled: true
+datalake.format: hudi
+datalake.hudi.mode: dfs
+datalake.hudi.catalog.path: s3a://fluss/hudi
+datalake.hudi.hadoop.fs.s3a.endpoint: http://rustfs:9000
+datalake.hudi.hadoop.fs.s3a.access.key: rustfsadmin
+datalake.hudi.hadoop.fs.s3a.secret.key: rustfsadmin
+datalake.hudi.hadoop.fs.s3a.path.style.access: true
+datalake.hudi.hadoop.fs.s3a.connection.ssl.enabled: false
+datalake.hudi.hadoop.fs.s3a.impl.disable.cache: true
+datalake.hudi.hadoop.fs.s3a.aws.credentials.provider: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
+datalake.hudi.hoodie.write.lock.provider: org.apache.hudi.client.transaction.lock.InProcessLockProvider
+
+fluss.tiering.poll.table.interval: 1s
Review Comment:
Only hudi and lance set this, iceberg and paimon get the 30s default. Intentional?
##########
devkit/docker-compose.yml:
##########
@@ -0,0 +1,181 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: fluss-devkit
+
+x-fluss-common: &fluss-common
+ image: ${FLUSS_DEVKIT_IMAGE:-eclipse-temurin:17-jre-noble}
+ init: true
+ restart: unless-stopped
+ stop_grace_period: 2m
+ volumes:
+ - ../build-target:/opt/fluss:ro
+ - ${FLUSS_DEVKIT_CONFIG:-./profiles/core/server.yaml}:/opt/fluss/conf/server.yaml:ro
+ - shared-data:/tmp/fluss
+ environment:
+ FLUSS_LOG_DIR: /tmp/fluss-logs
+ logging:
+ driver: json-file
+ options:
+ max-size: 10m
+ max-file: "3"
+
+x-tablet-common: &tablet-common
+ <<: *fluss-common
+ entrypoint: ["/opt/fluss/bin/tablet-server.sh", "start-foreground"]
+ depends_on:
+ coordinator-server:
+ condition: service_healthy
+ environment:
+ FLUSS_LOG_DIR: /tmp/fluss-logs
+ READINESS_TCP_PORT: "9123"
+ READINESS_HEALTH_CHECK_TIMEOUT_SECONDS: "180"
+ healthcheck:
+ test: ["CMD-SHELL", "bash /opt/fluss/bin/readiness-check.sh"]
+ interval: 5s
+ timeout: 15s
+ retries: 36
+ start_period: 10s
+
+services:
+ # Flink runs as the flink user while Fluss creates this shared volume as root.
+ shared-data-init:
+ image: ${FLUSS_DEVKIT_IMAGE:-eclipse-temurin:17-jre-noble}
+ restart: "no"
+ volumes:
+ - shared-data:/tmp/fluss
+ entrypoint: ["/bin/bash", "-c"]
+ command: ["chmod 0777 /tmp/fluss"]
+
+ zookeeper:
+ image: zookeeper:3.9.2
+ restart: unless-stopped
+ healthcheck:
+ test: ["CMD-SHELL", "zkServer.sh status"]
+ interval: 5s
+ timeout: 5s
+ retries: 20
+ start_period: 5s
+ volumes:
+ - zookeeper-data:/data
+ - zookeeper-log:/datalog
+
+ coordinator-server:
+ <<: *fluss-common
+ entrypoint: ["/opt/fluss/bin/coordinator-server.sh", "start-foreground"]
+ depends_on:
+ shared-data-init:
+ condition: service_completed_successfully
+ zookeeper:
+ condition: service_healthy
+ command:
+ - -Dzookeeper.address=zookeeper:2181
+ - -Dbind.listeners=INTERNAL://coordinator-server:0,CLIENT://coordinator-server:9123,DEVKIT://coordinator-server:19123
+ - -Dadvertised.listeners=CLIENT://localhost:${COORDINATOR_PORT:-9123}
+ - -Dinternal.listener.name=INTERNAL
+ - -Ddefault.replication.factor=${FLUSS_REPLICATION_FACTOR:-1}
+ - -Dmetrics.reporters=prometheus
+ - -Dmetrics.reporter.prometheus.port=9249
+ environment:
+ FLUSS_LOG_DIR: /tmp/fluss-logs
+ FLUSS_ENV_JAVA_OPTS: -Xms512m -Xmx512m -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005
+ ports:
+ - "${COORDINATOR_PORT:-9123}:9123"
+ - "${COORDINATOR_DEBUG_PORT:-15005}:5005"
+ - "${COORDINATOR_METRICS_PORT:-9249}:9249"
+ healthcheck:
+ test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/coordinator-server/9123'"]
+ interval: 5s
+ timeout: 5s
+ retries: 24
+ start_period: 10s
+
+ tablet-server-0:
+ <<: *tablet-common
+ profiles: [single, cluster]
+ command:
+ - -Dzookeeper.address=zookeeper:2181
+ - -Dbind.listeners=INTERNAL://tablet-server-0:0,CLIENT://tablet-server-0:9123,DEVKIT://tablet-server-0:19123
+ - -Dadvertised.listeners=CLIENT://localhost:${TABLET_SERVER_0_PORT:-9124}
+ - -Dinternal.listener.name=INTERNAL
+ - -Dtablet-server.id=0
+ - -Dkv.snapshot.interval=0s
Review Comment:
0s disables periodic KV snapshots as PeriodicSnapshotManager.start() only schedules when the
interval is > 0. PK tables will never snapshot and a restart replays the full log. Shall we use smth like 10s?
##########
devkit/docker-compose.yml:
##########
@@ -0,0 +1,181 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: fluss-devkit
+
+x-fluss-common: &fluss-common
+ image: ${FLUSS_DEVKIT_IMAGE:-eclipse-temurin:17-jre-noble}
+ init: true
+ restart: unless-stopped
+ stop_grace_period: 2m
+ volumes:
+ - ../build-target:/opt/fluss:ro
+ - ${FLUSS_DEVKIT_CONFIG:-./profiles/core/server.yaml}:/opt/fluss/conf/server.yaml:ro
+ - shared-data:/tmp/fluss
+ environment:
+ FLUSS_LOG_DIR: /tmp/fluss-logs
Review Comment:
Does this block ever apply? Every service declares its own `environment:`, which replaces it
entirely.
##########
devkit/profiles/hudi/server.urls:
##########
@@ -0,0 +1,21 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+https://repo.maven.apache.org/maven2/io/trino/hadoop/hadoop-apache/3.3.5-2/hadoop-apache-3.3.5-2.jar
+https://repo.maven.apache.org/maven2/org/apache/flink/flink-core/1.20.1/flink-core-1.20.1.jar
Review Comment:
Are these here because `plugins/hudi/` doesn't work on its own? Loading it with only the jar the
assembly ships throws:
NoClassDefFoundError: org/apache/flink/table/catalog/exceptions/DatabaseNotExistException
at org.apache.fluss.lake.hudi.HudiLakeStorage.createLakeCatalog(HudiLakeStorage.java:46)
`plugins.xml` only ships extra jars for paimon, so hudi gets nothing.
Hudi isn't in a release yet, so I assume this is unfinished rather than a regression.
@fhan688 can you help here to reason here, pls?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4232529736
##########
devkit/examples/lake/lance/create-table.sql:
##########
@@ -0,0 +1,32 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+CREATE CATALOG fluss_catalog WITH (
+ 'type' = 'fluss',
+ 'bootstrap.servers' = 'coordinator-server:19123'
+);
+
+USE CATALOG fluss_catalog;
+
+CREATE TABLE lake_table (
+ id BIGINT,
+ payload STRING,
+ PRIMARY KEY (id) NOT ENFORCED
Review Comment:
Thanks for pointing this out. Lance currently supports tiering Log Tables only, so I updated the DevKit example to use an append-only Log Table. I also found the same inconsistency in the official Lance documentation and opened [PR #4581](https://github.com/apache/fluss/pull/4581) to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on PR #3851:
URL: https://github.com/apache/fluss/pull/3851#issuecomment-6087652322
Hi @fresh-borzoni, apologies for the delayed follow-up, and thank you again for the detailed review. I’ve addressed the points you raised and pushed the changes to the PR. Please take another look when convenient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4232650026
##########
devkit/compose/tiering.yml:
##########
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+x-flink-common: &flink-common
+ image: ${FLUSS_DEVKIT_FLINK_IMAGE:-flink:1.20.3-scala_2.12-java17}
+ entrypoint:
+ - /bin/bash
+ - -c
+ - |
+ cp /opt/devkit-jars/lib/*.jar /opt/flink/lib/
+ chmod a+r /opt/flink/lib/*.jar
+ exec /docker-entrypoint.sh "$$@"
+ - --
+ volumes:
+ - ./.deps/flink/active:/opt/devkit-jars:ro
+ - ${FLUSS_DEVKIT_CONFIG:-./profiles/core/server.yaml}:/opt/devkit-config/server.yaml:ro
+ - shared-data:/tmp/fluss
+
+services:
+ jobmanager:
+ <<: *flink-common
+ profiles: [flink]
+ command: ["jobmanager"]
+ depends_on:
+ coordinator-server:
+ condition: service_healthy
+ environment:
+ FLINK_PROPERTIES: |
+ jobmanager.rpc.address: jobmanager
+ jobmanager.memory.process.size: 1024m
+ parallelism.default: 1
+ rest.address: jobmanager
+ ports:
+ - "${FLINK_REST_PORT:-8083}:8081"
+ healthcheck:
+ test: ["CMD-SHELL", "curl -fsS http://localhost:8081/overview >/dev/null"]
+ interval: 5s
+ timeout: 5s
+ retries: 24
+ start_period: 10s
+
+ taskmanager:
+ <<: *flink-common
+ profiles: [flink]
+ command: ["taskmanager"]
+ depends_on:
+ jobmanager:
+ condition: service_healthy
+ environment:
+ FLINK_PROPERTIES: |
+ jobmanager.rpc.address: jobmanager
+ taskmanager.memory.process.size: 1536m
+ taskmanager.numberOfTaskSlots: 4
+ healthcheck:
+ test:
+ - CMD-SHELL
+ - curl -fsS http://jobmanager:8081/taskmanagers | grep -q '"id"'
+ interval: 5s
+ timeout: 5s
+ retries: 24
+ start_period: 10s
+
+ tiering-submit:
+ <<: *flink-common
+ profiles: [tiering]
+ restart: "no"
Review Comment:
Yes, the Tiering Job is submitted by a one-shot CLI container during startup, so it is not submitted again after the JobManager restarts. I added `just tiering-restart` as an explicit way to resubmit it and documented it in the README.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4232745615
##########
AGENTS.md:
##########
@@ -196,6 +196,8 @@ Apache Fluss follows a layered Maven module architecture: `fluss-common` (founda
**`fluss-dist`** - Binary distribution with `bin/` scripts and `conf/` templates
+**`devkit/`** - Local development toolkit for running and testing Fluss across core and lakehouse scenarios
Review Comment:
Addressed. DevKit has been moved out of the Maven module list into Development Tooling, with a short note on using it for local manual end-to-end testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on PR #3851:
URL: https://github.com/apache/fluss/pull/3851#issuecomment-5194202543
Hi @luoyuxia and @fresh-borzoni, I put together this DevKit based on my experience getting started with Fluss, mainly to make local startup and validation easier. When you have time, could you please take a look at the overall direction and scope? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4232602079
##########
devkit/docker-compose.yml:
##########
@@ -0,0 +1,179 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: fluss-devkit
+
+x-fluss-common: &fluss-common
+ image: ${FLUSS_DEVKIT_IMAGE:-eclipse-temurin:17-jre-noble}
+ init: true
+ restart: unless-stopped
+ stop_grace_period: 2m
+ volumes:
+ - ../build-target:/opt/fluss:ro
+ - ${FLUSS_DEVKIT_CONFIG:-./profiles/core/server.yaml}:/opt/fluss/conf/server.yaml:ro
+ - shared-data:/tmp/fluss
+ logging:
+ driver: json-file
+ options:
+ max-size: 10m
+ max-file: "3"
+
+x-tablet-common: &tablet-common
+ <<: *fluss-common
+ entrypoint: ["/opt/fluss/bin/tablet-server.sh", "start-foreground"]
+ depends_on:
+ coordinator-server:
+ condition: service_healthy
+ environment:
Review Comment:
Sorry, I forgot to remove this earlier. I also took a quick look at some of the other shared Compose configurations, and I don’t think there are any similar ineffective ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r3846360033
##########
devkit/compose/tiering.yml:
##########
@@ -0,0 +1,114 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+x-flink-common: &flink-common
+ image: flink:1.20.0-scala_2.12-java17
Review Comment:
Addressed. The DevKit Flink image and related dependencies are now aligned on 1.20.3, matching the fluss-flink-1.20 POM configuration. The Hudi profile has been removed for now, so the old 1.20.1 dependency is no longer present.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4093799773
##########
devkit/examples/lake/lance/create-table.sql:
##########
@@ -0,0 +1,32 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+CREATE CATALOG fluss_catalog WITH (
+ 'type' = 'fluss',
+ 'bootstrap.servers' = 'coordinator-server:19123'
+);
+
+USE CATALOG fluss_catalog;
+
+CREATE TABLE lake_table (
+ id BIGINT,
+ payload STRING,
+ PRIMARY KEY (id) NOT ENFORCED
Review Comment:
This fails on every run: InvalidTableException: Currently, we don't support tiering a primary key table to Lance. Lance only tiers Log Tables. I ran the same file without the PK and it tiered fine, and the data showed up in RustFS.
Shall we drop the PK here?
##########
devkit/docker-compose.yml:
##########
@@ -0,0 +1,179 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: fluss-devkit
+
+x-fluss-common: &fluss-common
+ image: ${FLUSS_DEVKIT_IMAGE:-eclipse-temurin:17-jre-noble}
+ init: true
+ restart: unless-stopped
+ stop_grace_period: 2m
+ volumes:
+ - ../build-target:/opt/fluss:ro
+ - ${FLUSS_DEVKIT_CONFIG:-./profiles/core/server.yaml}:/opt/fluss/conf/server.yaml:ro
+ - shared-data:/tmp/fluss
+ logging:
+ driver: json-file
+ options:
+ max-size: 10m
+ max-file: "3"
+
+x-tablet-common: &tablet-common
+ <<: *fluss-common
+ entrypoint: ["/opt/fluss/bin/tablet-server.sh", "start-foreground"]
+ depends_on:
+ coordinator-server:
+ condition: service_healthy
+ environment:
Review Comment:
This one is still dead: every tablet server declares its own environment:, so this block never reaches them.
##########
devkit/compose/tiering.yml:
##########
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+x-flink-common: &flink-common
+ image: ${FLUSS_DEVKIT_FLINK_IMAGE:-flink:1.20.3-scala_2.12-java17}
+ entrypoint:
+ - /bin/bash
+ - -c
+ - |
+ cp /opt/devkit-jars/lib/*.jar /opt/flink/lib/
+ chmod a+r /opt/flink/lib/*.jar
+ exec /docker-entrypoint.sh "$$@"
+ - --
+ volumes:
+ - ./.deps/flink/active:/opt/devkit-jars:ro
+ - ${FLUSS_DEVKIT_CONFIG:-./profiles/core/server.yaml}:/opt/devkit-config/server.yaml:ro
+ - shared-data:/tmp/fluss
+
+services:
+ jobmanager:
+ <<: *flink-common
+ profiles: [flink]
+ command: ["jobmanager"]
+ depends_on:
+ coordinator-server:
+ condition: service_healthy
+ environment:
+ FLINK_PROPERTIES: |
+ jobmanager.rpc.address: jobmanager
+ jobmanager.memory.process.size: 1024m
+ parallelism.default: 1
+ rest.address: jobmanager
+ ports:
+ - "${FLINK_REST_PORT:-8083}:8081"
+ healthcheck:
+ test: ["CMD-SHELL", "curl -fsS http://localhost:8081/overview >/dev/null"]
+ interval: 5s
+ timeout: 5s
+ retries: 24
+ start_period: 10s
+
+ taskmanager:
+ <<: *flink-common
+ profiles: [flink]
+ command: ["taskmanager"]
+ depends_on:
+ jobmanager:
+ condition: service_healthy
+ environment:
+ FLINK_PROPERTIES: |
+ jobmanager.rpc.address: jobmanager
+ taskmanager.memory.process.size: 1536m
+ taskmanager.numberOfTaskSlots: 4
+ healthcheck:
+ test:
+ - CMD-SHELL
+ - curl -fsS http://jobmanager:8081/taskmanagers | grep -q '"id"'
+ interval: 5s
+ timeout: 5s
+ retries: 24
+ start_period: 10s
+
+ tiering-submit:
+ <<: *flink-common
+ profiles: [tiering]
+ restart: "no"
Review Comment:
If the JobManager restarts, the tiering job is gone and nothing resubmits it. Can we add a just recipe to resubmit it, or at least a note in the README?
##########
.github/workflows/devkit-smoke.yaml:
##########
@@ -0,0 +1,82 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+name: DevKit Smoke
+
+permissions:
+ contents: read
+
+on:
+ pull_request:
+ paths:
+ - 'devkit/**'
+ push:
+ branches:
+ - main
+ - release-**
+ paths:
+ - 'devkit/**'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }}
+ cancel-in-progress: true
+
+jobs:
+ smoke:
+ runs-on: ubuntu-latest
+ timeout-minutes: 45
+ steps:
+ - uses: actions/checkout@v6
+
+ - name: Set up JDK 11
+ uses: actions/setup-java@v5
+ with:
+ java-version: '11'
+ distribution: temurin
+
+ - name: Install just
+ uses: taiki-e/install-action@v2
+ with:
+ tool: just@1.40.0
+
+ - name: Validate managed profile dependency versions
+ working-directory: devkit
+ run: |
+ set -euo pipefail
+ paimon_version=$(sed -n 's:.*<paimon.version>\([^<]*\)</paimon.version>.*:\1:p' ../pom.xml)
+ iceberg_version=$(sed -n 's:.*<iceberg.version>\([^<]*\)</iceberg.version>.*:\1:p' ../pom.xml)
+ grep -Fq "/paimon-s3/${paimon_version}/paimon-s3-${paimon_version}.jar" profiles/paimon/jars.urls
+ grep -Fq "/paimon-flink-1.20/${paimon_version}/paimon-flink-1.20-${paimon_version}.jar" profiles/paimon/flink.urls
+ grep -Fq "/iceberg-aws/${iceberg_version}/iceberg-aws-${iceberg_version}.jar" profiles/iceberg/jars.urls
+ grep -Fq "/iceberg-aws-bundle/${iceberg_version}/iceberg-aws-bundle-${iceberg_version}.jar" profiles/iceberg/jars.urls
+ grep -Fq "/iceberg-flink-runtime-1.20/${iceberg_version}/iceberg-flink-runtime-1.20-${iceberg_version}.jar" profiles/iceberg/flink.urls
+
+ - name: Run Gateway and Paimon smoke tests
+ working-directory: devkit
+ run: |
+ set -euo pipefail
+ trap 'just clean >/dev/null 2>&1 || true' EXIT
+ just build gateway
+ just up gateway
+ curl --fail --silent --show-error http://localhost:8080/v1/clusters/default/databases
+ just down
+ just build paimon
+ just up paimon
+ just run-sql examples/lake/paimon/setup.sql
+ just run-sql examples/lake/paimon/union-read.sql
+ just run-sql examples/lake/paimon/lake-only-read.sql
Review Comment:
This passes even if tiering is broken. run-sql only fails on [ERROR], and an empty $lake result has no error in it. It went green this time because tiering happened to finish before the query. Can we wait for the rows and check the count?
##########
devkit/examples/lake/paimon/query-lake.sql:
##########
@@ -0,0 +1,35 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
nit: create-table, write-data and query-lake aren't used anywhere; the README and smoke use setup / union-read / lake-only-read. Is it expected?
##########
devkit/justfile:
##########
@@ -0,0 +1,514 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set dotenv-load := false
+
+# --- Paths --------------------------------------------------------------------
+
+DEVKIT_DIR := justfile_directory()
+FLUSS_DEV_HOME := DEVKIT_DIR + "/.."
+PROFILES_DIR := DEVKIT_DIR + "/profiles"
+DEPS_DIR := DEVKIT_DIR + "/.deps"
+CACHE_DIR := DEPS_DIR + "/cache"
+ACTIVE_PROFILE_FILE := DEPS_DIR + "/active-profile"
+
+# --- Docker Compose commands --------------------------------------------------
+
+COMPOSE := "docker compose -f " + DEVKIT_DIR + "/docker-compose.yml"
+ALL_COMPOSE := COMPOSE + " -f " + DEVKIT_DIR + "/compose/tiering.yml -f " + DEVKIT_DIR + "/compose/gateway.yml -f " + DEVKIT_DIR + "/compose/lake-common.yml -f " + DEVKIT_DIR + "/compose/iceberg.yml"
+ALL_PROFILES := "--profile single --profile cluster --profile flink --profile tiering --profile gateway"
+
+_default:
+ @just --list
+
+[doc("Prepare every artifact required by a profile")]
+build profile="core":
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ just validate-profile "$PROFILE"
+ TARGETS=$(just profile-runtime-targets "$PROFILE")
+ just deps "$PROFILE"
+ clean_dist() {
+ cd "{{ FLUSS_DEV_HOME }}"
+ ./mvnw -pl fluss-dist clean
+ }
+ build_core() {
+ clean_dist
+ cd "{{ FLUSS_DEV_HOME }}"
+ ./mvnw -DskipTests -pl fluss-dist,fluss-flink/fluss-flink-1.20 -am package
+ }
+ build_tiering() {
+ clean_dist
+ cd "{{ FLUSS_DEV_HOME }}"
+ ./mvnw -DskipTests -pl fluss-dist,fluss-flink/fluss-flink-1.20,fluss-flink/fluss-flink-tiering -am package
+ }
+ build_gateway() {
+ mkdir -p "{{ DEPS_DIR }}/cargo" "{{ DEPS_DIR }}/gateway"
+ docker run --rm \
+ --user "$(id -u):$(id -g)" \
+ -e CARGO_HOME=/cache \
+ -e CARGO_TARGET_DIR=/output \
+ -e HOME=/tmp \
+ -e RUSTUP_TOOLCHAIN=1.88.0 \
+ -v "{{ FLUSS_DEV_HOME }}:/workspace:ro" \
+ -v "{{ DEPS_DIR }}/cargo:/cache" \
+ -v "{{ DEPS_DIR }}/gateway:/output" \
+ -w /workspace/fluss-gateway \
+ rust:1.88-bookworm \
+ cargo build --locked --bin fluss-gateway
+ }
+ if grep -Fxq tiering <<<"$TARGETS"; then
+ build_tiering
+ elif grep -Eq '^(core|flink)$' <<<"$TARGETS"; then
+ build_core
+ fi
+ if grep -Fxq gateway <<<"$TARGETS"; then
+ build_gateway
+ fi
+
+[private]
+ensure-dist:
+ #!/usr/bin/env bash
+ set -e
+ DIST_HOME="{{ FLUSS_DEV_HOME }}/build-target"
+ if [ ! -x "$DIST_HOME/bin/coordinator-server.sh" ] || \
+ [ ! -x "$DIST_HOME/bin/tablet-server.sh" ] || \
+ [ ! -f "$DIST_HOME/conf/server.yaml" ] || \
+ [ ! -d "$DIST_HOME/lib" ]; then
+ echo "Complete local distribution not found. Run: just build <profile>"
+ exit 1
+ fi
+
+[private]
+ensure-gateway:
+ #!/usr/bin/env bash
+ if [ ! -x "{{ DEPS_DIR }}/gateway/debug/fluss-gateway" ]; then
+ echo "Local Gateway executable not found. Run: just build gateway"
+ exit 1
+ fi
+
+[private]
+validate-profile profile:
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ if [[ ! "$PROFILE" =~ ^[a-z0-9][a-z0-9-]*$ ]]; then
+ echo "profile must match [a-z0-9][a-z0-9-]*"
+ exit 1
+ fi
+ if [ ! -f "{{ PROFILES_DIR }}/$PROFILE/server.yaml" ]; then
+ echo "Unknown profile: $PROFILE"
+ exit 1
+ fi
+ if [ ! -f "{{ PROFILES_DIR }}/$PROFILE/runtime.targets" ]; then
+ echo "Profile $PROFILE does not declare runtime.targets"
+ exit 1
+ fi
+
+[private]
+profile-runtime-targets profile:
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ just validate-profile "$PROFILE"
+ TARGETS=$(sed 's/#.*//; /^[[:space:]]*$/d' "{{ PROFILES_DIR }}/$PROFILE/runtime.targets")
+ SEEN=""
+ for TARGET in $TARGETS; do
+ case "$TARGET" in
+ core|flink|gateway|tiering) ;;
+ *) echo "Unknown runtime target in profile $PROFILE: $TARGET" >&2; exit 1 ;;
+ esac
+ case " $SEEN " in
+ *" $TARGET "*)
+ echo "Duplicate runtime target in profile $PROFILE: $TARGET" >&2
+ exit 1
+ ;;
+ esac
+ SEEN="$SEEN $TARGET"
+ done
+ has_target() {
+ [[ " $SEEN " == *" $1 "* ]]
+ }
+ require_target() {
+ if has_target "$1" && ! has_target "$2"; then
+ echo "Profile $PROFILE requires runtime target: $2 for $1" >&2
+ exit 1
+ fi
+ }
+ require_target flink core
+ require_target tiering flink
+ require_target gateway core
+ if ! has_target core; then
+ echo "Profile $PROFILE must include runtime target: core" >&2
+ exit 1
+ fi
+ for TARGET in $SEEN; do
+ echo "$TARGET"
+ done
+
+[private]
+profile-format profile:
+ #!/usr/bin/env bash
+ PROFILE={{ quote(profile) }}
+ sed -n 's/^[[:space:]]*datalake\.format[[:space:]]*: \([^#]*\).*$/\1/p' \
+ "{{ PROFILES_DIR }}/$PROFILE/server.yaml" | sed 's/[[:space:]]*$//' | tail -n 1
+
+[private]
+compose profile topology *ARGS:
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ TOPOLOGY={{ quote(topology) }}
+ ARGS=( {{ ARGS }} )
+ just validate-profile "$PROFILE"
+ case "$TOPOLOGY" in
+ single) REPLICATION_FACTOR=1 ;;
+ cluster) REPLICATION_FACTOR=3 ;;
+ *) echo "Unknown topology: $TOPOLOGY"; exit 1 ;;
+ esac
+ PROFILE_DIR="{{ PROFILES_DIR }}/$PROFILE"
+ COMPOSE_ARGS=(-f "{{ DEVKIT_DIR }}/docker-compose.yml")
+ PROFILES=(--profile "$TOPOLOGY")
+ for TARGET in $(just profile-runtime-targets "$PROFILE"); do
+ case "$TARGET" in
+ core) ;;
+ flink)
+ COMPOSE_ARGS+=(-f "{{ DEVKIT_DIR }}/compose/tiering.yml")
+ PROFILES+=(--profile flink)
+ ;;
+ tiering) PROFILES+=(--profile tiering) ;;
+ gateway)
+ COMPOSE_ARGS+=(-f "{{ DEVKIT_DIR }}/compose/gateway.yml")
+ PROFILES+=(--profile gateway)
+ ;;
+ esac
+ done
+ if [ -f "$PROFILE_DIR/compose.files" ]; then
+ while IFS= read -r FILE; do
+ FILE="${FILE%%#*}"
+ FILE="${FILE//[[:space:]]/}"
+ [ -n "$FILE" ] || continue
+ if [ ! -f "{{ DEVKIT_DIR }}/$FILE" ]; then
+ echo "Compose file not found: $FILE"
+ exit 1
+ fi
+ COMPOSE_ARGS+=(-f "{{ DEVKIT_DIR }}/$FILE")
+ done < "$PROFILE_DIR/compose.files"
+ fi
+ export FLUSS_DEVKIT_CONFIG="$PROFILE_DIR/server.yaml"
+ export FLUSS_REPLICATION_FACTOR="$REPLICATION_FACTOR"
+ docker compose "${COMPOSE_ARGS[@]}" "${PROFILES[@]}" "${ARGS[@]}"
+
+[private]
+compose-active *ARGS:
+ #!/usr/bin/env bash
+ set -e
+ ARGS=( {{ ARGS }} )
+ if [ -f "{{ ACTIVE_PROFILE_FILE }}" ]; then
+ read -r PROFILE TOPOLOGY < "{{ ACTIVE_PROFILE_FILE }}"
+ if just validate-profile "$PROFILE" >/dev/null 2>&1; then
+ just compose "$PROFILE" "$TOPOLOGY" "${ARGS[@]}"
+ exit 0
+ fi
+ fi
+ {{ ALL_COMPOSE }} {{ ALL_PROFILES }} "${ARGS[@]}"
+
+[private]
+deps profile:
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ just validate-profile "$PROFILE"
+ PROFILE_DIR="{{ PROFILES_DIR }}/$PROFILE"
+ mkdir -p "{{ CACHE_DIR }}"
+ for FILE in "$PROFILE_DIR"/{jars,server,flink}.urls; do
+ [ -f "$FILE" ] || continue
+ while IFS= read -r URL; do
+ NAME="$(basename "${URL%%\?*}")"
+ TARGET="{{ CACHE_DIR }}/$NAME"
+ [ -s "$TARGET" ] && continue
+ echo "Downloading $(basename "$TARGET")..."
+ curl -fL --retry 3 -o "${TARGET}.tmp" "$URL"
+ mv "${TARGET}.tmp" "$TARGET"
+ done < <(sed 's/#.*//; s/[[:space:]]//g; /^$/d' "$FILE")
+ done
+
+[private]
+ensure-flink:
+ #!/usr/bin/env bash
+ set -e
+ ACTIVE_DIR="{{ DEPS_DIR }}/flink/active"
+ MATCHES=( {{ FLUSS_DEV_HOME }}/fluss-flink/fluss-flink-1.20/target/fluss-flink-1.20-*.jar )
+ if [ "${#MATCHES[@]}" -ne 1 ] || [ ! -f "${MATCHES[0]}" ]; then
+ echo "Expected one Flink connector artifact"
+ echo "Run: just build <profile>"
+ exit 1
+ fi
+ mkdir -p "$ACTIVE_DIR/lib" "$ACTIVE_DIR/jobs"
+ rm -f "$ACTIVE_DIR/lib/"*.jar "$ACTIVE_DIR/jobs/"*.jar
+ cp "${MATCHES[0]}" "$ACTIVE_DIR/lib/fluss-flink-1.20.jar"
+ chmod -R a+rX "$ACTIVE_DIR"
+
+[private]
+stage-profile-jars profile:
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ PROFILE_DIR="{{ PROFILES_DIR }}/$PROFILE"
+ FORMAT=$(just profile-format "$PROFILE")
+ SERVER_PLUGIN_DIR="{{ FLUSS_DEV_HOME }}/build-target/plugins/${FORMAT:-$PROFILE}"
+ rm -f "$SERVER_PLUGIN_DIR"/devkit-*.jar
+ copy_urls() {
+ local FILE="$1"
+ local TARGET="$2"
+ local PREFIX="${3:-}"
+ local URL NAME SOURCE
+ [ -f "$FILE" ] || return 0
+ while IFS= read -r URL; do
+ NAME="$(basename "${URL%%\?*}")"
+ SOURCE="{{ CACHE_DIR }}/$NAME"
+ if [ ! -s "$SOURCE" ]; then
+ echo "Profile dependency not found: $NAME"
+ echo "Run: just build $PROFILE"
+ exit 1
+ fi
+ mkdir -p "$TARGET"
+ cp "$SOURCE" "$TARGET/$PREFIX$NAME"
+ done < <(sed 's/#.*//; s/[[:space:]]//g; /^$/d' "$FILE")
+ }
+ ACTIVE_DIR="{{ DEPS_DIR }}/flink/active"
+ mkdir -p "$ACTIVE_DIR/lib" "$ACTIVE_DIR/jobs"
+ copy_urls "$PROFILE_DIR/jars.urls" "$SERVER_PLUGIN_DIR" "devkit-"
+ copy_urls "$PROFILE_DIR/jars.urls" "$ACTIVE_DIR/lib"
+ copy_urls "$PROFILE_DIR/server.urls" "$SERVER_PLUGIN_DIR" "devkit-"
+ copy_urls "$PROFILE_DIR/flink.urls" "$ACTIVE_DIR/lib"
+ chmod -R a+rX "$ACTIVE_DIR"
+
+[private]
+ensure-tiering profile:
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ just validate-profile "$PROFILE"
+ PROFILE_DIR="{{ PROFILES_DIR }}/$PROFILE"
+ FORMAT=$(just profile-format "$PROFILE")
+ if [ -z "$FORMAT" ]; then
+ echo "Profile $PROFILE does not configure datalake.format"
+ exit 1
+ fi
+ stage() {
+ local PATTERN="$1"
+ local TARGET="$2"
+ MATCHES=( $PATTERN )
+ if [ "${#MATCHES[@]}" -ne 1 ] || [ ! -f "${MATCHES[0]}" ]; then
+ echo "Expected one build artifact: $PATTERN"
+ echo "Run: just build $PROFILE"
+ exit 1
+ fi
+ cp "${MATCHES[0]}" "$TARGET"
+ }
+ ACTIVE_DIR="{{ DEPS_DIR }}/flink/active"
+ stage "{{ FLUSS_DEV_HOME }}/fluss-lake/fluss-lake-$FORMAT/target/fluss-lake-$FORMAT-*.jar" "$ACTIVE_DIR/lib/fluss-lake-$FORMAT.jar"
+ stage "{{ FLUSS_DEV_HOME }}/fluss-flink/fluss-flink-tiering/target/fluss-flink-tiering-*.jar" "$ACTIVE_DIR/jobs/fluss-flink-tiering.jar"
+
+ SERVER_PLUGIN_DIR="{{ FLUSS_DEV_HOME }}/build-target/plugins/$FORMAT"
+ mkdir -p "$SERVER_PLUGIN_DIR"
+ # Filesystem plugins must stay isolated from lake plugin classpaths.
+ rm -f "$SERVER_PLUGIN_DIR"/fluss-fs-*.jar
+ REMOTE_DATA_DIR=$(sed -n 's/^[[:space:]]*remote\.data\.dir[[:space:]]*: \([^#]*\).*$/\1/p' "$PROFILE_DIR/server.yaml" | sed 's/[[:space:]]*$//' | tail -n 1)
+ SCHEME="${REMOTE_DATA_DIR%%:*}"
+ if [ "$SCHEME" != "$REMOTE_DATA_DIR" ] && [ -d "{{ FLUSS_DEV_HOME }}/build-target/plugins/$SCHEME" ]; then
+ cp "{{ FLUSS_DEV_HOME }}/build-target/plugins/$SCHEME/"*.jar "$ACTIVE_DIR/lib/"
+ fi
+
+ chmod -R a+rX "$ACTIVE_DIR"
+
+[doc("Start a built profile from the profiles directory")]
+up profile="core" nodes="1":
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ just validate-profile "$PROFILE"
+ case "{{ nodes }}" in
+ 1) TOPOLOGY=single ;;
+ 3) TOPOLOGY=cluster ;;
+ *) echo "nodes must be 1 or 3"; exit 1 ;;
+ esac
+ ENABLE_TIERING=false
+ for TARGET in $(just profile-runtime-targets "$PROFILE"); do
+ case "$TARGET" in
+ core) just ensure-dist ;;
+ flink)
+ just ensure-flink
+ ;;
+ tiering) ENABLE_TIERING=true ;;
+ gateway)
+ just ensure-gateway
+ ;;
+ esac
+ done
+ just stage-profile-jars "$PROFILE"
+ if [ "$ENABLE_TIERING" = "true" ]; then
+ just ensure-tiering "$PROFILE"
+ fi
+ just compose-active down --remove-orphans >/dev/null 2>&1 || true
Review Comment:
Switching profiles keeps the volumes. I went from iceberg to lance and the Iceberg table was still in ZooKeeper, now running under the Lance config. Shall we print a warning when the profile changes?
##########
.github/workflows/devkit-smoke.yaml:
##########
@@ -0,0 +1,82 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+name: DevKit Smoke
+
+permissions:
+ contents: read
+
+on:
+ pull_request:
+ paths:
+ - 'devkit/**'
Review Comment:
The version check only runs when devkit/** changes, but it's there to catch pom bumps. #4093 (Paimon 2.0.0) changed only pom.xml, so it would have skipped this job.
Shall we add pom.xml here? Also, iceberg and lance never run in CI. That's how the Lance breakage got in.
##########
devkit/justfile:
##########
@@ -0,0 +1,514 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set dotenv-load := false
+
+# --- Paths --------------------------------------------------------------------
+
+DEVKIT_DIR := justfile_directory()
+FLUSS_DEV_HOME := DEVKIT_DIR + "/.."
+PROFILES_DIR := DEVKIT_DIR + "/profiles"
+DEPS_DIR := DEVKIT_DIR + "/.deps"
+CACHE_DIR := DEPS_DIR + "/cache"
+ACTIVE_PROFILE_FILE := DEPS_DIR + "/active-profile"
+
+# --- Docker Compose commands --------------------------------------------------
+
+COMPOSE := "docker compose -f " + DEVKIT_DIR + "/docker-compose.yml"
+ALL_COMPOSE := COMPOSE + " -f " + DEVKIT_DIR + "/compose/tiering.yml -f " + DEVKIT_DIR + "/compose/gateway.yml -f " + DEVKIT_DIR + "/compose/lake-common.yml -f " + DEVKIT_DIR + "/compose/iceberg.yml"
+ALL_PROFILES := "--profile single --profile cluster --profile flink --profile tiering --profile gateway"
+
+_default:
+ @just --list
+
+[doc("Prepare every artifact required by a profile")]
+build profile="core":
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ just validate-profile "$PROFILE"
+ TARGETS=$(just profile-runtime-targets "$PROFILE")
+ just deps "$PROFILE"
+ clean_dist() {
+ cd "{{ FLUSS_DEV_HOME }}"
+ ./mvnw -pl fluss-dist clean
+ }
+ build_core() {
+ clean_dist
+ cd "{{ FLUSS_DEV_HOME }}"
+ ./mvnw -DskipTests -pl fluss-dist,fluss-flink/fluss-flink-1.20 -am package
+ }
+ build_tiering() {
+ clean_dist
+ cd "{{ FLUSS_DEV_HOME }}"
+ ./mvnw -DskipTests -pl fluss-dist,fluss-flink/fluss-flink-1.20,fluss-flink/fluss-flink-tiering -am package
+ }
+ build_gateway() {
+ mkdir -p "{{ DEPS_DIR }}/cargo" "{{ DEPS_DIR }}/gateway"
+ docker run --rm \
+ --user "$(id -u):$(id -g)" \
+ -e CARGO_HOME=/cache \
+ -e CARGO_TARGET_DIR=/output \
+ -e HOME=/tmp \
+ -e RUSTUP_TOOLCHAIN=1.88.0 \
+ -v "{{ FLUSS_DEV_HOME }}:/workspace:ro" \
+ -v "{{ DEPS_DIR }}/cargo:/cache" \
+ -v "{{ DEPS_DIR }}/gateway:/output" \
+ -w /workspace/fluss-gateway \
+ rust:1.88-bookworm \
+ cargo build --locked --bin fluss-gateway
+ }
+ if grep -Fxq tiering <<<"$TARGETS"; then
+ build_tiering
+ elif grep -Eq '^(core|flink)$' <<<"$TARGETS"; then
+ build_core
+ fi
+ if grep -Fxq gateway <<<"$TARGETS"; then
+ build_gateway
+ fi
+
+[private]
+ensure-dist:
+ #!/usr/bin/env bash
+ set -e
+ DIST_HOME="{{ FLUSS_DEV_HOME }}/build-target"
+ if [ ! -x "$DIST_HOME/bin/coordinator-server.sh" ] || \
+ [ ! -x "$DIST_HOME/bin/tablet-server.sh" ] || \
+ [ ! -f "$DIST_HOME/conf/server.yaml" ] || \
+ [ ! -d "$DIST_HOME/lib" ]; then
+ echo "Complete local distribution not found. Run: just build <profile>"
+ exit 1
+ fi
+
+[private]
+ensure-gateway:
+ #!/usr/bin/env bash
+ if [ ! -x "{{ DEPS_DIR }}/gateway/debug/fluss-gateway" ]; then
+ echo "Local Gateway executable not found. Run: just build gateway"
+ exit 1
+ fi
+
+[private]
+validate-profile profile:
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ if [[ ! "$PROFILE" =~ ^[a-z0-9][a-z0-9-]*$ ]]; then
+ echo "profile must match [a-z0-9][a-z0-9-]*"
+ exit 1
+ fi
+ if [ ! -f "{{ PROFILES_DIR }}/$PROFILE/server.yaml" ]; then
+ echo "Unknown profile: $PROFILE"
+ exit 1
+ fi
+ if [ ! -f "{{ PROFILES_DIR }}/$PROFILE/runtime.targets" ]; then
+ echo "Profile $PROFILE does not declare runtime.targets"
+ exit 1
+ fi
+
+[private]
+profile-runtime-targets profile:
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ just validate-profile "$PROFILE"
+ TARGETS=$(sed 's/#.*//; /^[[:space:]]*$/d' "{{ PROFILES_DIR }}/$PROFILE/runtime.targets")
+ SEEN=""
+ for TARGET in $TARGETS; do
+ case "$TARGET" in
+ core|flink|gateway|tiering) ;;
+ *) echo "Unknown runtime target in profile $PROFILE: $TARGET" >&2; exit 1 ;;
+ esac
+ case " $SEEN " in
+ *" $TARGET "*)
+ echo "Duplicate runtime target in profile $PROFILE: $TARGET" >&2
+ exit 1
+ ;;
+ esac
+ SEEN="$SEEN $TARGET"
+ done
+ has_target() {
+ [[ " $SEEN " == *" $1 "* ]]
+ }
+ require_target() {
+ if has_target "$1" && ! has_target "$2"; then
+ echo "Profile $PROFILE requires runtime target: $2 for $1" >&2
+ exit 1
+ fi
+ }
+ require_target flink core
+ require_target tiering flink
+ require_target gateway core
+ if ! has_target core; then
+ echo "Profile $PROFILE must include runtime target: core" >&2
+ exit 1
+ fi
+ for TARGET in $SEEN; do
+ echo "$TARGET"
+ done
+
+[private]
+profile-format profile:
+ #!/usr/bin/env bash
+ PROFILE={{ quote(profile) }}
+ sed -n 's/^[[:space:]]*datalake\.format[[:space:]]*: \([^#]*\).*$/\1/p' \
+ "{{ PROFILES_DIR }}/$PROFILE/server.yaml" | sed 's/[[:space:]]*$//' | tail -n 1
+
+[private]
+compose profile topology *ARGS:
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ TOPOLOGY={{ quote(topology) }}
+ ARGS=( {{ ARGS }} )
+ just validate-profile "$PROFILE"
+ case "$TOPOLOGY" in
+ single) REPLICATION_FACTOR=1 ;;
+ cluster) REPLICATION_FACTOR=3 ;;
+ *) echo "Unknown topology: $TOPOLOGY"; exit 1 ;;
+ esac
+ PROFILE_DIR="{{ PROFILES_DIR }}/$PROFILE"
+ COMPOSE_ARGS=(-f "{{ DEVKIT_DIR }}/docker-compose.yml")
+ PROFILES=(--profile "$TOPOLOGY")
+ for TARGET in $(just profile-runtime-targets "$PROFILE"); do
+ case "$TARGET" in
+ core) ;;
+ flink)
+ COMPOSE_ARGS+=(-f "{{ DEVKIT_DIR }}/compose/tiering.yml")
+ PROFILES+=(--profile flink)
+ ;;
+ tiering) PROFILES+=(--profile tiering) ;;
+ gateway)
+ COMPOSE_ARGS+=(-f "{{ DEVKIT_DIR }}/compose/gateway.yml")
+ PROFILES+=(--profile gateway)
+ ;;
+ esac
+ done
+ if [ -f "$PROFILE_DIR/compose.files" ]; then
+ while IFS= read -r FILE; do
+ FILE="${FILE%%#*}"
+ FILE="${FILE//[[:space:]]/}"
+ [ -n "$FILE" ] || continue
+ if [ ! -f "{{ DEVKIT_DIR }}/$FILE" ]; then
+ echo "Compose file not found: $FILE"
+ exit 1
+ fi
+ COMPOSE_ARGS+=(-f "{{ DEVKIT_DIR }}/$FILE")
+ done < "$PROFILE_DIR/compose.files"
+ fi
+ export FLUSS_DEVKIT_CONFIG="$PROFILE_DIR/server.yaml"
+ export FLUSS_REPLICATION_FACTOR="$REPLICATION_FACTOR"
+ docker compose "${COMPOSE_ARGS[@]}" "${PROFILES[@]}" "${ARGS[@]}"
+
+[private]
+compose-active *ARGS:
+ #!/usr/bin/env bash
+ set -e
+ ARGS=( {{ ARGS }} )
+ if [ -f "{{ ACTIVE_PROFILE_FILE }}" ]; then
+ read -r PROFILE TOPOLOGY < "{{ ACTIVE_PROFILE_FILE }}"
+ if just validate-profile "$PROFILE" >/dev/null 2>&1; then
+ just compose "$PROFILE" "$TOPOLOGY" "${ARGS[@]}"
+ exit 0
+ fi
+ fi
+ {{ ALL_COMPOSE }} {{ ALL_PROFILES }} "${ARGS[@]}"
+
+[private]
+deps profile:
+ #!/usr/bin/env bash
+ set -e
+ PROFILE={{ quote(profile) }}
+ just validate-profile "$PROFILE"
+ PROFILE_DIR="{{ PROFILES_DIR }}/$PROFILE"
+ mkdir -p "{{ CACHE_DIR }}"
+ for FILE in "$PROFILE_DIR"/{jars,server,flink}.urls; do
+ [ -f "$FILE" ] || continue
+ while IFS= read -r URL; do
+ NAME="$(basename "${URL%%\?*}")"
+ TARGET="{{ CACHE_DIR }}/$NAME"
+ [ -s "$TARGET" ] && continue
Review Comment:
The cache is keyed only on the filename, and nothing checks the downloads. Central publishes .sha512 files. WDYT about verifying against them?
##########
AGENTS.md:
##########
@@ -196,6 +196,8 @@ Apache Fluss follows a layered Maven module architecture: `fluss-common` (founda
**`fluss-dist`** - Binary distribution with `bin/` scripts and `conf/` templates
+**`devkit/`** - Local development toolkit for running and testing Fluss across core and lakehouse scenarios
Review Comment:
nit: this list is Maven modules, and devkit isn't one. Shall we move it under Build?
##########
devkit/docker-compose.yml:
##########
@@ -0,0 +1,179 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: fluss-devkit
+
+x-fluss-common: &fluss-common
+ image: ${FLUSS_DEVKIT_IMAGE:-eclipse-temurin:17-jre-noble}
+ init: true
+ restart: unless-stopped
+ stop_grace_period: 2m
+ volumes:
+ - ../build-target:/opt/fluss:ro
+ - ${FLUSS_DEVKIT_CONFIG:-./profiles/core/server.yaml}:/opt/fluss/conf/server.yaml:ro
+ - shared-data:/tmp/fluss
+ logging:
+ driver: json-file
+ options:
+ max-size: 10m
+ max-file: "3"
+
+x-tablet-common: &tablet-common
+ <<: *fluss-common
+ entrypoint: ["/opt/fluss/bin/tablet-server.sh", "start-foreground"]
+ depends_on:
+ coordinator-server:
+ condition: service_healthy
+ environment:
+ FLUSS_LOG_DIR: /tmp/fluss-logs
+ READINESS_TCP_PORT: "9123"
+ READINESS_HEALTH_CHECK_TIMEOUT_SECONDS: "180"
+ healthcheck:
+ test: ["CMD-SHELL", "bash /opt/fluss/bin/readiness-check.sh"]
+ interval: 5s
+ timeout: 15s
+ retries: 36
+ start_period: 10s
+
+services:
+ # Flink runs as the flink user while Fluss creates this shared volume as root.
+ shared-data-init:
+ image: ${FLUSS_DEVKIT_IMAGE:-eclipse-temurin:17-jre-noble}
+ restart: "no"
+ volumes:
+ - shared-data:/tmp/fluss
+ entrypoint: ["/bin/bash", "-c"]
+ command: ["chmod 0777 /tmp/fluss"]
+
+ zookeeper:
+ image: zookeeper:3.9.2
+ restart: unless-stopped
+ healthcheck:
+ test: ["CMD-SHELL", "zkServer.sh status"]
+ interval: 5s
+ timeout: 5s
+ retries: 20
+ start_period: 5s
+ volumes:
+ - zookeeper-data:/data
+ - zookeeper-log:/datalog
+
+ coordinator-server:
+ <<: *fluss-common
+ entrypoint: ["/opt/fluss/bin/coordinator-server.sh", "start-foreground"]
+ depends_on:
+ shared-data-init:
+ condition: service_completed_successfully
+ zookeeper:
+ condition: service_healthy
+ command:
+ - -Dzookeeper.address=zookeeper:2181
+ - -Dbind.listeners=INTERNAL://coordinator-server:0,CLIENT://coordinator-server:9123,DEVKIT://coordinator-server:19123
+ - -Dadvertised.listeners=CLIENT://localhost:${COORDINATOR_PORT:-9123}
+ - -Dinternal.listener.name=INTERNAL
+ - -Ddefault.replication.factor=${FLUSS_REPLICATION_FACTOR:-1}
+ - -Dmetrics.reporters=prometheus
+ - -Dmetrics.reporter.prometheus.port=9249
+ environment:
+ FLUSS_LOG_DIR: /tmp/fluss-logs
+ FLUSS_ENV_JAVA_OPTS: -Xms512m -Xmx512m -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005
+ ports:
+ - "${COORDINATOR_PORT:-9123}:9123"
+ - "${COORDINATOR_DEBUG_PORT:-15005}:5005"
Review Comment:
All published ports bind to every interface, since none of the mappings names a host address. That includes JDWP on 15005-15008, which has no auth. Anyone who can reach the laptop can attach a debugger and run commands in the server containers. RustFS (fixed creds) and the Gateway (trust auth) are exposed the same way.
Shall we bind to 127.0.0.1 by default, maybe with a variable to override it? It feels cleaner this way. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub) via issues" <is...@fluss.apache.org>.
Gezi-lzq commented on PR #3851:
URL: https://github.com/apache/fluss/pull/3851#issuecomment-6663652033
> Thanks for putting this together. I see real value in this PR. Beyond making local development and validation easier, what I value most is the foundation it provides for E2E testing suite in CI, an important gap in Fluss’s current test coverage.
Thanks, I agree that this could be a useful foundation for E2E testing in CI. The current smoke tests already run a few small end-to-end flows against real services and check their results.
I haven’t yet worked out how to structure a broader E2E suite or systematically cover Fluss’s different features. I think these profiles and smoke cases are a good starting point, and we can build on them as the testing needs become clearer. I also expect DevKit to be useful in day-to-day development for starting Fluss, reproducing issues, and validating changes locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4233106823
##########
devkit/examples/core/02-streaming-scan.sql:
##########
@@ -0,0 +1,35 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
You are right. Some stale Paimon files were left behind after the examples were reorganized, so they no longer matched the README and smoke workflow. The stale files have been removed, and the remaining Paimon examples now follow the current workflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4233137345
##########
devkit/docker-compose.yml:
##########
@@ -0,0 +1,179 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: fluss-devkit
+
+x-fluss-common: &fluss-common
+ image: ${FLUSS_DEVKIT_IMAGE:-eclipse-temurin:17-jre-noble}
+ init: true
+ restart: unless-stopped
+ stop_grace_period: 2m
+ volumes:
+ - ../build-target:/opt/fluss:ro
+ - ${FLUSS_DEVKIT_CONFIG:-./profiles/core/server.yaml}:/opt/fluss/conf/server.yaml:ro
+ - shared-data:/tmp/fluss
+ logging:
+ driver: json-file
+ options:
+ max-size: 10m
+ max-file: "3"
+
+x-tablet-common: &tablet-common
+ <<: *fluss-common
+ entrypoint: ["/opt/fluss/bin/tablet-server.sh", "start-foreground"]
+ depends_on:
+ coordinator-server:
+ condition: service_healthy
+ environment:
+ FLUSS_LOG_DIR: /tmp/fluss-logs
+ READINESS_TCP_PORT: "9123"
+ READINESS_HEALTH_CHECK_TIMEOUT_SECONDS: "180"
+ healthcheck:
+ test: ["CMD-SHELL", "bash /opt/fluss/bin/readiness-check.sh"]
+ interval: 5s
+ timeout: 15s
+ retries: 36
+ start_period: 10s
+
+services:
+ # Flink runs as the flink user while Fluss creates this shared volume as root.
+ shared-data-init:
+ image: ${FLUSS_DEVKIT_IMAGE:-eclipse-temurin:17-jre-noble}
+ restart: "no"
+ volumes:
+ - shared-data:/tmp/fluss
+ entrypoint: ["/bin/bash", "-c"]
+ command: ["chmod 0777 /tmp/fluss"]
+
+ zookeeper:
+ image: zookeeper:3.9.2
+ restart: unless-stopped
+ healthcheck:
+ test: ["CMD-SHELL", "zkServer.sh status"]
+ interval: 5s
+ timeout: 5s
+ retries: 20
+ start_period: 5s
+ volumes:
+ - zookeeper-data:/data
+ - zookeeper-log:/datalog
+
+ coordinator-server:
+ <<: *fluss-common
+ entrypoint: ["/opt/fluss/bin/coordinator-server.sh", "start-foreground"]
+ depends_on:
+ shared-data-init:
+ condition: service_completed_successfully
+ zookeeper:
+ condition: service_healthy
+ command:
+ - -Dzookeeper.address=zookeeper:2181
+ - -Dbind.listeners=INTERNAL://coordinator-server:0,CLIENT://coordinator-server:9123,DEVKIT://coordinator-server:19123
+ - -Dadvertised.listeners=CLIENT://localhost:${COORDINATOR_PORT:-9123}
+ - -Dinternal.listener.name=INTERNAL
+ - -Ddefault.replication.factor=${FLUSS_REPLICATION_FACTOR:-1}
+ - -Dmetrics.reporters=prometheus
+ - -Dmetrics.reporter.prometheus.port=9249
+ environment:
+ FLUSS_LOG_DIR: /tmp/fluss-logs
+ FLUSS_ENV_JAVA_OPTS: -Xms512m -Xmx512m -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005
+ ports:
+ - "${COORDINATOR_PORT:-9123}:9123"
+ - "${COORDINATOR_DEBUG_PORT:-15005}:5005"
Review Comment:
Good point. All published ports, including JDWP, Gateway, RustFS, Flink, metrics, and Fluss server ports, are now bound to 127.0.0.1 by default. The host port numbers remain configurable through the existing environment variables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r3846365645
##########
devkit/docker-compose.yml:
##########
@@ -0,0 +1,181 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: fluss-devkit
+
+x-fluss-common: &fluss-common
+ image: ${FLUSS_DEVKIT_IMAGE:-eclipse-temurin:17-jre-noble}
+ init: true
+ restart: unless-stopped
+ stop_grace_period: 2m
+ volumes:
+ - ../build-target:/opt/fluss:ro
+ - ${FLUSS_DEVKIT_CONFIG:-./profiles/core/server.yaml}:/opt/fluss/conf/server.yaml:ro
+ - shared-data:/tmp/fluss
+ environment:
+ FLUSS_LOG_DIR: /tmp/fluss-logs
+ logging:
+ driver: json-file
+ options:
+ max-size: 10m
+ max-file: "3"
+
+x-tablet-common: &tablet-common
+ <<: *fluss-common
+ entrypoint: ["/opt/fluss/bin/tablet-server.sh", "start-foreground"]
+ depends_on:
+ coordinator-server:
+ condition: service_healthy
+ environment:
+ FLUSS_LOG_DIR: /tmp/fluss-logs
+ READINESS_TCP_PORT: "9123"
+ READINESS_HEALTH_CHECK_TIMEOUT_SECONDS: "180"
+ healthcheck:
+ test: ["CMD-SHELL", "bash /opt/fluss/bin/readiness-check.sh"]
+ interval: 5s
+ timeout: 15s
+ retries: 36
+ start_period: 10s
+
+services:
+ # Flink runs as the flink user while Fluss creates this shared volume as root.
+ shared-data-init:
+ image: ${FLUSS_DEVKIT_IMAGE:-eclipse-temurin:17-jre-noble}
+ restart: "no"
+ volumes:
+ - shared-data:/tmp/fluss
+ entrypoint: ["/bin/bash", "-c"]
+ command: ["chmod 0777 /tmp/fluss"]
+
+ zookeeper:
+ image: zookeeper:3.9.2
+ restart: unless-stopped
+ healthcheck:
+ test: ["CMD-SHELL", "zkServer.sh status"]
+ interval: 5s
+ timeout: 5s
+ retries: 20
+ start_period: 5s
+ volumes:
+ - zookeeper-data:/data
+ - zookeeper-log:/datalog
+
+ coordinator-server:
+ <<: *fluss-common
+ entrypoint: ["/opt/fluss/bin/coordinator-server.sh", "start-foreground"]
+ depends_on:
+ shared-data-init:
+ condition: service_completed_successfully
+ zookeeper:
+ condition: service_healthy
+ command:
+ - -Dzookeeper.address=zookeeper:2181
+ - -Dbind.listeners=INTERNAL://coordinator-server:0,CLIENT://coordinator-server:9123,DEVKIT://coordinator-server:19123
+ - -Dadvertised.listeners=CLIENT://localhost:${COORDINATOR_PORT:-9123}
+ - -Dinternal.listener.name=INTERNAL
+ - -Ddefault.replication.factor=${FLUSS_REPLICATION_FACTOR:-1}
+ - -Dmetrics.reporters=prometheus
+ - -Dmetrics.reporter.prometheus.port=9249
+ environment:
+ FLUSS_LOG_DIR: /tmp/fluss-logs
+ FLUSS_ENV_JAVA_OPTS: -Xms512m -Xmx512m -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005
+ ports:
+ - "${COORDINATOR_PORT:-9123}:9123"
+ - "${COORDINATOR_DEBUG_PORT:-15005}:5005"
+ - "${COORDINATOR_METRICS_PORT:-9249}:9249"
+ healthcheck:
+ test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/coordinator-server/9123'"]
+ interval: 5s
+ timeout: 5s
+ retries: 24
+ start_period: 10s
+
+ tablet-server-0:
+ <<: *tablet-common
+ profiles: [single, cluster]
+ command:
+ - -Dzookeeper.address=zookeeper:2181
+ - -Dbind.listeners=INTERNAL://tablet-server-0:0,CLIENT://tablet-server-0:9123,DEVKIT://tablet-server-0:19123
+ - -Dadvertised.listeners=CLIENT://localhost:${TABLET_SERVER_0_PORT:-9124}
+ - -Dinternal.listener.name=INTERNAL
+ - -Dtablet-server.id=0
+ - -Dkv.snapshot.interval=0s
Review Comment:
Addressed. The KV snapshot interval is now set to 10s, so Primary Key Tables periodically persist KV snapshots and do not need to replay the entire log after a restart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub)" <gi...@apache.org>.
Gezi-lzq commented on PR #3851:
URL: https://github.com/apache/fluss/pull/3851#issuecomment-5211332673
Hi @luoyuxia and @fresh-borzoni, I opened this PR based on my experience getting started with Fluss. I’d appreciate your thoughts on the overall direction and scope when you have time. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "wuchong (via GitHub) via issues" <is...@fluss.apache.org>.
wuchong commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4441831277
##########
.github/workflows/ci.yaml:
##########
@@ -49,6 +51,8 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v6
+ - name: Validate DevKit managed dependency versions
+ run: bash devkit/validate-managed-dependency-versions.sh
Review Comment:
Why do we need to validate the DevKit dependency versions in the Java 8 compilation job? This seems unrelated to Java compilation. Changes to the root `pom.xml` already trigger `devkit-smoke.yaml`, which runs `validate-managed-dependency-versions.sh`, so could we remove this duplicate check here?
##########
devkit/profiles/core/server.yaml:
##########
@@ -0,0 +1,19 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+remote.data.dir: file:///tmp/fluss/remote-data
Review Comment:
Is there a particular reason why the `core` and `gateway` profiles use a local `file://` path for `remote.data.dir`, while Paimon, Iceberg, and Lance use RustFS? Would it make sense to use RustFS for these two profiles as well, so these tests use a consistent storage setup and also exercise the S3-backed remote-storage path?
##########
.github/workflows/devkit-smoke.yaml:
##########
@@ -0,0 +1,175 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+name: DevKit Smoke
+
+permissions:
+ contents: read
+
+on:
+ pull_request:
+ paths:
+ - 'devkit/**'
+ - 'pom.xml'
+ - '.github/workflows/devkit-smoke.yaml'
+ push:
+ branches:
+ - main
+ - release-**
+ paths:
+ - 'devkit/**'
+ - 'pom.xml'
+ - '.github/workflows/devkit-smoke.yaml'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }}
+ cancel-in-progress: true
+
+jobs:
+ smoke:
+ runs-on: ubuntu-latest
+ timeout-minutes: 45
+ steps:
+ - uses: actions/checkout@v6
+
+ - name: Set up JDK 11
+ uses: actions/setup-java@v5
+ with:
+ java-version: '11'
+ distribution: temurin
+
+ - name: Install just
+ uses: taiki-e/install-action@v2
+ with:
+ tool: just@1.58.0
+
+ - name: Validate managed profile dependency versions
+ run: bash devkit/validate-managed-dependency-versions.sh
+
+ - name: Core smoke
+ working-directory: devkit
+ run: |
+ set -euo pipefail
+ just build core
+ just up core
+ just run-sql examples/core/setup.sql
+ core_output=$(just run-sql examples/core/batch-scan.sql)
+ printf '%s\n' "$core_output"
+ grep -Fq 'Alice' <<<"$core_output"
+ grep -Fq 'Bob' <<<"$core_output"
+ grep -Fq 'alpha' <<<"$core_output"
+ grep -Fq 'beta' <<<"$core_output"
+ just clean
+
+ - name: Gateway smoke
+ working-directory: devkit
+ run: |
+ set -euo pipefail
+ just build gateway
+ just up gateway
+ curl -fsS -X POST \
Review Comment:
Could we also assert the HTTP response bodies in the Gateway smoke test, for example the returned table metadata and append result? `curl -fsS` catches transport and HTTP errors, but a successful response with incorrect or empty content would still pass. Adding these assertions would make the smoke test verify the expected Gateway behavior.
##########
.github/workflows/devkit-smoke.yaml:
##########
@@ -0,0 +1,175 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+name: DevKit Smoke
+
+permissions:
+ contents: read
+
+on:
+ pull_request:
+ paths:
+ - 'devkit/**'
+ - 'pom.xml'
+ - '.github/workflows/devkit-smoke.yaml'
+ push:
+ branches:
+ - main
+ - release-**
+ paths:
+ - 'devkit/**'
+ - 'pom.xml'
+ - '.github/workflows/devkit-smoke.yaml'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }}
+ cancel-in-progress: true
+
+jobs:
+ smoke:
+ runs-on: ubuntu-latest
+ timeout-minutes: 45
+ steps:
+ - uses: actions/checkout@v6
+
+ - name: Set up JDK 11
+ uses: actions/setup-java@v5
+ with:
+ java-version: '11'
+ distribution: temurin
+
+ - name: Install just
+ uses: taiki-e/install-action@v2
+ with:
+ tool: just@1.58.0
+
+ - name: Validate managed profile dependency versions
+ run: bash devkit/validate-managed-dependency-versions.sh
+
+ - name: Core smoke
+ working-directory: devkit
+ run: |
+ set -euo pipefail
+ just build core
+ just up core
+ just run-sql examples/core/setup.sql
+ core_output=$(just run-sql examples/core/batch-scan.sql)
+ printf '%s\n' "$core_output"
+ grep -Fq 'Alice' <<<"$core_output"
+ grep -Fq 'Bob' <<<"$core_output"
+ grep -Fq 'alpha' <<<"$core_output"
+ grep -Fq 'beta' <<<"$core_output"
+ just clean
+
+ - name: Gateway smoke
+ working-directory: devkit
+ run: |
+ set -euo pipefail
+ just build gateway
+ just up gateway
+ curl -fsS -X POST \
+ -H 'Content-Type: application/json' \
+ http://localhost:8080/v1/clusters/default/databases/fluss/tables \
+ -d '{
+ "table_name": "gateway_events",
+ "columns": [
+ {"name": "event_id", "data_type": {"type": "BIGINT"}, "nullable": false},
+ {"name": "message", "data_type": {"type": "STRING"}, "nullable": false}
+ ],
+ "distribution": {"bucket_count": 1, "bucket_keys": []}
+ }'
+ curl -fsS \
+ http://localhost:8080/v1/clusters/default/databases/fluss/tables/gateway_events
+ curl -fsS -X POST \
+ -H 'Content-Type: application/json' \
+ http://localhost:8080/v1/clusters/default/databases/fluss/tables/gateway_events/records \
+ -d '{
+ "entries": [
+ {"id": "event-1", "append": {"event_id": "1", "message": "hello from Gateway"}}
+ ]
+ }'
+ just clean
+
+ - name: Paimon smoke
+ working-directory: devkit
+ run: |
+ set -euo pipefail
+ just build paimon
+ just up paimon
+ just run-sql examples/lake/paimon/setup.sql
+ just run-sql examples/lake/paimon/union-read.sql
+ for attempt in {1..30}; do
+ lake_output=$(just run-sql examples/lake/paimon/lake-only-read.sql || true)
+ printf '%s\n' "$lake_output"
+ if grep -Fq 'Alice' <<<"$lake_output" && grep -Fq 'Bob' <<<"$lake_output"; then
+ break
+ fi
+ if [ "$attempt" -eq 30 ]; then
+ echo 'Paimon lake-only query did not return Alice and Bob' >&2
+ just logs tiering-submit 200
+ exit 1
+ fi
+ sleep 2
+ done
+ just clean
+
+ - name: Iceberg smoke
+ working-directory: devkit
+ run: |
+ set -euo pipefail
+ just build iceberg
+ just up iceberg
+ just run-sql examples/lake/iceberg/create-table.sql
+ just run-sql examples/lake/iceberg/write-data.sql
+ iceberg_output=''
+ for attempt in {1..30}; do
+ iceberg_output=$(just run-sql examples/lake/iceberg/query-lake.sql || true)
+ printf '%s\n' "$iceberg_output"
+ if grep -Eq '3.*6.*3' <<<"$iceberg_output"; then
Review Comment:
Could we assert the complete expected result row, including column boundaries, instead of only matching `3.*6.*3`? The current pattern can match unrelated output, such as an error message containing job ID `3e6d3`. Combined with `|| true` ignoring the query's exit status, this can make a failed query pass the smoke test. We should require query success and match the full row (allowing for whitespace differences).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Add a local development kit for contributors [fluss]
Posted by "Gezi-lzq (via GitHub) via issues" <is...@fluss.apache.org>.
Gezi-lzq commented on code in PR #3851:
URL: https://github.com/apache/fluss/pull/3851#discussion_r4482873252
##########
.github/workflows/devkit-smoke.yaml:
##########
@@ -0,0 +1,175 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+name: DevKit Smoke
+
+permissions:
+ contents: read
+
+on:
+ pull_request:
+ paths:
+ - 'devkit/**'
+ - 'pom.xml'
+ - '.github/workflows/devkit-smoke.yaml'
+ push:
+ branches:
+ - main
+ - release-**
+ paths:
+ - 'devkit/**'
+ - 'pom.xml'
+ - '.github/workflows/devkit-smoke.yaml'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }}
+ cancel-in-progress: true
+
+jobs:
+ smoke:
+ runs-on: ubuntu-latest
+ timeout-minutes: 45
+ steps:
+ - uses: actions/checkout@v6
+
+ - name: Set up JDK 11
+ uses: actions/setup-java@v5
+ with:
+ java-version: '11'
+ distribution: temurin
+
+ - name: Install just
+ uses: taiki-e/install-action@v2
+ with:
+ tool: just@1.58.0
+
+ - name: Validate managed profile dependency versions
+ run: bash devkit/validate-managed-dependency-versions.sh
+
+ - name: Core smoke
+ working-directory: devkit
+ run: |
+ set -euo pipefail
+ just build core
+ just up core
+ just run-sql examples/core/setup.sql
+ core_output=$(just run-sql examples/core/batch-scan.sql)
+ printf '%s\n' "$core_output"
+ grep -Fq 'Alice' <<<"$core_output"
+ grep -Fq 'Bob' <<<"$core_output"
+ grep -Fq 'alpha' <<<"$core_output"
+ grep -Fq 'beta' <<<"$core_output"
+ just clean
+
+ - name: Gateway smoke
+ working-directory: devkit
+ run: |
+ set -euo pipefail
+ just build gateway
+ just up gateway
+ curl -fsS -X POST \
+ -H 'Content-Type: application/json' \
+ http://localhost:8080/v1/clusters/default/databases/fluss/tables \
+ -d '{
+ "table_name": "gateway_events",
+ "columns": [
+ {"name": "event_id", "data_type": {"type": "BIGINT"}, "nullable": false},
+ {"name": "message", "data_type": {"type": "STRING"}, "nullable": false}
+ ],
+ "distribution": {"bucket_count": 1, "bucket_keys": []}
+ }'
+ curl -fsS \
+ http://localhost:8080/v1/clusters/default/databases/fluss/tables/gateway_events
+ curl -fsS -X POST \
+ -H 'Content-Type: application/json' \
+ http://localhost:8080/v1/clusters/default/databases/fluss/tables/gateway_events/records \
+ -d '{
+ "entries": [
+ {"id": "event-1", "append": {"event_id": "1", "message": "hello from Gateway"}}
+ ]
+ }'
+ just clean
+
+ - name: Paimon smoke
+ working-directory: devkit
+ run: |
+ set -euo pipefail
+ just build paimon
+ just up paimon
+ just run-sql examples/lake/paimon/setup.sql
+ just run-sql examples/lake/paimon/union-read.sql
+ for attempt in {1..30}; do
+ lake_output=$(just run-sql examples/lake/paimon/lake-only-read.sql || true)
+ printf '%s\n' "$lake_output"
+ if grep -Fq 'Alice' <<<"$lake_output" && grep -Fq 'Bob' <<<"$lake_output"; then
+ break
+ fi
+ if [ "$attempt" -eq 30 ]; then
+ echo 'Paimon lake-only query did not return Alice and Bob' >&2
+ just logs tiering-submit 200
+ exit 1
+ fi
+ sleep 2
+ done
+ just clean
+
+ - name: Iceberg smoke
+ working-directory: devkit
+ run: |
+ set -euo pipefail
+ just build iceberg
+ just up iceberg
+ just run-sql examples/lake/iceberg/create-table.sql
+ just run-sql examples/lake/iceberg/write-data.sql
+ iceberg_output=''
+ for attempt in {1..30}; do
+ iceberg_output=$(just run-sql examples/lake/iceberg/query-lake.sql || true)
+ printf '%s\n' "$iceberg_output"
+ if grep -Eq '3.*6.*3' <<<"$iceberg_output"; then
Review Comment:
Get, I originally added these checks to catch DevKit profiles that stop working as the project changes. I’ve moved the scenarios into a small unittest script with shared setup and cleanup, so we can add focused checks when needed. For this PR, I’m keeping them as smoke checks rather than expanding them into a broader E2E suite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org