You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "Yohahaha (via GitHub)" <gi...@apache.org> on 2026/10/15 06:44:09 UTC
[PR] [build] Run only Spark CI for Spark-only pull requests [fluss]
Yohahaha opened a new pull request, #4654:
URL: https://github.com/apache/fluss/pull/4654
<!--
Generated-by: OpenAI Codex (GPT-6) following https://github.com/apache/fluss/blob/main/AGENTS.md
-->
### Purpose
Run only Spark CI for pull requests whose entire diff is confined to `fluss-spark/`, avoiding unrelated Java 8 compilation, core, Flink, and lake CI stages. Keep full CI for mixed changes and push events, and preserve the nightly Scala 2.13 matrix.
Linked issue: None identified; this is a CI optimization.
### Brief change log
- Detect Spark-only pull requests using the paginated pull request file API with read-only permissions. Treat cross-module renames and incomplete file lists as full CI changes.
- Skip the Java 8 compilation job for Spark-only pull requests and pass a boolean input to the reusable workflow to select Spark stages.
- Preserve Maven test profiles and the optional Spark Scala 2.13 stage.
- Temporary source comments are used to verify Spark-only and mixed Spark/client job selection and will be removed after validation.
### Tests
- `actionlint -shellcheck= .github/workflows/ci.yaml .github/workflows/ci-template.yaml .github/workflows/nightly.yaml` passed.
- Thirteen local routing cases passed, including mixed changes, cross-module renames, pagination, empty diffs, and truncated API results.
- All four Spark-only/Scala 2.13 matrix combinations passed.
- `./mvnw -B --no-transfer-progress -N verify` passed (root Checkstyle, Spotless, and RAT checks; no Java tests in the root project).
- Full-reactor `./mvnw -B --no-transfer-progress validate spotless:check` passed for all 48 modules.
- Full `./mvnw verify` was attempted and stopped after stale server test bytecode produced `IncompatibleClassChangeError` in `RemoteLogTabletTest`. A clean targeted recheck is in progress. No application code is changed by this PR.
- Live GitHub Actions trigger verification is in progress. The main PR includes workflow changes and therefore correctly selects full CI. An isolated fork PR will verify Spark-only and mixed module diffs against a base containing the workflow changes.
### API and Format
No public API, serialization, or storage format changes. The reusable CI workflow gains an optional `spark-only` boolean input that defaults to `false`.
### Documentation
No user documentation changes are required.
Generative AI disclosure: OpenAI Codex (GPT-6) assisted with implementation and validation. Human review is pending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Extract Spark CI into a separate workflow [fluss]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on PR #4654:
URL: https://github.com/apache/fluss/pull/4654#issuecomment-6197435002
@wuchong @luoyuxia @fresh-borzoni PTAL!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Extract Spark CI into a separate workflow [fluss]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni merged PR #4654:
URL: https://github.com/apache/fluss/pull/4654
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Extract Spark CI into a separate workflow [fluss]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on code in PR #4654:
URL: https://github.com/apache/fluss/pull/4654#discussion_r4291047556
##########
.github/workflows/ci-spark.yaml:
##########
@@ -0,0 +1,93 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+name: Spark Connector CI
+
+permissions:
+ contents: read
+
+on:
+ push:
+ branches:
+ - main
+ - release-**
+ - ci-**
+ paths:
+ - 'fluss-spark/**'
+ - 'fluss-client/**'
+ - 'fluss-common/**'
+ - 'fluss-rpc/**'
+ - 'fluss-server/**'
+ - 'fluss-test-utils/**'
+ - 'fluss-metrics/**'
+ - 'fluss-protogen/**'
+ - 'fluss-lake/**'
+ - 'fluss-flink/**'
+ - 'fluss-filesystems/**'
+ - 'fluss-test-coverage/**'
+ - 'fluss-docgen/**'
+ - 'pom.xml'
+ - '.mvn/**'
+ - 'mvnw'
+ - 'tools/maven/**'
+ - 'tools/ci/**'
+ - '.scalafmt.conf'
+ - 'copyright.txt'
+ - '.github/workflows/ci-spark.yaml'
+ - '.github/workflows/ci-template.yaml'
+ - '.github/workflows/stage.sh'
+ - '!**/*.md'
+
+ pull_request:
+ paths:
+ - 'fluss-spark/**'
+ - 'fluss-client/**'
+ - 'fluss-common/**'
+ - 'fluss-rpc/**'
+ - 'fluss-server/**'
+ - 'fluss-test-utils/**'
+ - 'fluss-metrics/**'
+ - 'fluss-protogen/**'
+ - 'fluss-lake/**'
+ - 'fluss-flink/**'
+ - 'fluss-filesystems/**'
+ - 'fluss-test-coverage/**'
+ - 'fluss-docgen/**'
+ - 'pom.xml'
+ - '.mvn/**'
+ - 'mvnw'
+ - 'tools/maven/**'
+ - 'tools/ci/**'
+ - '.scalafmt.conf'
+ - 'copyright.txt'
+ - '.github/workflows/ci-spark.yaml'
+ - '.github/workflows/ci-template.yaml'
+ - '.github/workflows/stage.sh'
+ - '!**/*.md'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }}
+ cancel-in-progress: true
+
+jobs:
+ build-on-jdk11:
+ name: "Java 11"
+ uses: ./.github/workflows/ci-template.yaml
+ with:
+ java-version: "11"
+ modules: '["spark3","spark3-lake","spark3-scala213"]'
Review Comment:
done, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [build] Extract Spark CI into a separate workflow [fluss]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on code in PR #4654:
URL: https://github.com/apache/fluss/pull/4654#discussion_r4289307072
##########
.github/workflows/ci-spark.yaml:
##########
@@ -0,0 +1,93 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+name: Spark Connector CI
+
+permissions:
+ contents: read
+
+on:
+ push:
+ branches:
+ - main
+ - release-**
+ - ci-**
+ paths:
+ - 'fluss-spark/**'
+ - 'fluss-client/**'
+ - 'fluss-common/**'
+ - 'fluss-rpc/**'
+ - 'fluss-server/**'
+ - 'fluss-test-utils/**'
+ - 'fluss-metrics/**'
+ - 'fluss-protogen/**'
+ - 'fluss-lake/**'
+ - 'fluss-flink/**'
+ - 'fluss-filesystems/**'
+ - 'fluss-test-coverage/**'
+ - 'fluss-docgen/**'
+ - 'pom.xml'
+ - '.mvn/**'
+ - 'mvnw'
+ - 'tools/maven/**'
+ - 'tools/ci/**'
+ - '.scalafmt.conf'
+ - 'copyright.txt'
+ - '.github/workflows/ci-spark.yaml'
+ - '.github/workflows/ci-template.yaml'
+ - '.github/workflows/stage.sh'
+ - '!**/*.md'
+
+ pull_request:
+ paths:
+ - 'fluss-spark/**'
+ - 'fluss-client/**'
+ - 'fluss-common/**'
+ - 'fluss-rpc/**'
+ - 'fluss-server/**'
+ - 'fluss-test-utils/**'
+ - 'fluss-metrics/**'
+ - 'fluss-protogen/**'
+ - 'fluss-lake/**'
+ - 'fluss-flink/**'
+ - 'fluss-filesystems/**'
+ - 'fluss-test-coverage/**'
+ - 'fluss-docgen/**'
+ - 'pom.xml'
+ - '.mvn/**'
+ - 'mvnw'
+ - 'tools/maven/**'
+ - 'tools/ci/**'
+ - '.scalafmt.conf'
+ - 'copyright.txt'
+ - '.github/workflows/ci-spark.yaml'
+ - '.github/workflows/ci-template.yaml'
+ - '.github/workflows/stage.sh'
+ - '!**/*.md'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }}
+ cancel-in-progress: true
+
+jobs:
+ build-on-jdk11:
+ name: "Java 11"
+ uses: ./.github/workflows/ci-template.yaml
+ with:
+ java-version: "11"
+ modules: '["spark3","spark3-lake","spark3-scala213"]'
Review Comment:
The flaky test reporter only looks at ci.yaml and nightly.yaml (tools/ci/unstable_test_reporter.py:56). Could we add ci-spark.yaml there, so Spark failures on main still get reported?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org