You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "stankiewicz (via GitHub)" <gi...@apache.org> on 2026/10/07 14:47:25 UTC
[PR] Export runtimeOnly dependencies to Maven POM for unshaded modules [beam]
stankiewicz opened a new pull request, #40450:
URL: https://github.com/apache/beam/pull/40450
### Summary
`BeamModulePlugin.groovy` manually constructs the `<dependencies>` section of published Maven POMs in `pom.withXml` (rather than using `from components.java` due to Beam's shading model) and previously only exported `shadow`/`implementation` (`compile` scope) and `provided` (`provided` scope). As a result, `runtimeOnly` dependencies in unshaded modules were silently omitted from published POMs, requiring either `implementation` + `permitUnusedDeclared` workarounds or forcing downstream Maven users to manually redeclare runtime dependencies.
Simply calling `generateDependenciesFromConfiguration(configuration: 'runtimeOnly', scope: 'runtime')` had three blockers:
1. **Shaded modules (`shadowClosure`):** Shaded modules bundle `runtimeClasspath` into `shadowJar`, so exporting `runtimeOnly` would leak shaded dependencies into the POM.
2. **Configuration inheritance (`runtimeOnly.extendsFrom(provided)`):** Iterating `runtimeOnly.allDependencies` without deduplication duplicates every `provided` dependency (and BOM) under `<scope>runtime</scope>`.
3. **Unpublished `ProjectDependency` targets & Configure-on-Demand:** Some modules declare `runtimeOnly` dependencies on internal subprojects with `publish: false` (`includeInJavaBom == false`), and running `generatePomFileForMavenJavaPublication` with `org.gradle.configureondemand=true` leaves `ProjectDependency` target subprojects unconfigured unless evaluated before POM generation.
### Changes in `BeamModulePlugin.groovy`
1. **Unshaded `runtimeOnly` export:** Export `runtimeOnly` with `<scope>runtime</scope>` only when `!configuration.shadowClosure`.
2. **Cross-scope and BOM deduplication:** Track emitted `[groupId, artifactId, classifiers]` coordinates (`addedDependencies`) and BOM `[group, name]` coordinates (`addedBoms`) across `compile`, `provided`, and `runtime` scopes so `provided` or `compile` dependencies and BOMs are never duplicated in `<scope>runtime</scope>`.
3. **Unpublished `ProjectDependency` filtering & Configure-on-Demand safety:**
- Skip `ProjectDependency` entries where `dependencyProject.ext.has('includeInJavaBom') && !dependencyProject.ext.includeInJavaBom` (`publish: false`).
- Attach a lightweight `evaluateForPom` task dependency to `generatePomFileForMavenJavaPublication` for `ProjectDependency` targets in published configurations so they are configured on demand without triggering circular `evaluationDependsOn` errors.
### Impact on Published POMs (`2.78.0-SNAPSHOT` diff audit)
- **`beam-sdks-java-io-iceberg` (`:sdks:java:io:iceberg`):**
- Now exports its 8 `runtimeOnly` dependencies under `<scope>runtime</scope>`: `iceberg-gcp`, `iceberg-aws`, `iceberg-aws-bundle`, `iceberg-azure`, `iceberg-azure-bundle`, `gcs-connector`, `util-hadoop`, and `hadoop-client`.
- `org.immutables:value` remains once under `<scope>provided</scope>` (no duplication).
- **`beam-runners-google-cloud-dataflow-java` (`:runners:google-cloud-dataflow-java`):**
- Allows `opentelemetry-exporter-otlp`, `opentelemetry-sdk-extension-autoconfigure`, and `beam-sdks-java-extensions-opentelemetry-gcp-auth-extension` to be declared as `runtimeOnly` (exported under `<scope>runtime</scope>`) without `implementation` + `permitUnusedDeclared` workarounds.
- `beam-sdks-java-io-kafka` remains once under `<scope>provided</scope>`, and `com.google.cloud:libraries-bom` appears once in `<dependencyManagement>`.
- **`beam-sdks-java-extensions-sql-iceberg` (`:sdks:java:extensions:sql:iceberg`):**
- **No POM diff:** Unpublished `runtimeOnly` project dependencies (`:sdks:java:io:iceberg:bqms` and `:sdks:java:io:iceberg:hive`, which have `publish: false`) are automatically excluded from the POM.
- **Shaded modules (e.g., `beam-sdks-java-core`, expansion service fat JARs, Dataflow worker):**
- **No POM diff:** Guarded by `!configuration.shadowClosure`.
- **Other unshaded published modules declaring `runtimeOnly`:**
- `beam-sdks-java-io-delta`: Exports `org.apache.hadoop:hadoop-aws` under `<scope>runtime</scope>`.
- `beam-sdks-java-io-pulsar`: Exports `org.apache.pulsar:pulsar-client` and `org.apache.pulsar:pulsar-client-admin` under `<scope>runtime</scope>`.
- `beam-sdks-java-io-azure` & `beam-sdks-java-io-azure-cosmos`: Export `io.netty:netty-all` under `<scope>runtime</scope>`.
- `beam-sdks-java-expansion-service`: Exports `com.github.jbellis:jamm` and `org.slf4j:slf4j-jdk14` under `<scope>runtime</scope>`.
- `beam-examples-java`: Exports `beam-runners-direct-java`, `hadoop-client`, and `gcs-connector` under `<scope>runtime</scope>`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Export runtimeOnly dependencies to Maven POM for unshaded modules [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #40450:
URL: https://github.com/apache/beam/pull/40450#issuecomment-6043553029
> Impact on Published POMs
It appears there are scenarios where those dependencies should be `provided` scope, for example, specific implementation (iceberg-gcp, iceberg-aws, iceberg-aws-bundle, iceberg-azure, iceberg-azure-bundle, gcs-connector); client libraries (org.apache.pulsar:pulsar-client, org.apache.pulsar:pulsar-client-admin), as current it is, becomes exported.
And I agree this sounds "adding things when I think we really want to be subtracting thing". As long as "permitUnusedDeclared" works I'm leaning to the current workaround than adding further custom logic + actual changing dependency tree side effect
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Export runtimeOnly dependencies to Maven POM for unshaded modules [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #40450:
URL: https://github.com/apache/beam/pull/40450#issuecomment-6147443669
Note:`permitUnusedDeclared` and `permitUsedUndeclared` are everywhere in the codebase because we added dependency checking and had someone just suppress all the warnings while we take care of them a little at a time. We should assume that these are just signs of incorrect deps, until each case has been analyzed.
Since gradle does not have any concept of `provided` we already roll our own and can use that where we need to (FWIW the reason Gradle doesn't have it is that it is highly questionable). That is different than `runtimeOnly` which maps exactly to maven's `runtime` scope. All the various suppressions are probably allowing a lot of deps to just be specified wrong and have surprising downstream effects once in a while.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Export runtimeOnly dependencies to Maven POM for unshaded modules [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on code in PR #40450:
URL: https://github.com/apache/beam/pull/40450#discussion_r4209128071
##########
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy:
##########
@@ -1972,6 +1969,34 @@ class BeamModulePlugin implements Plugin<Project> {
if ((isRelease(project) || project.hasProperty('publishing')) && configuration.publish) {
project.apply plugin: "maven-publish"
+ // Ensure project dependencies in published configurations are evaluated even when
+ // org.gradle.configureondemand=true is enabled, without calling evaluationDependsOn during
+ // project evaluation (which can trigger circular evaluation between subprojects).
+ if (!project.tasks.names.contains('evaluateForPom')) {
Review Comment:
This line is totally beyond me
##########
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy:
##########
@@ -2145,12 +2170,42 @@ class BeamModulePlugin implements Plugin<Project> {
// BOMs, declared with 'platform' or 'enforced-platform', appear in <dependencyManagement> section
def boms = []
+ def addedBoms = [] as Set
+ // Track emitted (groupId, artifactId, classifiers) coordinates across scopes so that
+ // configuration inheritance (e.g. runtimeOnly.extendsFrom(provided)) or duplicate
+ // declarations across configurations do not produce duplicate <dependency> nodes.
+ def addedDependencies = [] as Set
def generateDependenciesFromConfiguration = { param ->
project.configurations."${param.configuration}".allDependencies.each {
String category = it.getAttributes().getAttribute(Category.CATEGORY_ATTRIBUTE)
if (Category.ENFORCED_PLATFORM == category || Category.REGULAR_PLATFORM == category) {
- boms.add(it)
+ if (addedBoms.add([it.group, it.name])) {
+ boms.add(it)
Review Comment:
This part seems generally useful, I guess it avoids duplicate boms?
##########
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy:
##########
@@ -2145,12 +2170,42 @@ class BeamModulePlugin implements Plugin<Project> {
// BOMs, declared with 'platform' or 'enforced-platform', appear in <dependencyManagement> section
def boms = []
+ def addedBoms = [] as Set
+ // Track emitted (groupId, artifactId, classifiers) coordinates across scopes so that
+ // configuration inheritance (e.g. runtimeOnly.extendsFrom(provided)) or duplicate
+ // declarations across configurations do not produce duplicate <dependency> nodes.
+ def addedDependencies = [] as Set
def generateDependenciesFromConfiguration = { param ->
project.configurations."${param.configuration}".allDependencies.each {
String category = it.getAttributes().getAttribute(Category.CATEGORY_ATTRIBUTE)
if (Category.ENFORCED_PLATFORM == category || Category.REGULAR_PLATFORM == category) {
- boms.add(it)
+ if (addedBoms.add([it.group, it.name])) {
+ boms.add(it)
+ }
+ return
+ }
+
+ if (it instanceof ProjectDependency) {
+ def dependencyProject = it.getDependencyProject()
+ if (dependencyProject.ext.has('includeInJavaBom') && !dependencyProject.ext.includeInJavaBom) {
Review Comment:
I feel like this is hacking into gradle internal magic strings also?
##########
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy:
##########
@@ -2204,6 +2259,9 @@ class BeamModulePlugin implements Plugin<Project> {
generateDependenciesFromConfiguration(
configuration: (configuration.shadowClosure ? 'shadow' : 'implementation'), scope: 'compile')
generateDependenciesFromConfiguration(configuration: 'provided', scope: 'provided')
+ if (!configuration.shadowClosure) {
+ generateDependenciesFromConfiguration(configuration: 'runtimeOnly', scope: 'runtime')
Review Comment:
this kind of seems like the line that actually does the work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Export runtimeOnly dependencies to Maven POM for unshaded modules [beam]
Posted by "stankiewicz (via GitHub)" <gi...@apache.org>.
stankiewicz commented on PR #40450:
URL: https://github.com/apache/beam/pull/40450#issuecomment-6144662189
thanks for the feedback.I've added my workaround so I'm ok leaving it as is, I've suggested @ahmedabu98 to take a look at iceberg dependencies so he can apply similar approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Export runtimeOnly dependencies to Maven POM for unshaded modules [beam]
Posted by "stankiewicz (via GitHub)" <gi...@apache.org>.
stankiewicz closed pull request #40450: Export runtimeOnly dependencies to Maven POM for unshaded modules
URL: https://github.com/apache/beam/pull/40450
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Export runtimeOnly dependencies to Maven POM for unshaded modules [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40450:
URL: https://github.com/apache/beam/pull/40450#issuecomment-6040527140
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Export runtimeOnly dependencies to Maven POM for unshaded modules [beam]
Posted by "stankiewicz (via GitHub)" <gi...@apache.org>.
stankiewicz commented on PR #40450:
URL: https://github.com/apache/beam/pull/40450#issuecomment-6040496334
R: @kennknowles
R: @ahmedabu98 as it introduces some dependencies to iceberg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org