You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by yh...@apache.org on 2026/09/03 16:01:41 UTC
(beam) branch master updated: Update IO support matrix after Beam 2.76.0 release (#39982)
This is an automated email from the ASF dual-hosted git repository.
Abacn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new ffcbd7e464f Update IO support matrix after Beam 2.76.0 release (#39982)
ffcbd7e464f is described below
commit ffcbd7e464f8244bd623d23e7fa9fcc750e81bbb
Author: Yi Hu <ya...@google.com>
AuthorDate: Thu Sep 3 12:01:32 2026 -0400
Update IO support matrix after Beam 2.76.0 release (#39982)
* Update IO support matrix after Beam 2.76.0 release
* Fix CHANGES.md wrong version
* Fix Yamldoc empty for certain transforms due to expansion service is not present
* Remove item that not affect any release
---
CHANGES.md | 10 +-
sdks/python/build.gradle | 8 +
website/www/site/content/en/blog/beam-2.76.0.md | 4 -
.../site/content/en/documentation/io/connectors.md | 181 +++++++--------------
website/www/site/data/capability_matrix.yaml | 1 -
5 files changed, 71 insertions(+), 133 deletions(-)
diff --git a/CHANGES.md b/CHANGES.md
index 365605c7065..f0d5d06b9d9 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -76,6 +76,9 @@
* (Java/Python) `Watch` can bound its deduplication state by event time, retiring an output key once the greatest emitted timestamp has moved more than the allowed lateness past it. Java adds `Watch.growthOf(...).withTimestampCursor()`. Python adds `allowed_lateness` for the existing `timestamp_cursor` option ([#18459](https://github.com/apache/beam/issues/18459)).
* (Java) Spark Structured Streaming runner: stateful ParDo with state, timers, `@RequiresTimeSortedInput` and tagged outputs is now supported in batch mode ([#39779](https://github.com/apache/beam/issues/39779)).
* (Python) Added support for Vertex AI Model Monitoring V2 in RunInference ([#39738](https://github.com/apache/beam/issues/39738)).
+* [Flink Runner] Added opt-in static round-robin split assignment for small bounded sources via the new `sourceStaticSplitThresholdMb` pipeline option. The default of 0 keeps the existing lazy pull-based assignment ([#39873](https://github.com/apache/beam/issues/39873)).
+* Added automatic caching of bounded, single-pane side-input views for classic Java Flink DataStream execution ([#39866](https://github.com/apache/beam/issues/39866)).
+* (Python) Added `Sample.Any`, the Python equivalent of Java's `Sample.any`, which returns up to n arbitrary elements from a PCollection ([#18552](https://github.com/apache/beam/issues/18552)).
## Breaking Changes
@@ -89,7 +92,6 @@
## Bugfixes
-* (Java) Restored binary compatibility for `CoderTranslatorRegistrar` implementations compiled against Beam 2.76 and earlier ([#38714](https://github.com/apache/beam/issues/38714)).
* (Java) Fixed the Spark runner firing processing-time timers in reverse timestamp order ([#39824](https://github.com/apache/beam/issues/39824)).
* (Python) Fixed incorrect profiler options handling on portable runners ([#39613](https://github.com/apache/beam/issues/39613)).
* (Java) KafkaIO dynamic reads no longer require the obsolete `beam_fn_api` experiment ([#29998](https://github.com/apache/beam/issues/29998)).
@@ -124,8 +126,6 @@
## New Features / Improvements
-* [Flink Runner] Added opt-in static round-robin split assignment for small bounded sources via the new `sourceStaticSplitThresholdMb` pipeline option. The default of 0 keeps the existing lazy pull-based assignment ([#39873](https://github.com/apache/beam/issues/39873)).
-* Added automatic caching of bounded, single-pane side-input views for classic Java Flink DataStream execution ([#39866](https://github.com/apache/beam/issues/39866)).
* Added `GroupIntoBatches` transform and the standard
`beam:coder:sharded_key:v1` coder to the Go SDK, along with
`beam.Coder.IsDeterministic`, `beam.PCollection.WindowingStrategy`,
@@ -146,7 +146,6 @@
* (Python) Added `Watch`, a transform that polls a growing set of outputs for each input element, deduplicates outputs across poll rounds, and stops per a user-supplied termination condition
([#21521](https://github.com/apache/beam/issues/21521)).
* (Python) Added support to analyze core dumps created after python worker segmentation faults with `pystack` (or `gdb` if installed) using the `--profiler_agent=coredump` pipeline option. ([#39484](https://github.com/apache/beam/issues/39484)).
-* (Python) Added `Sample.Any`, the Python equivalent of Java's `Sample.any`, which returns up to n arbitrary elements from a PCollection ([#18552](https://github.com/apache/beam/issues/18552)).
## Breaking Changes
@@ -161,9 +160,6 @@
Use pipeline option `--updateCompatibilityVersion=2.75.0` (or any older version) to keep the old behavior ([#39344](https://github.com/apache/beam/issues/39344)).
* `DoFn.process` returning a `str`, `bytes`, or `dict` (instead of an iterable wrapping one) now raises a `TypeError` rather than silently iterating per-character/byte/key (Python) ([#18712](https://github.com/apache/beam/issues/18712)).
* (Java) Added `DRAINING` and `DRAINED` states to `PipelineResult`, including runner state mappings and Dataflow update handling ([#39020](https://github.com/apache/beam/issues/39020)).
-* (Python) Typehints of dataclass fields are honored during type inferences. To restore the behavior of fallback-to-any,
- use pipeline option `--exclude_infer_dataclass_field_type` ([#38797](https://github.com/apache/beam/issues/38797)).
- However fixing forward is recommended.
* (Java) IcebergIO and projects that use it must now be built with Java 17 or later as a result of Iceberg 1.11.0 upgrade ([#38925](https://github.com/apache/beam/issues/38925)).
## Bugfixes
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index 75afa5d7c96..339d87c76cb 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -107,16 +107,24 @@ tasks.register("prepareExpansionServicesForYamlDocs") {
dependsOn ":sdks:java:extensions:schemaio-expansion-service:shadowJar"
dependsOn ":sdks:java:extensions:sql:expansion-service:shadowJar"
+ dependsOn ":sdks:java:io:amazon-web-services2:expansion-service:shadowJar"
+ dependsOn ":sdks:java:io:debezium:expansion-service:shadowJar"
dependsOn ":sdks:java:io:expansion-service:shadowJar"
dependsOn ":sdks:java:io:google-cloud-platform:expansion-service:shadowJar"
+ dependsOn ":sdks:java:io:messaging-expansion-service:shadowJar"
+ dependsOn ":sdks:java:io:snowflake:expansion-service:shadowJar"
doLast {
// Copy expansion service jar into cache path (.apache_beam/cache/jars/).
copy {
from project.tasks.findByPath(":sdks:java:extensions:schemaio-expansion-service:shadowJar")
from project.tasks.findByPath(":sdks:java:extensions:sql:expansion-service:shadowJar")
+ from project.tasks.findByPath(":sdks:java:io:amazon-web-services2:expansion-service:shadowJar")
+ from project.tasks.findByPath(":sdks:java:io:debezium:expansion-service:shadowJar")
from project.tasks.findByPath(":sdks:java:io:expansion-service:shadowJar")
from project.tasks.findByPath(":sdks:java:io:google-cloud-platform:expansion-service:shadowJar")
+ from project.tasks.findByPath(":sdks:java:io:messaging-expansion-service:shadowJar")
+ from project.tasks.findByPath(":sdks:java:io:snowflake:expansion-service:shadowJar")
into "${System.getProperty('user.home')}/.apache_beam/cache/jars/"
}
}
diff --git a/website/www/site/content/en/blog/beam-2.76.0.md b/website/www/site/content/en/blog/beam-2.76.0.md
index cea653eaade..240c961cbd9 100644
--- a/website/www/site/content/en/blog/beam-2.76.0.md
+++ b/website/www/site/content/en/blog/beam-2.76.0.md
@@ -63,7 +63,6 @@ For more information on changes in 2.76.0, check out the [detailed release notes
* (Python) Added `Watch`, a transform that polls a growing set of outputs for each input element, deduplicates outputs across poll rounds, and stops per a user-supplied termination condition
([#21521](https://github.com/apache/beam/issues/21521)).
* (Python) Added support to analyze core dumps created after python worker segmentation faults with `pystack` (or `gdb` if installed) using the `--profiler_agent=coredump` pipeline option. ([#39484](https://github.com/apache/beam/issues/39484)).
-* (Python) Added `Sample.Any`, the Python equivalent of Java's `Sample.any`, which returns up to n arbitrary elements from a PCollection ([#18552](https://github.com/apache/beam/issues/18552)).
### Breaking Changes
@@ -78,9 +77,6 @@ For more information on changes in 2.76.0, check out the [detailed release notes
Use pipeline option `--updateCompatibilityVersion=2.75.0` (or any older version) to keep the old behavior ([#39344](https://github.com/apache/beam/issues/39344)).
* `DoFn.process` returning a `str`, `bytes`, or `dict` (instead of an iterable wrapping one) now raises a `TypeError` rather than silently iterating per-character/byte/key (Python) ([#18712](https://github.com/apache/beam/issues/18712)).
* (Java) Added `DRAINING` and `DRAINED` states to `PipelineResult`, including runner state mappings and Dataflow update handling ([#39020](https://github.com/apache/beam/issues/39020)).
-* (Python) Typehints of dataclass fields are honored during type inferences. To restore the behavior of fallback-to-any,
- use pipeline option `--exclude_infer_dataclass_field_type` ([#38797](https://github.com/apache/beam/issues/38797)).
- However fixing forward is recommended.
* (Java) IcebergIO and projects that use it must now be built with Java 17 or later as a result of Iceberg 1.11.0 upgrade ([#38925](https://github.com/apache/beam/issues/38925)).
### Bugfixes
diff --git a/website/www/site/content/en/documentation/io/connectors.md b/website/www/site/content/en/documentation/io/connectors.md
index 679b6bf1e0e..f3078c6b378 100644
--- a/website/www/site/content/en/documentation/io/connectors.md
+++ b/website/www/site/content/en/documentation/io/connectors.md
@@ -429,7 +429,10 @@ This table provides a consolidated, at-a-glance overview of the available built-
✔
<a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jms/JmsIO.html">native</a>
</td>
- <td>Not available</td>
+ <td class="present">
+ ✔
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.xlang.io.html#apache_beam.transforms.xlang.io.ReadFromJms">via X-language</a>
+ </td>
<td>Not available</td>
<td>Not available</td>
<td>Not available</td>
@@ -790,7 +793,11 @@ This table provides a consolidated, at-a-glance overview of the available built-
<a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang/debeziumio">via X-language</a>
</td>
<td>Not available</td>
- <td>Not available</td>
+ <td class="present">
+ ✔
+ <br>
+ <a href="https://beam.apache.org/releases/yamldoc/current/#readfromdebezium">read</a>
+ </td>
<td class="present">✔</td>
<td class="present">✔</td>
<td class="absent">✘</td>
@@ -1322,18 +1329,25 @@ This table provides a consolidated, at-a-glance overview of the available built-
<tr>
<td>Delta Lake</td>
<td class="present">✔</td>
- <td class="present">✘</td>
+ <td class="absent">✘</td>
<td class="present">
✔
<a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/managed/Managed.html">native</a>
</td>
+ <td class="present">
+ ✔
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.managed.html#module-apache_beam.transforms.managed">via X-language</a>
+ </td>
<td>Not available</td>
<td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
+ <td class="present">
+ ✔
+ <br>
+ <a href="https://beam.apache.org/releases/yamldoc/current/#readfromdelta">read</a>
+ </td>
<td class="present">✔</td>
<td class="absent">✘</td>
- <td class="absent">✔</td>
+ <td class="present">✔</td>
</tr>
</table>
</div>
@@ -1346,205 +1360,130 @@ This table provides a consolidated, at-a-glance overview of the available built-
<th>Connector Name</th>
<th>Source Supported</th>
<th>Sink Supported</th>
- <th>Java</th>
- <th>Python</th>
- <th>Go</th>
- <th>Typescript</th>
- <th>Yaml</th>
+ <th>Language</th>
<th>Batch Supported</th>
<th>Streaming Supported</th>
</tr>
<tr>
<td>
- <a href="https://github.com/google/hana-bq-beam-connector">SAP Hana to Google BigQuery</a>
+ <a href="https://cloud.google.com/bigtable/docs/hbase-dataflow-java">Cloud Bigtable (HBase based)</a>
</td>
<td class="present">✔</td>
<td class="present">✔</td>
- <td class="present">
- ✔
- native
- </td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
+ <td>Java</td>
<td class="present">✔</td>
<td class="absent">✘</td>
</tr>
<tr>
<td>
- <a href="https://github.com/esakik/beam-mysql-connector">MySQL</a>
+ <a href="https://github.com/cognitedata/cdf-beam-connector-java">Cognite Data Fusion</a>
</td>
<td class="present">✔</td>
<td class="present">✔</td>
- <td>Not available</td>
- <td class="present">
- ✔
- native
- </td>
- <td>Not available</td>
- <td>Not available</td>
- <td class="present">
- ✔
- <br>
- <a href="https://beam.apache.org/releases/yamldoc/current/#readfrommysql">read</a> / <a href="https://beam.apache.org/releases/yamldoc/current/#writetomysql">write</a>
- </td>
+ <td>Java</td>
+ <td class="present">✔</td>
<td class="present">✔</td>
- <td class="absent">✘</td>
</tr>
<tr>
<td>
- <a href="https://github.com/Refinitiv/trep-websockets-beam-io">TrepWsIO</a>
+ <a href="https://github.com/kineticadb/kinetica-connector-beam">KineticaDB</a>
</td>
<td class="present">✔</td>
- <td class="absent">✘</td>
- <td class="present">
- ✔
- native
- </td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
<td class="present">✔</td>
+ <td>Java</td>
<td class="present">✔</td>
+ <td class="absent">✘</td>
</tr>
<tr>
<td>
- <a href="https://github.com/kineticadb/kinetica-connector-beam">KineticaDB</a>
+ <a href="https://github.com/google/hana-bq-beam-connector">SAP Hana to Google BigQuery</a>
</td>
<td class="present">✔</td>
<td class="present">✔</td>
- <td class="present">
- ✔
- native
- </td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
+ <td>Java</td>
<td class="present">✔</td>
<td class="absent">✘</td>
</tr>
<tr>
<td>
- <a href="https://github.com/cognitedata/cdf-beam-connector-java">Cognite Data Fusion</a>
+ <a href="https://github.com/Refinitiv/trep-websockets-beam-io">TrepWsIO</a>
</td>
<td class="present">✔</td>
- <td class="present">✔</td>
- <td class="present">
- ✔
- native
- </td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
+ <td class="absent">✘</td>
+ <td>Java</td>
<td class="present">✔</td>
<td class="present">✔</td>
</tr>
<tr>
<td>
- <a href="https://github.com/314e/beam-pyodbc-connector">Pyodbc</a>
+ <a href="https://github.com/beam-pyio">Beam PyIO (Collection of Python IO connectors)</a>
</td>
<td class="present">✔</td>
<td class="present">✔</td>
- <td>Not available</td>
- <td class="present">
- ✔
- native
- </td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
+ <td>Python</td>
+ <td class="present">✔</td>
<td class="present">✔</td>
- <td class="absent">✘</td>
</tr>
<tr>
<td>
- <a href="https://github.com/amient/goconnect">Go Connect</a>
+ <a href="https://github.com/jccatrinck/dataflow-cloud-sql-python">Cloud SQL</a>
</td>
<td class="present">✔</td>
- <td class="present">✔</td>
<td class="absent">✘</td>
- <td class="absent">✘</td>
- <td class="present">
- ✔
- native
- </td>
- <td>Not available</td>
- <td>Not available</td>
- <td class="present">✔</td>
+ <td>Python</td>
<td class="present">✔</td>
+ <td class="absent">✘</td>
</tr>
<tr>
<td>
- <a href="https://github.com/tinybirdco/tinybird-beam">Tinybird</a>
+ <a href="https://github.com/esakik/beam-mysql-connector">MySQL</a>
</td>
- <td class="absent">✘</td>
<td class="present">✔</td>
- <td>Not available</td>
- <td class="present">
- ✔
- native
- </td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
<td class="present">✔</td>
+ <td>Python</td>
<td class="present">✔</td>
+ <td class="absent">✘</td>
</tr>
<tr>
<td>
- <a href="https://github.com/jccatrinck/dataflow-cloud-sql-python">Cloud SQL</a>
+ <a href="https://github.com/314e/beam-pyodbc-connector">Pyodbc</a>
</td>
<td class="present">✔</td>
- <td class="absent">✘</td>
- <td>Not available</td>
- <td class="present">
- ✔
- native
- </td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
+ <td class="present">✔</td>
+ <td>Python</td>
<td class="present">✔</td>
<td class="absent">✘</td>
</tr>
<tr>
<td>
- <a href="https://cloud.google.com/bigtable/docs/hbase-dataflow-java">Cloud Bigtable (HBase based)</a>
+ <a href="https://github.com/tinybirdco/tinybird-beam">Tinybird</a>
</td>
+ <td class="absent">✘</td>
<td class="present">✔</td>
+ <td>Python</td>
<td class="present">✔</td>
- <td class="present">
- ✔
- native
- </td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
<td class="present">✔</td>
- <td class="absent">✘</td>
</tr>
<tr>
<td>
- <a href="https://github.com/beam-pyio">Beam PyIO (Collection of Python IO connectors)</a>
+ <a href="https://github.com/amient/goconnect">Go Connect</a>
</td>
<td class="present">✔</td>
<td class="present">✔</td>
- <td>Not available</td>
- <td class="present">
- ✔
- native
+ <td>Go</td>
+ <td class="present">✔</td>
+ <td class="present">✔</td>
+ </tr>
+ <tr>
+ <td>
+ <a href="https://beam.apache.org/releases/yamldoc/current/#readfrommysql">MySQL, PostgreSQL, Oracle, SQL Server</a>
</td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
<td class="present">✔</td>
<td class="present">✔</td>
+ <td>Yaml</td>
+ <td class="present">✔</td>
+ <td class="absent">✘</td>
</tr>
</table>
</div>
diff --git a/website/www/site/data/capability_matrix.yaml b/website/www/site/data/capability_matrix.yaml
index e4db68c0cbe..3f3bf39594c 100644
--- a/website/www/site/data/capability_matrix.yaml
+++ b/website/www/site/data/capability_matrix.yaml
@@ -1736,7 +1736,6 @@ capability-matrix:
l1: "Unverified"
l2:
l3:
-
- class: kafka-streams
l1: "No"
l2: not implemented