You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Abacn (via GitHub)" <gi...@apache.org> on 2026/05/18 17:02:01 UTC
[PR] Update managed-io.md for release 2.74.0-RC1 [beam]
Abacn opened a new pull request, #38527:
URL: https://github.com/apache/beam/pull/38527
Content generated from release 2.74.0-RC1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update managed-io.md for release 2.74.0-RC1 [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #38527:
URL: https://github.com/apache/beam/pull/38527#discussion_r3260669271
##########
website/www/site/content/en/documentation/io/managed-io.md:
##########
@@ -418,228 +421,228 @@ and Beam SQL is invoked via the Managed API under the hood.
<code style="color: green">str</code>
</td>
<td>
- A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. | Format: host1:port1,host2:port2,...
+ A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form `host1:port1,host2:port2,...`
Review Comment:
The description format for `bootstrap_servers` here is inconsistent with the one used in the `KAFKA Write` section (line 590). It is better to use a consistent format across the documentation.
```suggestion
A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. | Format: host1:port1,host2:port2,...
```
##########
website/www/site/content/en/documentation/io/managed-io.md:
##########
@@ -418,228 +421,228 @@ and Beam SQL is invoked via the Managed API under the hood.
<code style="color: green">str</code>
</td>
<td>
- A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. | Format: host1:port1,host2:port2,...
+ A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form `host1:port1,host2:port2,...`
</td>
</tr>
<tr>
<td>
- <strong>format</strong>
+ <strong>topic</strong>
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- The encoding format for the data stored in Kafka. Valid options are: RAW,JSON,AVRO,PROTO
+ n/a
Review Comment:
The description for the required field `topic` is "n/a". This should be updated with a meaningful description.
```suggestion
The Kafka topic to read from.
```
##########
website/www/site/content/en/documentation/io/managed-io.md:
##########
@@ -809,6 +823,19 @@ and Beam SQL is invoked via the Managed API under the hood.
For a streaming pipeline, sets the limit for lifting bundles into the direct write path.
</td>
</tr>
+ <tr>
+ <td>
+ distribution_mode
+ </td>
+ <td>
+ <code style="color: green">str</code>
+ </td>
+ <td>
+ Defines distribution of write data. Supported distributions:
+- none: don't shuffle rows (default)
+- hash: shuffle rows by partition key before writing data
Review Comment:
The markdown list inside the HTML `<td>` tag might not render correctly because it lacks proper indentation and line breaks. For consistency with other entries in this file, consider using `<br>` for line breaks and ensuring the list items are indented.
```suggestion
Defines distribution of write data. Supported distributions:<br>
- none: don't shuffle rows (default)<br>
- hash: shuffle rows by partition key before writing data
```
##########
website/www/site/content/en/documentation/io/managed-io.md:
##########
@@ -864,6 +891,18 @@ and Beam SQL is invoked via the Managed API under the hood.
For more information on partition transforms, please visit https://iceberg.apache.org/spec/#partition-transforms.
</td>
</tr>
+ <tr>
+ <td>
+ sort_fields
+ </td>
+ <td>
+ <code>list[<span style="color: green;">str</span>]</code>
+ </td>
+ <td>
+ Fields used to set the table's sort order, applied when the table is created. Each entry has the form `<term> [asc|desc] [nulls first|nulls last]`, where `<term>` is a field name or one of the partition transforms (e.g. `bucket(col, 4)`, `day(ts)`). Direction defaults to ascending; null order defaults to nulls-first for ascending and nulls-last for descending. Note: this sets the table's declared sort order as metadata; it does not cause Beam to physically sort records before writing.
+For more information on sort orders, please visit https://iceberg.apache.org/spec/#sort-orders.
Review Comment:
There is a raw newline between the description and the link. This can cause rendering issues within an HTML table cell in some markdown processors. Consider using a `<br>` and maintaining consistent indentation.
```suggestion
Fields used to set the table's sort order, applied when the table is created. Each entry has the form `<term> [asc|desc] [nulls first|nulls last]`, where `<term>` is a field name or one of the partition transforms (e.g. `bucket(col, 4)`, `day(ts)`). Direction defaults to ascending; null order defaults to nulls-first for ascending and nulls-last for descending. Note: this sets the table's declared sort order as metadata; it does not cause Beam to physically sort records before writing.<br>
For more information on sort orders, please visit https://iceberg.apache.org/spec/#sort-orders.
```
##########
website/www/site/content/en/documentation/io/managed-io.md:
##########
@@ -1076,17 +1104,6 @@ For more information on table properties, please visit https://iceberg.apache.or
n/a
Review Comment:
The description for the required field `jdbc_url` is "n/a". This should be updated with a meaningful description.
```suggestion
Connection URL for the JDBC sink.
```
##########
website/www/site/content/en/documentation/io/managed-io.md:
##########
@@ -650,7 +653,7 @@ and Beam SQL is invoked via the Managed API under the hood.
<code style="color: green">str</code>
</td>
<td>
- The schema in which the data is encoded in the Kafka topic. For AVRO data, this is a schema defined with AVRO schema syntax (https://avro.apache.org/docs/1.10.2/spec.html#schemas). For JSON data, this is a schema defined with JSON-schema syntax (https://json-schema.org/). If a URL to Confluent Schema Registry is provided, then this field is ignored, and the schema is fetched from Confluent Schema Registry.
+ n/a
Review Comment:
The description for `schema` has been replaced with "n/a", which is a regression from the previous version. It should be restored to provide necessary context for users.
```suggestion
The schema in which the data is encoded in the Kafka topic. For AVRO data, this is a schema defined with AVRO schema syntax (https://avro.apache.org/docs/1.10.2/spec.html#schemas). For JSON data, this is a schema defined with JSON-schema syntax (https://json-schema.org/). If a URL to Confluent Schema Registry is provided, then this field is ignored, and the schema is fetched from Confluent Schema Registry.
```
##########
website/www/site/content/en/documentation/io/managed-io.md:
##########
@@ -418,228 +421,228 @@ and Beam SQL is invoked via the Managed API under the hood.
<code style="color: green">str</code>
</td>
<td>
- A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. | Format: host1:port1,host2:port2,...
+ A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form `host1:port1,host2:port2,...`
</td>
</tr>
<tr>
<td>
- <strong>format</strong>
+ <strong>topic</strong>
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- The encoding format for the data stored in Kafka. Valid options are: RAW,JSON,AVRO,PROTO
+ n/a
</td>
</tr>
<tr>
<td>
- <strong>topic</strong>
+ allow_duplicates
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: orange">boolean</code>
</td>
<td>
- n/a
+ If the Kafka read allows duplicates.
</td>
</tr>
<tr>
<td>
- file_descriptor_path
+ confluent_schema_registry_subject
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- The path to the Protocol Buffer File Descriptor Set file. This file is used for schema definition and message serialization.
+ n/a
</td>
</tr>
<tr>
<td>
- message_name
+ confluent_schema_registry_url
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- The name of the Protocol Buffer message to be used for schema extraction and data conversion.
+ n/a
</td>
</tr>
<tr>
<td>
- producer_config_updates
+ consumer_config_updates
</td>
<td>
<code>map[<span style="color: green;">str</span>, <span style="color: green;">str</span>]</code>
</td>
<td>
- A list of key-value pairs that act as configuration parameters for Kafka producers. Most of these configurations will not be needed, but if you need to customize your Kafka producer, you may use this. See a detailed list: https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html
+ A list of key-value pairs that act as configuration parameters for Kafka consumers. Most of these configurations will not be needed, but if you need to customize your Kafka consumer, you may use this. See a detailed list: https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html
</td>
</tr>
<tr>
<td>
- schema
+ file_descriptor_path
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- n/a
+ The path to the Protocol Buffer File Descriptor Set file. This file is used for schema definition and message serialization.
</td>
</tr>
- </table>
-</div>
-
-### `KAFKA` Read
-
-<div class="table-container-wrapper">
- <table class="table table-bordered">
- <tr>
- <th>Configuration</th>
- <th>Type</th>
- <th>Description</th>
- </tr>
<tr>
<td>
- <strong>bootstrap_servers</strong>
+ format
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form `host1:port1,host2:port2,...`
+ The encoding format for the data stored in Kafka. Valid options are: RAW,STRING,AVRO,JSON,PROTO
</td>
</tr>
<tr>
<td>
- <strong>topic</strong>
+ message_name
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- n/a
+ The name of the Protocol Buffer message to be used for schema extraction and data conversion.
</td>
</tr>
<tr>
<td>
- allow_duplicates
+ offset_deduplication
</td>
<td>
<code style="color: orange">boolean</code>
</td>
<td>
- If the Kafka read allows duplicates.
+ If the redistribute is using offset deduplication mode.
</td>
</tr>
<tr>
<td>
- confluent_schema_registry_subject
+ redistribute_by_record_key
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: orange">boolean</code>
</td>
<td>
- n/a
+ If the redistribute keys by the Kafka record key.
</td>
</tr>
<tr>
<td>
- confluent_schema_registry_url
+ redistribute_num_keys
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: #f54251">int32</code>
</td>
<td>
- n/a
+ The number of keys for redistributing Kafka inputs.
</td>
</tr>
<tr>
<td>
- consumer_config_updates
+ redistributed
</td>
<td>
- <code>map[<span style="color: green;">str</span>, <span style="color: green;">str</span>]</code>
+ <code style="color: orange">boolean</code>
</td>
<td>
- A list of key-value pairs that act as configuration parameters for Kafka consumers. Most of these configurations will not be needed, but if you need to customize your Kafka consumer, you may use this. See a detailed list: https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html
+ If the Kafka read should be redistributed.
</td>
</tr>
<tr>
<td>
- file_descriptor_path
+ schema
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- The path to the Protocol Buffer File Descriptor Set file. This file is used for schema definition and message serialization.
+ The schema in which the data is encoded in the Kafka topic. For AVRO data, this is a schema defined with AVRO schema syntax (https://avro.apache.org/docs/1.10.2/spec.html#schemas). For JSON data, this is a schema defined with JSON-schema syntax (https://json-schema.org/). If a URL to Confluent Schema Registry is provided, then this field is ignored, and the schema is fetched from Confluent Schema Registry.
</td>
</tr>
+ </table>
+</div>
+
+### `KAFKA` Write
+
+<div class="table-container-wrapper">
+ <table class="table table-bordered">
+ <tr>
+ <th>Configuration</th>
+ <th>Type</th>
+ <th>Description</th>
+ </tr>
<tr>
<td>
- format
+ <strong>bootstrap_servers</strong>
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- The encoding format for the data stored in Kafka. Valid options are: RAW,STRING,AVRO,JSON,PROTO
+ A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. | Format: host1:port1,host2:port2,...
</td>
</tr>
<tr>
<td>
- message_name
+ <strong>format</strong>
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- The name of the Protocol Buffer message to be used for schema extraction and data conversion.
+ The encoding format for the data stored in Kafka. Valid options are: RAW,JSON,AVRO,PROTO
</td>
</tr>
<tr>
<td>
- offset_deduplication
+ <strong>topic</strong>
</td>
<td>
- <code style="color: orange">boolean</code>
+ <code style="color: green">str</code>
</td>
<td>
- If the redistribute is using offset deduplication mode.
+ n/a
Review Comment:
The description for the required field `topic` is "n/a". This should be updated with a meaningful description.
```suggestion
The Kafka topic to write to.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update managed-io.md for release 2.74.0-RC1 [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on PR #38527:
URL: https://github.com/apache/beam/pull/38527#issuecomment-4479960095
## Summary of Changes
Hello, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request updates the Managed IO documentation for the Apache Beam 2.74.0-RC1 release. The changes primarily involve adding new configuration options for various IO connectors and correcting existing documentation to accurately reflect the current capabilities and parameter requirements for supported databases and services.
### Highlights
* **Documentation Updates**: Updated the managed-io.md documentation to reflect changes for the 2.74.0-RC1 release, including new configuration options and corrected table definitions.
* **Configuration Additions**: Added new configuration parameters such as autosharding, distribution_mode, and sort_fields to various IO connectors to improve flexibility and performance.
* **Connector Refactoring**: Reorganized and corrected the documentation for SQLSERVER, BIGQUERY, MYSQL, and POSTGRES connectors to ensure accuracy regarding supported parameters and read/write operations.
<details>
<summary><b>New Features</b></summary>
🧠 You can now enable **Memory** (public preview) to help **Gemini Code Assist** learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. **Click [here](https://codeassist.google/code-review/login) to enable Memory in your admin console.**
</details>
<details>
<summary><b>Using Gemini Code Assist</b></summary>
<br>
The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips.
<b>Invoking Gemini</b>
You can request assistance from Gemini at any point by creating a comment using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands on the current page.
Feature | Command | Description
--- | --- | ---
Code Review | `/gemini review` | Performs a code review for the current pull request in its current state.
Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state.
Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help | `/gemini help` | Displays a list of available commands.
<b>Customization</b>
To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github).
<b>Limitations & Feedback</b>
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up [here](https://google.qualtrics.com/jfe/form/SV_2cyuGuTWsEw84yG).
</details>
[^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update managed-io.md for release 2.74.0-RC3 [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #38527:
URL: https://github.com/apache/beam/pull/38527
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update managed-io.md for release 2.74.0-RC2 [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #38527:
URL: https://github.com/apache/beam/pull/38527#issuecomment-4522715856
No doc changes between RC1 and RC2, reusing PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update managed-io.md for release 2.74.0-RC1 [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38527:
URL: https://github.com/apache/beam/pull/38527#issuecomment-4480529266
Assigning reviewers:
R: @shunping for label website.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org