You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "ahmedabu98 (via GitHub)" <gi...@apache.org> on 2024/05/03 15:47:18 UTC
[PR] Support Kafka Managed IO [beam]
ahmedabu98 opened a new pull request, #31172:
URL: https://github.com/apache/beam/pull/31172
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Kafka Managed IO [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #31172:
URL: https://github.com/apache/beam/pull/31172#discussion_r1595703461
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformProvider.java:
##########
@@ -202,7 +203,26 @@ Row getConfigurationRow() {
static Row getRowConfig(ManagedConfig config, Schema transformSchema) {
// May return an empty row (perhaps the underlying transform doesn't have any required
// parameters)
- return YamlUtils.toBeamRow(config.resolveUnderlyingConfig(), transformSchema, false);
+ String yamlConfig = config.resolveUnderlyingConfig();
+ Map<String, Object> configMap = YamlUtils.yamlStringToMap(yamlConfig);
+
+ // The config Row object will be used to build the underlying SchemaTransform.
+ // If a mapping for the SchemaTransform exists, we use it to update parameter names and align
+ // with the underlying config schema
+ Map<String, String> mapping = MAPPINGS.get(config.getTransformIdentifier());
+ if (mapping != null && configMap != null) {
+ Map<String, Object> remappedConfig = new HashMap<>();
+ for (Map.Entry<String, Object> entry : configMap.entrySet()) {
+ String paramName = entry.getKey();
+ if (mapping.containsKey(paramName)) {
Review Comment:
Ah got it - I think leaving it is fine, thanks
> But agreed some logging would be helpful. I'll add a log showing the final Row configuration used to build the underlying transform
Sounds good, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Kafka Managed IO [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #31172:
URL: https://github.com/apache/beam/pull/31172#discussion_r1595687475
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformProvider.java:
##########
@@ -202,7 +203,26 @@ Row getConfigurationRow() {
static Row getRowConfig(ManagedConfig config, Schema transformSchema) {
// May return an empty row (perhaps the underlying transform doesn't have any required
// parameters)
- return YamlUtils.toBeamRow(config.resolveUnderlyingConfig(), transformSchema, false);
+ String yamlConfig = config.resolveUnderlyingConfig();
+ Map<String, Object> configMap = YamlUtils.yamlStringToMap(yamlConfig);
+
+ // The config Row object will be used to build the underlying SchemaTransform.
+ // If a mapping for the SchemaTransform exists, we use it to update parameter names and align
+ // with the underlying config schema
+ Map<String, String> mapping = MAPPINGS.get(config.getTransformIdentifier());
+ if (mapping != null && configMap != null) {
+ Map<String, Object> remappedConfig = new HashMap<>();
+ for (Map.Entry<String, Object> entry : configMap.entrySet()) {
+ String paramName = entry.getKey();
+ if (mapping.containsKey(paramName)) {
Review Comment:
P.S. alternatively we can make it mandatory to include all parameter names in the mapping, regardless if they need to be updated or not. (i'm open to this option but i find it unnecessarily strict)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Kafka Managed IO [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #31172:
URL: https://github.com/apache/beam/pull/31172#discussion_r1595705163
##########
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProviderTest.java:
##########
@@ -18,16 +18,25 @@
package org.apache.beam.sdk.io.kafka;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
Review Comment:
It would also be good to add a basic Kafka IT (though this is about testing Managed as much as it is Kafka). I'm fine deferring that to a future PR where we can address this together with Iceberg or including it here, up to you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Kafka Managed IO [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #31172:
URL: https://github.com/apache/beam/pull/31172#discussion_r1595690110
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformProvider.java:
##########
@@ -202,7 +203,26 @@ Row getConfigurationRow() {
static Row getRowConfig(ManagedConfig config, Schema transformSchema) {
// May return an empty row (perhaps the underlying transform doesn't have any required
// parameters)
- return YamlUtils.toBeamRow(config.resolveUnderlyingConfig(), transformSchema, false);
+ String yamlConfig = config.resolveUnderlyingConfig();
+ Map<String, Object> configMap = YamlUtils.yamlStringToMap(yamlConfig);
+
+ // The config Row object will be used to build the underlying SchemaTransform.
+ // If a mapping for the SchemaTransform exists, we use it to update parameter names and align
+ // with the underlying config schema
+ Map<String, String> mapping = MAPPINGS.get(config.getTransformIdentifier());
Review Comment:
Similar to the reasoning above, a mapping exists only if the transform needs one.
In the Iceberg case, we are already producing a configuration schema with `snake_case` convention, so no remapping of names is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Kafka Managed IO [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31172:
URL: https://github.com/apache/beam/pull/31172#issuecomment-2093420927
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Kafka Managed IO [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #31172:
URL: https://github.com/apache/beam/pull/31172#discussion_r1594853864
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformProvider.java:
##########
@@ -202,7 +203,26 @@ Row getConfigurationRow() {
static Row getRowConfig(ManagedConfig config, Schema transformSchema) {
// May return an empty row (perhaps the underlying transform doesn't have any required
// parameters)
- return YamlUtils.toBeamRow(config.resolveUnderlyingConfig(), transformSchema, false);
+ String yamlConfig = config.resolveUnderlyingConfig();
+ Map<String, Object> configMap = YamlUtils.yamlStringToMap(yamlConfig);
+
+ // The config Row object will be used to build the underlying SchemaTransform.
+ // If a mapping for the SchemaTransform exists, we use it to update parameter names and align
+ // with the underlying config schema
+ Map<String, String> mapping = MAPPINGS.get(config.getTransformIdentifier());
Review Comment:
What happens if the transform identifier isn't found - semes like Iceberg already falls into that case, I'm also worried about typos though if folks don't use the enum (maybe rare)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Kafka Managed IO [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 merged PR #31172:
URL: https://github.com/apache/beam/pull/31172
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Kafka Managed IO [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #31172:
URL: https://github.com/apache/beam/pull/31172#issuecomment-2096727888
R: @chamikaramj
R: @damccorm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Kafka Managed IO [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #31172:
URL: https://github.com/apache/beam/pull/31172#discussion_r1595687475
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformProvider.java:
##########
@@ -202,7 +203,26 @@ Row getConfigurationRow() {
static Row getRowConfig(ManagedConfig config, Schema transformSchema) {
// May return an empty row (perhaps the underlying transform doesn't have any required
// parameters)
- return YamlUtils.toBeamRow(config.resolveUnderlyingConfig(), transformSchema, false);
+ String yamlConfig = config.resolveUnderlyingConfig();
+ Map<String, Object> configMap = YamlUtils.yamlStringToMap(yamlConfig);
+
+ // The config Row object will be used to build the underlying SchemaTransform.
+ // If a mapping for the SchemaTransform exists, we use it to update parameter names and align
+ // with the underlying config schema
+ Map<String, String> mapping = MAPPINGS.get(config.getTransformIdentifier());
+ if (mapping != null && configMap != null) {
+ Map<String, Object> remappedConfig = new HashMap<>();
+ for (Map.Entry<String, Object> entry : configMap.entrySet()) {
+ String paramName = entry.getKey();
+ if (mapping.containsKey(paramName)) {
Review Comment:
P.S. alternatively we can make it mandatory to include all parameter names in the mapping, regardless if they need to be updated or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Kafka Managed IO [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #31172:
URL: https://github.com/apache/beam/pull/31172#discussion_r1594853381
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformProvider.java:
##########
@@ -202,7 +203,26 @@ Row getConfigurationRow() {
static Row getRowConfig(ManagedConfig config, Schema transformSchema) {
// May return an empty row (perhaps the underlying transform doesn't have any required
// parameters)
- return YamlUtils.toBeamRow(config.resolveUnderlyingConfig(), transformSchema, false);
+ String yamlConfig = config.resolveUnderlyingConfig();
+ Map<String, Object> configMap = YamlUtils.yamlStringToMap(yamlConfig);
+
+ // The config Row object will be used to build the underlying SchemaTransform.
+ // If a mapping for the SchemaTransform exists, we use it to update parameter names and align
+ // with the underlying config schema
+ Map<String, String> mapping = MAPPINGS.get(config.getTransformIdentifier());
+ if (mapping != null && configMap != null) {
+ Map<String, Object> remappedConfig = new HashMap<>();
+ for (Map.Entry<String, Object> entry : configMap.entrySet()) {
+ String paramName = entry.getKey();
+ if (mapping.containsKey(paramName)) {
Review Comment:
Right now any parameters that aren't recognized are silently dropped. I think we should be at least warning here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Kafka Managed IO [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31172:
URL: https://github.com/apache/beam/pull/31172#issuecomment-2096729679
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Kafka Managed IO [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #31172:
URL: https://github.com/apache/beam/pull/31172#discussion_r1595683176
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformProvider.java:
##########
@@ -202,7 +203,26 @@ Row getConfigurationRow() {
static Row getRowConfig(ManagedConfig config, Schema transformSchema) {
// May return an empty row (perhaps the underlying transform doesn't have any required
// parameters)
- return YamlUtils.toBeamRow(config.resolveUnderlyingConfig(), transformSchema, false);
+ String yamlConfig = config.resolveUnderlyingConfig();
+ Map<String, Object> configMap = YamlUtils.yamlStringToMap(yamlConfig);
+
+ // The config Row object will be used to build the underlying SchemaTransform.
+ // If a mapping for the SchemaTransform exists, we use it to update parameter names and align
+ // with the underlying config schema
+ Map<String, String> mapping = MAPPINGS.get(config.getTransformIdentifier());
+ if (mapping != null && configMap != null) {
+ Map<String, Object> remappedConfig = new HashMap<>();
+ for (Map.Entry<String, Object> entry : configMap.entrySet()) {
+ String paramName = entry.getKey();
+ if (mapping.containsKey(paramName)) {
Review Comment:
The mapping contains parameter names that should be updated. If a parameter is not included in the mapping, we assume its original name is correct.
But agreed some logging would be helpful. I'll add a log showing the final Row configuration used to build the underlying transform
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Kafka Managed IO [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #31172:
URL: https://github.com/apache/beam/pull/31172#discussion_r1624528691
##########
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProviderTest.java:
##########
@@ -18,16 +18,25 @@
package org.apache.beam.sdk.io.kafka;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
Review Comment:
Done in #31362. We have existing integration tests for Kafka SchemaTransforms and I just switched them to use Managed API instead (which ultimately uses the schematransforms)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org