You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@auron.apache.org by "Tartarus0zm (via GitHub)" <gi...@apache.org> on 2026/05/13 08:57:54 UTC
[PR] [AURON #2266] Fix KafkaScanExec error when using the JSON format [auron]
Tartarus0zm opened a new pull request, #2267:
URL: https://github.com/apache/auron/pull/2267
# Which issue does this PR close?
Closes #2266
# Rationale for this change
Since the JSON format does not rely on the `pb_desc_file`, `root_message_name`, and `skip_fields` information, an error will occur when attempting to retrieve these values from parser_config.
# What changes are included in this PR?
Optimize the retrieval of the `pb_desc_file`, `root_message_name`, and `skip_fields` fields
# Are there any user-facing changes?
No
# How was this patch tested?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2266] Fix KafkaScanExec error when using the JSON format [auron]
Posted by "Tartarus0zm (via GitHub)" <gi...@apache.org>.
Tartarus0zm commented on PR #2267:
URL: https://github.com/apache/auron/pull/2267#issuecomment-4440616355
@richox CI is green, PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2266] Fix KafkaScanExec error when using the JSON format [auron]
Posted by "Tartarus0zm (via GitHub)" <gi...@apache.org>.
Tartarus0zm merged PR #2267:
URL: https://github.com/apache/auron/pull/2267
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2266] Fix KafkaScanExec error when using the JSON format [auron]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2267:
URL: https://github.com/apache/auron/pull/2267#discussion_r3234634685
##########
native-engine/datafusion-ext-plans/src/flink/kafka_scan_exec.rs:
##########
@@ -517,17 +502,35 @@ fn parse_records(
}
}
- let local_pb_desc_file = env::var("PWD").expect("PWD env var is not set") + "/" + &pb_desc_file;
- log::info!("load desc from {local_pb_desc_file}");
- let file_descriptor_bytes = fs::read(local_pb_desc_file).expect("Failed to read file");
- let skip_fields_vec: Vec<String> = skip_fields
- .split(",")
- .map(|s| s.to_string())
- .collect::<Vec<String>>();
+ let (file_descriptor_bytes, root_message_name, skip_fields_vec) = if data_format != 0 {
+ let pb_desc_file = parser_config
+ .get("pb_desc_file")
+ .and_then(|v| v.as_str())
+ .expect("pb_desc_file is not valid string")
+ .to_string();
+ let root_message_name = parser_config
+ .get("root_message_name")
+ .and_then(|v| v.as_str())
+ .expect("root_message_name is not valid string")
+ .to_string();
+ let skip_fields = parser_config
+ .get("skip_fields")
+ .and_then(|v| v.as_str())
+ .expect("skip_fields is not valid string")
+ .to_string();
+ let local_pb_desc_file =
+ env::var("PWD").expect("PWD env var is not set") + "/" + &pb_desc_file;
+ log::info!("load desc from {local_pb_desc_file}");
+ let file_descriptor_bytes = fs::read(local_pb_desc_file).expect("Failed to read file");
+ let skip_fields_vec: Vec<String> = skip_fields.split(",").map(|s| s.to_string()).collect();
+ (file_descriptor_bytes, root_message_name, skip_fields_vec)
+ } else {
+ (vec![], String::new(), vec![])
+ };
Review Comment:
The new conditional parsing of `pb_desc_file` / `root_message_name` / `skip_fields` fixes the JSON format crash, but there’s no regression test ensuring JSON `data_format == 0` works when `format_config_json` omits protobuf-only fields. Adding a unit/integration test that exercises `parse_records` (or the planner->exec path) with JSON format and a minimal parser_config would help prevent this from recurring.
This issue also appears on line 505 of the same file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org