You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "aokolnychyi (via GitHub)" <gi...@apache.org> on 2024/05/10 17:25:00 UTC
[PR] Spark 3.5: Remove obsolete conf parsing logic [iceberg]
aokolnychyi opened a new pull request, #10309:
URL: https://github.com/apache/iceberg/pull/10309
This PR removes some conf parsing logic that predates the split of Spark 2 and Spark 3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.5: Remove obsolete conf parsing logic [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #10309:
URL: https://github.com/apache/iceberg/pull/10309#discussion_r1597026246
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkConfParser.java:
##########
@@ -220,14 +228,10 @@ public ThisT tableProperty(String name) {
}
protected T parse(Function<String, T> conversion, T defaultValue) {
- if (!optionNames.isEmpty()) {
Review Comment:
The if here is redundant. We initially had `!= null` check before we added a list of options that is never null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.5: Remove obsolete conf parsing logic [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #10309:
URL: https://github.com/apache/iceberg/pull/10309#discussion_r1597026741
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkConfParser.java:
##########
@@ -220,14 +228,10 @@ public ThisT tableProperty(String name) {
}
protected T parse(Function<String, T> conversion, T defaultValue) {
- if (!optionNames.isEmpty()) {
- for (String optionName : optionNames) {
- // use lower case comparison as DataSourceOptions.asMap() in Spark 2 returns a lower case
- // map
- String optionValue = options.get(optionName.toLowerCase(Locale.ROOT));
- if (optionValue != null) {
- return conversion.apply(optionValue);
- }
+ for (String optionName : optionNames) {
+ String optionValue = options.get(optionName);
Review Comment:
Removed the explicit conversion to lower case and obsolete comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.5: Remove obsolete conf parsing logic [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #10309:
URL: https://github.com/apache/iceberg/pull/10309#discussion_r1597024872
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkConfParser.java:
##########
@@ -70,6 +70,14 @@ public DurationConfParser durationConf() {
return new DurationConfParser();
}
+ private static CaseInsensitiveStringMap asCaseInsensitiveStringMap(Map<String, String> map) {
Review Comment:
Prior to split of Spark 2 and Spark 3, we had to use `Map<String, String>` as Spark 3 used one class for options and Spark 2 used another one. It is always `CaseInsensitiveStringMap` in Spark 3 but I am checking here to be safe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.5: Remove obsolete conf parsing logic [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #10309:
URL: https://github.com/apache/iceberg/pull/10309#issuecomment-2105954175
Thanks, @nastra!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.5: Remove obsolete conf parsing logic [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi merged PR #10309:
URL: https://github.com/apache/iceberg/pull/10309
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org