You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "amogh-jahagirdar (via GitHub)" <gi...@apache.org> on 2024/09/24 16:57:49 UTC
[PR] Core: Replace use of CharSequenceMap in DeleteFileIndex with String [iceberg]
amogh-jahagirdar opened a new pull request, #11199:
URL: https://github.com/apache/iceberg/pull/11199
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Replace use of CharSequenceMap in DeleteFileIndex with Map [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #11199:
URL: https://github.com/apache/iceberg/pull/11199#discussion_r1776090183
##########
core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -49,7 +50,21 @@ public static <F extends ContentFile<K>, K> K copy(
}
}
+ /**
+ * @deprecated since 1.7.0, will be removed in 2.0.0; use {@link
+ * #referencedDataFileLocation(DeleteFile)} instead.
+ */
+ @Deprecated
public static CharSequence referencedDataFile(DeleteFile deleteFile) {
+ return referencedLocation(deleteFile);
+ }
+
+ public static String referencedDataFileLocation(DeleteFile deleteFile) {
+ CharBuffer location = referencedLocation(deleteFile);
+ return location != null ? location.toString() : null;
+ }
+
+ private static <T> T referencedLocation(DeleteFile deleteFile) {
Review Comment:
I think we can migrate now as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Replace use of CharSequenceMap in DeleteFileIndex with Map [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #11199:
URL: https://github.com/apache/iceberg/pull/11199#discussion_r1773993913
##########
core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -49,7 +50,21 @@ public static <F extends ContentFile<K>, K> K copy(
}
}
+ /**
+ * @deprecated since 1.7.0, will be removed in 2.0.0; use {@link
+ * #referencedDataFileLocation(DeleteFile)} instead.
+ */
+ @Deprecated
public static CharSequence referencedDataFile(DeleteFile deleteFile) {
+ return referencedLocation(deleteFile);
+ }
+
+ public static String referencedDataFileLocation(DeleteFile deleteFile) {
+ CharBuffer location = referencedLocation(deleteFile);
+ return location != null ? location.toString() : null;
+ }
+
+ private static <T> T referencedLocation(DeleteFile deleteFile) {
Review Comment:
The logic in referencedLocation will change once we track referenced location per manifest entry (Since then there's no need for upperbound lowerbound checks), but that depends on the spec change. It does appear like for now there's a performance benefit to getting off the CharSequenceMap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Replace use of CharSequenceMap in DeleteFileIndex with Map [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar merged PR #11199:
URL: https://github.com/apache/iceberg/pull/11199
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Replace use of CharSequenceMap in DeleteFileIndex with Map [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #11199:
URL: https://github.com/apache/iceberg/pull/11199#discussion_r1776090858
##########
core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -49,7 +50,21 @@ public static <F extends ContentFile<K>, K> K copy(
}
}
+ /**
+ * @deprecated since 1.7.0, will be removed in 2.0.0; use {@link
+ * #referencedDataFileLocation(DeleteFile)} instead.
+ */
+ @Deprecated
public static CharSequence referencedDataFile(DeleteFile deleteFile) {
+ return referencedLocation(deleteFile);
+ }
+
+ public static String referencedDataFileLocation(DeleteFile deleteFile) {
+ CharBuffer location = referencedLocation(deleteFile);
+ return location != null ? location.toString() : null;
+ }
+
+ private static <T> T referencedLocation(DeleteFile deleteFile) {
Review Comment:
Is there value in a new method? Why not simply call `referencedDataFile` in the new method?
```
public static String referencedDataFileLocation(DeleteFile deleteFile) {
CharSequence location = referencedDataFile(deleteFile);
return location != null ? location.toString() : null;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Replace use of CharSequenceMap in DeleteFileIndex with String [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #11199:
URL: https://github.com/apache/iceberg/pull/11199#discussion_r1773720327
##########
core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -75,4 +76,32 @@ public static CharSequence referencedDataFile(DeleteFile deleteFile) {
return null;
}
}
+
+ public static String referencedDataFileLocation(DeleteFile deleteFile) {
Review Comment:
I'll get rid of this duplication, was just me being lazy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Replace use of CharSequenceMap in DeleteFileIndex with Map [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #11199:
URL: https://github.com/apache/iceberg/pull/11199#discussion_r1776092317
##########
core/src/main/java/org/apache/iceberg/DeleteFileIndex.java:
##########
@@ -458,14 +457,14 @@ DeleteFileIndex build() {
}
private void add(
- CharSequenceMap<PositionDeletes> deletesByPath,
+ Map<String, PositionDeletes> deletesByPath,
PartitionMap<PositionDeletes> deletesByPartition,
DeleteFile file) {
- CharSequence path = ContentFileUtil.referencedDataFile(file);
+ String path = ContentFileUtil.referencedDataFileLocation(file);
PositionDeletes deletes;
if (path != null) {
- deletes = deletesByPath.computeIfAbsent(path, PositionDeletes::new);
+ deletes = deletesByPath.computeIfAbsent(path, somePath -> new PositionDeletes());
Review Comment:
Minor: `somePath` -> `ignored` or `key`? We use this style in a few places, I think it highlights the key is not used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Replace use of CharSequenceMap in DeleteFileIndex with Map [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #11199:
URL: https://github.com/apache/iceberg/pull/11199#discussion_r1776091909
##########
core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -49,7 +50,21 @@ public static <F extends ContentFile<K>, K> K copy(
}
}
+ /**
+ * @deprecated since 1.7.0, will be removed in 2.0.0; use {@link
Review Comment:
Agree, we can drop it in 1.8.
I wouldn't also mind not deprecating this for now to see if there would be any use cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Replace use of CharSequenceMap in DeleteFileIndex with Map [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #11199:
URL: https://github.com/apache/iceberg/pull/11199#discussion_r1773993913
##########
core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -49,7 +50,21 @@ public static <F extends ContentFile<K>, K> K copy(
}
}
+ /**
+ * @deprecated since 1.7.0, will be removed in 2.0.0; use {@link
+ * #referencedDataFileLocation(DeleteFile)} instead.
+ */
+ @Deprecated
public static CharSequence referencedDataFile(DeleteFile deleteFile) {
+ return referencedLocation(deleteFile);
+ }
+
+ public static String referencedDataFileLocation(DeleteFile deleteFile) {
+ CharBuffer location = referencedLocation(deleteFile);
+ return location != null ? location.toString() : null;
+ }
+
+ private static <T> T referencedLocation(DeleteFile deleteFile) {
Review Comment:
The logic in referencedLocation will change once we track referenced location per manifest entry (Since then there's no need for upperbound lowerbound checks), but that depends on the spec change. It does appear like for now there's a performance benefit to getting off the CharSequenceMap so I don't think we need to wait for that? CC @aokolnychyi @rdblue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Replace use of CharSequenceMap in DeleteFileIndex with Map [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #11199:
URL: https://github.com/apache/iceberg/pull/11199#discussion_r1774786143
##########
core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -49,7 +50,21 @@ public static <F extends ContentFile<K>, K> K copy(
}
}
+ /**
+ * @deprecated since 1.7.0, will be removed in 2.0.0; use {@link
Review Comment:
I think it's fine to remove this in 1.8.0 instead of 2.0.0 since this is iceberg-core?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Replace use of CharSequenceMap in DeleteFileIndex with Map [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #11199:
URL: https://github.com/apache/iceberg/pull/11199#discussion_r1776090858
##########
core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -49,7 +50,21 @@ public static <F extends ContentFile<K>, K> K copy(
}
}
+ /**
+ * @deprecated since 1.7.0, will be removed in 2.0.0; use {@link
+ * #referencedDataFileLocation(DeleteFile)} instead.
+ */
+ @Deprecated
public static CharSequence referencedDataFile(DeleteFile deleteFile) {
+ return referencedLocation(deleteFile);
+ }
+
+ public static String referencedDataFileLocation(DeleteFile deleteFile) {
+ CharBuffer location = referencedLocation(deleteFile);
+ return location != null ? location.toString() : null;
+ }
+
+ private static <T> T referencedLocation(DeleteFile deleteFile) {
Review Comment:
Is there value in a new method? Why not simply call `referencedDataFile` in the added method?
```
public static String referencedDataFileLocation(DeleteFile deleteFile) {
CharSequence location = referencedDataFile(deleteFile);
return location != null ? location.toString() : null;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Replace use of CharSequenceMap in DeleteFileIndex with Map [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #11199:
URL: https://github.com/apache/iceberg/pull/11199#discussion_r1777434631
##########
core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -49,7 +50,21 @@ public static <F extends ContentFile<K>, K> K copy(
}
}
+ /**
+ * @deprecated since 1.7.0, will be removed in 2.0.0; use {@link
+ * #referencedDataFileLocation(DeleteFile)} instead.
+ */
+ @Deprecated
public static CharSequence referencedDataFile(DeleteFile deleteFile) {
+ return referencedLocation(deleteFile);
+ }
+
+ public static String referencedDataFileLocation(DeleteFile deleteFile) {
+ CharBuffer location = referencedLocation(deleteFile);
+ return location != null ? location.toString() : null;
+ }
+
+ private static <T> T referencedLocation(DeleteFile deleteFile) {
Review Comment:
Good point, no not really. Just how I refactored it, in my head I was just isolating the extraction of the location to a separate method. Since there's no real value add (re-ran benchmarking just to be sure), we may as well make the change more minimal.
##########
core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -49,7 +50,21 @@ public static <F extends ContentFile<K>, K> K copy(
}
}
+ /**
+ * @deprecated since 1.7.0, will be removed in 2.0.0; use {@link
Review Comment:
I thought about this a bit more, I originally deprecated it because I did think there's no need to have both methods. However, after more thought I think it makes sense to not deprecate for now since it isn't super obvious (at least to me) we will never need this.
If at a certain point we realize it really isn't needed we can easily deprecate at that point (especially since it's in core like @nastra mentioned). However, if we deprecate and realize we need it in some cases then we're in an awkward state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Replace use of CharSequenceMap in DeleteFileIndex with Map [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #11199:
URL: https://github.com/apache/iceberg/pull/11199#discussion_r1777434631
##########
core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -49,7 +50,21 @@ public static <F extends ContentFile<K>, K> K copy(
}
}
+ /**
+ * @deprecated since 1.7.0, will be removed in 2.0.0; use {@link
+ * #referencedDataFileLocation(DeleteFile)} instead.
+ */
+ @Deprecated
public static CharSequence referencedDataFile(DeleteFile deleteFile) {
+ return referencedLocation(deleteFile);
+ }
+
+ public static String referencedDataFileLocation(DeleteFile deleteFile) {
+ CharBuffer location = referencedLocation(deleteFile);
+ return location != null ? location.toString() : null;
+ }
+
+ private static <T> T referencedLocation(DeleteFile deleteFile) {
Review Comment:
Good point, no not really. Just how I refactored it, in my head I was just isolating the extraction of the location to a separate method. Since there's no real value add, we may as well make the change more minimal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Replace use of CharSequenceMap in DeleteFileIndex with Map [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #11199:
URL: https://github.com/apache/iceberg/pull/11199#issuecomment-2377571892
Thanks for reviewing @nastra @aokolnychyi !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Replace use of CharSequenceMap in DeleteFileIndex with Map [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #11199:
URL: https://github.com/apache/iceberg/pull/11199#discussion_r1773993913
##########
core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java:
##########
@@ -49,7 +50,21 @@ public static <F extends ContentFile<K>, K> K copy(
}
}
+ /**
+ * @deprecated since 1.7.0, will be removed in 2.0.0; use {@link
+ * #referencedDataFileLocation(DeleteFile)} instead.
+ */
+ @Deprecated
public static CharSequence referencedDataFile(DeleteFile deleteFile) {
+ return referencedLocation(deleteFile);
+ }
+
+ public static String referencedDataFileLocation(DeleteFile deleteFile) {
+ CharBuffer location = referencedLocation(deleteFile);
+ return location != null ? location.toString() : null;
+ }
+
+ private static <T> T referencedLocation(DeleteFile deleteFile) {
Review Comment:
The logic in referencedLocation will change once we track referenced location per manifest entry, but that depends on the spec change. It does appear like for now there's a performance benefit to getting off the CharSequenceMap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org