You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "nastra (via GitHub)" <gi...@apache.org> on 2026/09/29 14:37:22 UTC
[PR] API, Core, Parquet, Spark: Update avg_value_size_in_bytes to total_bytes [iceberg]
nastra opened a new pull request, #18310:
URL: https://github.com/apache/iceberg/pull/18310
This updates all the places that were using `avg_value_size_in_bytes` to the new `total_bytes` field
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core, Parquet, Spark: Update avg_value_size_in_bytes to total_bytes [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #18310:
URL: https://github.com/apache/iceberg/pull/18310#discussion_r4146645493
##########
core/src/main/java/org/apache/iceberg/GeometryFieldMetrics.java:
##########
@@ -69,15 +69,14 @@ public void addValue(ByteBuffer wkb) {
public GeometryFieldMetrics build() {
// build() returns null bounds when a dimension has no value or a value could not be bounded,
- // which leaves the field with counts but no bounds. The average size is still reported: it
- // does not depend on whether a box could be formed.
+ // which leaves the field with counts but no bounds. The total is still reported and does not
Review Comment:
Minor: `total` is ambigous here.
##########
core/src/main/java/org/apache/iceberg/GeometryFieldMetrics.java:
##########
@@ -69,15 +69,14 @@ public void addValue(ByteBuffer wkb) {
public GeometryFieldMetrics build() {
// build() returns null bounds when a dimension has no value or a value could not be bounded,
- // which leaves the field with counts but no bounds. The average size is still reported: it
- // does not depend on whether a box could be formed.
+ // which leaves the field with counts but no bounds. The total is still reported and does not
+ // depend on whether a box could be formed.
Review Comment:
```suggestion
// which leaves the field with counts but no bounds. The total bytes are still reported
// and do not depend on whether a box could be formed.
```
##########
format/spec.md:
##########
@@ -827,7 +827,7 @@ Each stats struct holds statistics for one table field. It may contain the follo
| _optional_ | 4 | `value_count` | `long` | all | Number of values in the column (including null and NaN values) |
| _optional_ | 5 | `null_value_count` | `long` | optional fields | Number of null values in the column |
| _optional_ | 6 | `nan_value_count` | `long` | `float`, `double` | Number of NaN values in the column |
-| _optional_ | 7 | `avg_value_size_in_bytes` | `int` | `string`, `binary`, `variant`, `geometry`, `geography` | Avg value size in memory (uncompressed) in bytes over non-null values to estimate memory consumption |
Review Comment:
Is this stacked on top of #18308? We need to drop this otherwise, as it's conflicting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core, Parquet, Spark: Update avg_value_size_in_bytes to total_bytes [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #18310:
URL: https://github.com/apache/iceberg/pull/18310#issuecomment-5935053566
Yup agree with all the previous points, totally good with breaking API compatibility on this, no practical concerns since nobody can really be setting this at this point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core, Parquet, Spark: Update avg_value_size_in_bytes to total_bytes [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #18310:
URL: https://github.com/apache/iceberg/pull/18310#discussion_r4152225361
##########
.palantir/revapi.yml:
##########
@@ -674,7 +674,7 @@ acceptedBreaks:
\ be implemented outside the project. Promoting them to public reads as an\
\ existing type gaining methods, not as a new type appearing."
- code: "java.method.addedToInterface"
- new: "method java.lang.Integer org.apache.iceberg.FieldStats<T>::avgValueSizeInBytes()"
+ new: "method java.lang.Long org.apache.iceberg.FieldStats<T>::totalBytes()"
Review Comment:
hm yeah so this worked before we released 1.12.0 obviously, but now I believe we need to deprecate all of the avgValueSizes() methods across those classes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core, Parquet, Spark: Update avg_value_size_in_bytes to total_bytes [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on PR #18310:
URL: https://github.com/apache/iceberg/pull/18310#issuecomment-5933211463
> Given that we just released 1.12.0 this PR isn't a pure rename of avgValueSize() methods to totalBytes() in all of the places anymore. We now actually have to do the changes in a backwards compatible way. I'll set the PR to WIP until I have the correct changes
those are v4 APIs where nobody should be using. can we just note the breaking change in revapi exclusion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core, Parquet, Spark: Update avg_value_size_in_bytes to total_bytes [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on PR #18310:
URL: https://github.com/apache/iceberg/pull/18310#issuecomment-5933522359
> > Given that we just released 1.12.0 this PR isn't a pure rename of avgValueSize() methods to totalBytes() in all of the places anymore. We now actually have to do the changes in a backwards compatible way. I'll set the PR to WIP until I have the correct changes
>
> those are v4 APIs where nobody should be using. can we just note the breaking change in revapi exclusion?
@stevenzwu we can certainly do that if people agree. It is kind of a bad timing for this rename since `avgValueSizes()` was added to a few central places and just released, like `ContentFile`/ `Metrics`/`BaseFile`/`ContentFileParser` and also to the REST spec.
Would be good to also hear from others here /cc @rdblue @RussellSpitzer @pvary @amogh-jahagirdar @singhpk234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core, Parquet, Spark: Update avg_value_size_in_bytes to total_bytes [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #18310:
URL: https://github.com/apache/iceberg/pull/18310#discussion_r4150403784
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -198,7 +198,7 @@ default Long firstRowId() {
* to copy data without stats when collecting files.
*
* @return a copy of this data file, without lower bounds, upper bounds, value counts, null value
- * counts, nan value counts, or average value sizes
+ * counts, nan value counts, or total bytes
Review Comment:
I am wondering if we should stop enumerating every metric. just say `a copy of this data file without column stats`?
Same for two other Javadoc comments in this file.
##########
.palantir/revapi.yml:
##########
@@ -674,7 +674,7 @@ acceptedBreaks:
\ be implemented outside the project. Promoting them to public reads as an\
\ existing type gaining methods, not as a new type appearing."
- code: "java.method.addedToInterface"
- new: "method java.lang.Integer org.apache.iceberg.FieldStats<T>::avgValueSizeInBytes()"
+ new: "method java.lang.Long org.apache.iceberg.FieldStats<T>::totalBytes()"
Review Comment:
This swap covers `FieldStats.totalBytes()`, but CI is still failing on the 1.12.0 removals of `ContentFile.avgValueSizes()` and `Metrics.avgValueSizes()` (`java.method.removed`, source + binary). We probably need to add `java.method.removed` entries for those two.
##########
api/src/main/java/org/apache/iceberg/ContentFile.java:
##########
@@ -100,13 +100,13 @@ default String location() {
Map<Integer, ByteBuffer> upperBounds();
/**
- * Returns if collected, map from column ID to its average value size in memory (uncompressed) in
- * bytes over non-null values, null otherwise.
+ * Returns if collected, map from column ID to the total uncompressed size in memory in bytes of
Review Comment:
I would move the `if collected` next to `, null otherwise`. like
```
Returns map from column ID to the total uncompressed size in memory in bytes of
non-null values if collected, null otherwise.
```
##########
core/src/main/java/org/apache/iceberg/GeometryFieldMetrics.java:
##########
@@ -69,15 +69,14 @@ public void addValue(ByteBuffer wkb) {
public GeometryFieldMetrics build() {
// build() returns null bounds when a dimension has no value or a value could not be bounded,
- // which leaves the field with counts but no bounds. The average size is still reported: it
- // does not depend on whether a box could be formed.
+ // which leaves the field with counts but no bounds. The total is still reported and does not
Review Comment:
+1. maybe "the total bytes" here.
##########
api/src/main/java/org/apache/iceberg/Metrics.java:
##########
@@ -150,8 +150,8 @@ public Metrics(
* @param nanValueCounts a map of field id to the number of NaN values, or null if unknown
* @param lowerBounds a map of field id to the lower bound of the column, or null if unknown
* @param upperBounds a map of field id to the upper bound of the column, or null if unknown
- * @param avgValueSizes a map of field id to the average size in bytes of the column's non-null
- * values, or null if unknown
+ * @param totalBytes a map of field id to the total uncompressed size in bytes of the column's
+ * non-null values, or null if unknown
Review Comment:
nit: `, or null if not collected` to be consistent with another Javadoc earlier?
##########
core/src/main/java/org/apache/iceberg/MetricsUtil.java:
##########
@@ -57,14 +56,14 @@ public static Metrics copyWithoutFieldCounts(Metrics metrics, Set<Integer> exclu
copyWithoutKeys(metrics.nanValueCounts(), excludedFieldIds),
metrics.lowerBounds(),
metrics.upperBounds(),
- copyWithoutKeys(metrics.avgValueSizes(), excludedFieldIds),
+ copyWithoutKeys(metrics.totalBytes(), excludedFieldIds),
metrics.originalTypes());
}
/**
- * Copies a metrics object without counts, average value sizes, and bounds for given fields.
+ * Copies a metrics object without counts, total bytes, and bounds for given fields.
*
- * @param excludedFieldIds field IDs for which the counts, average value sizes, and bounds must be
+ * @param excludedFieldIds field IDs for which the counts, total bytes, and bounds must be dropped
* dropped
Review Comment:
left a duplicate `dropped` word here.
##########
core/src/test/java/org/apache/iceberg/TestFieldStatsStruct.java:
##########
@@ -357,14 +357,15 @@ public void variantSerialization(RoundTripSerializer<FieldStatsStruct<?>> serial
int size = metadata.dictionarySize() + lowerObject.sizeInBytes();
Review Comment:
nit: just define `long size` to do one type cast.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core, Parquet, Spark: Update avg_value_size_in_bytes to total_bytes [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #18310:
URL: https://github.com/apache/iceberg/pull/18310#issuecomment-5934792215
I'm not super worried because no one can reasonably be filling that in or recording it at the moment. We could do the standard deprecation if folks are worried. Or even just keep the public methods and have them do the calculation if the other values (total size and non-null count) are present.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core, Parquet, Spark: Update avg_value_size_in_bytes to total_bytes [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #18310:
URL: https://github.com/apache/iceberg/pull/18310#discussion_r4208058072
##########
core/src/main/java/org/apache/iceberg/FieldStats.java:
##########
@@ -61,11 +61,35 @@ public interface FieldStats<T> {
/** The total NaN value count, defined only when {@link #hasNanValueCount()}. */
long nanValueCount();
+ /** Whether total bytes are tracked for this field. */
+ default boolean hasTotalBytes() {
+ return false;
+ }
+
+ /**
+ * The total uncompressed size in memory in bytes of non-null values, for variable-length types
+ * (string, binary, variant, geometry, geography).
+ */
+ default long totalBytes() {
+ return 0L;
+ }
+
/**
* The avg value size in memory (uncompressed) in bytes for variable-length types (string, binary,
* variant, geometry, geography)
+ *
+ * @deprecated since 1.13.0, will be removed in 1.14.0; use {@link #totalBytes()} instead.
*/
- Integer avgValueSizeInBytes();
+ @Deprecated
Review Comment:
we may not need to deprecate this API because it is just a cheap derived value computation. we can provide it as a convenient method if this is useful information to caller.
I guess one down side of keeping it in the signature. if we maintain consistency as other numeric types, this should return a primitive `int` and we need another `hasAvgValueSizeInBytes` API.
##########
api/src/main/java/org/apache/iceberg/Metrics.java:
##########
@@ -244,14 +244,50 @@ public Map<Integer, ByteBuffer> upperBounds() {
return upperBounds;
}
+ /**
+ * Get the total uncompressed size in memory in bytes of non-null values, for all fields where it
+ * was collected.
+ *
+ * @return a Map of fieldId to total uncompressed size in bytes
+ */
+ public Map<Integer, Long> totalBytes() {
+ return totalBytes;
+ }
+
/**
* Get the average value size in memory (uncompressed) in bytes over non-null values, for all
* fields where it was collected.
*
* @return a Map of fieldId to average value size in bytes
+ * @deprecated since 1.13.0, will be removed in 2.0.0; use {@link #totalBytes()} instead.
*/
+ @Deprecated
public Map<Integer, Integer> avgValueSizes() {
- return avgValueSizes;
+ return avgValueSizes(totalBytes, valueCounts, nullValueCounts);
Review Comment:
this is the API that we can probably drop without maintaining compatibility because it requires relatively expensive computation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core, Parquet, Spark: Update avg_value_size_in_bytes to total_bytes [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #18310:
URL: https://github.com/apache/iceberg/pull/18310#discussion_r4228009249
##########
api/src/main/java/org/apache/iceberg/Metrics.java:
##########
@@ -244,14 +244,50 @@ public Map<Integer, ByteBuffer> upperBounds() {
return upperBounds;
}
+ /**
+ * Get the total uncompressed size in memory in bytes of non-null values, for all fields where it
+ * was collected.
+ *
+ * @return a Map of fieldId to total uncompressed size in bytes
+ */
+ public Map<Integer, Long> totalBytes() {
+ return totalBytes;
+ }
+
/**
* Get the average value size in memory (uncompressed) in bytes over non-null values, for all
* fields where it was collected.
*
* @return a Map of fieldId to average value size in bytes
+ * @deprecated since 1.13.0, will be removed in 2.0.0; use {@link #totalBytes()} instead.
*/
+ @Deprecated
public Map<Integer, Integer> avgValueSizes() {
- return avgValueSizes;
+ return avgValueSizes(totalBytes, valueCounts, nullValueCounts);
Review Comment:
I'm ok either way but I think if we want to be consistent with how we typically deal with APIs, it's probably fine to deprecate it and then remove it in the next release, regardless of whether this calculation is expensive or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core, Parquet, Spark: Update avg_value_size_in_bytes to total_bytes [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on PR #18310:
URL: https://github.com/apache/iceberg/pull/18310#issuecomment-6036497281
@amogh-jahagirdar @stevenzwu @RussellSpitzer @anoopj can you guys please re-review? I've tried to maintain API backwards compatibility and left `avgValueSizes()` in place and just derived it in order to reduce RevAPI failures
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core, Parquet, Spark: Update avg_value_size_in_bytes to total_bytes [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #18310:
URL: https://github.com/apache/iceberg/pull/18310#discussion_r4208058072
##########
core/src/main/java/org/apache/iceberg/FieldStats.java:
##########
@@ -61,11 +61,35 @@ public interface FieldStats<T> {
/** The total NaN value count, defined only when {@link #hasNanValueCount()}. */
long nanValueCount();
+ /** Whether total bytes are tracked for this field. */
+ default boolean hasTotalBytes() {
+ return false;
+ }
+
+ /**
+ * The total uncompressed size in memory in bytes of non-null values, for variable-length types
+ * (string, binary, variant, geometry, geography).
+ */
+ default long totalBytes() {
+ return 0L;
+ }
+
/**
* The avg value size in memory (uncompressed) in bytes for variable-length types (string, binary,
* variant, geometry, geography)
+ *
+ * @deprecated since 1.13.0, will be removed in 1.14.0; use {@link #totalBytes()} instead.
*/
- Integer avgValueSizeInBytes();
+ @Deprecated
Review Comment:
we may not need to deprecate this API because it is just a cheap derived value computation. we can provide it as a convenient method if this is useful information to caller.
I guess one down side of keeping it in the signature. if we maintain consistency as other numeric types, this should return a primitive `int` and we need another `hasAvgValueSizeInBytes` API.
with all things considered (including dropping it in the ContentFile and Metrics), I am ok with dropping it here too and not maintaining the API compatability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core, Parquet, Spark: Update avg_value_size_in_bytes to total_bytes [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #18310:
URL: https://github.com/apache/iceberg/pull/18310#discussion_r4227996284
##########
core/src/main/java/org/apache/iceberg/FieldStats.java:
##########
@@ -61,11 +61,35 @@ public interface FieldStats<T> {
/** The total NaN value count, defined only when {@link #hasNanValueCount()}. */
long nanValueCount();
+ /** Whether total bytes are tracked for this field. */
+ default boolean hasTotalBytes() {
+ return false;
+ }
+
+ /**
+ * The total uncompressed size in memory in bytes of non-null values, for variable-length types
+ * (string, binary, variant, geometry, geography).
+ */
+ default long totalBytes() {
+ return 0L;
+ }
+
/**
* The avg value size in memory (uncompressed) in bytes for variable-length types (string, binary,
* variant, geometry, geography)
+ *
+ * @deprecated since 1.13.0, will be removed in 1.14.0; use {@link #totalBytes()} instead.
*/
- Integer avgValueSizeInBytes();
+ @Deprecated
Review Comment:
I would also be ok dropping it in all places, but I think just deprecating it and dropping it after the next release is just aligning better with our general strategy of maintaining APIs. Therefore I probably wouldn't want to include another `hasAvgValueSizeInBytes` just to mark it deprecated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core, Parquet, Spark: Update avg_value_size_in_bytes to total_bytes [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #18310:
URL: https://github.com/apache/iceberg/pull/18310#discussion_r4208058072
##########
core/src/main/java/org/apache/iceberg/FieldStats.java:
##########
@@ -61,11 +61,35 @@ public interface FieldStats<T> {
/** The total NaN value count, defined only when {@link #hasNanValueCount()}. */
long nanValueCount();
+ /** Whether total bytes are tracked for this field. */
+ default boolean hasTotalBytes() {
+ return false;
+ }
+
+ /**
+ * The total uncompressed size in memory in bytes of non-null values, for variable-length types
+ * (string, binary, variant, geometry, geography).
+ */
+ default long totalBytes() {
+ return 0L;
+ }
+
/**
* The avg value size in memory (uncompressed) in bytes for variable-length types (string, binary,
* variant, geometry, geography)
+ *
+ * @deprecated since 1.13.0, will be removed in 1.14.0; use {@link #totalBytes()} instead.
*/
- Integer avgValueSizeInBytes();
+ @Deprecated
Review Comment:
we may not need to deprecate this API because it is just a cheap derived value computation. we can provide it as a convenient method if this is useful information to caller.
I guess one down side of keeping it in the signature. if we maintain consistency as other numeric types, this should return a primitive `int` and we need another `hasAvgValueSizeInBytes` API.
Since I am leaning toward dropping it in the ContentFile and Metrics, I am also ok with dropping it here too and not maintaining the API compatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core, Parquet, Spark: Update avg_value_size_in_bytes to total_bytes [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu merged PR #18310:
URL: https://github.com/apache/iceberg/pull/18310
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core, Parquet, Spark: Update avg_value_size_in_bytes to total_bytes [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on PR #18310:
URL: https://github.com/apache/iceberg/pull/18310#issuecomment-5930724652
Given that we just released 1.12.0 this PR isn't a pure rename of avgValueSize() methods to totalBytes() in all of the places. We now actually have to do the changes in a backwards compatible way. I'll set the PR to WIP until I have the correct changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core, Parquet, Spark: Update avg_value_size_in_bytes to total_bytes [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #18310:
URL: https://github.com/apache/iceberg/pull/18310#discussion_r4146905015
##########
format/spec.md:
##########
@@ -827,7 +827,7 @@ Each stats struct holds statistics for one table field. It may contain the follo
| _optional_ | 4 | `value_count` | `long` | all | Number of values in the column (including null and NaN values) |
| _optional_ | 5 | `null_value_count` | `long` | optional fields | Number of null values in the column |
| _optional_ | 6 | `nan_value_count` | `long` | `float`, `double` | Number of NaN values in the column |
-| _optional_ | 7 | `avg_value_size_in_bytes` | `int` | `string`, `binary`, `variant`, `geometry`, `geography` | Avg value size in memory (uncompressed) in bytes over non-null values to estimate memory consumption |
Review Comment:
yes this is stacked and I've just rebased
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org