You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2025/09/16 00:06:02 UTC
[PR] [MINOR[] Fixing record size estimator for avro record [hudi]
nsivabalan opened a new pull request, #13896:
URL: https://github.com/apache/hudi/pull/13896
### Change Logs
Fixing record size estimator for avro record. Sometimes we might use records w/o meta fields while deserializing (COW merge paths), while compaction might use it w/ meta fields. We happened to instantiate AvroRecordSizeEstimator with a schema containing meta fields, but `SizeEstimator.sizeEstimate()` is invoked for a BufferedRecord w/o meta fields which could lead to negative size estimations
### Impact
Robust record size estimation for avro records.
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fixing record size estimator for avro record [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13896:
URL: https://github.com/apache/hudi/pull/13896#issuecomment-3294446032
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8dac723c778bb51ec90db80251b2b95b5ef20a49",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8546",
"triggerID" : "8dac723c778bb51ec90db80251b2b95b5ef20a49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8dac723c778bb51ec90db80251b2b95b5ef20a49 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8546)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fixing record size estimator for avro record [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13896:
URL: https://github.com/apache/hudi/pull/13896#issuecomment-3294385143
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8dac723c778bb51ec90db80251b2b95b5ef20a49",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dac723c778bb51ec90db80251b2b95b5ef20a49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8dac723c778bb51ec90db80251b2b95b5ef20a49 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fixing record size estimator for avro record [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13896:
URL: https://github.com/apache/hudi/pull/13896#issuecomment-3305327615
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8dac723c778bb51ec90db80251b2b95b5ef20a49",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8546",
"triggerID" : "8dac723c778bb51ec90db80251b2b95b5ef20a49",
"triggerType" : "PUSH"
}, {
"hash" : "95056652c203e6050d9151bfd99a95ccab2d679e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8613",
"triggerID" : "95056652c203e6050d9151bfd99a95ccab2d679e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8dac723c778bb51ec90db80251b2b95b5ef20a49 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8546)
* 95056652c203e6050d9151bfd99a95ccab2d679e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8613)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fixing record size estimator for avro record [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13896:
URL: https://github.com/apache/hudi/pull/13896#discussion_r2357442844
##########
hudi-common/src/main/java/org/apache/hudi/avro/AvroRecordSizeEstimator.java:
##########
@@ -45,6 +49,8 @@ public long sizeEstimate(BufferedRecord<IndexedRecord> record) {
return sizeOfRecord;
}
// do not contain size of Avro schema as the schema is reused among records
+ Schema recordSchema = record.getRecord().getSchema();
+ long sizeOfSchema = sizeOfSchemaMap.computeIfAbsent(recordSchema.getFields().size(), arity -> ObjectSizeCalculator.getObjectSize(recordSchema));
Review Comment:
cc @cshuo to review this, guess there is no much perf regression after the change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fixing record size estimator for avro record [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13896:
URL: https://github.com/apache/hudi/pull/13896#issuecomment-3294388603
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8dac723c778bb51ec90db80251b2b95b5ef20a49",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8546",
"triggerID" : "8dac723c778bb51ec90db80251b2b95b5ef20a49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8dac723c778bb51ec90db80251b2b95b5ef20a49 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8546)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fixing record size estimator for avro record [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13896:
URL: https://github.com/apache/hudi/pull/13896#issuecomment-3305378375
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8dac723c778bb51ec90db80251b2b95b5ef20a49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8546",
"triggerID" : "8dac723c778bb51ec90db80251b2b95b5ef20a49",
"triggerType" : "PUSH"
}, {
"hash" : "95056652c203e6050d9151bfd99a95ccab2d679e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8613",
"triggerID" : "95056652c203e6050d9151bfd99a95ccab2d679e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 95056652c203e6050d9151bfd99a95ccab2d679e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8613)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fixing record size estimator for avro record [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #13896:
URL: https://github.com/apache/hudi/pull/13896#discussion_r2357644952
##########
hudi-common/src/main/java/org/apache/hudi/avro/AvroRecordSizeEstimator.java:
##########
@@ -45,6 +49,8 @@ public long sizeEstimate(BufferedRecord<IndexedRecord> record) {
return sizeOfRecord;
}
// do not contain size of Avro schema as the schema is reused among records
+ Schema recordSchema = record.getRecord().getSchema();
+ long sizeOfSchema = sizeOfSchemaMap.computeIfAbsent(recordSchema.getFields().size(), arity -> ObjectSizeCalculator.getObjectSize(recordSchema));
Review Comment:
yes, for cow write path, there can be 2 schemas, incoming record do not contain schema fields. Regarding the performance, I think it'll not incur much degradation, since the size estimating is performed through sampling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fixing record size estimator for avro record [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed pull request #13896: [MINOR] Fixing record size estimator for avro record
URL: https://github.com/apache/hudi/pull/13896
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fixing record size estimator for avro record [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13896:
URL: https://github.com/apache/hudi/pull/13896#discussion_r2357413010
##########
hudi-common/src/main/java/org/apache/hudi/avro/AvroRecordSizeEstimator.java:
##########
@@ -45,6 +80,11 @@ public long sizeEstimate(BufferedRecord<IndexedRecord> record) {
return sizeOfRecord;
}
// do not contain size of Avro schema as the schema is reused among records
- return sizeOfRecord - sizeOfSchema + 8;
+ long toReturn = sizeOfRecord - sizeOfSchema + 8;
+ if (toReturn < 0) {
Review Comment:
This check is invalid if the record size is huge itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fixing record size estimator for avro record [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13896:
URL: https://github.com/apache/hudi/pull/13896#issuecomment-3305324727
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8dac723c778bb51ec90db80251b2b95b5ef20a49",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8546",
"triggerID" : "8dac723c778bb51ec90db80251b2b95b5ef20a49",
"triggerType" : "PUSH"
}, {
"hash" : "95056652c203e6050d9151bfd99a95ccab2d679e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "95056652c203e6050d9151bfd99a95ccab2d679e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8dac723c778bb51ec90db80251b2b95b5ef20a49 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8546)
* 95056652c203e6050d9151bfd99a95ccab2d679e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org