You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/03/26 14:48:50 UTC
[GitHub] [hudi] xiarixiaoyao opened a new pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
xiarixiaoyao opened a new pull request #5137:
URL: https://github.com/apache/hudi/pull/5137
…iting
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars removed a comment on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars removed a comment on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079727554
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079727554
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079838919
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
}, {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401",
"triggerID" : "1079727554",
"triggerType" : "MANUAL"
}, {
"hash" : "dcec725d12526135dac3be0073fddadf25d654e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dcec725d12526135dac3be0073fddadf25d654e9",
"triggerType" : "PUSH"
}, {
"hash" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7406",
"triggerID" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dcec725d12526135dac3be0073fddadf25d654e9 UNKNOWN
* eaf08d6f752baa9ca28cc3db1ff35c99e99bc207 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7406)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
YannByron commented on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079839012
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079737555
@xiarixiaoyao @xushiyan let's also think about how we can prevent such regressions in the future. Ideally, we should check in the test that you used to validate it as a smoke test (checking in that it completes, say, w/in 10s).
Alternatively, we can also reshape and commit it as a benchmark (also committing its output as a reference) so that we can at least verify it manually by running it periodically and comparing against the baseline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079727822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
}, {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401",
"triggerID" : "1079727554",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#discussion_r835784601
##########
File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala
##########
@@ -62,10 +62,10 @@ object AvroConversionUtils {
* @param rootCatalystType Catalyst [[StructType]] to be transformed into
* @return converter accepting Avro payload and transforming it into a Catalyst one (in the form of [[InternalRow]])
*/
- def createAvroToInternalRowConverter(rootAvroType: Schema, rootCatalystType: StructType): GenericRecord => Option[InternalRow] =
- record => sparkAdapter.createAvroDeserializer(rootAvroType, rootCatalystType)
Review comment:
@xiarixiaoyao that's a very sneaky issue
Please leave a note in a comment explaining what was the issue and how you've addressed it.
However, i'd suggest to just pull out SparkAdapter from the closure keeping the API of this method intact: we don't want to push casting (to InternalRow/GenericRecord) onto the users
```
def createAvroToInternalRowConverter(rootAvroType: Schema, rootCatalystType: StructType): GenericRecord => Option[InternalRow] = {
val deserilizer = sparkAdapter.createAvroDeserializer(rootAvroType, rootCatalystType)
record => deserializer.deserialize(record).map(_.asInstanceOf[InternalRow])
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079726338
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079872420
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
}, {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401",
"triggerID" : "1079727554",
"triggerType" : "MANUAL"
}, {
"hash" : "dcec725d12526135dac3be0073fddadf25d654e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dcec725d12526135dac3be0073fddadf25d654e9",
"triggerType" : "PUSH"
}, {
"hash" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7406",
"triggerID" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"triggerType" : "PUSH"
}, {
"hash" : "88ccdb5a456e683b4bcf6d87ded558b70d1aee32",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7412",
"triggerID" : "88ccdb5a456e683b4bcf6d87ded558b70d1aee32",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dcec725d12526135dac3be0073fddadf25d654e9 UNKNOWN
* eaf08d6f752baa9ca28cc3db1ff35c99e99bc207 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7406)
* 88ccdb5a456e683b4bcf6d87ded558b70d1aee32 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7412)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079838919
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
}, {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401",
"triggerID" : "1079727554",
"triggerType" : "MANUAL"
}, {
"hash" : "dcec725d12526135dac3be0073fddadf25d654e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dcec725d12526135dac3be0073fddadf25d654e9",
"triggerType" : "PUSH"
}, {
"hash" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7406",
"triggerID" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dcec725d12526135dac3be0073fddadf25d654e9 UNKNOWN
* eaf08d6f752baa9ca28cc3db1ff35c99e99bc207 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7406)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079887376
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
}, {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401",
"triggerID" : "1079727554",
"triggerType" : "MANUAL"
}, {
"hash" : "dcec725d12526135dac3be0073fddadf25d654e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dcec725d12526135dac3be0073fddadf25d654e9",
"triggerType" : "PUSH"
}, {
"hash" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7406",
"triggerID" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"triggerType" : "PUSH"
}, {
"hash" : "88ccdb5a456e683b4bcf6d87ded558b70d1aee32",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7412",
"triggerID" : "88ccdb5a456e683b4bcf6d87ded558b70d1aee32",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dcec725d12526135dac3be0073fddadf25d654e9 UNKNOWN
* 88ccdb5a456e683b4bcf6d87ded558b70d1aee32 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7412)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#discussion_r835843426
##########
File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala
##########
@@ -62,10 +62,10 @@ object AvroConversionUtils {
* @param rootCatalystType Catalyst [[StructType]] to be transformed into
* @return converter accepting Avro payload and transforming it into a Catalyst one (in the form of [[InternalRow]])
*/
- def createAvroToInternalRowConverter(rootAvroType: Schema, rootCatalystType: StructType): GenericRecord => Option[InternalRow] =
- record => sparkAdapter.createAvroDeserializer(rootAvroType, rootCatalystType)
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079829847
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
}, {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401",
"triggerID" : "1079727554",
"triggerType" : "MANUAL"
}, {
"hash" : "dcec725d12526135dac3be0073fddadf25d654e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dcec725d12526135dac3be0073fddadf25d654e9",
"triggerType" : "PUSH"
}, {
"hash" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401)
* dcec725d12526135dac3be0073fddadf25d654e9 UNKNOWN
* eaf08d6f752baa9ca28cc3db1ff35c99e99bc207 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079872420
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
}, {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401",
"triggerID" : "1079727554",
"triggerType" : "MANUAL"
}, {
"hash" : "dcec725d12526135dac3be0073fddadf25d654e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dcec725d12526135dac3be0073fddadf25d654e9",
"triggerType" : "PUSH"
}, {
"hash" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7406",
"triggerID" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"triggerType" : "PUSH"
}, {
"hash" : "88ccdb5a456e683b4bcf6d87ded558b70d1aee32",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7412",
"triggerID" : "88ccdb5a456e683b4bcf6d87ded558b70d1aee32",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dcec725d12526135dac3be0073fddadf25d654e9 UNKNOWN
* eaf08d6f752baa9ca28cc3db1ff35c99e99bc207 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7406)
* 88ccdb5a456e683b4bcf6d87ded558b70d1aee32 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7412)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079711532
@xushiyan @YannByron @leesf @alexeykudinkin
could you pls help me review this pr, thanks
its a serious bug
before pacth: 295553 ms
after patch: 5279 ms
```
val dfx = spark.range(0, 50000000).toDF("id")
.withColumn("c1", lit("dsfsdfsafsasdfa"))
.withColumn("c2", lit(12.99d))
.withColumn("c3", lit(1))
val avroSchemax = AvroConversionUtils.convertStructTypeToAvroSchema(dfx.schema, "record", "my")
val sparkSchema = dfx.schema
spark.sparkContext.getConf.registerAvroSchemas(avroSchemax)
val testRDD = HoodieSparkUtils.createRdd(dfx,"record", "my", Some(avroSchemax))
// warm up
dfx.count()
spark.time(testRDD.foreach(f => f))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079709263
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079748188
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
}, {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401",
"triggerID" : "1079727554",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079709263
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079709771
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079829534
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
}, {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401",
"triggerID" : "1079727554",
"triggerType" : "MANUAL"
}, {
"hash" : "dcec725d12526135dac3be0073fddadf25d654e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dcec725d12526135dac3be0073fddadf25d654e9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401)
* dcec725d12526135dac3be0073fddadf25d654e9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079862346
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
}, {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401",
"triggerID" : "1079727554",
"triggerType" : "MANUAL"
}, {
"hash" : "dcec725d12526135dac3be0073fddadf25d654e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dcec725d12526135dac3be0073fddadf25d654e9",
"triggerType" : "PUSH"
}, {
"hash" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7406",
"triggerID" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"triggerType" : "PUSH"
}, {
"hash" : "88ccdb5a456e683b4bcf6d87ded558b70d1aee32",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88ccdb5a456e683b4bcf6d87ded558b70d1aee32",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dcec725d12526135dac3be0073fddadf25d654e9 UNKNOWN
* eaf08d6f752baa9ca28cc3db1ff35c99e99bc207 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7406)
* 88ccdb5a456e683b4bcf6d87ded558b70d1aee32 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079838219
> @xiarixiaoyao thanks for fixing this!
>
> Can you please also add the test that you've used as a benchmark (based on JMH)?
i will add benchmark to cover it. not jmh.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079830211
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
}, {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401",
"triggerID" : "1079727554",
"triggerType" : "MANUAL"
}, {
"hash" : "dcec725d12526135dac3be0073fddadf25d654e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dcec725d12526135dac3be0073fddadf25d654e9",
"triggerType" : "PUSH"
}, {
"hash" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7406",
"triggerID" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401)
* dcec725d12526135dac3be0073fddadf25d654e9 UNKNOWN
* eaf08d6f752baa9ca28cc3db1ff35c99e99bc207 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7406)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079748188
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
}, {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401",
"triggerID" : "1079727554",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1080737427
thanks for finding the regression and fixing it. good job!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan merged pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
xushiyan merged pull request #5137:
URL: https://github.com/apache/hudi/pull/5137
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079732495
@xiarixiaoyao also, please update the PR description so that whoever will be reviewing this PR will not need to go to JIRA to understand what it is about
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079834292
@xiarixiaoyao thanks for fixing this!
Can you please also add the test that you've used as a benchmark (based on JMH)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079709771
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079829847
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
}, {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401",
"triggerID" : "1079727554",
"triggerType" : "MANUAL"
}, {
"hash" : "dcec725d12526135dac3be0073fddadf25d654e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dcec725d12526135dac3be0073fddadf25d654e9",
"triggerType" : "PUSH"
}, {
"hash" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401)
* dcec725d12526135dac3be0073fddadf25d654e9 UNKNOWN
* eaf08d6f752baa9ca28cc3db1ff35c99e99bc207 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079830211
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
}, {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401",
"triggerID" : "1079727554",
"triggerType" : "MANUAL"
}, {
"hash" : "dcec725d12526135dac3be0073fddadf25d654e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dcec725d12526135dac3be0073fddadf25d654e9",
"triggerType" : "PUSH"
}, {
"hash" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7406",
"triggerID" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401)
* dcec725d12526135dac3be0073fddadf25d654e9 UNKNOWN
* eaf08d6f752baa9ca28cc3db1ff35c99e99bc207 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7406)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079862346
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
}, {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401",
"triggerID" : "1079727554",
"triggerType" : "MANUAL"
}, {
"hash" : "dcec725d12526135dac3be0073fddadf25d654e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dcec725d12526135dac3be0073fddadf25d654e9",
"triggerType" : "PUSH"
}, {
"hash" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7406",
"triggerID" : "eaf08d6f752baa9ca28cc3db1ff35c99e99bc207",
"triggerType" : "PUSH"
}, {
"hash" : "88ccdb5a456e683b4bcf6d87ded558b70d1aee32",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88ccdb5a456e683b4bcf6d87ded558b70d1aee32",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dcec725d12526135dac3be0073fddadf25d654e9 UNKNOWN
* eaf08d6f752baa9ca28cc3db1ff35c99e99bc207 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7406)
* 88ccdb5a456e683b4bcf6d87ded558b70d1aee32 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079726338
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079727822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
}, {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401",
"triggerID" : "1079727554",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5137:
URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079829534
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398",
"triggerID" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"triggerType" : "PUSH"
}, {
"hash" : "6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401",
"triggerID" : "1079727554",
"triggerType" : "MANUAL"
}, {
"hash" : "dcec725d12526135dac3be0073fddadf25d654e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dcec725d12526135dac3be0073fddadf25d654e9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7398) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7401)
* dcec725d12526135dac3be0073fddadf25d654e9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org