You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Alowator (via GitHub)" <gi...@apache.org> on 2025/01/29 08:38:33 UTC
[PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Alowator opened a new pull request, #12729:
URL: https://github.com/apache/hudi/pull/12729
### Change Logs
Inspired by RFC-84 [HUDI-8920](https://issues.apache.org/jira/browse/HUDI-8920): there is an opinion Avro is not the best choice for Hudi. It requires an extra ser/de operations not only between Flink operators (will be fixed by RFC-84).
I decided to benchmark a POC version with native Flink's RowData writer for Hudi. It was simple enough, because Hudi already has native RowData to Parquet writer used by append mode, I reused this writer to create log blocks and two bottlenecks were found:
1. Hudi performs a lot of Avro ser/de operations in writer runtime.
2. Hudi stores Avro recrods as List<HoodieRecord>, it causes a GC pressure on writer runtime, on my benchmarks garbage collection is about 30% of all hudi writer runtime.
### Impact
As a result I reduced write time from ~4min to ~1min 20sec (x3 write performance boost):
I have a POC version we are already testing in our cloud environment, key improvements:
1. Write native RowData to parquet log blocks (eliminate unnecessary ser/de)
2. Recrods are stored in BinaryInMemorySortBuffer (reduce GC pressure)
3. Records are sorted by new QuickSort().sort(sortBuffer) before writing log block (maybe it's possible to preform data compaction without sorting? this sort performs fast enough so it doesn't affect write performance.)
4. RowDataStreamWriteFunction flushes bucket async (reduces previous operators backpressure)
### Risk level (write none, low medium or high below)
None
### Documentation Update
None
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2680019834
@vinothchandar
> Can someone help understand any overlaps of this effort vs RFC-84 and @geserdugarov 's PR that's nearing landing..
RFC-84 focus on SE/DE performance between Flink operators, this one focus on writers.
> Can this work layer on top and add more speedup?
Definitely yes, they are orthogonal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "Alowator (via GitHub)" <gi...@apache.org>.
Alowator commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2680945328
> ## CI report:
> * [76a89c2](https://github.com/apache/hudi/commit/76a89c23a22a754b1852d00a32bfb40981d4c35f) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3877)
>
> Bot commands
> @hudi-bot supports the following commands:
> * `@hudi-bot run azure` re-run the last Azure build
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2677323700
> > > > @Alowator @cshuo @danny0405 great to see this effort. Do we agree on two separate rfcs?
> > > > Let's make a call and either land this PR to claim 87 or add @Alowator to 88?
> > >
> > >
> > > As discussed offline, it's ok to keep the flink optimizing stuff separate, so as to make it possible to land solving avro performance issue in release 1.1. @Alowator Could you briefly outline the scope of RFC-87, so that optimization works about other aspect of Flink integration can start simultaneously, such as reading/compaction..just make sure we are not doing the same things.
> >
> >
> > @cshuo I agree with your point. To ensure we don't duplicate efforts, here’s the plan I propose:
> >
> > 1. RFC-88 will focus solely on defining new abstractions with old avro writers.
> > 2. RFC-87 will be dedicated to performance improvements that leverage those abstractions.
> > Since RFC-88 is big, I can assist by implementing the abstractions for the writer as part of that RFC. Then, I will move forward with implementing Avro elimination in RFC-87.
> >
> > For RFC-87, I can start drafting the design for the performance improvements. However, the actual implementation of this design will require the abstractions from RFC-88 to be completed first.
> > For RFC-87 it does not affect the reader's logic directly. However, conflicts could arise in the compactor logic, for writers which utilize it. These potential conflicts should be clearly outlined in the design of RFC-87.
> > TL;DR: It’s possible to start designing RFC-87 without RFC-88, but full implementation depends on completing the abstractions in RFC-88 first.
>
> Hi @Alowator, after discussing with @danny0405, we think the optimizing of Flink reading/writing has a higher priority, at least for release 1.1. In summary, we have the following suggestions:
>
> * Push 87 forward without depending on RFC-88 at all, including the abstraction of writers/readers. As the abstraction of reader/writer in RFC-88 has other dependencies, for eg, Data Type. So let's keep RFC-87 orthogonal, and make it possible to land without any blocker.
> * As for the scope of this RFC, it's ok this RFC focuses on writing part of Flink integration, and for design choice wise, it'd be better that data is kept as `RowData` all the way in the writing path without any conversions, e.g., HoodieRecord, to achieve optimal write performance.
@Alowator kindly reminder, hope we can moving fast to solving the flink read/writing performance issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "Alowator (via GitHub)" <gi...@apache.org>.
Alowator commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2667844385
> > @Alowator @cshuo @danny0405 great to see this effort. Do we agree on two separate rfcs?
> > Let's make a call and either land this PR to claim 87 or add @Alowator to 88?
>
> As discussed offline, it's ok to keep the flink optimizing stuff separate, so as to make it possible to land solving avro performance issue in release 1.1. @Alowator Could you briefly outline the scope of RFC-87, so that optimization works about other aspect of Flink integration can start simultaneously, such as reading/compaction..just make sure we are not doing the same things.
@cshuo I agree with your point. To ensure we don't duplicate efforts, here’s the plan I propose:
1. RFC-88 will focus solely on defining new abstractions with old avro writers.
2. RFC-87 will be dedicated to performance improvements that leverage those abstractions.
Since RFC-88 is big, I can assist by implementing the abstractions for the writer as part of that RFC. Then, I will move forward with implementing Avro elimination in RFC-87.
For RFC-87, I can start drafting the design for the performance improvements. However, the actual implementation of this design will require the abstractions from RFC-88 to be completed first.
For RFC-87 it does not affect the reader's logic directly. However, conflicts could arise in the compactor logic, for writers which utilize it. These potential conflicts should be clearly outlined in the design of RFC-87.
TL;DR: It’s possible to start designing RFC-87 without RFC-88, but full implementation depends on completing the abstractions in RFC-88 first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2681197195
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fa4a3d1a4a35c0cbc641691368a360c8ba442b71",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3250",
"triggerID" : "fa4a3d1a4a35c0cbc641691368a360c8ba442b71",
"triggerType" : "PUSH"
}, {
"hash" : "76a89c23a22a754b1852d00a32bfb40981d4c35f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3877",
"triggerID" : "76a89c23a22a754b1852d00a32bfb40981d4c35f",
"triggerType" : "PUSH"
}, {
"hash" : "76a89c23a22a754b1852d00a32bfb40981d4c35f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3880",
"triggerID" : "2680945328",
"triggerType" : "MANUAL"
}, {
"hash" : "76a89c23a22a754b1852d00a32bfb40981d4c35f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3877",
"triggerID" : "2680945328",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 76a89c23a22a754b1852d00a32bfb40981d4c35f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3880) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3877)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2680651611
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fa4a3d1a4a35c0cbc641691368a360c8ba442b71",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3250",
"triggerID" : "fa4a3d1a4a35c0cbc641691368a360c8ba442b71",
"triggerType" : "PUSH"
}, {
"hash" : "76a89c23a22a754b1852d00a32bfb40981d4c35f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3877",
"triggerID" : "76a89c23a22a754b1852d00a32bfb40981d4c35f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fa4a3d1a4a35c0cbc641691368a360c8ba442b71 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3250)
* 76a89c23a22a754b1852d00a32bfb40981d4c35f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3877)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2623452592
> > @cshuo is working on a RFC to introduce basic infras to Hudi, such as schema/data type/expressions, one of the goles is to integrate with engine native "row" for both reader and writer, welcome to contribute to it, I will cc you once the RFC is comming out.
>
> So do you think it’s better to do it as a part of another RFC ? I think creation of engine native row for Flink is a very big task to make it as part of another RFC, so it’s better to clearly separate this work
That's my thought, @cshuo 's RFC is huge and it's great if you and @geserdugarov can get involved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2621000477
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fa4a3d1a4a35c0cbc641691368a360c8ba442b71",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3250",
"triggerID" : "fa4a3d1a4a35c0cbc641691368a360c8ba442b71",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fa4a3d1a4a35c0cbc641691368a360c8ba442b71 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3250)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "Alowator (via GitHub)" <gi...@apache.org>.
Alowator commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2677631672
> > > > > @Alowator @cshuo @danny0405 great to see this effort. Do we agree on two separate rfcs?
> > > > > Let's make a call and either land this PR to claim 87 or add @Alowator to 88?
> > > >
> > > >
> > > > As discussed offline, it's ok to keep the flink optimizing stuff separate, so as to make it possible to land solving avro performance issue in release 1.1. @Alowator Could you briefly outline the scope of RFC-87, so that optimization works about other aspect of Flink integration can start simultaneously, such as reading/compaction..just make sure we are not doing the same things.
> > >
> > >
> > > @cshuo I agree with your point. To ensure we don't duplicate efforts, here’s the plan I propose:
> > >
> > > 1. RFC-88 will focus solely on defining new abstractions with old avro writers.
> > > 2. RFC-87 will be dedicated to performance improvements that leverage those abstractions.
> > > Since RFC-88 is big, I can assist by implementing the abstractions for the writer as part of that RFC. Then, I will move forward with implementing Avro elimination in RFC-87.
> > >
> > > For RFC-87, I can start drafting the design for the performance improvements. However, the actual implementation of this design will require the abstractions from RFC-88 to be completed first.
> > > For RFC-87 it does not affect the reader's logic directly. However, conflicts could arise in the compactor logic, for writers which utilize it. These potential conflicts should be clearly outlined in the design of RFC-87.
> > > TL;DR: It’s possible to start designing RFC-87 without RFC-88, but full implementation depends on completing the abstractions in RFC-88 first.
> >
> >
> > Hi @Alowator, after discussing with @danny0405, we think the optimizing of Flink reading/writing has a higher priority, at least for release 1.1. In summary, we have the following suggestions:
> >
> > * Push 87 forward without depending on RFC-88 at all, including the abstraction of writers/readers. As the abstraction of reader/writer in RFC-88 has other dependencies, for eg, Data Type. So let's keep RFC-87 orthogonal, and make it possible to land without any blocker.
> > * As for the scope of this RFC, it's ok this RFC focuses on writing part of Flink integration, and for design choice wise, it'd be better that data is kept as `RowData` all the way in the writing path without any conversions, e.g., HoodieRecord, to achieve optimal write performance.
>
> @Alowator kindly reminder, hope we can moving fast to solving the flink read/writing performance issue.
Ok, lets merge this RFC claim I will try to write a design of these changes in few days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2680020319
@Alowator Can you fix the conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2679991334
Can someone help understand any overlaps of this effort vs RFC-84 and @geserdugarov 's PR that's nearing landing..
Can this work layer on top and add more speedup?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "Alowator (via GitHub)" <gi...@apache.org>.
Alowator commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2680671714
> @Alowator Can you fix the conflicts?
Done.
> Can someone help understand any overlaps of this effort vs RFC-84 and @geserdugarov 's PR that's nearing landing..
>
> Can this work layer on top and add more speedup?
#12796 already merged so we can start to work on this RFC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2623428178
@cshuo is working on a RFC to introduce basic infras to Hudi, such as schema/data type/expressions, one of the goles is to integrate with engine native "row" for both reader and writer, welcome to contribute to it, I will cc you once the RFC is comming out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2680041938
> Can someone help understand any overlaps of this effort vs RFC-84 and @geserdugarov 's PR that's nearing landing..
>
>
>
> Can this work layer on top and add more speedup?
I would say that this part should be done on top of RFC-84, because it expects that we already have Flink native RowData in writers.
And if it done this way, it will lead to great optimization. RFC-84 ensures that there is no slowdown in operators before writers. Flink uses backpressure to adapt processing speed of separate operators in the pipeline. So, after RFC-84 the most slowing down part is writing. And any optimization of it, would lead to significant improvement of total optimization, because Flink would reduce backpressure on previous operators until we faced the next bottleneck.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2667219232
@Alowator @cshuo @danny0405 great to see this effort. Do we agree on two separate rfcs?
Let's make a call and either land this PR to claim 87 or add @Alowator to 88?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "Alowator (via GitHub)" <gi...@apache.org>.
Alowator commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2623658467
> > > @cshuo is working on a RFC to introduce basic infras to Hudi, such as schema/data type/expressions, one of the goles is to integrate with engine native "row" for both reader and writer, welcome to contribute to it, I will cc you once the RFC is comming out.
> >
> >
> > So do you think it’s better to do it as a part of another RFC ? I think creation of engine native row for Flink is a very big task to make it as part of another RFC, so it’s better to clearly separate this work
>
> That's my thought, @cshuo 's RFC is huge and it's great if you and @geserdugarov can get involved.
Ok, will look forward for @cshuo 's RFC. This work requires a scrutiny related to Flink write performance, including moments I presented in this ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2667569542
> I can take this job and do it in this RFC-87.
@Alowator did you mean all the writers for all the writer scenarios?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2668598958
> > > @Alowator @cshuo @danny0405 great to see this effort. Do we agree on two separate rfcs?
> > > Let's make a call and either land this PR to claim 87 or add @Alowator to 88?
> >
> >
> > As discussed offline, it's ok to keep the flink optimizing stuff separate, so as to make it possible to land solving avro performance issue in release 1.1. @Alowator Could you briefly outline the scope of RFC-87, so that optimization works about other aspect of Flink integration can start simultaneously, such as reading/compaction..just make sure we are not doing the same things.
>
> @cshuo I agree with your point. To ensure we don't duplicate efforts, here’s the plan I propose:
>
> 1. RFC-88 will focus solely on defining new abstractions with old avro writers.
> 2. RFC-87 will be dedicated to performance improvements that leverage those abstractions.
> Since RFC-88 is big, I can assist by implementing the abstractions for the writer as part of that RFC. Then, I will move forward with implementing Avro elimination in RFC-87.
>
> For RFC-87, I can start drafting the design for the performance improvements. However, the actual implementation of this design will require the abstractions from RFC-88 to be completed first.
>
> For RFC-87 it does not affect the reader's logic directly. However, conflicts could arise in the compactor logic, for writers which utilize it. These potential conflicts should be clearly outlined in the design of RFC-87.
>
> TL;DR: It’s possible to start designing RFC-87 without RFC-88, but full implementation depends on completing the abstractions in RFC-88 first.
Hi @Alowator, after discussing with @danny0405, we think the optimizing of Flink reading/writing has a higher priority, at least for release 1.1. In summary, we have the following recommendations:
* Push 87 forward without depending on RFC-88 at all, including the abstraction of writers/readers. As the abstraction of reader/writer in RFC-88 has other dependencies, for eg, Data Type. So just keep RFC-87 orthogonal, and make it possible to land without any blocker.
* As for the scope of this RFC, it's ok this RFC focuses on writing part of Flink integration, and for design choice wise, it'd be better that data is kept as `RowData` all the way in the writing path, without any conversions, e.g., HoodieRecord, to achieve optimal write performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #12729:
URL: https://github.com/apache/hudi/pull/12729
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2667673991
> @Alowator @cshuo @danny0405 great to see this effort. Do we agree on two separate rfcs?
>
> Let's make a call and either land this PR to claim 87 or add @Alowator to 88?
As discussed offline, it's ok to keep the flink optimizing stuff separate, so as to make it possible to land solving avro performance issue in release 1.1.
@Alowator Could you briefly outline the scope of RFC-87, so that optimization works about other aspect of Flink integration can start simultaneously, such as reading/compaction..just make sure we are not doing the same things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2667484818
@Alowator Can you share your the scope you want to take, becase we want to rewrite all the writers/readers/compactors/clustering ..., that is a lot of work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2620997527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fa4a3d1a4a35c0cbc641691368a360c8ba442b71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fa4a3d1a4a35c0cbc641691368a360c8ba442b71",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fa4a3d1a4a35c0cbc641691368a360c8ba442b71 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2680948660
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fa4a3d1a4a35c0cbc641691368a360c8ba442b71",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3250",
"triggerID" : "fa4a3d1a4a35c0cbc641691368a360c8ba442b71",
"triggerType" : "PUSH"
}, {
"hash" : "76a89c23a22a754b1852d00a32bfb40981d4c35f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3877",
"triggerID" : "76a89c23a22a754b1852d00a32bfb40981d4c35f",
"triggerType" : "PUSH"
}, {
"hash" : "76a89c23a22a754b1852d00a32bfb40981d4c35f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3880",
"triggerID" : "2680945328",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 76a89c23a22a754b1852d00a32bfb40981d4c35f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3877) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3880)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "Alowator (via GitHub)" <gi...@apache.org>.
Alowator commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2667499807
@danny0405 @vinothchandar In this RFC, I only mention writers, and I already have a prototype of such writer, so I think it's logical to decompose this work and implement new writers as RFC-87 to focus on write performance implementing infras proposed in RFC-87.
I can take this job and do it in this RFC-87.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2635497376
@Alowator Great benchmark work! The redundant Ser/De of Avro in the hudi read/write path has been a long-standing problem. As @danny0405 mentioned, I'm working on a RFC to introduce some basic infras, which will come out soon, and it's great if you can get involved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2680890689
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fa4a3d1a4a35c0cbc641691368a360c8ba442b71",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3250",
"triggerID" : "fa4a3d1a4a35c0cbc641691368a360c8ba442b71",
"triggerType" : "PUSH"
}, {
"hash" : "76a89c23a22a754b1852d00a32bfb40981d4c35f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3877",
"triggerID" : "76a89c23a22a754b1852d00a32bfb40981d4c35f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 76a89c23a22a754b1852d00a32bfb40981d4c35f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3877)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2680954607
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fa4a3d1a4a35c0cbc641691368a360c8ba442b71",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3250",
"triggerID" : "fa4a3d1a4a35c0cbc641691368a360c8ba442b71",
"triggerType" : "PUSH"
}, {
"hash" : "76a89c23a22a754b1852d00a32bfb40981d4c35f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3877",
"triggerID" : "76a89c23a22a754b1852d00a32bfb40981d4c35f",
"triggerType" : "PUSH"
}, {
"hash" : "76a89c23a22a754b1852d00a32bfb40981d4c35f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3880",
"triggerID" : "2680945328",
"triggerType" : "MANUAL"
}, {
"hash" : "76a89c23a22a754b1852d00a32bfb40981d4c35f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3877",
"triggerID" : "2680945328",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 76a89c23a22a754b1852d00a32bfb40981d4c35f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3880) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3877)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2639724373
@Alowator here is RFC pr #12795, welcome to review and get involved to collaborate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2680649267
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fa4a3d1a4a35c0cbc641691368a360c8ba442b71",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3250",
"triggerID" : "fa4a3d1a4a35c0cbc641691368a360c8ba442b71",
"triggerType" : "PUSH"
}, {
"hash" : "76a89c23a22a754b1852d00a32bfb40981d4c35f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "76a89c23a22a754b1852d00a32bfb40981d4c35f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fa4a3d1a4a35c0cbc641691368a360c8ba442b71 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3250)
* 76a89c23a22a754b1852d00a32bfb40981d4c35f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2621194582
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fa4a3d1a4a35c0cbc641691368a360c8ba442b71",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3250",
"triggerID" : "fa4a3d1a4a35c0cbc641691368a360c8ba442b71",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fa4a3d1a4a35c0cbc641691368a360c8ba442b71 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3250)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8934] Claim of RFC-87. Avro elimination for Flink writer [hudi]
Posted by "Alowator (via GitHub)" <gi...@apache.org>.
Alowator commented on PR #12729:
URL: https://github.com/apache/hudi/pull/12729#issuecomment-2623434249
> @cshuo is working on a RFC to introduce basic infras to Hudi, such as schema/data type/expressions, one of the goles is to integrate with engine native "row" for both reader and writer, welcome to contribute to it, I will cc you once the RFC is comming out.
So do you think it’s better to do it as a part of another RFC ? I think creation of engine native row for Flink is a big enough task to make it as part of another RFC .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org