You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "beyond1920 (via GitHub)" <gi...@apache.org> on 2024/04/08 07:06:07 UTC
[I] [SUPPORT] Compaction job become much more slower after upgrade from 0.10 version to 0.14 version [hudi]
beyond1920 opened a new issue, #10978:
URL: https://github.com/apache/hudi/issues/10978
Dear community,
After upgrade a hudi table from 0.10 version to 0.14 version, the compaction job become much more slower.
The hudi table is a MOR table without partition field. And the hudi table does not do any schema evolution.
The compaction job would finished in 52 minutes using 0.14 version.
<img width="2011" alt="image" src="https://github.com/apache/hudi/assets/1525333/f825da15-5319-4ab2-9f0c-97741f4ea4f7">
The compaction job would finished in 25 minutes using 0.10 version.
<img width="1974" alt="image" src="https://github.com/apache/hudi/assets/1525333/223c3fc2-7991-40a0-8a86-5a949821c55e">
And in the 0.14 version, the task jstack become much more complex. Including the following content:
<img width="1433" alt="image" src="https://github.com/apache/hudi/assets/1525333/9394a3b4-3074-4ba5-bd07-7c73f195085f">
After compare 0.14 and 0.10 version, we found there is a difference when copy the old record from old base file to new base file.
In 0.14 version, the cost is much more heavy.
<img width="1259" alt="image" src="https://github.com/apache/hudi/assets/1525333/879b0f8e-dbc8-458b-9b45-afdced25580c">
<img width="1354" alt="image" src="https://github.com/apache/hudi/assets/1525333/d22835b2-7d6c-44ae-aaf1-967d1622c9ae">
<img width="1438" alt="image" src="https://github.com/apache/hudi/assets/1525333/438984f7-5d3f-4635-ae64-d3221d73cc34">
<img width="1627" alt="image" src="https://github.com/apache/hudi/assets/1525333/e1d5ddb4-1544-4f17-b9f9-6193765c8bed">
In 0.10 version, the copy is more simple.
<img width="1421" alt="image" src="https://github.com/apache/hudi/assets/1525333/28eb2af7-e0f2-43b7-bfc7-f174e30cd944">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Compaction job become much more slower after upgrade from 0.10 version to 0.14 version [hudi]
Posted by "beyond1920 (via GitHub)" <gi...@apache.org>.
beyond1920 commented on issue #10978:
URL: https://github.com/apache/hudi/issues/10978#issuecomment-2042010815
I think rewriting all fields value of each old record is not necessary, update new file path value and metadata fields are enough.
I simplify the copy old record logical, the compaction job in 0.14 version is as fast as 0.10 version.
Any suggestion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Compaction job become much more slower after upgrade from 0.10 version to 0.14 version [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #10978: [SUPPORT] Compaction job become much more slower after upgrade from 0.10 version to 0.14 version
URL: https://github.com/apache/hudi/issues/10978
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Compaction job become much more slower after upgrade from 0.10 version to 0.14 version [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10978:
URL: https://github.com/apache/hudi/issues/10978#issuecomment-2114887810
Closing this issue as got fixed in https://github.com/apache/hudi/pull/11028
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Compaction job become much more slower after upgrade from 0.10 version to 0.14 version [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10978:
URL: https://github.com/apache/hudi/issues/10978#issuecomment-2042211389
Reasonable, will take some time to check the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org