You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/08/08 17:18:35 UTC
[GitHub] [hudi] NikhilCollooru opened a new issue, #6332: Avoid creating Configuration copies in Hudi
NikhilCollooru opened a new issue, #6332:
URL: https://github.com/apache/hudi/issues/6332
We observed that creating Configuration copies is consuming a lot of CPU.
We made changes in Presto to use a wrapper instead of creating configuration copies. But we were told that the presto change is breaking Hudi.
So one suggestion is to avoid creating copies in places like this:
https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/config/SerializableConfiguration.java#L37
we can instead simply do
```
public SerializableConfiguration(Configuration configuration) {
this.configuration = configuration;
}
```
The breaking Presto PR: https://github.com/prestodb/presto/pull/18115
https://github.com/prestodb/presto/issues/17736
@pratyakshsharma @7c00 do you the code suggestion above makes sense ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on issue #6332: Avoid creating Configuration copies in Hudi
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on issue #6332:
URL: https://github.com/apache/hudi/issues/6332#issuecomment-1210434264
I see the changes were introduced in this commit - https://github.com/apache/hudi/pull/755. Hence tagging @bvaradar here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #6332: Avoid creating Configuration copies in Hudi
Posted by GitBox <gi...@apache.org>.
codope commented on issue #6332:
URL: https://github.com/apache/hudi/issues/6332#issuecomment-1210443411
I think avoiding copies may not be that trivial. I understand this might slow down queries via presto-hudi connector but I still want to understand the breaking change in detail. @pratyakshsharma @NikhilCollooru do you have some time this week to sync up and discuss this? I am `sagar sumit` on Hudi Slack.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on issue #6332: Avoid creating Configuration copies in Hudi
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on issue #6332:
URL: https://github.com/apache/hudi/issues/6332#issuecomment-1263352404
@codope Let us connect sometime next week to discuss this. This has been pending for some time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed issue #6332: Avoid creating Configuration copies in Hudi
Posted by GitBox <gi...@apache.org>.
codope closed issue #6332: Avoid creating Configuration copies in Hudi
URL: https://github.com/apache/hudi/issues/6332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #6332: Avoid creating Configuration copies in Hudi
Posted by GitBox <gi...@apache.org>.
codope commented on issue #6332:
URL: https://github.com/apache/hudi/issues/6332#issuecomment-1210435469
The reason for creating copies was to avoid ConcurrentModificationException encountered while long-running deltastreamer jobs. To avoid current modification, table meta client and file system get exclusive copies of the configuration. See https://github.com/apache/hudi/pull/755 for context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed issue #6332: Avoid creating Configuration copies in Hudi
Posted by GitBox <gi...@apache.org>.
codope closed issue #6332: Avoid creating Configuration copies in Hudi
URL: https://github.com/apache/hudi/issues/6332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #6332: Avoid creating Configuration copies in Hudi
Posted by GitBox <gi...@apache.org>.
codope commented on issue #6332:
URL: https://github.com/apache/hudi/issues/6332#issuecomment-1265613884
Synced up with @pratyakshsharma regarding this issue. First of all, the issue affects hudi tables queries via presto-hive connector. We need to see if we can use the config provided by the engine itself while instantiating the meta client. Created HUDI-4974 to track that issue. For now, we have a mitigation. We will unwrap the wrapper config object and pass that.
Closing the issue as it has been triaged and we have an interim solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] 7c00 commented on issue #6332: Avoid creating Configuration copies in Hudi
Posted by GitBox <gi...@apache.org>.
7c00 commented on issue #6332:
URL: https://github.com/apache/hudi/issues/6332#issuecomment-1208834476
The changed is to be taken in Hudi.
@vinothchandar @codope How do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6332: Avoid creating Configuration copies in Hudi
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6332:
URL: https://github.com/apache/hudi/issues/6332#issuecomment-1229260882
@codope : any updates on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org