You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/07/15 17:13:16 UTC
[GitHub] [hudi] zuyanton opened a new issue #1837: [SUPPORT]S3 file listing causing compaction to get eventually slow
zuyanton opened a new issue #1837:
URL: https://github.com/apache/hudi/issues/1837
We are running incremental updates to our MoR table on S3. We are running updates every 10 minutes. We compact every 10 commits (every ~1.5 hour). we have noticed that if we want to keep history for longer then few hours (set cleanup to clean after 50 commits ) , then compaction time starts increasing as number of files in s3 increase. Chart below shows time taken to upsert incremental changes to the table, spikes indicate the commit when inline compaction got triggered.
when looking into logs we have noticed that majority of the time is spend listing recursively all files in tables S3 folder. more specifically, logs contain following lines:
```
20/07/15 13:58:19 INFO HoodieMergeOnReadTableCompactor: Compacting s3://bucket/table with commit 20200715135819
20/07/15 14:36:04 INFO HoodieMergeOnReadTableCompactor: Compaction looking for files to compact in [0, 1, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 2, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 3, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 4, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 5, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 6, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 7, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 8, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 9, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99] partitions
```
the code lines that gets executed between those two log lines are:
https://github.com/apache/hudi/blob/master/hudi-client/src/main/java/org/apache/hudi/table/action/compact/HoodieMergeOnReadTableCompactor.java#L181-L194
I put log lines around various parts of that code to measure time and was able to narrow down to this function
https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java#L225
as a meter of fact compaction that took 50+ minutes, 38 of that 50+ minutes was executing that function, which looks like mostly recursively list files in S3 table location.
This issue observed on all tables, however it most noticeable at tables where incremental updates update large number of partitions (50% of all partitions).
**some table stats**
100 partitions, initial size 100gb, initial file count 6k, we observed 50+ minutes compaction after table grew to 300gb, 20k files.
**Environment Description**
* Hudi version : master branch
* Spark version : 2.4.4
* Hive version : 2.3.6
* Hadoop version : 2.8.5
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar closed issue #1837: [SUPPORT]S3 file listing causing compaction to get eventually slow
Posted by GitBox <gi...@apache.org>.
vinothchandar closed issue #1837:
URL: https://github.com/apache/hudi/issues/1837
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] steveloughran commented on issue #1837: [SUPPORT]S3 file listing causing compaction to get eventually slow
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1837:
URL: https://github.com/apache/hudi/issues/1837#issuecomment-671523549
The issue here is that treewalking is pathologically bad for S3. Asking for a deep listing is often more efficient; filesystem.listFiles(path, recursive=true) will do this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #1837: [SUPPORT]S3 file listing causing compaction to get eventually slow
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1837:
URL: https://github.com/apache/hudi/issues/1837#issuecomment-660408124
> This means that we need to move out these listing logic our of hudi-common if we want to parallelize it with spark context.
I will be landing a PR over the weekend, that avoids listings for rollbacks.. consequently, I moved the place you are changing into hudi-client already.. So it should be simple to redo on top of that.
Overall, We already have a `StorageSchemes` class that does different things for S3/GCS etc and HDFS/Ignite.. based on append support.. As a more elegant fix, I feel if we can take a pass at listing usages and do different forms of listing based on storage schemes..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #1837: [SUPPORT]S3 file listing causing compaction to get eventually slow
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1837:
URL: https://github.com/apache/hudi/issues/1837#issuecomment-855190520
Metadata table is out for couple releases now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #1837: [SUPPORT]S3 file listing causing compaction to get eventually slow
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1837:
URL: https://github.com/apache/hudi/issues/1837#issuecomment-659889727
Thanks @zuyanton for reporting this issue.
@umehrot2 We can solve this If we can generalize this PR : https://github.com/apache/hudi/pull/1768 to also parallelize FSUtils.getAllPartitionPaths. Would you be able to make the change ?
Jira : https://issues.apache.org/jira/browse/HUDI-1108
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] umehrot2 commented on issue #1837: [SUPPORT]S3 file listing causing compaction to get eventually slow
Posted by GitBox <gi...@apache.org>.
umehrot2 commented on issue #1837:
URL: https://github.com/apache/hudi/issues/1837#issuecomment-660392013
@bvaradar this was on our radar. I would ideally like to keep the whole **marker optimizations** separate. Also I received certain comments in my PR where folks do not want `hudi-common` to depend on `spark`. This means that we need to move out these listing logic our of `hudi-common` if we want to parallelize it with `spark context`. So lets target this in a separate PR.
One of my teammates was interested in taking this up. Let me check with him if he can pick this and if not I will. For the meanwhile, I have assigned it to myself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #1837: [SUPPORT]S3 file listing causing compaction to get eventually slow
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1837:
URL: https://github.com/apache/hudi/issues/1837#issuecomment-672430487
Thanks @steveloughran : Good to know. We are looking at an approach using consolidated metadata to avoid file listing (RFC-15) in the first place. @umehrot2 : What are your thoughts on this ? Do you think this would significantly help S3 case in the interim ? We do list all partitions for compaction scheduling currently. I am wondering if this is worth looking at 0.6.1 timeframe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #1837: [SUPPORT]S3 file listing causing compaction to get eventually slow
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1837:
URL: https://github.com/apache/hudi/issues/1837#issuecomment-672560925
and this is the last such place. (cleaner, rollback are all incremental now) . cc @prashantwason to the rescue ;)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar edited a comment on issue #1837: [SUPPORT]S3 file listing causing compaction to get eventually slow
Posted by GitBox <gi...@apache.org>.
vinothchandar edited a comment on issue #1837:
URL: https://github.com/apache/hudi/issues/1837#issuecomment-660408124
> This means that we need to move out these listing logic our of hudi-common if we want to parallelize it with spark context.
I will be landing a PR over the weekend, that avoids listings for rollbacks.. consequently, I moved the place you are changing into hudi-client already.. So it should be simple to redo on top of that.
Overall, We already have a `StorageSchemes` class that does different things for S3/GCS etc and HDFS/Ignite.. based on append support.. As a more elegant fix, I feel if we can take a pass at listing usages and do different forms of listing based on storage schemes..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org