You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/01/08 20:09:48 UTC
[GitHub] [hudi] prashanthvg89 commented on pull request #2069: [WIP][HUDI-945] Cleanup Spillable map files eagerly for DiskBasedMap
prashanthvg89 commented on pull request #2069:
URL: https://github.com/apache/hudi/pull/2069#issuecomment-756969267
I saw this failure in Spark streaming job writing to Hudi. Is this still being worked on and is there an ETA for this?
Hudi version: 0.6.0
Spark version: 2.4.0
EMR version: 5.29.0
Hudi storage type: Copy-On-Write
Number of partitions specified using partition path field: 100
Number of Hudi records currently: 463 million
Caused by: org.apache.hudi.exception.HoodieIOException: Cannot instantiate an ExternalSpillableMap
at org.apache.hudi.io.HoodieMergeHandle.init(HoodieMergeHandle.java:155)
at org.apache.hudi.io.HoodieMergeHandle.<init>(HoodieMergeHandle.java:75)
at org.apache.hudi.table.action.commit.CommitActionExecutor.getUpdateHandle(CommitActionExecutor.java:101)
at org.apache.hudi.table.action.commit.CommitActionExecutor.handleUpdate(CommitActionExecutor.java:72)
at org.apache.hudi.table.action.commit.BaseCommitActionExecutor.handleUpsertPartition(BaseCommitActionExecutor.java:257)
... 30 more
Caused by: java.io.IOException: Too many open files
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1014)
at org.apache.hudi.common.util.collection.DiskBasedMap.initFile(DiskBasedMap.java:115)
at org.apache.hudi.common.util.collection.DiskBasedMap.<init>(DiskBasedMap.java:81)
at org.apache.hudi.common.util.collection.ExternalSpillableMap.<init>(ExternalSpillableMap.java:86)
at org.apache.hudi.io.HoodieMergeHandle.init(HoodieMergeHandle.java:152)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org