You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/01/08 20:09:48 UTC

[GitHub] [hudi] prashanthvg89 commented on pull request #2069: [WIP][HUDI-945] Cleanup Spillable map files eagerly for DiskBasedMap

prashanthvg89 commented on pull request #2069:
URL: https://github.com/apache/hudi/pull/2069#issuecomment-756969267


   I saw this failure in Spark streaming job writing to Hudi. Is this still being worked on and is there an ETA for this?
   
   Hudi version: 0.6.0
   Spark version: 2.4.0
   EMR version: 5.29.0
   Hudi storage type: Copy-On-Write
   Number of partitions specified using partition path field: 100
   Number of Hudi records currently: 463 million
   
   Caused by: org.apache.hudi.exception.HoodieIOException: Cannot instantiate an ExternalSpillableMap
           at org.apache.hudi.io.HoodieMergeHandle.init(HoodieMergeHandle.java:155)
           at org.apache.hudi.io.HoodieMergeHandle.<init>(HoodieMergeHandle.java:75)
           at org.apache.hudi.table.action.commit.CommitActionExecutor.getUpdateHandle(CommitActionExecutor.java:101)
           at org.apache.hudi.table.action.commit.CommitActionExecutor.handleUpdate(CommitActionExecutor.java:72)
           at org.apache.hudi.table.action.commit.BaseCommitActionExecutor.handleUpsertPartition(BaseCommitActionExecutor.java:257)
           ... 30 more
   Caused by: java.io.IOException: Too many open files
           at java.io.UnixFileSystem.createFileExclusively(Native Method)
           at java.io.File.createNewFile(File.java:1014)
           at org.apache.hudi.common.util.collection.DiskBasedMap.initFile(DiskBasedMap.java:115)
           at org.apache.hudi.common.util.collection.DiskBasedMap.<init>(DiskBasedMap.java:81)
           at org.apache.hudi.common.util.collection.ExternalSpillableMap.<init>(ExternalSpillableMap.java:86)
           at org.apache.hudi.io.HoodieMergeHandle.init(HoodieMergeHandle.java:152)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org