[GitHub] [incubator-hudi] garyli1019 opened a new issue #768: No Space Left On Device for upsert

[GitBox <gi...@apache.org>]

garyli1019 opened a new issue #768: No Space Left On Device for upsert
URL: https://github.com/apache/incubator-hudi/issues/768
 
 
   Hello, I am experiencing `No Space Left On Device` error on upserting and it always happened in one stage. I believe this might caused by `/tmp` is too small. I tried to change `spark.local.dir` to somewhere larger but it didn't help. Based on this answer: https://community.hortonworks.com/content/supportkb/186401/why-is-spark-filling-the-tmp-folder.html, I might need to ask my IT team to increase the size of it, but before that, I'd like to see what Hudi was doing on that stage, cause the shuffling size of the data was really small, which seems strange to cause `No Space Left On Device` error.
   
   The error was in job 8 from https://hudi.apache.org/admin_guide.html#spark-ui, `count at HoodieSparkSqlWriter.scala:148`. Inside the stage, it seems like the failing task is `MapPartitionsRDD [888] [Cached]flatMap at HoodieWriteClient.java:444`. I'd like to know what Hudi is doing in this stage and why relatively small `Shuffle Read Size`(Only 1000MB for my case) will lead to `No Space Left On Device` error.
   
   Thank you so much!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services