You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "cbg-wx (via GitHub)" <gi...@apache.org> on 2024/01/16 02:50:46 UTC
[I] [SUPPORT]HoodieIOException:Unable to create /data13/hadoop/yam/local/usercache/D4046B698E12F32FEF9F968718893E9D/appcache/applicatfon_1701998877139_1409559/hudi- BITCASK-8549d240-78a0-43af-a30c-0ef6d6327cd0 [hudi]
cbg-wx opened a new issue, #10502:
URL: https://github.com/apache/hudi/issues/10502
spark-2.4.6,hudi-0.13.1,spark write hudi cow table exception in below:
Caused by: org.apache.hudi.exception.HoodieIOException:Unable to create /data13/hadoop/yam/local/usercache/D4046B698E12F32FEF9F968718893E9D/appcache/application_1701998877139_1409559/hudi-
BITCASK-8549d240-78a0-43af-a30c-0ef6d6327cd0
at org.apache.hudi.common.utl.collection.ExternalSpillableMap.getDiskBasedMap(ExtermalSpillableMapjava:121)
at org.apache.hudi.common.util.collection.External factors SpillableMap.getDiskB asedMapNumEntries(ExternalSpillableMap.java.140)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]HoodieIOException:Unable to create /data13/hadoop/yam/local/usercache/D4046B698E12F32FEF9F968718893E9D/appcache/applicatfon_1701998877139_1409559/hudi- BITCASK-8549d240-78a0-43af-a30c-0ef6d6327cd0 [hudi]
Posted by "cbg-wx (via GitHub)" <gi...@apache.org>.
cbg-wx commented on issue #10502:
URL: https://github.com/apache/hudi/issues/10502#issuecomment-4885522979
This issue was caused by an inconsistency in the yarn.nodemanager.local-dirs configuration parameter in the yarn-site.xml file before and after the Hadoop cluster expansion, where the values differed between the clusters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]HoodieIOException:Unable to create /data13/hadoop/yam/local/usercache/D4046B698E12F32FEF9F968718893E9D/appcache/applicatfon_1701998877139_1409559/hudi- BITCASK-8549d240-78a0-43af-a30c-0ef6d6327cd0 [hudi]
Posted by "cbg-wx (via GitHub)" <gi...@apache.org>.
cbg-wx commented on issue #10502:
URL: https://github.com/apache/hudi/issues/10502#issuecomment-1892994059
public static void mkdir(File directory) throws IOException {
if (!directory.exists()) {
directory.mkdirs();
}
if (!directory.isDirectory()) {
throw new IOException("Unable to create :" + directory);
}
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]HoodieIOException:Unable to create /data13/hadoop/yam/local/usercache/D4046B698E12F32FEF9F968718893E9D/appcache/applicatfon_1701998877139_1409559/hudi- BITCASK-8549d240-78a0-43af-a30c-0ef6d6327cd0 [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10502:
URL: https://github.com/apache/hudi/issues/10502#issuecomment-1907471894
@cbg-wx Can you give more context on when you are getting this error? Is it interim or you are getting this consistently.
Did you tried configuring this to a different directory - hoodie.filesystem.view.spillable.dir
Also, when you see this error, can you see anything alarming like disk space is full.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]HoodieIOException:Unable to create /data13/hadoop/yam/local/usercache/D4046B698E12F32FEF9F968718893E9D/appcache/applicatfon_1701998877139_1409559/hudi- BITCASK-8549d240-78a0-43af-a30c-0ef6d6327cd0 [hudi]
Posted by "cbg-wx (via GitHub)" <gi...@apache.org>.
cbg-wx commented on issue #10502:
URL: https://github.com/apache/hudi/issues/10502#issuecomment-1951600206
> hoodie.filesystem.view.spillable.dir
i user default configuration, upsert mor hudi table, hoodie.filesystem.view.spillable.dir is default value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]HoodieIOException:Unable to create /data13/hadoop/yam/local/usercache/D4046B698E12F32FEF9F968718893E9D/appcache/applicatfon_1701998877139_1409559/hudi- BITCASK-8549d240-78a0-43af-a30c-0ef6d6327cd0 [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10502:
URL: https://github.com/apache/hudi/issues/10502#issuecomment-1907472736
@cbg-wx Let us know the writer configuration you are using?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]HoodieIOException:Unable to create /data13/hadoop/yam/local/usercache/D4046B698E12F32FEF9F968718893E9D/appcache/applicatfon_1701998877139_1409559/hudi- BITCASK-8549d240-78a0-43af-a30c-0ef6d6327cd0 [hudi]
Posted by "cbg-wx (via GitHub)" <gi...@apache.org>.
cbg-wx closed issue #10502: [SUPPORT]HoodieIOException:Unable to create /data13/hadoop/yam/local/usercache/D4046B698E12F32FEF9F968718893E9D/appcache/applicatfon_1701998877139_1409559/hudi- BITCASK-8549d240-78a0-43af-a30c-0ef6d6327cd0
URL: https://github.com/apache/hudi/issues/10502
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org