You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "westonpace (via GitHub)" <gi...@apache.org> on 2023/05/15 18:50:46 UTC
[GitHub] [arrow] westonpace commented on issue #35318: [Python][GcsFileSystem][Parquet] fails to create ParquetFile from GCS after a few hundred files
westonpace commented on issue #35318:
URL: https://github.com/apache/arrow/issues/35318#issuecomment-1548384073
From https://github.com/curl/curl/issues/8921 it would seem that too many open file descriptors is indeed a very likely culprit.
@zpz can you show what you get from `ulimit -a`?
Another possible cause is that the kernel is running out of memory. @zpz can you share the value of `cat /proc/sys/vm/overcommit_memory`? If overcommit is disabled (i.e. if that command returns `2`) then it is possible the kernel will decide it is out of memory well before it actually uses all physical memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org