You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "0x26res (via GitHub)" <gi...@apache.org> on 2024/01/08 13:26:54 UTC
[I] Transient issue with PyGILState_Release [arrow]
0x26res opened a new issue, #39503:
URL: https://github.com/apache/arrow/issues/39503
### Describe the bug, including details regarding any error messages, version, and platform.
Since updating from pyarrow==12.0.0 to pyarrow=14.0.1, I have transient issues with some of my python jobs.
I have simple ETL jobs, they load a few parquet files from s3, manipulate the data with a mix of pyarrow and pandas, and save one parquet file to s3.
Since updating to pyarrow==14.0.1, some of my jobs (something like 1 out of a 1000) are failing. Here are the symptoms:
- I see this message in stderr:
```
Fatal Python error: PyGILState_Release: auto-releasing thread-state, but no thread-state for this thread
Python runtime state: finalizing (tstate=0x000056046142faa0)
```
- My job orchestrator (aws batch) report a return code of 139 which stands for :
> 139 – Occurs when a segmentation fault is experienced. Likely, the application tried to access a memory region which it not available, or there is an unset or invalid environment variable.
- From looking at the full log and the expected side effect of my job (aka the s3 file that is saved), it looks like the job is sucesfull
So it pretty much looks like the job fails while releasing resources after it's done running.
I can't create a reproducible example, simply because the error is transient. Also I can't really isolate which update is causing the issue. I was updating fsspec and s3fs at the same time as I was updating pyarrow. But I'm not sure they use `PyGILState_Release`.
I looked at relevant issues and code in the repo, but couldn't find any relevant change. It could be related to this https://github.com/apache/arrow/issues/38626 but I doubt it (I don't use `atexit.register`).
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Transient issue with PyGILState_Release with S3 [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #39503:
URL: https://github.com/apache/arrow/issues/39503#issuecomment-1881612638
cc @jorisvandenbossche @pitrou
I remember there were some issues with S3 finalization on C++ but I am not sure if this could be related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Transient issue with PyGILState_Release with S3 [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #39503:
URL: https://github.com/apache/arrow/issues/39503#issuecomment-1892569941
> In your opinion, could it be because of fsspec or s3fs (do these library use `PyGILState_Release`)?
Well, this is the Apache Arrow issue tracker, so you'll probably won't find people with the specific knowledge you're asking for. The fsspec and s3fs projects are not related to us, and they have dedicated issue trackers where you can ask that question.
I'm also going to point out that Arrow (and PyArrow) comes with its own [S3 filesystem wrapper](https://arrow.apache.org/docs/python/filesystems.html#filesystem-s3), but you're using a third-party one here. Of course, there can be legitimate reasons to that, but it also complicates the picture quite a bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Transient issue with PyGILState_Release with S3 [arrow]
Posted by "0x26res (via GitHub)" <gi...@apache.org>.
0x26res commented on issue #39503:
URL: https://github.com/apache/arrow/issues/39503#issuecomment-1883109419
We have plans to migrate to 3.11, which should happen in the coming weeks. 3.12 isn't possible at the moment (because of pytorch).
Also given the issue is transient, and I have to run a lot of jobs before it happens, it's hard to pin point. But I can keep you updated as I upgrade dependencies, if I see the issue disappear.
In your opinion, could it be because of fsspec or s3fs (do these library use `PyGILState_Release`)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Transient issue with PyGILState_Release with S3 [arrow]
Posted by "0x26res (via GitHub)" <gi...@apache.org>.
0x26res commented on issue #39503:
URL: https://github.com/apache/arrow/issues/39503#issuecomment-1914470777
I haven't had the issue since updating to 14.0.2 (though it's hard to be 100% sure that the issue is gone).
I'll close the issue, thanks for your help/suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Transient issue with PyGILState_Release with S3 [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #39503:
URL: https://github.com/apache/arrow/issues/39503#issuecomment-1883033109
Is it possible for you to check with a more recent Python version such as 3.12.1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Transient issue with PyGILState_Release with S3 [arrow]
Posted by "0x26res (via GitHub)" <gi...@apache.org>.
0x26res commented on issue #39503:
URL: https://github.com/apache/arrow/issues/39503#issuecomment-1883015628
@pitrou I'm using 3.10.13 (more specifically I'm running it on a docker image built from python:3.10.13-slim-bullseye)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Transient issue with PyGILState_Release with S3 [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #39503:
URL: https://github.com/apache/arrow/issues/39503#issuecomment-1882986929
@0x26res Which Python version are you using?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Transient issue with PyGILState_Release with S3 [arrow]
Posted by "0x26res (via GitHub)" <gi...@apache.org>.
0x26res closed issue #39503: [Python][C++] Transient issue with PyGILState_Release with S3
URL: https://github.com/apache/arrow/issues/39503
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Transient issue with PyGILState_Release with S3 [arrow]
Posted by "jmahlik (via GitHub)" <gi...@apache.org>.
jmahlik commented on issue #39503:
URL: https://github.com/apache/arrow/issues/39503#issuecomment-1892564520
Is the code in question doing something similar to the reproducer in https://github.com/apache/arrow/issues/36980? That could be another potential cause of segfault in acquiring the GIL after interpreter shutdown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org