You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "CurtHagenlocher (via GitHub)" <gi...@apache.org> on 2024/04/26 17:01:08 UTC

[I] [C#] Tests fail on MacOS arm64 with a stack overflow [arrow]

CurtHagenlocher opened a new issue, #41397:
URL: https://github.com/apache/arrow/issues/41397

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   Importing pyarrow into PythonNET seems to trigger a stack overflow. At first it seemed that this was triggered by the move to .NET 8 but the failure can be repro'd locally against the maint-15.0.0 branch (which uses 7.0) so there's clearly something else going on.
   
   ```
   The active test run was aborted. Reason: Test host process crashed : Stack overflow.
      at Python.Runtime.Runtime.PyImport_ImportModule(System.String)
      at Python.Runtime.PyModule.Import(System.String)
      at Python.Runtime.Py.Import(System.String)
      at Apache.Arrow.Tests.CDataSchemaPythonTest.ImportRecordBatchFromBuffer()
      at System.RuntimeMethodHandle.InvokeMethod(System.Object, Void**, System.Signature, Boolean)
      at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(System.Object, System.Reflection.BindingFlags)
      at Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].CallTestMethod(System.Object)
   ```
   
   ### Component(s)
   
   C#


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C#] Tests fail on MacOS arm64 with a stack overflow [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #41397:
URL: https://github.com/apache/arrow/issues/41397#issuecomment-2146044280

   Issue resolved by pull request 41934
   https://github.com/apache/arrow/pull/41934


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C#] Tests fail on MacOS arm64 with a stack overflow [arrow]

Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #41397:
URL: https://github.com/apache/arrow/issues/41397#issuecomment-2099992648

   The problem is that at the moment the nuget job and all the verification jobs for the 16.1.0 release that involves C# are failing. We can't generate new packages and can't verify the release with this failure. We can't release at the moment.
   See the nuget failure here: https://github.com/ursacomputing/crossbow/actions/runs/8985979418/job/24718232619
   and verification job failures here: https://github.com/apache/arrow/pull/41572#issuecomment-2098372912


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C#] Tests fail on MacOS arm64 with a stack overflow [arrow]

Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #41397:
URL: https://github.com/apache/arrow/issues/41397#issuecomment-2100232890

   Ok, cherry-picking https://github.com/apache/arrow/commit/48a9639bb0bae600adf846fbdcba77aaca151fc0 seems to fix the issue on the maintenance branch:
   https://github.com/apache/arrow/pull/41437#issuecomment-2100199219
   and
   https://github.com/apache/arrow/pull/41437#issuecomment-2100215816
   I will validate once the Release Candidate 1 is created.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C#] Tests fail on MacOS arm64 with a stack overflow [arrow]

Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #41397:
URL: https://github.com/apache/arrow/issues/41397#issuecomment-2102231916

   I've validated that RC1 jobs are successful with the commit mentioned above. I am closing the issue as this is not happening on main for nuget or verification jobs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C#] Tests fail on MacOS arm64 with a stack overflow [arrow]

Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on issue #41397:
URL: https://github.com/apache/arrow/issues/41397#issuecomment-2102979570

   I'm still seeing this both in PR builds and locally so I'm reopening. See #41603 for a recent PR build that exhibits this problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C#] Tests fail on MacOS arm64 with a stack overflow [arrow]

Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on issue #41397:
URL: https://github.com/apache/arrow/issues/41397#issuecomment-2138436383

   Also, this has been reported as a bug in PythonNET at https://github.com/pythonnet/pythonnet/issues/2396, so this doesn't look specific to pyarrow. 
   
   And I should add that I didn't test `macos-14-large` as that needs a paid account, so it's not clear whether it was the upgrade to MacOS 14 that broke things or the change to ARM.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C#] Tests fail on MacOS arm64 with a stack overflow [arrow]

Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on issue #41397:
URL: https://github.com/apache/arrow/issues/41397#issuecomment-2145183405

   Thanks @adamreeve! This looks like the best thing to do in the near term.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C#] Tests fail on MacOS arm64 with a stack overflow [arrow]

Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on issue #41397:
URL: https://github.com/apache/arrow/issues/41397#issuecomment-2138608782

   This is the change I tested, which only affected the C# jobs: https://github.com/adamreeve/arrow/commit/449a8b3a76346f37c6beac1b3e26e04a0b7c3083. Looking more closely at the failures in https://github.com/apache/arrow/issues/41397#issuecomment-2099992648, those appear to be a different issue and it makes sense that those were fixed by 48a9639bb0bae600adf846fbdcba77aaca151fc0, so I think maybe the verification jobs aren't actually affected by this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C#] Tests fail on MacOS arm64 with a stack overflow [arrow]

Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on issue #41397:
URL: https://github.com/apache/arrow/issues/41397#issuecomment-2138428609

   Would it make sense to downgrade the runner image to `macos-13` or one of the [other non-ARM images](https://github.com/actions/runner-images?tab=readme-ov-file#available-images) to keep the tests passing until this can be fixed? It appears that this issue only started when GitHub changed `macos-latest` to be an ARM image. I tested downgrading to `macos-13` and that does seem to fix the issue; I ran the C# actions a few times on my fork without any failures.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C#] Tests fail on MacOS arm64 with a stack overflow [arrow]

Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on issue #41397:
URL: https://github.com/apache/arrow/issues/41397#issuecomment-2138598915

   It definitely worked on my M1 Mac before the OS upgrade.
   
   Are the OS images shared between tests or can the runner be downgraded just for the C# runs?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C#] Tests fail on MacOS arm64 with a stack overflow [arrow]

Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #41397:
URL: https://github.com/apache/arrow/issues/41397#issuecomment-2092720524

   I am marking this as a blocker for 16.1.0. If I can't get any news around the issue in the next 3-4 days I will remove all C# commits from the 16.1.0 branch in order to unblock the release. Hopefully this is reasonable. Do let me know if I should wait more time. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C#] Tests fail on MacOS arm64 with a stack overflow [arrow]

Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on issue #41397:
URL: https://github.com/apache/arrow/issues/41397#issuecomment-2094299141

   I don't think this should be a blocker because I get the same error when running against the 15.0.0 branch. To the best of my ability to tell, the stack overflow is coming from inside the kernel so maybe this reflects some kind of OS change? This would also explain why we don't see the same error on e.g. Linux.
   
   Unfortunately, most of my hardcore debugging experience is on Windows and the toolset is just entirely different. I'm currently approaching this by trying to create a minimal repro.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C#] Tests fail on MacOS arm64 with a stack overflow [arrow]

Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #41397:
URL: https://github.com/apache/arrow/issues/41397#issuecomment-2100164544

   On a different note, I can see that currently non of the C# verification nor the nuget jobs are failing on the nightly tasks on main so we might just have to find what is the issue that solved it on main and cherry-pick for the 16.1.0 release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C#] Tests fail on MacOS arm64 with a stack overflow [arrow]

Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd closed issue #41397: [C#] Tests fail on MacOS arm64 with a stack overflow
URL: https://github.com/apache/arrow/issues/41397


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C#] Tests fail on MacOS arm64 with a stack overflow [arrow]

Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on issue #41397:
URL: https://github.com/apache/arrow/issues/41397#issuecomment-2138603031

   We could downgrade the C# jobs only, the only thing we need to keep in mind is the binary jobs for the releases and if want/need to build nuget binaries on arm64 mac.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C#] Tests fail on MacOS arm64 with a stack overflow [arrow]

Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on issue #41397:
URL: https://github.com/apache/arrow/issues/41397#issuecomment-2138608579

   The build environment shouldn't matter when building the packages as they're all purely MSIL; it would only matter when testing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C#] Tests fail on MacOS arm64 with a stack overflow [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou closed issue #41397: [C#] Tests fail on MacOS arm64 with a stack overflow
URL: https://github.com/apache/arrow/issues/41397


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org