[GitHub] [arrow] phalexo commented on issue #35876: [C++] Problem when used from "qlora", huggingface. Does not clean up, and dumps core.

["phalexo (via GitHub)" <gi...@apache.org>]

phalexo commented on issue #35876:
URL: https://github.com/apache/arrow/issues/35876#issuecomment-1585767266

   > This requirement comes from S3 and is documented [here](https://sdk.amazonaws.com/cpp/api/LATEST/root/html/index.html)
   > 
   > > The AWS SDK for C++ must be initialized by calling Aws::InitAPI. Before the application terminates, the SDK must be shut down by calling Aws::ShutdownAPI. Each method accepts an argument of [Aws::SDKOptions](https://sdk.amazonaws.com/cpp/api/LATEST/aws-cpp-sdk-core/html/struct_aws_1_1_s_d_k_options.html)
   > > All other calls to the SDK can be performed between these two method calls.
   > > All AWS SDK for C++ calls performed between Aws::InitAPI and Aws::ShutdownAPI should either to be contained within a pair of curly braces or should be invoked by functions called between the two methods.
   > 
   > The `FinalizeS3` method must be called once and it must be called before the application exits (e.g. before any objects with "static storage duration" are destroyed)
   
   Yes, so arrow, pyarrow should clean the stuff up properly. I looked at the arrow code, and it is not clear to me where this should happen. Putting the call before the connection is severed did not fix the problem.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org