You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "OliLay (via GitHub)" <gi...@apache.org> on 2025/03/11 15:30:08 UTC

[I] [C++] Thread pool performance behaves counter-intuitive for I/O intensive operations (e.g. file system) [arrow]

OliLay opened a new issue, #45749:
URL: https://github.com/apache/arrow/issues/45749

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   Hey guys,
   
   we noticed something with using S3/Azure Blob and the arrow file system implementation. We are doing a lot of concurrent file operations and use a larger thread pool size than the default (32 instead of 8, kind of like it is recommended [here](https://arrow.apache.org/docs/cpp/threading.html#cpu-vs-i-o)).
   
   When using a larger thread pool size and dispatching a similar amount of async operations, the time until these async operations finish is significantly higher than it is with a smaller thread pool. [This](https://github.com/OliLay/arrow/blob/7d28657ae758f2df42454fa096bd5224eacc7a97/cpp/src/arrow/filesystem/s3fs_benchmark.cc#L58) benchmark reproduces the issue. It basically tries to open 32 files asynchronously with different thread pool sizes against files in a real S3 bucket (not MinIO).
   
   The results are quite surprising (first parameter is # files, second one is # threads)
   ```
   ---------------------------------------------------------------------------------------
   Benchmark                                             Time             CPU   Iterations
   ---------------------------------------------------------------------------------------
   OpenFileAsync/32/4/iterations:1/process_time      0.254 s         0.139 s             1
   OpenFileAsync/32/8/iterations:1/process_time      0.186 s         0.329 s             1
   OpenFileAsync/32/16/iterations:1/process_time      0.181 s         0.799 s             1
   OpenFileAsync/32/32/iterations:1/process_time      0.328 s          1.63 s             1
   OpenFileAsync/32/64/iterations:1/process_time      0.385 s          1.71 s             1
   ```
   In contrary to what you would expect with I/O tasks, you can see that with 32 threads we take almost double the time as with 8 or 16 threads. One could work around this issue by configuring the thread pool dynamically according to the workload (e.g. decrease the size to 16 threads if we see we only have 32 files), but this is cumbersome to do for every use-case and not possible everywhere (e.g. for concurrent usage of thread pools by different use-cases).
   
   I verified this holds true for the Azure file system as well as the S3 file system, hence I think this has to do with how arrow's thread pool works. I did not get any additional insights by perf'ing this (the flamegraph looks almost identical); I assume there must be some lock contention in the thread pool.
   
   I just wanted to report this and ask if you have seen similar issues already? I am not too familiar with the thread pool code, could you maybe point me into a direction how we can improve this without dynamically adjusting the thread pool size?
   
   Note: To reproduce the issue the best, it makes sense to only run the benchmark executable for a parameter at a time as I assume there is some caching ongoing when executing all the parameters in the same run.
   
   
   
   ### Component(s)
   
   C++


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++] Thread pool performance behaves counter-intuitive for I/O intensive operations (e.g. file system) [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #45749:
URL: https://github.com/apache/arrow/issues/45749#issuecomment-2717859812

   > I assume there must be some lock contention in the thread pool.
   
   The fact that it seems to take a lot of CPU time makes this explanation unlikely (some lock contention would not result in such CPU usage).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++] Increasing thread count leads to worse perforamnce for object storage file systems (S3, Azure Blob) [arrow]

Posted by "OliLay (via GitHub)" <gi...@apache.org>.
OliLay commented on issue #45749:
URL: https://github.com/apache/arrow/issues/45749#issuecomment-2720399318

   After further investigation it seems like I have found the root cause. 
   When using a larger thread pool, arrow [explicitly sets](https://github.com/apache/arrow/blob/c3e399af43c4a2e384a41fad0619589baa9045f0/cpp/src/arrow/filesystem/s3fs.cc#L1178) the number of connections in the connection pool to the number of threads in the arrow thread pool. 
   This means a new socket connect + TLS handshake will have to be done for every single request. When only having metadata requests (as in this case), opening a new connection for every very short running request (as there is barely any data involved) seems to be expensive, and doing 32 TLS handshakes in parallel seems to take ~1sec. I suspect this may be some throttling on the server-side also, but not sure. 
   When using a smaller connection pool size, the requests run through in <200ms because the AWS SDK implements curl connection handle re-use, hence the number of TLS handshakes equals the number of connections in the pool.
   
   I wonder if arrow should really set the number of connections to the number of threads. I think this should only be done when you know you have large data to transfer and one time overhead of opening a connection is marginal; but for metadata requests that doesn't seem viable. Either we can make the connection pool size configurable via `S3Options`; or let the user give a hint about how the file system is used (e.g. metadata, or large data transfer); and then decide based on that how many connections we want to spawn.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++] Increasing thread count leads to worse perforamnce for object storage file systems (S3, Azure Blob) [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #45749:
URL: https://github.com/apache/arrow/issues/45749#issuecomment-2721034279

   Well, large data transfers is usually what people do with Arrow filesystems (the Arrow filesystem layer was designed primarily for reading and writing datasets). So at least the default setting sounds fine to me.
   
   Having the number of IO operations equal to the number of threads is probably a worst case, since each connection will only be used once. Otherwise, if the number of IO operations is significantly than the number of threads, connection reuse will amortized the TLS handshake cost over many operations.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++] Thread pool performance behaves counter-intuitive for I/O intensive operations (e.g. file system) [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #45749:
URL: https://github.com/apache/arrow/issues/45749#issuecomment-2718028461

   Could you change the issue title to reflect your findings?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++] Thread pool performance behaves counter-intuitive for I/O intensive operations (e.g. file system) [arrow]

Posted by "OliLay (via GitHub)" <gi...@apache.org>.
OliLay commented on issue #45749:
URL: https://github.com/apache/arrow/issues/45749#issuecomment-2717994328

   > 
   > The fact that it seems to take a lot of CPU time makes this explanation unlikely (some lock contention would not result in such CPU usage).
   
   Yes, further investigation also revealed that my initial assumption is not correct.
   In fact, debugging a bit further, it seems like the actual call to `HeadObject` (in `ObjectInputFile::Init`) into the AWS SDK takes so long.
   Debugging further yield the result that setting the [maxConnections](https://github.com/apache/arrow/blob/9b36c709a52caabd3579d006bb92379e1b263e52/cpp/src/arrow/filesystem/s3fs.cc#L1178) triggers that behavior. For some reason the AWS SDK code [interacting with curl ](https://github.com/aws/aws-sdk-cpp/blob/main/src/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp) seems to be problematic in that case. As Azure also uses curl as underlying library, this may be the common denominator. 
   Using the [AWS SDK CRT HTTP](https://github.com/aws/aws-sdk-cpp/blob/109022594cb4bcd589ee0e8a12a0d11e42b9f558/docs/CMake_Parameters.md#use_crt_http_client) client also resolves the issue. Not sure if it is a deeper problem inside curl; or with how curl is used.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++] Thread pool performance behaves counter-intuitive for I/O intensive operations (e.g. file system) [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #45749:
URL: https://github.com/apache/arrow/issues/45749#issuecomment-2718026047

   > Using the [AWS SDK CRT HTTP](https://github.com/aws/aws-sdk-cpp/blob/109022594cb4bcd589ee0e8a12a0d11e42b9f558/docs/CMake_Parameters.md#use_crt_http_client) client also resolves the issue.
   
   This may also be related to https://github.com/apache/arrow/issues/18923 . 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org