You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Tom-Newton (via GitHub)" <gi...@apache.org> on 2024/12/02 11:30:05 UTC

[PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Tom-Newton opened a new pull request, #44897:
URL: https://github.com/apache/arrow/pull/44897

   <!--
   Thanks for opening a pull request!
   If this is your first pull request you can find detailed information on how 
   to contribute here:
     * [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
     * [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
   
   
   If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
   
   Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
   
   Then could you also rename the pull request title in the following format?
   
       GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
   
   or
   
       MINOR: [${COMPONENT}] ${SUMMARY}
   
   In the case of PARQUET issues on JIRA the title also supports:
   
       PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
   
   -->
   
   ### Rationale for this change
   Currently it deadlocks when trying to upload more files than we have IO threads. 
   
   ### What changes are included in this PR?
   Will fill in the description shortly, but I want to start a CI run :)
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   ### Are these changes tested?
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   ### Are there any user-facing changes?
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking.
   -->
   <!-- **This PR includes breaking changes to public APIs.** -->
   
   <!--
   Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious.
   -->
   <!-- **This PR contains a "Critical Fix".** -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: [C++] Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on PR #44897:
URL: https://github.com/apache/arrow/pull/44897#issuecomment-2546863832

   > Ok, thanks a lot for doing this @Tom-Newton !
   
   Thanks for reviewing 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #44897:
URL: https://github.com/apache/arrow/pull/44897#issuecomment-2545377149

   Revision: 7573d1128e00fe389b5d97306168caf05b5c2444
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-92edf8328c](https://github.com/ursacomputing/crossbow/branches/all?query=actions-92edf8328c)
   
   |Task|Status|
   |----|------|
   |example-cpp-minimal-build-static|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-example-cpp-minimal-build-static)](https://github.com/ursacomputing/crossbow/actions/runs/12351759258/job/34467375292)|
   |example-cpp-minimal-build-static-system-dependency|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-example-cpp-minimal-build-static-system-dependency)](https://github.com/ursacomputing/crossbow/actions/runs/12351759146/job/34467374468)|
   |example-cpp-tutorial|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-example-cpp-tutorial)](https://github.com/ursacomputing/crossbow/actions/runs/12351759153/job/34467374483)|
   |test-alpine-linux-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-alpine-linux-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/12351758802/job/34467373132)|
   |test-build-cpp-fuzz|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-build-cpp-fuzz)](https://github.com/ursacomputing/crossbow/actions/runs/12351758793/job/34467372612)|
   |test-conda-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-conda-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/12351758961/job/34467373564)|
   |test-conda-cpp-valgrind|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-conda-cpp-valgrind)](https://github.com/ursacomputing/crossbow/actions/runs/12351759283/job/34467375309)|
   |test-cuda-cpp-ubuntu-20.04-cuda-11.2.2|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-cuda-cpp-ubuntu-20.04-cuda-11.2.2)](https://github.com/ursacomputing/crossbow/actions/runs/12351758825/job/34467372577)|
   |test-cuda-cpp-ubuntu-22.04-cuda-11.7.1|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-cuda-cpp-ubuntu-22.04-cuda-11.7.1)](https://github.com/ursacomputing/crossbow/actions/runs/12351759066/job/34467374476)|
   |test-debian-12-cpp-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-debian-12-cpp-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/12351759031/job/34467374159)|
   |test-debian-12-cpp-i386|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-debian-12-cpp-i386)](https://github.com/ursacomputing/crossbow/actions/runs/12351759023/job/34467374139)|
   |test-fedora-39-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-fedora-39-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/12351759145/job/34467374492)|
   |test-ubuntu-20.04-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-ubuntu-20.04-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/12351758966/job/34467373618)|
   |test-ubuntu-20.04-cpp-bundled|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-ubuntu-20.04-cpp-bundled)](https://github.com/ursacomputing/crossbow/actions/runs/12351758941/job/34467372890)|
   |test-ubuntu-22.04-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-ubuntu-22.04-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/12351758904/job/34467372573)|
   |test-ubuntu-22.04-cpp-20|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-ubuntu-22.04-cpp-20)](https://github.com/ursacomputing/crossbow/actions/runs/12351759344/job/34467375582)|
   |test-ubuntu-22.04-cpp-emscripten|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-ubuntu-22.04-cpp-emscripten)](https://github.com/ursacomputing/crossbow/actions/runs/12351759038/job/34467374122)|
   |test-ubuntu-22.04-cpp-no-threading|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-ubuntu-22.04-cpp-no-threading)](https://github.com/ursacomputing/crossbow/actions/runs/12351759560/job/34467376168)|
   |test-ubuntu-24.04-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-ubuntu-24.04-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/12351758833/job/34467372583)|
   |test-ubuntu-24.04-cpp-bundled-offline|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-ubuntu-24.04-cpp-bundled-offline)](https://github.com/ursacomputing/crossbow/actions/runs/12351759050/job/34467374132)|
   |test-ubuntu-24.04-cpp-gcc-13-bundled|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-ubuntu-24.04-cpp-gcc-13-bundled)](https://github.com/ursacomputing/crossbow/actions/runs/12351759239/job/34467375308)|
   |test-ubuntu-24.04-cpp-gcc-14|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-ubuntu-24.04-cpp-gcc-14)](https://github.com/ursacomputing/crossbow/actions/runs/12351758949/job/34467373157)|
   |test-ubuntu-24.04-cpp-minimal-with-formats|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-ubuntu-24.04-cpp-minimal-with-formats)](https://github.com/ursacomputing/crossbow/actions/runs/12351759314/job/34467375559)|
   |test-ubuntu-24.04-cpp-thread-sanitizer|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92edf8328c-github-test-ubuntu-24.04-cpp-thread-sanitizer)](https://github.com/ursacomputing/crossbow/actions/runs/12351759229/job/34467375305)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44897:
URL: https://github.com/apache/arrow/pull/44897#discussion_r1870010307


##########
cpp/src/arrow/util/thread_pool.cc:
##########
@@ -449,8 +468,8 @@ static void WorkerLoop(std::shared_ptr<ThreadPool::State> state,
 
       DCHECK_GE(state->tasks_queued_or_running_, 0);
       {
-        Task task = std::move(state->pending_tasks_.front());
-        state->pending_tasks_.pop_front();
+        Task task = std::move(const_cast<Task&>(state->pending_tasks_.top().task));

Review Comment:
   Yes, I don't think `Task` can be copied, that's why we need to move-construct it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #44897:
URL: https://github.com/apache/arrow/pull/44897#issuecomment-2545849598

   Revision: 5774bf2506f3f99b14d1e0c16f013ad05411b22e
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-372e4c30db](https://github.com/ursacomputing/crossbow/branches/all?query=actions-372e4c30db)
   
   |Task|Status|
   |----|------|
   |example-cpp-minimal-build-static|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-example-cpp-minimal-build-static)](https://github.com/ursacomputing/crossbow/actions/runs/12355248182/job/34478429510)|
   |example-cpp-minimal-build-static-system-dependency|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-example-cpp-minimal-build-static-system-dependency)](https://github.com/ursacomputing/crossbow/actions/runs/12355248344/job/34478432271)|
   |example-cpp-tutorial|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-example-cpp-tutorial)](https://github.com/ursacomputing/crossbow/actions/runs/12355248392/job/34478433243)|
   |test-alpine-linux-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-alpine-linux-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/12355248249/job/34478432262)|
   |test-build-cpp-fuzz|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-build-cpp-fuzz)](https://github.com/ursacomputing/crossbow/actions/runs/12355248457/job/34478433893)|
   |test-conda-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-conda-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/12355248451/job/34478433849)|
   |test-conda-cpp-valgrind|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-conda-cpp-valgrind)](https://github.com/ursacomputing/crossbow/actions/runs/12355248703/job/34478434967)|
   |test-cuda-cpp-ubuntu-20.04-cuda-11.2.2|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-cuda-cpp-ubuntu-20.04-cuda-11.2.2)](https://github.com/ursacomputing/crossbow/actions/runs/12355248271/job/34478432289)|
   |test-cuda-cpp-ubuntu-22.04-cuda-11.7.1|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-cuda-cpp-ubuntu-22.04-cuda-11.7.1)](https://github.com/ursacomputing/crossbow/actions/runs/12355248157/job/34478429478)|
   |test-debian-12-cpp-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-debian-12-cpp-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/12355248354/job/34478432326)|
   |test-debian-12-cpp-i386|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-debian-12-cpp-i386)](https://github.com/ursacomputing/crossbow/actions/runs/12355248180/job/34478429547)|
   |test-fedora-39-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-fedora-39-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/12355248079/job/34478428947)|
   |test-ubuntu-20.04-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-ubuntu-20.04-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/12355247952/job/34478428256)|
   |test-ubuntu-20.04-cpp-bundled|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-ubuntu-20.04-cpp-bundled)](https://github.com/ursacomputing/crossbow/actions/runs/12355248187/job/34478430022)|
   |test-ubuntu-22.04-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-ubuntu-22.04-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/12355248358/job/34478433223)|
   |test-ubuntu-22.04-cpp-20|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-ubuntu-22.04-cpp-20)](https://github.com/ursacomputing/crossbow/actions/runs/12355248510/job/34478434895)|
   |test-ubuntu-22.04-cpp-emscripten|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-ubuntu-22.04-cpp-emscripten)](https://github.com/ursacomputing/crossbow/actions/runs/12355248509/job/34478434880)|
   |test-ubuntu-22.04-cpp-no-threading|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-ubuntu-22.04-cpp-no-threading)](https://github.com/ursacomputing/crossbow/actions/runs/12355248405/job/34478434361)|
   |test-ubuntu-24.04-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-ubuntu-24.04-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/12355248742/job/34478434924)|
   |test-ubuntu-24.04-cpp-bundled-offline|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-ubuntu-24.04-cpp-bundled-offline)](https://github.com/ursacomputing/crossbow/actions/runs/12355248033/job/34478428225)|
   |test-ubuntu-24.04-cpp-gcc-13-bundled|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-ubuntu-24.04-cpp-gcc-13-bundled)](https://github.com/ursacomputing/crossbow/actions/runs/12355248136/job/34478429945)|
   |test-ubuntu-24.04-cpp-gcc-14|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-ubuntu-24.04-cpp-gcc-14)](https://github.com/ursacomputing/crossbow/actions/runs/12355247942/job/34478428228)|
   |test-ubuntu-24.04-cpp-minimal-with-formats|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-ubuntu-24.04-cpp-minimal-with-formats)](https://github.com/ursacomputing/crossbow/actions/runs/12355248087/job/34478429154)|
   |test-ubuntu-24.04-cpp-thread-sanitizer|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-372e4c30db-github-test-ubuntu-24.04-cpp-thread-sanitizer)](https://github.com/ursacomputing/crossbow/actions/runs/12355248784/job/34478435242)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: [C++] Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #44897:
URL: https://github.com/apache/arrow/pull/44897#issuecomment-2581480992

   After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 894f5fb721b4849700eadebb41f7e04a837131ac.
   
   There were no benchmark performance regressions. 🎉
   
   The [full Conbench report](https://github.com/apache/arrow/runs/35403707229) has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #44897:
URL: https://github.com/apache/arrow/pull/44897#issuecomment-2530768583

   Revision: 0a3832f9562609ebe398289e2b905cf08c389a92
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-a42b04ee01](https://github.com/ursacomputing/crossbow/branches/all?query=actions-a42b04ee01)
   
   |Task|Status|
   |----|------|
   |example-cpp-minimal-build-static|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-example-cpp-minimal-build-static)](https://github.com/ursacomputing/crossbow/actions/runs/12252062979/job/34177908853)|
   |example-cpp-minimal-build-static-system-dependency|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-example-cpp-minimal-build-static-system-dependency)](https://github.com/ursacomputing/crossbow/actions/runs/12252062641/job/34177907379)|
   |example-cpp-tutorial|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-example-cpp-tutorial)](https://github.com/ursacomputing/crossbow/actions/runs/12252063070/job/34177909131)|
   |test-alpine-linux-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-alpine-linux-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/12252063140/job/34177909600)|
   |test-build-cpp-fuzz|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-build-cpp-fuzz)](https://github.com/ursacomputing/crossbow/actions/runs/12252062692/job/34177908052)|
   |test-conda-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-conda-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/12252062594/job/34177907587)|
   |test-conda-cpp-valgrind|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-conda-cpp-valgrind)](https://github.com/ursacomputing/crossbow/actions/runs/12252062684/job/34177907922)|
   |test-cuda-cpp-ubuntu-20.04-cuda-11.2.2|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-cuda-cpp-ubuntu-20.04-cuda-11.2.2)](https://github.com/ursacomputing/crossbow/actions/runs/12252063201/job/34177909881)|
   |test-cuda-cpp-ubuntu-22.04-cuda-11.7.1|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-cuda-cpp-ubuntu-22.04-cuda-11.7.1)](https://github.com/ursacomputing/crossbow/actions/runs/12252062867/job/34177908391)|
   |test-debian-12-cpp-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-debian-12-cpp-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/12252062541/job/34177907633)|
   |test-debian-12-cpp-i386|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-debian-12-cpp-i386)](https://github.com/ursacomputing/crossbow/actions/runs/12252063184/job/34177909604)|
   |test-fedora-39-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-fedora-39-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/12252062969/job/34177908879)|
   |test-ubuntu-20.04-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-ubuntu-20.04-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/12252062615/job/34177907358)|
   |test-ubuntu-20.04-cpp-bundled|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-ubuntu-20.04-cpp-bundled)](https://github.com/ursacomputing/crossbow/actions/runs/12252063096/job/34177909482)|
   |test-ubuntu-22.04-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-ubuntu-22.04-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/12252062817/job/34177908404)|
   |test-ubuntu-22.04-cpp-20|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-ubuntu-22.04-cpp-20)](https://github.com/ursacomputing/crossbow/actions/runs/12252063035/job/34177909234)|
   |test-ubuntu-22.04-cpp-emscripten|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-ubuntu-22.04-cpp-emscripten)](https://github.com/ursacomputing/crossbow/actions/runs/12252062525/job/34177907309)|
   |test-ubuntu-22.04-cpp-no-threading|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-ubuntu-22.04-cpp-no-threading)](https://github.com/ursacomputing/crossbow/actions/runs/12252062577/job/34177907368)|
   |test-ubuntu-24.04-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-ubuntu-24.04-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/12252063127/job/34177909609)|
   |test-ubuntu-24.04-cpp-bundled-offline|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-ubuntu-24.04-cpp-bundled-offline)](https://github.com/ursacomputing/crossbow/actions/runs/12252063239/job/34177910096)|
   |test-ubuntu-24.04-cpp-gcc-13-bundled|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-ubuntu-24.04-cpp-gcc-13-bundled)](https://github.com/ursacomputing/crossbow/actions/runs/12252062879/job/34177908551)|
   |test-ubuntu-24.04-cpp-gcc-14|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-ubuntu-24.04-cpp-gcc-14)](https://github.com/ursacomputing/crossbow/actions/runs/12252062977/job/34177908918)|
   |test-ubuntu-24.04-cpp-minimal-with-formats|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-ubuntu-24.04-cpp-minimal-with-formats)](https://github.com/ursacomputing/crossbow/actions/runs/12252063164/job/34177909872)|
   |test-ubuntu-24.04-cpp-thread-sanitizer|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-a42b04ee01-github-test-ubuntu-24.04-cpp-thread-sanitizer)](https://github.com/ursacomputing/crossbow/actions/runs/12252062663/job/34177907985)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on code in PR #44897:
URL: https://github.com/apache/arrow/pull/44897#discussion_r1876327652


##########
cpp/src/arrow/util/thread_pool_test.cc:
##########
@@ -578,6 +578,57 @@ TEST_F(TestThreadPool, Spawn) {
   SpawnAdds(pool.get(), 7, task_add<int>);
 }
 
+TEST_F(TestThreadPool, TasksRunInPriorityOrder) {
+  auto pool = this->MakeThreadPool(1);
+  constexpr int kNumTasks = 10;
+  auto recorded_times = std::vector<std::chrono::steady_clock::time_point>(kNumTasks);
+  auto futures = std::vector<Future<int>>(kNumTasks);
+  auto sleep_task = []() { SleepABit(); };
+
+  // Spawn a sleep task to block the pool while we add the other tasks. This
+  // ensures all the tasks are queued before any of them start running, so that
+  // their running order is fully determined by their priority.
+  ASSERT_OK(pool->Spawn(sleep_task));

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #44897:
URL: https://github.com/apache/arrow/pull/44897#issuecomment-2546853942

   Ok, thanks a lot for doing this @Tom-Newton !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on code in PR #44897:
URL: https://github.com/apache/arrow/pull/44897#discussion_r1869468022


##########
cpp/src/arrow/filesystem/filesystem.cc:
##########
@@ -642,12 +645,31 @@ Status CopyFiles(const std::vector<FileLocator>& sources,
     ARROW_ASSIGN_OR_RAISE(auto destination, destinations[i].filesystem->OpenOutputStream(
                                                 destinations[i].path, metadata));
     RETURN_NOT_OK(internal::CopyStream(source, destination, chunk_size, io_context));
-    return destination->Close();
+    // Using the blocking Close() here can cause reduced performance and deadlocks because
+    // FileSystem implementations that implement background_writes need to queue and wait
+    // for other IO thread(s). There is a risk that most or all the threads in the IO
+    // thread pool are blocking on a call Close(), leaving no IO threads left to actually
+    // fulfil the background writes.
+    return destination->CloseAsync();
   };
 
-  return ::arrow::internal::OptionalParallelFor(
-      use_threads, static_cast<int>(sources.size()), std::move(copy_one_file),
-      io_context.executor());
+  // Spawn copy_one_file less urgently than default, so that background_writes are done
+  // with higher priority. Otherwise copy_one_file will keep buffering more data in memory
+  // without giving the background_writes any chance to upload the data and drop it from
+  // memory. Therefore, without this large copies would cause OOMs.

Review Comment:
   Yes, that is why I updated `cpp/src/arrow/util/thread_pool.cc` to make use of `TaskHints`... I've been trying to get some opinion from maintainers on whether this is a reasonable thing to do. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #44897:
URL: https://github.com/apache/arrow/pull/44897#issuecomment-2545371107

   @github-actions crossbow submit -g cpp


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #44897:
URL: https://github.com/apache/arrow/pull/44897#issuecomment-2546102345

   Yes, I've restarted those CI builds.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: [C++] Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #44897:
URL: https://github.com/apache/arrow/pull/44897


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #44897:
URL: https://github.com/apache/arrow/pull/44897#issuecomment-2545842472

   @github-actions crossbow submit -g cpp


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44897:
URL: https://github.com/apache/arrow/pull/44897#discussion_r1876172195


##########
cpp/src/arrow/filesystem/test_util.cc:
##########
@@ -578,6 +578,67 @@ void GenericFileSystemTest::TestCopyFile(FileSystem* fs) {
   AssertAllFiles(fs, {"AB/abc", "EF/ghi", "def"});
 }
 
+void GenericFileSystemTest::TestCopyFiles(FileSystem* fs) {
+#if defined(ADDRESS_SANITIZER) || defined(ARROW_VALGRIND)
+  if (have_false_positive_memory_leak_with_async_close()) {
+    GTEST_SKIP() << "Filesystem have false positive memory leak with generator";
+  }
+#endif
+  auto io_thread_pool =
+      static_cast<arrow::internal::ThreadPool*>(fs->io_context().executor());
+  auto original_threads = io_thread_pool->GetCapacity();
+  // Needs to be smaller than the number of files we test with to catch GH-15233
+  ASSERT_OK(io_thread_pool->SetCapacity(2));
+  // Ensure the thread pool capacity is set back to the original value after the test
+  auto reset_thread_pool = [io_thread_pool, original_threads](void*) {
+    ASSERT_OK(io_thread_pool->SetCapacity(original_threads));
+  };
+  std::unique_ptr<void, decltype(reset_thread_pool)> resetThreadPoolGuard(

Review Comment:
   Nit
   ```suggestion
     std::unique_ptr<void, decltype(reset_thread_pool)> reset_thread_pool_guard(
   ```



##########
cpp/src/arrow/util/thread_pool.cc:
##########
@@ -52,6 +52,24 @@ struct Task {
   Executor::StopCallback stop_callback;
 };
 
+struct QueuedTask {
+  Task task;
+  int32_t priority;
+  uint64_t spawn_index;
+
+  // Implement comparison so that std::priority_queue will pop the low priorities more
+  // urgently.
+  bool operator<(const QueuedTask& other) const {
+    if (priority == other.priority) {
+      // Maintain spawn order for tasks with the same priority. TODO: Decide if this is
+      // really needed. Currently several test cases in arrow-acero-hash-aggregate-test
+      // depend on it.

Review Comment:
   I think this is preferrable for deterministic execution, so we can probably remove the TODO.



##########
cpp/src/arrow/util/thread_pool_test.cc:
##########
@@ -578,6 +578,57 @@ TEST_F(TestThreadPool, Spawn) {
   SpawnAdds(pool.get(), 7, task_add<int>);
 }
 
+TEST_F(TestThreadPool, TasksRunInPriorityOrder) {
+  auto pool = this->MakeThreadPool(1);
+  constexpr int kNumTasks = 10;
+  auto recorded_times = std::vector<std::chrono::steady_clock::time_point>(kNumTasks);
+  auto futures = std::vector<Future<int>>(kNumTasks);
+  auto sleep_task = []() { SleepABit(); };
+
+  // Spawn a sleep task to block the pool while we add the other tasks. This
+  // ensures all the tasks are queued before any of them start running, so that
+  // their running order is fully determined by their priority.
+  ASSERT_OK(pool->Spawn(sleep_task));

Review Comment:
   This may well fail to fulfill its purpose if run on a slow or busy machine (which unfortunately can happen with public CI infrastructure). 
   
   Using a synchronization primitive would be better, something like:
   1) sleep_task waits for a condition
   2) main thread spawns all record_time tasks, which can't run until sleep_task finishes
   3) main thread sets the condition so that sleep_task returns, unblocking the thread pool



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on code in PR #44897:
URL: https://github.com/apache/arrow/pull/44897#discussion_r1869614409


##########
cpp/src/arrow/util/thread_pool.cc:
##########
@@ -449,8 +468,8 @@ static void WorkerLoop(std::shared_ptr<ThreadPool::State> state,
 
       DCHECK_GE(state->tasks_queued_or_running_, 0);
       {
-        Task task = std::move(state->pending_tasks_.front());
-        state->pending_tasks_.pop_front();
+        Task task = std::move(const_cast<Task&>(state->pending_tasks_.top().task));

Review Comment:
   Hmm... it looks like `c` is protected so it doesn't seem to like it 
   ```
   /home/tomnewton/arrow/cpp/src/arrow/util/thread_pool.cc:471:53: error: ‘std::vector<arrow::internal::{anonymous}::QueuedTask, std::allocator<arrow::internal::{anonymous}::QueuedTask> > std::priority_queue<arrow::internal::{anonymous}::QueuedTask>::c’ is protected within this context
   ```



##########
cpp/src/arrow/filesystem/test_util.cc:
##########
@@ -578,6 +578,67 @@ void GenericFileSystemTest::TestCopyFile(FileSystem* fs) {
   AssertAllFiles(fs, {"AB/abc", "EF/ghi", "def"});
 }
 
+void GenericFileSystemTest::TestCopyFiles(FileSystem* fs) {
+#if defined(ADDRESS_SANITIZER) || defined(ARROW_VALGRIND)
+  if (have_false_positive_memory_leak_with_async_close()) {
+    GTEST_SKIP() << "Filesystem have false positive memory leak with generator";
+  }
+#endif
+  auto originalThreads = io::GetIOThreadPoolCapacity();
+  // Needs to be smaller than the number of files we test with to catch GH-15233
+  ASSERT_OK(io::SetIOThreadPoolCapacity(2));
+  // Ensure the thread pool capacity is set back to the original value after the test
+  auto resetThreadPool = [originalThreads](void*) {
+    ASSERT_OK(io::SetIOThreadPoolCapacity(originalThreads));
+  };
+  std::unique_ptr<void, decltype(resetThreadPool)> resetThreadPoolGuard(nullptr,
+                                                                        resetThreadPool);
+
+  auto mock_fs = std::make_shared<arrow::fs::internal::MockFileSystem>(
+      std::chrono::system_clock::now());
+  std::shared_ptr<FileSystem> shared_ptr_fs(fs, [](FileSystem*) {});

Review Comment:
   Yeah, that's just because I don't know what I'm doing with C++ :sweat-smile:



##########
cpp/src/arrow/filesystem/test_util.cc:
##########
@@ -578,6 +578,67 @@ void GenericFileSystemTest::TestCopyFile(FileSystem* fs) {
   AssertAllFiles(fs, {"AB/abc", "EF/ghi", "def"});
 }
 
+void GenericFileSystemTest::TestCopyFiles(FileSystem* fs) {
+#if defined(ADDRESS_SANITIZER) || defined(ARROW_VALGRIND)
+  if (have_false_positive_memory_leak_with_async_close()) {
+    GTEST_SKIP() << "Filesystem have false positive memory leak with generator";
+  }
+#endif
+  auto originalThreads = io::GetIOThreadPoolCapacity();

Review Comment:
   > Also it seems that we should actually call fs->io_context().executor()->capacity()?
   
   
   :+1: I've also changed the SetCapacity to be done in a similar way. Hopefully that makes sense.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #44897:
URL: https://github.com/apache/arrow/pull/44897#issuecomment-2530870796

   Hmm, it looks like we'll either have to fix the serial executor to also take priorities into account (if that's at all possible?), or simply disable the priority test when threading is disabled.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on PR #44897:
URL: https://github.com/apache/arrow/pull/44897#issuecomment-2541800131

   @pitrou do you have any opinion on extra tests for this? If you are happy please can you trigger the crossbow tests again. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on PR #44897:
URL: https://github.com/apache/arrow/pull/44897#issuecomment-2545428556

   Sorry, it looks like there are some bits that I missed when converting `SerialExecutor` to priority queue because I never tested it locally with `ARROW_ENABLE_THREADING=OFF` 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44897:
URL: https://github.com/apache/arrow/pull/44897#discussion_r1869409753


##########
cpp/src/arrow/filesystem/filesystem.cc:
##########
@@ -642,12 +645,31 @@ Status CopyFiles(const std::vector<FileLocator>& sources,
     ARROW_ASSIGN_OR_RAISE(auto destination, destinations[i].filesystem->OpenOutputStream(
                                                 destinations[i].path, metadata));
     RETURN_NOT_OK(internal::CopyStream(source, destination, chunk_size, io_context));
-    return destination->Close();
+    // Using the blocking Close() here can cause reduced performance and deadlocks because
+    // FileSystem implementations that implement background_writes need to queue and wait
+    // for other IO thread(s). There is a risk that most or all the threads in the IO
+    // thread pool are blocking on a call Close(), leaving no IO threads left to actually
+    // fulfil the background writes.
+    return destination->CloseAsync();
   };
 
-  return ::arrow::internal::OptionalParallelFor(
-      use_threads, static_cast<int>(sources.size()), std::move(copy_one_file),
-      io_context.executor());
+  // Spawn copy_one_file less urgently than default, so that background_writes are done
+  // with higher priority. Otherwise copy_one_file will keep buffering more data in memory
+  // without giving the background_writes any chance to upload the data and drop it from
+  // memory. Therefore, without this large copies would cause OOMs.

Review Comment:
   Note that TaskHints are ignored by the default executor implementation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44897:
URL: https://github.com/apache/arrow/pull/44897#discussion_r1869522078


##########
cpp/src/arrow/util/thread_pool_test.cc:
##########
@@ -578,6 +578,55 @@ TEST_F(TestThreadPool, Spawn) {
   SpawnAdds(pool.get(), 7, task_add<int>);
 }
 
+TEST_F(TestThreadPool, TasksRunInPriorityOrder) {
+  auto pool = this->MakeThreadPool(1);
+  auto recorded_times = std::vector<std::chrono::system_clock::time_point>(10);

Review Comment:
   Let's avoid hardcoding 10 everywhere in this test?
   ```suggestion
     constexpr int kNumTasks = 10;
     auto recorded_times = std::vector<std::chrono::system_clock::time_point>(kNumTasks);
   ```
   (etc.)



##########
cpp/src/arrow/util/thread_pool.cc:
##########
@@ -449,8 +468,8 @@ static void WorkerLoop(std::shared_ptr<ThreadPool::State> state,
 
       DCHECK_GE(state->tasks_queued_or_running_, 0);
       {
-        Task task = std::move(state->pending_tasks_.front());
-        state->pending_tasks_.pop_front();
+        Task task = std::move(const_cast<Task&>(state->pending_tasks_.top().task));

Review Comment:
   AFAICT we can remove the `const_cast` hack by writing:
   ```suggestion
           Task task = std::move(state->pending_tasks_.c.front().task);
   ```



##########
cpp/src/arrow/util/thread_pool.cc:
##########
@@ -52,6 +52,24 @@ struct Task {
   Executor::StopCallback stop_callback;
 };
 
+struct QueuedTask {
+  Task task;
+  TaskHints hints;

Review Comment:
   Storing priority would be sufficient I think.



##########
cpp/src/arrow/filesystem/test_util.cc:
##########
@@ -578,6 +578,67 @@ void GenericFileSystemTest::TestCopyFile(FileSystem* fs) {
   AssertAllFiles(fs, {"AB/abc", "EF/ghi", "def"});
 }
 
+void GenericFileSystemTest::TestCopyFiles(FileSystem* fs) {
+#if defined(ADDRESS_SANITIZER) || defined(ARROW_VALGRIND)
+  if (have_false_positive_memory_leak_with_async_close()) {
+    GTEST_SKIP() << "Filesystem have false positive memory leak with generator";
+  }
+#endif
+  auto originalThreads = io::GetIOThreadPoolCapacity();
+  // Needs to be smaller than the number of files we test with to catch GH-15233
+  ASSERT_OK(io::SetIOThreadPoolCapacity(2));
+  // Ensure the thread pool capacity is set back to the original value after the test
+  auto resetThreadPool = [originalThreads](void*) {
+    ASSERT_OK(io::SetIOThreadPoolCapacity(originalThreads));
+  };
+  std::unique_ptr<void, decltype(resetThreadPool)> resetThreadPoolGuard(nullptr,
+                                                                        resetThreadPool);
+
+  auto mock_fs = std::make_shared<arrow::fs::internal::MockFileSystem>(
+      std::chrono::system_clock::now());
+  std::shared_ptr<FileSystem> shared_ptr_fs(fs, [](FileSystem*) {});

Review Comment:
   Uh, what is this for? Normally you can just use `fs->shared_from_this()`.



##########
cpp/src/arrow/util/thread_pool_test.cc:
##########
@@ -578,6 +578,55 @@ TEST_F(TestThreadPool, Spawn) {
   SpawnAdds(pool.get(), 7, task_add<int>);
 }
 
+TEST_F(TestThreadPool, TasksRunInPriorityOrder) {
+  auto pool = this->MakeThreadPool(1);
+  auto recorded_times = std::vector<std::chrono::system_clock::time_point>(10);
+  auto futures = std::vector<Future<int>>(10);
+  auto sleep_task = []() { SleepABit(); };
+
+  // Spawn a sleep task to block the pool while we add the other tasks. This
+  // ensures all the tasks are queued before any of them start running, so that
+  // their running order is fully determined by their priority.
+  ASSERT_OK(pool->Spawn(sleep_task));
+
+  for (int i = 0; i < 10; ++i) {
+    auto record_time = [&recorded_times, i]() {
+      recorded_times[i] = std::chrono::system_clock::now();

Review Comment:
   We should use `steady_clock` to ensure that we get monotonic time measurements.



##########
cpp/src/arrow/filesystem/test_util.cc:
##########
@@ -578,6 +578,67 @@ void GenericFileSystemTest::TestCopyFile(FileSystem* fs) {
   AssertAllFiles(fs, {"AB/abc", "EF/ghi", "def"});
 }
 
+void GenericFileSystemTest::TestCopyFiles(FileSystem* fs) {
+#if defined(ADDRESS_SANITIZER) || defined(ARROW_VALGRIND)
+  if (have_false_positive_memory_leak_with_async_close()) {
+    GTEST_SKIP() << "Filesystem have false positive memory leak with generator";
+  }
+#endif
+  auto originalThreads = io::GetIOThreadPoolCapacity();
+  // Needs to be smaller than the number of files we test with to catch GH-15233
+  ASSERT_OK(io::SetIOThreadPoolCapacity(2));
+  // Ensure the thread pool capacity is set back to the original value after the test
+  auto resetThreadPool = [originalThreads](void*) {

Review Comment:
   Here as well 



##########
cpp/src/arrow/filesystem/test_util.cc:
##########
@@ -578,6 +578,67 @@ void GenericFileSystemTest::TestCopyFile(FileSystem* fs) {
   AssertAllFiles(fs, {"AB/abc", "EF/ghi", "def"});
 }
 
+void GenericFileSystemTest::TestCopyFiles(FileSystem* fs) {
+#if defined(ADDRESS_SANITIZER) || defined(ARROW_VALGRIND)
+  if (have_false_positive_memory_leak_with_async_close()) {
+    GTEST_SKIP() << "Filesystem have false positive memory leak with generator";
+  }
+#endif
+  auto originalThreads = io::GetIOThreadPoolCapacity();

Review Comment:
   Also it seems that we should actually call `fs->io_context().executor()->capacity()`?



##########
cpp/src/arrow/filesystem/filesystem.cc:
##########
@@ -642,12 +645,31 @@ Status CopyFiles(const std::vector<FileLocator>& sources,
     ARROW_ASSIGN_OR_RAISE(auto destination, destinations[i].filesystem->OpenOutputStream(
                                                 destinations[i].path, metadata));
     RETURN_NOT_OK(internal::CopyStream(source, destination, chunk_size, io_context));
-    return destination->Close();
+    // Using the blocking Close() here can cause reduced performance and deadlocks because
+    // FileSystem implementations that implement background_writes need to queue and wait
+    // for other IO thread(s). There is a risk that most or all the threads in the IO
+    // thread pool are blocking on a call Close(), leaving no IO threads left to actually
+    // fulfil the background writes.
+    return destination->CloseAsync();
   };
 
-  return ::arrow::internal::OptionalParallelFor(
-      use_threads, static_cast<int>(sources.size()), std::move(copy_one_file),
-      io_context.executor());
+  // Spawn copy_one_file less urgently than default, so that background_writes are done
+  // with higher priority. Otherwise copy_one_file will keep buffering more data in memory
+  // without giving the background_writes any chance to upload the data and drop it from
+  // memory. Therefore, without this large copies would cause OOMs.

Review Comment:
   Thanks, taking a look now.



##########
cpp/src/arrow/filesystem/test_util.cc:
##########
@@ -578,6 +578,67 @@ void GenericFileSystemTest::TestCopyFile(FileSystem* fs) {
   AssertAllFiles(fs, {"AB/abc", "EF/ghi", "def"});
 }
 
+void GenericFileSystemTest::TestCopyFiles(FileSystem* fs) {
+#if defined(ADDRESS_SANITIZER) || defined(ARROW_VALGRIND)
+  if (have_false_positive_memory_leak_with_async_close()) {
+    GTEST_SKIP() << "Filesystem have false positive memory leak with generator";
+  }
+#endif
+  auto originalThreads = io::GetIOThreadPoolCapacity();

Review Comment:
   Please let's keep consistent with the coding conventions (`snake_case` for local variables)
   



##########
cpp/src/arrow/io/interfaces.cc:
##########
@@ -68,8 +68,8 @@ Status SetIOThreadPoolCapacity(int threads) {
 FileInterface::~FileInterface() = default;
 
 Future<> FileInterface::CloseAsync() {
-  return DeferNotOk(
-      default_io_context().executor()->Submit([this]() { return Close(); }));
+  return DeferNotOk(default_io_context().executor()->Submit(
+      [self = shared_from_this()]() { return self->Close(); }));

Review Comment:
   +1



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on PR #44897:
URL: https://github.com/apache/arrow/pull/44897#issuecomment-2536599942

   > Hmm, it looks like we'll either have to fix the serial executor to also take priorities into account (if that's at all possible?), or simply disable the priority test when threading is disabled.
   
   I think we can implement priorities exactly the same on `SerialExecutor` as on `ThreadPool`, so I pushed a commit to do that. I feel like this should be covered by the normal CI though, without needing to build with `ARROW_ENABLE_THREADING=OFF`, so I'm going to see how easy it is to parameterise the new tests I added. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44897:
URL: https://github.com/apache/arrow/pull/44897#discussion_r1886659069


##########
cpp/src/arrow/util/thread_pool.cc:
##########
@@ -386,7 +405,8 @@ struct ThreadPool::State {
   std::list<std::thread> workers_;
   // Trashcan for finished threads
   std::vector<std::thread> finished_workers_;
-  std::deque<Task> pending_tasks_;
+  std::priority_queue<QueuedTask> pending_tasks_;
+  uint64_t spawned_tasks_count = 0;

Review Comment:
   Nit: naming
   ```suggestion
     uint64_t spawned_tasks_count_ = 0;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on code in PR #44897:
URL: https://github.com/apache/arrow/pull/44897#discussion_r1887162947


##########
cpp/src/arrow/util/thread_pool.cc:
##########
@@ -386,7 +405,8 @@ struct ThreadPool::State {
   std::list<std::thread> workers_;
   // Trashcan for finished threads
   std::vector<std::thread> finished_workers_;
-  std::deque<Task> pending_tasks_;
+  std::priority_queue<QueuedTask> pending_tasks_;
+  uint64_t spawned_tasks_count = 0;

Review Comment:
   I just realised I forgot about this and I also should have noticed the naming convention, when I originally wrote it. Pushing a correction now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on PR #44897:
URL: https://github.com/apache/arrow/pull/44897#issuecomment-2545672435

   Ok, this time I managed to test it locally `ARROW_ENABLE_THREADING=OFF` so hopefully I haven't missed anything else. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on code in PR #44897:
URL: https://github.com/apache/arrow/pull/44897#discussion_r1869653872


##########
cpp/src/arrow/filesystem/test_util.cc:
##########
@@ -578,6 +578,67 @@ void GenericFileSystemTest::TestCopyFile(FileSystem* fs) {
   AssertAllFiles(fs, {"AB/abc", "EF/ghi", "def"});
 }
 
+void GenericFileSystemTest::TestCopyFiles(FileSystem* fs) {
+#if defined(ADDRESS_SANITIZER) || defined(ARROW_VALGRIND)
+  if (have_false_positive_memory_leak_with_async_close()) {
+    GTEST_SKIP() << "Filesystem have false positive memory leak with generator";
+  }
+#endif
+  auto originalThreads = io::GetIOThreadPoolCapacity();
+  // Needs to be smaller than the number of files we test with to catch GH-15233
+  ASSERT_OK(io::SetIOThreadPoolCapacity(2));
+  // Ensure the thread pool capacity is set back to the original value after the test
+  auto resetThreadPool = [originalThreads](void*) {
+    ASSERT_OK(io::SetIOThreadPoolCapacity(originalThreads));
+  };
+  std::unique_ptr<void, decltype(resetThreadPool)> resetThreadPoolGuard(nullptr,
+                                                                        resetThreadPool);
+
+  auto mock_fs = std::make_shared<arrow::fs::internal::MockFileSystem>(
+      std::chrono::system_clock::now());
+  std::shared_ptr<FileSystem> shared_ptr_fs(fs, [](FileSystem*) {});

Review Comment:
   Yeah, that's just because I don't know what I'm doing with C++ :sweat_smile: 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on code in PR #44897:
URL: https://github.com/apache/arrow/pull/44897#discussion_r1869771031


##########
cpp/src/arrow/util/thread_pool.cc:
##########
@@ -449,8 +468,8 @@ static void WorkerLoop(std::shared_ptr<ThreadPool::State> state,
 
       DCHECK_GE(state->tasks_queued_or_running_, 0);
       {
-        Task task = std::move(state->pending_tasks_.front());
-        state->pending_tasks_.pop_front();
+        Task task = std::move(const_cast<Task&>(state->pending_tasks_.top().task));

Review Comment:
   Also maybe useful to mention the error without any cast is 
   ```
   function "arrow::internal::<unnamed>::Task::Task(const arrow::internal::<unnamed>::Task &)" (declared implicitly) cannot be referenced -- it is a deleted functionC/C++(1776)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on PR #44897:
URL: https://github.com/apache/arrow/pull/44897#issuecomment-2536655815

   >I feel like this should be covered by the normal CI though, without needing to build with `ARROW_ENABLE_THREADING=OFF`, so I'm going to see how easy it is to parameterise the new tests I added.
   
   I think it would at least require making the constructor of `SerialExecutor`, so maybe its not really desirable to add such a test?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on code in PR #44897:
URL: https://github.com/apache/arrow/pull/44897#discussion_r1876200719


##########
cpp/src/arrow/filesystem/test_util.cc:
##########
@@ -578,6 +578,67 @@ void GenericFileSystemTest::TestCopyFile(FileSystem* fs) {
   AssertAllFiles(fs, {"AB/abc", "EF/ghi", "def"});
 }
 
+void GenericFileSystemTest::TestCopyFiles(FileSystem* fs) {
+#if defined(ADDRESS_SANITIZER) || defined(ARROW_VALGRIND)
+  if (have_false_positive_memory_leak_with_async_close()) {
+    GTEST_SKIP() << "Filesystem have false positive memory leak with generator";
+  }
+#endif
+  auto io_thread_pool =
+      static_cast<arrow::internal::ThreadPool*>(fs->io_context().executor());
+  auto original_threads = io_thread_pool->GetCapacity();
+  // Needs to be smaller than the number of files we test with to catch GH-15233
+  ASSERT_OK(io_thread_pool->SetCapacity(2));
+  // Ensure the thread pool capacity is set back to the original value after the test
+  auto reset_thread_pool = [io_thread_pool, original_threads](void*) {
+    ASSERT_OK(io_thread_pool->SetCapacity(original_threads));
+  };
+  std::unique_ptr<void, decltype(reset_thread_pool)> resetThreadPoolGuard(

Review Comment:
   Sorry, I should have spotted that one myself. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on code in PR #44897:
URL: https://github.com/apache/arrow/pull/44897#discussion_r1875885152


##########
cpp/src/arrow/util/thread_pool.cc:
##########
@@ -449,8 +468,8 @@ static void WorkerLoop(std::shared_ptr<ThreadPool::State> state,
 
       DCHECK_GE(state->tasks_queued_or_running_, 0);
       {
-        Task task = std::move(state->pending_tasks_.front());
-        state->pending_tasks_.pop_front();
+        Task task = std::move(const_cast<Task&>(state->pending_tasks_.top().task));

Review Comment:
   That makes sense and its the same as before when it was a `std::dequeue`. The difference with `std::priority_queue` is that it returns a `const` reference and `std::move` requires a non-const reference. The internal container is also protected, so I don't really see any alternative to using `const_cast` and as far as I understand it should be safe in this case.  
   
   Do you think the current implementation with `const_cast` is an acceptable solution?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44897:
URL: https://github.com/apache/arrow/pull/44897#discussion_r1875895027


##########
cpp/src/arrow/util/thread_pool.cc:
##########
@@ -449,8 +468,8 @@ static void WorkerLoop(std::shared_ptr<ThreadPool::State> state,
 
       DCHECK_GE(state->tasks_queued_or_running_, 0);
       {
-        Task task = std::move(state->pending_tasks_.front());
-        state->pending_tasks_.pop_front();
+        Task task = std::move(const_cast<Task&>(state->pending_tasks_.top().task));

Review Comment:
   Yes, it's acceptable IMHO. I was just hoping there was a cleaner alternative :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #44897:
URL: https://github.com/apache/arrow/pull/44897#issuecomment-2530761154

   @github-actions crossbow submit -g cpp


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on PR #44897:
URL: https://github.com/apache/arrow/pull/44897#issuecomment-2545897573

   Thanks for triggering the extra tests again. That run is looking better but it looks like a couple failed because of a timeout downloading a dependency
   ```
       error: downloading 'https://archive.apache.org/dist/orc/orc-format-1.0.0/orc-format-1.0.0.tar.gz' failed
             status_code: 28
             status_string: "Timeout was reached"
             log:
             --- LOG BEGIN ---
               Trying 65.108.204.189:443...
       Trying 2a01:4f9:1a:a084::2:443...
   ```
   Presumably this is not related to me changes. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-15233: Fix CopyFiles when destination is a FileSystem with background_writes [arrow]

Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on code in PR #44897:
URL: https://github.com/apache/arrow/pull/44897#discussion_r1869771031


##########
cpp/src/arrow/util/thread_pool.cc:
##########
@@ -449,8 +468,8 @@ static void WorkerLoop(std::shared_ptr<ThreadPool::State> state,
 
       DCHECK_GE(state->tasks_queued_or_running_, 0);
       {
-        Task task = std::move(state->pending_tasks_.front());
-        state->pending_tasks_.pop_front();
+        Task task = std::move(const_cast<Task&>(state->pending_tasks_.top().task));

Review Comment:
   Also maybe useful to mention the error without any cast is 
   ```
   /home/tomnewton/arrow/cpp/src/arrow/util/thread_pool.cc:471:63: error: use of deleted function ‘arrow::internal::{anonymous}::Task::Task(const arrow::internal::{anonymous}::Task&)’
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org