You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "pitrou (via GitHub)" <gi...@apache.org> on 2023/05/17 16:52:10 UTC
[GitHub] [arrow] pitrou opened a new issue, #35641: [CI][C++] Bad compile/link times despite ccache
pitrou opened a new issue, #35641:
URL: https://github.com/apache/arrow/issues/35641
### Describe the bug, including details regarding any error messages, version, and platform.
Example here on the AMD64 Conda C++ build:
https://github.com/apache/arrow/actions/runs/5005430904/jobs/8969381538?pr=33634
```
=== ccache statistics before build
===
Cacheable calls: 284 / 894 (31.77%)
Hits: 14 / 284 ( 4.93%)
Direct: 14 / 14 (100.0%)
Preprocessed: 0 / 14 ( 0.00%)
Misses: 270 / 284 (95.07%)
Uncacheable calls: 610 / 894 (68.23%)
Local storage:
Cache size (GB): 0.1 / 1.0 (11.35%)
Hits: 14 / 284 ( 4.93%)
Misses: 270 / 284 (95.07%)
[ ... CMake runs ... ]
real 27m46.563s
user 49m43.936s
sys 4m28.705s
===
=== ccache statistics after build
===
Cacheable calls: 568 / 1788 (31.77%)
Hits: 298 / 568 (52.46%)
Direct: 233 / 298 (78.19%)
Preprocessed: 65 / 298 (21.81%)
Misses: 270 / 568 (47.54%)
Uncacheable calls: 1220 / 1788 (68.23%)
Local storage:
Cache size (GB): 0.1 / 1.0 (11.35%)
Hits: 298 / 568 (52.46%)
Misses: 270 / 568 (47.54%)
```
So ccache was completely successful in this build (0 misses during the entire CMake configure / build), yet CMake took 27 minutes.
We should investigate:
* disabling building static libraries (`ARROW_BUILD_STATIC`) to save on linking times
* why there are more than 600 uncacheable calls and also 280 calls, which is the majority of the ~860 compilation calls (the "Building CXX" lines in the logs)
Note that the "AMD64 Conda C++" build can be reproduced using [Archery](https://arrow.apache.org/docs/dev/developers/continuous_integration/docker.html) with the `conda-cpp` image:
```console
$ archery docker run conda-cpp
```
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #35641: [CI][C++] Bad compile/link times despite ccache
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #35641:
URL: https://github.com/apache/arrow/issues/35641#issuecomment-1553171746
@felipecrv Do you want to take a look at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on issue #35641: [CI][C++] Bad compile/link times despite ccache
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on issue #35641:
URL: https://github.com/apache/arrow/issues/35641#issuecomment-1552223776
hm interesting investigation, thanks @pitrou !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #35641: [CI][C++] Bad compile/link times despite ccache
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #35641:
URL: https://github.com/apache/arrow/issues/35641#issuecomment-1559517298
No problem, @felipecrv ! @benibus, are you interested in taking a look at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #35641: [CI][C++] Bad compile/link times despite ccache
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #35641:
URL: https://github.com/apache/arrow/issues/35641#issuecomment-1551775293
It seems "Could not use precompiled header" is the reason, at least on `conda-cpp`, for uncacheable compiler calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #35641: [CI][C++] Bad compile/link times despite ccache
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #35641:
URL: https://github.com/apache/arrow/issues/35641#issuecomment-1551750212
cc @kou @felipecrv
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou closed issue #35641: [CI][C++] Bad compile/link times despite ccache
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #35641: [CI][C++] Bad compile/link times despite ccache
URL: https://github.com/apache/arrow/issues/35641
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on issue #35641: [CI][C++] Bad compile/link times despite ccache
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on issue #35641:
URL: https://github.com/apache/arrow/issues/35641#issuecomment-1559514412
I'm unassigning myself for now because I have too many open workstreams that I want to close right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #35641: [CI][C++] Bad compile/link times despite ccache
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #35641:
URL: https://github.com/apache/arrow/issues/35641#issuecomment-1552215717
If precompiled header is a problem, we may be able to fix this by the following change:
```diff
diff --git a/docker-compose.yml b/docker-compose.yml
index 588146075..96b4290f5 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -281,7 +281,6 @@ services:
ARROW_EXTRA_ERROR_CONTEXT: "ON"
ARROW_MIMALLOC: "ON"
ARROW_USE_LD_GOLD: "ON"
- ARROW_USE_PRECOMPILED_HEADERS: "ON"
BUILD_DOCS_PYTHON: "ON"
volumes: &conda-volumes
- .:/arrow:delegated
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #35641: [CI][C++] Bad compile/link times despite ccache
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #35641:
URL: https://github.com/apache/arrow/issues/35641#issuecomment-1552762520
> If precompiled header is a problem, we may be able to fix this by the following change:
Yes, I think we should disable precompiled headers, and perhaps enable unity builds instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org