You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/07/08 18:11:16 UTC
[GitHub] [arrow] assignUser opened a new pull request, #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
assignUser opened a new pull request, #13556:
URL: https://github.com/apache/arrow/pull/13556
This is a first stab at enabling sccache, I used the R nightly job as an example as it compiles arrow on mac, win and linux.
I want to open this for feedback specifically with an eye towards the cmake changes I implemented which enable this as long as the correct envvars (AWS creds + bucket) are set and sccache is available. My hope was to keep the changes required to activate sccache for existing jobs as minimal as possible.
While working on this I also noticed that `ThirdpartyToolchain.cmake` is not build using (s)ccache, the flags had to be explicitly set. That fix alone should speed up CI across the board (in arrow and crossbow).
For the R task this results in a build time reduction of 40-70% 🚀
Thanks for your input!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
assignUser commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r917156431
##########
r/inst/build_arrow_static.sh:
##########
@@ -81,10 +96,14 @@ ${CMAKE} -DARROW_BOOST_USE_SHARED=OFF \
-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON \
-DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD:-OFF} \
-Dxsimd_SOURCE=${xsimd_SOURCE:-} \
+ -DARROW_USE_SCCACHE=ON \
${EXTRA_CMAKE_FLAGS} \
-G ${CMAKE_GENERATOR:-"Unix Makefiles"} \
${SOURCE_DIR}
${CMAKE} --build . --target install
+echo "=== sccache stats after the build ==="
+sccache --show-stats
Review Comment:
https://github.com/ursacomputing/crossbow/runs/7257889831?check_suite_focus=true#step:6:2578
The other Linux builds of that matrix and the wincpp job also displays them if you want to compare :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1179247146
https://issues.apache.org/jira/browse/ARROW-17021
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1189681905
@github-actions crossbow submit r-binary-packages conan-*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
raulcd commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r924288454
##########
dev/tasks/r/github.packages.yml:
##########
@@ -82,14 +82,24 @@ jobs:
{{ macros.github_checkout_arrow()|indent }}
{{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
{{ macros.github_install_archery()|indent }}
+
- name: Build libarrow
shell: bash
env:
- UBUNTU: {{ '${{ matrix.config.version}}' }}
+ SCCACHE_BUCKET: {{ '${{ secrets.SCCACHE_BUCKET }}' }}
+ SCCACHE_ID: {{ '${{ secrets.AWS_ACCESS_KEY_ID }}' }}
+ SCCACHE_KEY: {{' ${{ secrets.AWS_SECRET_ACCESS_KEY }}' }}
+ SCCACHE_PREFIX: sccache
Review Comment:
I am happier with this name `SCCACHE_S3_KEY_PREFIX` too as it gives me a little more context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1246563329
@github-actions crossbow submit r-binary-packages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1187091950
@github-actions crossbow submit r-binary-packages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1190282075
@github-actions crossbow submit -g cpp r-binary-packages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
kszucs commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1187217998
> For all the reasons above I think we shouldn't bother with `actions/cache` here.
Agree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
assignUser commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r972213827
##########
ci/scripts/install_sccache.sh:
##########
@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+if [ "$#" -lt 1 -o "$#" -gt 3 ]; then
+ echo "Usage: $0 <build> <prefix> <arch> <version>"
+ echo "Will default to arch=x86_64 and version=0.3.0 "
+ exit 1
+fi
+
+BUILD=$1
+PREFIX=$2
+ARCH=${3:-x86_64}
+VERSION=${4:-0.3.0}
+
+SCCACHE_URL="https://github.com/mozilla/sccache/releases/download/v0.3.0/sccache-v$VERSION-$ARCH-$BUILD.tar.gz"
+SCCACHE_ARCHIVE=sccache.tar.gz
+
+# Download archive and checksum
+curl -L $SCCACHE_URL --output $SCCACHE_ARCHIVE
+curl -L $SCCACHE_URL.sha256 --output $SCCACHE_ARCHIVE.sha256
Review Comment:
I initially used wget but that was missing from some runner (windows maybe? it's been a while ^^) so I switch to curl, only to then find out that curl is not installed in all images so wither would have been fine :shrug:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
assignUser commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r972214668
##########
ci/scripts/install_sccache.sh:
##########
@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+if [ "$#" -lt 1 -o "$#" -gt 3 ]; then
+ echo "Usage: $0 <build> <prefix> <arch> <version>"
+ echo "Will default to arch=x86_64 and version=0.3.0 "
+ exit 1
+fi
+
+BUILD=$1
+PREFIX=$2
+ARCH=${3:-x86_64}
+VERSION=${4:-0.3.0}
+
+SCCACHE_URL="https://github.com/mozilla/sccache/releases/download/v0.3.0/sccache-v$VERSION-$ARCH-$BUILD.tar.gz"
Review Comment:
Indeed, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
pitrou commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1180168860
> Can we change our approach so that we can use `actions/cache`?
We can, but we'll also run into capacity limits which may make caching poorly efficient ("A repository can have up to 10GB of caches").
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
pitrou commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1190502322
Caveat: looking at the source code, sccache only supports setting a cache size limit for local disk and for Redis. So our S3 bucket size may grow a lot if we are not careful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1189674005
Revision: 70c555b7b9ab739aa748b94205e19a7f993709c9
Submitted crossbow builds: [ursacomputing/crossbow @ actions-3a66b33d96](https://github.com/ursacomputing/crossbow/branches/all?query=actions-3a66b33d96)
|Task|Status|
|----|------|
|conan-maximum|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-3a66b33d96-github-conan-maximum)|
|conan-minimum|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-3a66b33d96-github-conan-minimum)|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-3a66b33d96-github-r-binary-packages)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1188557886
Revision: 2cae958857b5cf1bb6aab75aa2b9e13119dd5b48
Submitted crossbow builds: [ursacomputing/crossbow @ actions-f170114481](https://github.com/ursacomputing/crossbow/branches/all?query=actions-f170114481)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-f170114481-github-r-binary-packages)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1179247160
:warning: Ticket **has not been started in JIRA**, please click 'Start Progress'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
assignUser commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r972218322
##########
ci/scripts/install_sccache.sh:
##########
@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+if [ "$#" -lt 1 -o "$#" -gt 3 ]; then
+ echo "Usage: $0 <build> <prefix> <arch> <version>"
+ echo "Will default to arch=x86_64 and version=0.3.0 "
+ exit 1
+fi
+
+BUILD=$1
+PREFIX=$2
+ARCH=${3:-x86_64}
+VERSION=${4:-0.3.0}
+
+SCCACHE_URL="https://github.com/mozilla/sccache/releases/download/v0.3.0/sccache-v$VERSION-$ARCH-$BUILD.tar.gz"
+SCCACHE_ARCHIVE=sccache.tar.gz
+
+# Download archive and checksum
+curl -L $SCCACHE_URL --output $SCCACHE_ARCHIVE
+curl -L $SCCACHE_URL.sha256 --output $SCCACHE_ARCHIVE.sha256
+
+echo "$(cat $SCCACHE_ARCHIVE.sha256) $SCCACHE_ARCHIVE" | sha256sum --check --status
+
+if [ ! -d $PREFIX ]; then
+ mkdir -p $PREFIX
+fi
+
+tar -xzvf $SCCACHE_ARCHIVE --strip-component=1 --directory $PREFIX --wildcards sccache*/sccache*
+chmod u+x $PREFIX/sccache
+
+if [ "$GITHUB_ACTIONS" ]; then
Review Comment:
I use GITHUB_ACTION***S*** which is exactly for this use case:
> GITHUB_ACTIONS | Always set to true when GitHub Actions is running the workflow. You can use this variable to differentiate when tests are being run locally or by GitHub Actions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r971066116
##########
ci/scripts/install_sccache.sh:
##########
@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+if [ "$#" -lt 1 -o "$#" -gt 3 ]; then
+ echo "Usage: $0 <build> <prefix> <arch> <version>"
+ echo "Will default to arch=x86_64 and version=0.3.0 "
+ exit 1
+fi
+
+BUILD=$1
+PREFIX=$2
+ARCH=${3:-x86_64}
+VERSION=${4:-0.3.0}
+
+SCCACHE_URL="https://github.com/mozilla/sccache/releases/download/v0.3.0/sccache-v$VERSION-$ARCH-$BUILD.tar.gz"
Review Comment:
Did you mean
```suggestion
SCCACHE_URL="https://github.com/mozilla/sccache/releases/download/v$VERSION/sccache-v$VERSION-$ARCH-$BUILD.tar.gz"
```
##########
ci/scripts/install_sccache.sh:
##########
@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+if [ "$#" -lt 1 -o "$#" -gt 3 ]; then
+ echo "Usage: $0 <build> <prefix> <arch> <version>"
+ echo "Will default to arch=x86_64 and version=0.3.0 "
+ exit 1
+fi
+
+BUILD=$1
+PREFIX=$2
+ARCH=${3:-x86_64}
+VERSION=${4:-0.3.0}
+
+SCCACHE_URL="https://github.com/mozilla/sccache/releases/download/v0.3.0/sccache-v$VERSION-$ARCH-$BUILD.tar.gz"
+SCCACHE_ARCHIVE=sccache.tar.gz
+
+# Download archive and checksum
+curl -L $SCCACHE_URL --output $SCCACHE_ARCHIVE
+curl -L $SCCACHE_URL.sha256 --output $SCCACHE_ARCHIVE.sha256
Review Comment:
Just for the record, why not use `wget`?
##########
cpp/cmake_modules/DefineOptions.cmake:
##########
@@ -111,7 +111,10 @@ if(ARROW_DEFINE_OPTIONS)
define_option(ARROW_POSITION_INDEPENDENT_CODE
"Whether to create position-independent target" ON)
-
+
+ define_option(ARROW_USE_SCCACHE "Use sccache when compiling (if available),;\
+takes precdence over ccache if a storage backend is configured" ON)
Review Comment:
Also, move this below ARROW_USE_CCACHE?
##########
cpp/cmake_modules/DefineOptions.cmake:
##########
@@ -111,7 +111,10 @@ if(ARROW_DEFINE_OPTIONS)
define_option(ARROW_POSITION_INDEPENDENT_CODE
"Whether to create position-independent target" ON)
-
+
+ define_option(ARROW_USE_SCCACHE "Use sccache when compiling (if available),;\
+takes precdence over ccache if a storage backend is configured" ON)
Review Comment:
```suggestion
define_option(ARROW_USE_SCCACHE "Use sccache when compiling (if available), \
takes precedence over ccache if a storage backend is configured" ON)
```
##########
dev/tasks/tasks.yml:
##########
@@ -951,7 +951,7 @@ tasks:
custom_version: Unset
artifacts:
- r-lib__libarrow__bin__windows__arrow-[0-9\.]+\.zip
- - r-lib__libarrow__bin__centos-7__arrow-[0-9\.]+\.zip
+ #- r-lib__libarrow__bin__centos-7__arrow-[0-9\.]+\.zip
Review Comment:
Add a comment explaining why?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
kou commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r972418301
##########
ci/scripts/install_sccache.sh:
##########
@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+if [ "$#" -lt 1 -o "$#" -gt 3 ]; then
+ echo "Usage: $0 <build> <prefix> <arch> <version>"
+ echo "Will default to arch=x86_64 and version=0.3.0 "
+ exit 1
+fi
+
+BUILD=$1
+PREFIX=$2
+ARCH=${3:-x86_64}
+VERSION=${4:-0.3.0}
+
+SCCACHE_URL="https://github.com/mozilla/sccache/releases/download/v0.3.0/sccache-v$VERSION-$ARCH-$BUILD.tar.gz"
+SCCACHE_ARCHIVE=sccache.tar.gz
+
+# Download archive and checksum
+curl -L $SCCACHE_URL --output $SCCACHE_ARCHIVE
+curl -L $SCCACHE_URL.sha256 --output $SCCACHE_ARCHIVE.sha256
+
+echo "$(cat $SCCACHE_ARCHIVE.sha256) $SCCACHE_ARCHIVE" | sha256sum --check --status
+
+if [ ! -d $PREFIX ]; then
+ mkdir -p $PREFIX
+fi
+
+tar -xzvf $SCCACHE_ARCHIVE --strip-component=1 --directory $PREFIX --wildcards sccache*/sccache*
+chmod u+x $PREFIX/sccache
+
+if [ "$GITHUB_ACTIONS" ]; then
Review Comment:
Ah, sorry. I missed "S".
Then, `[ "${GITHUB_ACTIONS}" = "true" ]` is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1179330144
Revision: 2bdf9dc187087dd83c8aac9faba3ac47696c9e79
Submitted crossbow builds: [ursacomputing/crossbow @ actions-a68e8f2ff9](https://github.com/ursacomputing/crossbow/branches/all?query=actions-a68e8f2ff9)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-a68e8f2ff9-github-r-binary-packages)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r917728804
##########
cpp/cmake_modules/DefineOptions.cmake:
##########
@@ -111,7 +111,10 @@ if(ARROW_DEFINE_OPTIONS)
define_option(ARROW_POSITION_INDEPENDENT_CODE
"Whether to create position-independent target" ON)
-
+
+ define_option(ARROW_USE_SCCACHE "Use sccache when compiling (if available), takes precdent\
Review Comment:
```suggestion
define_option(ARROW_USE_SCCACHE "Use sccache when compiling (if available), takes precdence\
```
##########
dev/tasks/r/github.packages.yml:
##########
@@ -114,17 +124,38 @@ jobs:
- run: git config --global core.autocrlf false
{{ macros.github_checkout_arrow()|indent }}
{{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
-
- uses: r-lib/actions/setup-r@v2
with:
rtools-version: 40
r-version: "4.0"
Ncpus: 2
+ - name: Get sccache
+ shell: bash
+ run: |
+ SCCACHE_BUILD=pc-windows-msvc
+ SCCACHE_URL="https://github.com/mozilla/sccache/releases/download/v0.3.0/sccache-v0.3.0-x86_64-$SCCACHE_BUILD.tar.gz"
Review Comment:
Should there be a script e.g. `ci/script/install_sccache.sh` for this?
The syntax could be `ci/scripts/install_sccache.sh <build> <executable install dir>`
##########
dev/tasks/r/github.packages.yml:
##########
@@ -280,10 +319,33 @@ jobs:
with:
install-r: false
{{ macros.github_setup_local_r_repo(false, false)|indent }}
+ - name: Get sccache
+ shell: bash
+ run: |
+ SCCACHE_BUILD=unknown-linux-musl
+ SCCACHE_URL="https://github.com/mozilla/sccache/releases/download/v0.3.0/sccache-v0.3.0-x86_64-$SCCACHE_BUILD.tar.gz"
+
+ # Download archive and checksum
+ curl -L $SCCACHE_URL --output sccache.tar.gz
+ curl -L $SCCACHE_URL.sha256 --output sccache.tar.gz.sha256
+
+ SCCACHE_ARCHIVE=sccache.tar.gz
+ echo "$(cat $SCCACHE_ARCHIVE.sha256) $SCCACHE_ARCHIVE" | sha256sum --check --status
+
+ mkdir -p sccache
+ tar -xzvf $SCCACHE_ARCHIVE --strip-component=1 --directory sccache
+
+ echo "$(pwd)/sccache" >> $GITHUB_PATH
+ echo "SCCACHE_PATH=$(pwd)/sccache/sccache.exe" >> $GITHUB_ENV
+ - run: sudo apt update && sudo apt install libcurl4-openssl-dev
Review Comment:
Seems a bit weird indeed. Why would dev dependencies be neede for a compiled executable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r926586624
##########
docker-compose.yml:
##########
@@ -62,6 +62,12 @@ x-ccache: &ccache
CCACHE_MAXSIZE: 1G
CCACHE_DIR: /ccache
+x-sccache: &sccache
+ AWS_ACCESS_KEY_ID:
+ AWS_SECRET_ACCESS_KEY:
+ SCCACHE_BUCKET:
Review Comment:
Yes, saving us the work to add the mall manually via `-e`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1245691203
@github-actions crossbow submit conan-* test-conda-cpp test-conda-python-*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1246559639
Revision: cd0dbecbc452c381407f5b09bbc288ceaa3bd131
Submitted crossbow builds: [ursacomputing/crossbow @ actions-fc7864294d](https://github.com/ursacomputing/crossbow/branches/all?query=actions-fc7864294d)
|Task|Status|
|----|------|
|conan-maximum|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-fc7864294d-github-conan-maximum)|
|conan-minimum|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-fc7864294d-github-conan-minimum)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1246567090
Revision: 9a942589e9190743d8eb189a76957b78464d039c
Submitted crossbow builds: [ursacomputing/crossbow @ actions-f1372d9832](https://github.com/ursacomputing/crossbow/branches/all?query=actions-f1372d9832)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-f1372d9832-github-r-binary-packages)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1246854057
This is ready for review, the job above only failed due to the missing [centos artifact, ](https://github.com/ursacomputing/crossbow/actions/runs/3052342834/jobs/4924218048#step:10:37)I have fixed that in `task.yml`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r924255831
##########
cpp/cmake_modules/DefineOptions.cmake:
##########
@@ -111,7 +111,10 @@ if(ARROW_DEFINE_OPTIONS)
define_option(ARROW_POSITION_INDEPENDENT_CODE
"Whether to create position-independent target" ON)
-
+
+ define_option(ARROW_USE_SCCACHE "Use sccache when compiling (if available), takes precdence\
+ over ccache" ON)
Review Comment:
Hmm, I think we might want to leave this off by default, since sccache requires configuring unlike ccache? Just because the binary is installed on the machine doesn't mean it's actually usable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
pitrou commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1187216096
For all the reasons above I think we shouldn't bother with `actions/cache` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r923274369
##########
dev/tasks/r/github.packages.yml:
##########
@@ -280,10 +319,33 @@ jobs:
with:
install-r: false
{{ macros.github_setup_local_r_repo(false, false)|indent }}
+ - name: Get sccache
+ shell: bash
+ run: |
+ SCCACHE_BUILD=unknown-linux-musl
+ SCCACHE_URL="https://github.com/mozilla/sccache/releases/download/v0.3.0/sccache-v0.3.0-x86_64-$SCCACHE_BUILD.tar.gz"
+
+ # Download archive and checksum
+ curl -L $SCCACHE_URL --output sccache.tar.gz
+ curl -L $SCCACHE_URL.sha256 --output sccache.tar.gz.sha256
+
+ SCCACHE_ARCHIVE=sccache.tar.gz
+ echo "$(cat $SCCACHE_ARCHIVE.sha256) $SCCACHE_ARCHIVE" | sha256sum --check --status
+
+ mkdir -p sccache
+ tar -xzvf $SCCACHE_ARCHIVE --strip-component=1 --directory sccache
Review Comment:
Tested it, yes otherwise we get permission errors. See [here](https://github.com/ursacomputing/crossbow/runs/7388502640?check_suite_focus=true#step:6:1140)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
kou commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1189802135
> It would also be a **major** rework of all of the related archery functions, which is likely to happen at some point but sccache will help us _now_.
OK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1189681319
@github-actions crossbow submit r-binary-packages conan-*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1188493694
@github-actions crossbow submit r-binary-packages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1187161540
Revision: 7a1c9c447ce6924544b947826deeb6926eed63cb
Submitted crossbow builds: [ursacomputing/crossbow @ actions-a3056bf7fb](https://github.com/ursacomputing/crossbow/branches/all?query=actions-a3056bf7fb)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-a3056bf7fb-github-r-binary-packages)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1245327376
@github-actions crossbow submit conan-* test-conda-cpp test-conda-python-* test-conda-python-3.8-hypothesis r-binary-packages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on a diff in pull request #13556: ARROW-17021: [C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
assignUser commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r972874874
##########
ci/scripts/install_sccache.sh:
##########
@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+if [ "$#" -lt 1 -o "$#" -gt 3 ]; then
+ echo "Usage: $0 <build> <prefix> <arch> <version>"
+ echo "Will default to arch=x86_64 and version=0.3.0 "
+ exit 1
+fi
+
+BUILD=$1
+PREFIX=$2
+ARCH=${3:-x86_64}
+VERSION=${4:-0.3.0}
+
+SCCACHE_URL="https://github.com/mozilla/sccache/releases/download/v0.3.0/sccache-v$VERSION-$ARCH-$BUILD.tar.gz"
+SCCACHE_ARCHIVE=sccache.tar.gz
+
+# Download archive and checksum
+curl -L $SCCACHE_URL --output $SCCACHE_ARCHIVE
+curl -L $SCCACHE_URL.sha256 --output $SCCACHE_ARCHIVE.sha256
+
+echo "$(cat $SCCACHE_ARCHIVE.sha256) $SCCACHE_ARCHIVE" | sha256sum --check --status
+
+if [ ! -d $PREFIX ]; then
+ mkdir -p $PREFIX
+fi
+
+tar -xzvf $SCCACHE_ARCHIVE --strip-component=1 --directory $PREFIX --wildcards sccache*/sccache*
+chmod u+x $PREFIX/sccache
+
+if [ "$GITHUB_ACTIONS" ]; then
Review Comment:
Hm I guess people could set the envvar to false for their own scripts or something and that would be a false positive like this, ok 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
pitrou commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1248356008
By the way, compiling sccache from source only happened once, I hope (not on every CI run)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1179247389
@github-actions crossbow submit r-binary-packages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1179253961
@github-actions crossbow submit r-binary-packages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
kou commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1180019956
We want to use `sccache` because we can't use `actions/cache` of GitHub Actions with our "one branch per task" approach, right?
Can we change our approach so that we can use `actions/cache`? For example:
1. We always use the `master` branch for all tasks instead of creating one branch per task
2. We add `jobs/${JOB_ID}/tasks.yaml` (or something) that lists commit IDs for target tasks instead of using branch search feature of GitHub (that is too slow for many branches)
(We can keep only the latest N jobs because old logs in GitHub Actions are removed.)
3. We create a `${JOB_ID}-${TASK_NAME}` tag for each commit of task to create a release of GitHub Releases to upload artifacts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
assignUser commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r923216544
##########
dev/tasks/r/github.packages.yml:
##########
@@ -155,11 +186,19 @@ jobs:
rtools-version: {{ '${{ matrix.r_version.rtools }}' }}
Ncpus: 2
{{ macros.github_setup_local_r_repo(false, true)|indent }}
+ - name: Install sccache
+ if: startsWith(matrix.platform, 'macos')
+ run: brew install sccache
Review Comment:
Yes I was also already thinking of making this it's own action.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r923291276
##########
dev/tasks/r/github.packages.yml:
##########
@@ -280,10 +319,33 @@ jobs:
with:
install-r: false
{{ macros.github_setup_local_r_repo(false, false)|indent }}
+ - name: Get sccache
+ shell: bash
+ run: |
+ SCCACHE_BUILD=unknown-linux-musl
+ SCCACHE_URL="https://github.com/mozilla/sccache/releases/download/v0.3.0/sccache-v0.3.0-x86_64-$SCCACHE_BUILD.tar.gz"
+
+ # Download archive and checksum
+ curl -L $SCCACHE_URL --output sccache.tar.gz
+ curl -L $SCCACHE_URL.sha256 --output sccache.tar.gz.sha256
+
+ SCCACHE_ARCHIVE=sccache.tar.gz
+ echo "$(cat $SCCACHE_ARCHIVE.sha256) $SCCACHE_ARCHIVE" | sha256sum --check --status
+
+ mkdir -p sccache
+ tar -xzvf $SCCACHE_ARCHIVE --strip-component=1 --directory sccache
+
+ echo "$(pwd)/sccache" >> $GITHUB_PATH
+ echo "SCCACHE_PATH=$(pwd)/sccache/sccache.exe" >> $GITHUB_ENV
+ - run: sudo apt update && sudo apt install libcurl4-openssl-dev
Review Comment:
This is a source build and does not use the binaries, so these are required for aws/gcs bindings to be enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r924256471
##########
dev/tasks/r/github.packages.yml:
##########
@@ -82,14 +82,24 @@ jobs:
{{ macros.github_checkout_arrow()|indent }}
{{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
{{ macros.github_install_archery()|indent }}
+
- name: Build libarrow
shell: bash
env:
- UBUNTU: {{ '${{ matrix.config.version}}' }}
+ SCCACHE_BUCKET: {{ '${{ secrets.SCCACHE_BUCKET }}' }}
+ SCCACHE_ID: {{ '${{ secrets.AWS_ACCESS_KEY_ID }}' }}
+ SCCACHE_KEY: {{' ${{ secrets.AWS_SECRET_ACCESS_KEY }}' }}
Review Comment:
Also, I like that this is explicit that these are for sccache, not for unrelated purposes. So I would vote to keep the `SCCACHE` prefix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1190348323
Revision: 660df214541680f90a1ef9463df529f0176d2725
Submitted crossbow builds: [ursacomputing/crossbow @ actions-91ca0fff5c](https://github.com/ursacomputing/crossbow/branches/all?query=actions-91ca0fff5c)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-91ca0fff5c-github-r-binary-packages)|
|test-build-cpp-fuzz|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-91ca0fff5c-github-test-build-cpp-fuzz)|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-91ca0fff5c-github-test-conda-cpp)|
|test-conda-cpp-valgrind|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-91ca0fff5c-azure-test-conda-cpp-valgrind)|
|test-debian-10-cpp-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-91ca0fff5c-github-test-debian-10-cpp-amd64)|
|test-debian-10-cpp-i386|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-91ca0fff5c-github-test-debian-10-cpp-i386)|
|test-debian-11-cpp-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-91ca0fff5c-github-test-debian-11-cpp-amd64)|
|test-debian-11-cpp-i386|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-91ca0fff5c-github-test-debian-11-cpp-i386)|
|test-fedora-35-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-91ca0fff5c-github-test-fedora-35-cpp)|
|test-ubuntu-18.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-91ca0fff5c-github-test-ubuntu-18.04-cpp)|
|test-ubuntu-18.04-cpp-release|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-91ca0fff5c-github-test-ubuntu-18.04-cpp-release)|
|test-ubuntu-18.04-cpp-static|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-91ca0fff5c-github-test-ubuntu-18.04-cpp-static)|
|test-ubuntu-20.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-91ca0fff5c-github-test-ubuntu-20.04-cpp)|
|test-ubuntu-20.04-cpp-14|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-91ca0fff5c-github-test-ubuntu-20.04-cpp-14)|
|test-ubuntu-20.04-cpp-17|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-91ca0fff5c-github-test-ubuntu-20.04-cpp-17)|
|test-ubuntu-20.04-cpp-bundled|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-91ca0fff5c-github-test-ubuntu-20.04-cpp-bundled)|
|test-ubuntu-20.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-91ca0fff5c-github-test-ubuntu-20.04-cpp-thread-sanitizer)|
|test-ubuntu-22.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-91ca0fff5c-github-test-ubuntu-22.04-cpp)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1190410011
@pitrou see above, it is now very easy to activate sccache for tasks
I re-ran `test-conda-cpp` 32min -> 13min
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1246555843
@github-actions crossbow submit conan-*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #13556: ARROW-17021: [C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1250195690
Benchmark runs are scheduled for baseline = decddbbf1afbf620710f95e2c82b4b6f59a752da and contender = c58e6a3550a7eb887e2e7911079a3aa48a1f5bbf. c58e6a3550a7eb887e2e7911079a3aa48a1f5bbf is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/eeeed56221b9450ea980fd1328755fc2...253d62140a3741518b6ac974866f5355/)
[Failed :arrow_down:0.65% :arrow_up:0.03%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/0f1e98cee1c746ef9def5deaec1028fa...2c0eb132675b442d8bf76fd54e68dad5/)
[Failed :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/f74ef1b7398447fe872b8d6f91a9bc6f...06a21c379d7b4f4ea72163f71d2ab22f/)
[Finished :arrow_down:0.25% :arrow_up:0.0%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/845f53fc26a940cd8a6fa3c815af16ba...161cca0772184401aa9a2f5ff702493e/)
Buildkite builds:
[Finished] [`c58e6a35` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1510)
[Failed] [`c58e6a35` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1527)
[Failed] [`c58e6a35` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1510)
[Finished] [`c58e6a35` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1524)
[Finished] [`decddbbf` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1509)
[Failed] [`decddbbf` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1526)
[Failed] [`decddbbf` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1509)
[Finished] [`decddbbf` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1523)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1179248053
```
Failed to render template `r/github.packages.yml` with UndefinedError: 'matrix' is undefined
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/2637816571
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r924256758
##########
dev/tasks/r/github.packages.yml:
##########
@@ -82,14 +82,24 @@ jobs:
{{ macros.github_checkout_arrow()|indent }}
{{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
{{ macros.github_install_archery()|indent }}
+
- name: Build libarrow
shell: bash
env:
- UBUNTU: {{ '${{ matrix.config.version}}' }}
+ SCCACHE_BUCKET: {{ '${{ secrets.SCCACHE_BUCKET }}' }}
+ SCCACHE_ID: {{ '${{ secrets.AWS_ACCESS_KEY_ID }}' }}
+ SCCACHE_KEY: {{' ${{ secrets.AWS_SECRET_ACCESS_KEY }}' }}
+ SCCACHE_PREFIX: sccache
Review Comment:
SCCACHE_S3_KEY_PREFIX perhaps?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1188494299
Revision: cd1926268c1c2ac92b6cf12919c64090393dac2a
Submitted crossbow builds: [ursacomputing/crossbow @ actions-fa4002462d](https://github.com/ursacomputing/crossbow/branches/all?query=actions-fa4002462d)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-fa4002462d-github-r-binary-packages)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r923332641
##########
dev/tasks/r/github.packages.yml:
##########
@@ -82,14 +82,24 @@ jobs:
{{ macros.github_checkout_arrow()|indent }}
{{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
{{ macros.github_install_archery()|indent }}
+
- name: Build libarrow
shell: bash
env:
- UBUNTU: {{ '${{ matrix.config.version}}' }}
+ SCCACHE_BUCKET: {{ '${{ secrets.SCCACHE_BUCKET }}' }}
+ SCCACHE_ID: {{ '${{ secrets.AWS_ACCESS_KEY_ID }}' }}
+ SCCACHE_KEY: {{' ${{ secrets.AWS_SECRET_ACCESS_KEY }}' }}
Review Comment:
The idea was to make the archery command a bit more concise as archery emulates the docker `-e` param but does not support passing only the envvar name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r923276083
##########
cpp/CMakeLists.txt:
##########
@@ -215,18 +215,39 @@ else()
set(PYTHON_EXECUTABLE ${Python3_EXECUTABLE})
endif()
-if(ARROW_USE_CCACHE
- AND NOT CMAKE_C_COMPILER_LAUNCHER
- AND NOT CMAKE_CXX_COMPILER_LAUNCHER)
+if((ARROW_USE_SCCACHE)
+ AND NOT CMAKE_C_COMPILER_LAUNCHER
+ AND NOT CMAKE_CXX_COMPILER_LAUNCHER)
+
+ find_program(SCCACHE_FOUND sccache)
+
+ if(NOT SCCACHE_FOUND AND DEFINED ENV{SCCACHE_PATH})
+ # cmake has problems finding sccache from within mingw
Review Comment:
No it falls back to [ccache](https://github.com/ursacomputing/crossbow/runs/7388502838?check_suite_focus=true#step:7:5555) if we don't use the envvar hack. I would prefer to not need it too...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1189692687
@github-actions crossbow submit conan-*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1190264895
@github-actiuons crossbow submit -g cpp r-binary-packages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
raulcd commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r923327193
##########
dev/tasks/r/github.packages.yml:
##########
@@ -82,14 +82,24 @@ jobs:
{{ macros.github_checkout_arrow()|indent }}
{{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
{{ macros.github_install_archery()|indent }}
+
- name: Build libarrow
shell: bash
env:
- UBUNTU: {{ '${{ matrix.config.version}}' }}
+ SCCACHE_BUCKET: {{ '${{ secrets.SCCACHE_BUCKET }}' }}
+ SCCACHE_ID: {{ '${{ secrets.AWS_ACCESS_KEY_ID }}' }}
+ SCCACHE_KEY: {{' ${{ secrets.AWS_SECRET_ACCESS_KEY }}' }}
Review Comment:
wouldn't be better to call this `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. I am not sure the renaming adds anything and the environment variable names used on archery are already the original ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r925810577
##########
docker-compose.yml:
##########
@@ -62,6 +62,12 @@ x-ccache: &ccache
CCACHE_MAXSIZE: 1G
CCACHE_DIR: /ccache
+x-sccache: &sccache
+ AWS_ACCESS_KEY_ID:
+ AWS_SECRET_ACCESS_KEY:
+ SCCACHE_BUCKET:
Review Comment:
Does it mean they are inherited from the calling process environment?
##########
docker-compose.yml:
##########
@@ -245,8 +251,10 @@ services:
volumes: &conda-volumes
- .:/arrow:delegated
- ${DOCKER_VOLUME_PREFIX}conda-ccache:/ccache:delegated
- command:
- ["/arrow/ci/scripts/cpp_build.sh /arrow /build &&
+ command: &conda-cpp-command
+ ["/arrow/ci/scripts/install_sccache.sh unknown-linux-musl &&
Review Comment:
I'm curious: why "unknown-linux-musl"? Does our conda image really use musl?
##########
dev/tasks/r/github.packages.yml:
##########
@@ -82,10 +82,15 @@ jobs:
{{ macros.github_checkout_arrow()|indent }}
{{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
{{ macros.github_install_archery()|indent }}
+
- name: Build libarrow
shell: bash
env:
- UBUNTU: {{ '${{ matrix.config.version}}' }}
+ AWS_ACCESS_KEY_ID: {{ '${{ secrets.AWS_ACCESS_KEY_ID }}' }}
+ AWS_SECRET_ACCESS_KEY: {{' ${{ secrets.AWS_SECRET_ACCESS_KEY }}' }}
+ SCCACHE_BUCKET: {{ '${{ secrets.SCCACHE_BUCKET }}' }}
+ SCCACHE_S3_KEY_PREFIX: sccache
Review Comment:
This snippet is everywhere, can it be made a macro?
##########
docker-compose.yml:
##########
@@ -245,8 +251,10 @@ services:
volumes: &conda-volumes
- .:/arrow:delegated
- ${DOCKER_VOLUME_PREFIX}conda-ccache:/ccache:delegated
- command:
- ["/arrow/ci/scripts/cpp_build.sh /arrow /build &&
+ command: &conda-cpp-command
+ ["/arrow/ci/scripts/install_sccache.sh unknown-linux-musl &&
Review Comment:
Also, can the `install_sccache.sh` call instead be moved to the dockerfile?
##########
docker-compose.yml:
##########
@@ -245,8 +251,10 @@ services:
volumes: &conda-volumes
- .:/arrow:delegated
- ${DOCKER_VOLUME_PREFIX}conda-ccache:/ccache:delegated
- command:
- ["/arrow/ci/scripts/cpp_build.sh /arrow /build &&
+ command: &conda-cpp-command
+ ["/arrow/ci/scripts/install_sccache.sh unknown-linux-musl &&
+ export PATH=$$(pwd)/sccache:$$PATH &&
Review Comment:
Probably sccache can be installed to `/usr/local/bin` instead?
##########
r/inst/build_arrow_static.sh:
##########
@@ -87,4 +87,7 @@ ${CMAKE} -DARROW_BOOST_USE_SHARED=OFF \
${CMAKE} --build . --target install
+echo "=== sccache stats after the build ==="
+sccache --show-stats
Review Comment:
Can we add similar snippets to `ci/script/cpp_build.sh`, just as with ccache?
##########
ci/scripts/install_sccache.sh:
##########
@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+if [ "$#" -lt 1 -o "$#" -gt 3 ]; then
+ echo "Usage: $0 <build> <arch> <version>"
+ echo "Will default to arch=x86_64 and version=0.3.0 "
+ exit 1
+fi
+
+BUILD=$1
+ARCH=${2:-x86_64}
+VERSION=${3:-0.3.0}
+
+SCCACHE_URL="https://github.com/mozilla/sccache/releases/download/v0.3.0/sccache-v$VERSION-$ARCH-$BUILD.tar.gz"
+SCCACHE_ARCHIVE=sccache.tar.gz
+
+# Download archive and checksum
+curl -L $SCCACHE_URL --output $SCCACHE_ARCHIVE
+curl -L $SCCACHE_URL.sha256 --output $SCCACHE_ARCHIVE.sha256
+
+echo "$(cat $SCCACHE_ARCHIVE.sha256) $SCCACHE_ARCHIVE" | sha256sum --check --status
+
+mkdir -p sccache
+tar -xzvf $SCCACHE_ARCHIVE --strip-component=1 --directory sccache
+chmod u+x sccache/sccache
+
+SCCACHE_DIR=$(pwd)/sccache
Review Comment:
Can the install prefix instead be taken as an argument, just like for `install_minio.sh`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
raulcd commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r917246083
##########
dev/tasks/r/github.packages.yml:
##########
@@ -280,10 +319,33 @@ jobs:
with:
install-r: false
{{ macros.github_setup_local_r_repo(false, false)|indent }}
+ - name: Get sccache
+ shell: bash
+ run: |
+ SCCACHE_BUILD=unknown-linux-musl
+ SCCACHE_URL="https://github.com/mozilla/sccache/releases/download/v0.3.0/sccache-v0.3.0-x86_64-$SCCACHE_BUILD.tar.gz"
+
+ # Download archive and checksum
+ curl -L $SCCACHE_URL --output sccache.tar.gz
+ curl -L $SCCACHE_URL.sha256 --output sccache.tar.gz.sha256
+
+ SCCACHE_ARCHIVE=sccache.tar.gz
+ echo "$(cat $SCCACHE_ARCHIVE.sha256) $SCCACHE_ARCHIVE" | sha256sum --check --status
+
+ mkdir -p sccache
+ tar -xzvf $SCCACHE_ARCHIVE --strip-component=1 --directory sccache
+
+ echo "$(pwd)/sccache" >> $GITHUB_PATH
+ echo "SCCACHE_PATH=$(pwd)/sccache/sccache.exe" >> $GITHUB_ENV
+ - run: sudo apt update && sudo apt install libcurl4-openssl-dev
Review Comment:
why is this now required? Should be done on the previous step (install sscache)?
##########
cpp/cmake_modules/DefineOptions.cmake:
##########
@@ -111,7 +111,10 @@ if(ARROW_DEFINE_OPTIONS)
define_option(ARROW_POSITION_INDEPENDENT_CODE
"Whether to create position-independent target" ON)
-
+
+ define_option(ARROW_USE_SCCACHE "Use sccache when compiling (if available), takes precdent\
Review Comment:
```suggestion
define_option(ARROW_USE_SCCACHE "Use sccache when compiling (if available), takes precedent\
```
##########
dev/tasks/r/github.packages.yml:
##########
@@ -155,11 +186,19 @@ jobs:
rtools-version: {{ '${{ matrix.r_version.rtools }}' }}
Ncpus: 2
{{ macros.github_setup_local_r_repo(false, true)|indent }}
+ - name: Install sccache
+ if: startsWith(matrix.platform, 'macos')
+ run: brew install sccache
Review Comment:
could we install on windows following the same approach we don on line 132: `- name: Get sccache`?
Maybe we could create our own action for that. I am just thinking out loud, not required for this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
kou commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r917144917
##########
dev/tasks/r/github.packages.yml:
##########
@@ -280,10 +319,33 @@ jobs:
with:
install-r: false
{{ macros.github_setup_local_r_repo(false, false)|indent }}
+ - name: Get sccache
+ shell: bash
+ run: |
+ SCCACHE_BUILD=unknown-linux-musl
+ SCCACHE_URL="https://github.com/mozilla/sccache/releases/download/v0.3.0/sccache-v0.3.0-x86_64-$SCCACHE_BUILD.tar.gz"
+
+ # Download archive and checksum
+ curl -L $SCCACHE_URL --output sccache.tar.gz
+ curl -L $SCCACHE_URL.sha256 --output sccache.tar.gz.sha256
+
+ SCCACHE_ARCHIVE=sccache.tar.gz
+ echo "$(cat $SCCACHE_ARCHIVE.sha256) $SCCACHE_ARCHIVE" | sha256sum --check --status
+
+ mkdir -p sccache
+ tar -xzvf $SCCACHE_ARCHIVE --strip-component=1 --directory sccache
Review Comment:
Do we need `chmod +x sccache/sccache`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1248386889
> By the way, compiling sccache from source only happened once, I hope (not on every CI run)?
That shouldn't have happened at al 🤦 it was very specific to the use of the 10.13 runner which is eol -> no bottles. But yes it will only happen this one time, anything else would be crazy :D Should we add further 10.13 runners I will install it manually before the runner is connected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1189682362
Revision: 474f393b645589d68d658754c723dabcfad30abf
Submitted crossbow builds: [ursacomputing/crossbow @ actions-2a8ed4df38](https://github.com/ursacomputing/crossbow/branches/all?query=actions-2a8ed4df38)
|Task|Status|
|----|------|
|conan-maximum|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2a8ed4df38-github-conan-maximum)|
|conan-minimum|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2a8ed4df38-github-conan-minimum)|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2a8ed4df38-github-r-binary-packages)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1187204465
@kou adding to @pitrou's comment: We could rework archery to take advantage of `actions/cache` but the action itself is inefficient/broken for this use case because it does not update on a primary key hit, this would force us to create new, unique caches every time leading to the mentioned capacity issue.
It would also be a **major** rework of all of the related archery functions, which is likely to happen at some point but sccache will help us *now*.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
pitrou commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1188803256
For the record, it might be easier to start with the C++ builds? They are generally simpler (less steps) than the R builds...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r926557181
##########
docker-compose.yml:
##########
@@ -245,8 +251,10 @@ services:
volumes: &conda-volumes
- .:/arrow:delegated
- ${DOCKER_VOLUME_PREFIX}conda-ccache:/ccache:delegated
- command:
- ["/arrow/ci/scripts/cpp_build.sh /arrow /build &&
+ command: &conda-cpp-command
+ ["/arrow/ci/scripts/install_sccache.sh unknown-linux-musl &&
Review Comment:
> There are prebuilt x86-64 binaries available for Windows, **Linux (a portable binary compiled against musl),** and macOS
I haven't moved the installation to the dockerfiles for ease of testing but we can do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1245377499
crossbow submit conan-* test-conda-cpp test-conda-python-*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
assignUser commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r972237067
##########
cpp/cmake_modules/DefineOptions.cmake:
##########
@@ -111,7 +111,10 @@ if(ARROW_DEFINE_OPTIONS)
define_option(ARROW_POSITION_INDEPENDENT_CODE
"Whether to create position-independent target" ON)
-
+
+ define_option(ARROW_USE_SCCACHE "Use sccache when compiling (if available),;\
+takes precdence over ccache if a storage backend is configured" ON)
Review Comment:
I have moved ccache above this, the semicolon is not a typo though, it is required as a line break:
```
CMake Error at cmake_modules/DefineOptions.cmake:27 (message):
description for ARROW_USE_SCCACHE contained a
line 105 characters long!
(max is 80). Split it into more lines with semicolons
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
kou commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r971414553
##########
ci/docker/centos-7-cpp.dockerfile:
##########
@@ -20,6 +20,7 @@ FROM centos:centos7
RUN yum install -y \
diffutils \
gcc-c++ \
+ curl \
Review Comment:
Could you keep this list in alphabetical order?
##########
cpp/CMakeLists.txt:
##########
@@ -191,18 +191,47 @@ else()
set(PYTHON_EXECUTABLE ${Python3_EXECUTABLE})
endif()
-if(ARROW_USE_CCACHE
+if((ARROW_USE_SCCACHE)
AND NOT CMAKE_C_COMPILER_LAUNCHER
AND NOT CMAKE_CXX_COMPILER_LAUNCHER)
+
+ find_program(SCCACHE_FOUND sccache)
+
+ if(NOT SCCACHE_FOUND AND DEFINED ENV{SCCACHE_PATH})
+ # cmake has problems finding sccache from within mingw
+ message(STATUS "Did not find sccache, using envvar fallback.")
+ set(SCCACHE_FOUND $ENV{SCCACHE_PATH})
+ endif()
+
+ # Only use sccache if a storage backend is configured
+ if(SCCACHE_FOUND AND (
+ DEFINED ENV{SCCACHE_AZURE_BLOB_CONTAINER}
+ OR DEFINED ENV{SCCACHE_BUCKET}
+ OR DEFINED ENV{SCCACHE_DIR}
+ OR DEFINED ENV{SCCACHE_GCS_BUCKET}
+ OR DEFINED ENV{SCCACHE_MEMCACHED}
+ OR DEFINED ENV{SCCACHE_REDIS}
+ ))
+ message(STATUS "Using sccache: ${SCCACHE_FOUND}")
+ set(CMAKE_C_COMPILER_LAUNCHER ${SCCACHE_FOUND})
+ set(CMAKE_CXX_COMPILER_LAUNCHER ${SCCACHE_FOUND})
+ endif()
+endif()
+
+if((ARROW_USE_CCACHE)
Review Comment:
```suggestion
if(ARROW_USE_CCACHE
```
##########
dev/tasks/docker-tests/github.linux.yml:
##########
@@ -31,6 +31,8 @@ jobs:
- name: Execute Docker Build
shell: bash
+ env:
+ {{ macros.github_set_sccache_envvars()|indent(8) }}
Review Comment:
Do we need one more indent here?
```suggestion
{{ macros.github_set_sccache_envvars()|indent(10) }}
```
##########
docker-compose.yml:
##########
@@ -411,14 +416,16 @@ services:
- apparmor:unconfined
ulimits: *ulimits
environment:
- <<: *ccache
+ <<: [*ccache, *sccache]
ARROW_HOME: /arrow
ARROW_DEPENDENCY_SOURCE: BUNDLED
LIBARROW_MINIMAL: "false"
ARROW_MIMALLOC: "ON"
volumes: *ubuntu-volumes
- command: /bin/bash -c "
- cd /arrow && r/inst/build_arrow_static.sh"
+ command: &arrow-static-cmd
Review Comment:
Could you use the same naming convention others use?
```suggestion
command: &cpp-static-command
```
##########
cpp/CMakeLists.txt:
##########
@@ -191,18 +191,47 @@ else()
set(PYTHON_EXECUTABLE ${Python3_EXECUTABLE})
endif()
-if(ARROW_USE_CCACHE
+if((ARROW_USE_SCCACHE)
Review Comment:
```suggestion
if(ARROW_USE_SCCACHE
```
##########
ci/scripts/install_sccache.sh:
##########
@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+if [ "$#" -lt 1 -o "$#" -gt 3 ]; then
+ echo "Usage: $0 <build> <prefix> <arch> <version>"
+ echo "Will default to arch=x86_64 and version=0.3.0 "
+ exit 1
+fi
+
+BUILD=$1
+PREFIX=$2
+ARCH=${3:-x86_64}
+VERSION=${4:-0.3.0}
+
+SCCACHE_URL="https://github.com/mozilla/sccache/releases/download/v0.3.0/sccache-v$VERSION-$ARCH-$BUILD.tar.gz"
+SCCACHE_ARCHIVE=sccache.tar.gz
+
+# Download archive and checksum
+curl -L $SCCACHE_URL --output $SCCACHE_ARCHIVE
+curl -L $SCCACHE_URL.sha256 --output $SCCACHE_ARCHIVE.sha256
+
+echo "$(cat $SCCACHE_ARCHIVE.sha256) $SCCACHE_ARCHIVE" | sha256sum --check --status
+
+if [ ! -d $PREFIX ]; then
+ mkdir -p $PREFIX
+fi
+
+tar -xzvf $SCCACHE_ARCHIVE --strip-component=1 --directory $PREFIX --wildcards sccache*/sccache*
+chmod u+x $PREFIX/sccache
+
+if [ "$GITHUB_ACTIONS" ]; then
Review Comment:
```suggestion
if [ -n "$GITHUB_ACTIONS" ]; then
```
How about using `$GITHUB_PATH` or `$GITHUB_ENV` instead because `$GITHUB_ACTION` is for the action name not whether the current environment is GitHub Actions or not:
https://docs.github.com/en/actions/learn-github-actions/environment-variables#default-environment-variables
> GITHUB_ACTION | The name of the action currently running, or the id of a step. For example, for an action, __repo-owner_name-of-action-repo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1248395931
There is a lint error in R for which there already is a PR. I have fixed the cmake issues wit harchery.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1246527342
The failures above (except conan) are all tests failures, sccache works and is used:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1188557266
@github-actions crossbow submit r-binary-packages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1179255011
Revision: d751f944199cf6a5d8c53b8f0eff649e66420d69
Submitted crossbow builds: [ursacomputing/crossbow @ actions-15a1d9d2db](https://github.com/ursacomputing/crossbow/branches/all?query=actions-15a1d9d2db)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-15a1d9d2db-github-r-binary-packages)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
kou commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r917152318
##########
r/inst/build_arrow_static.sh:
##########
@@ -81,10 +96,14 @@ ${CMAKE} -DARROW_BOOST_USE_SHARED=OFF \
-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON \
-DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD:-OFF} \
-Dxsimd_SOURCE=${xsimd_SOURCE:-} \
+ -DARROW_USE_SCCACHE=ON \
${EXTRA_CMAKE_FLAGS} \
-G ${CMAKE_GENERATOR:-"Unix Makefiles"} \
${SOURCE_DIR}
${CMAKE} --build . --target install
+echo "=== sccache stats after the build ==="
+sccache --show-stats
Review Comment:
Could you show a build log that we can see this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1179258925
Revision: c6dc1f36aa916199627cf2520636af0ddcb48f1e
Submitted crossbow builds: [ursacomputing/crossbow @ actions-c356c46844](https://github.com/ursacomputing/crossbow/branches/all?query=actions-c356c46844)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c356c46844-github-r-binary-packages)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1179258134
@github-actions crossbow submit r-binary-packages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r925719565
##########
cpp/cmake_modules/DefineOptions.cmake:
##########
@@ -111,7 +111,10 @@ if(ARROW_DEFINE_OPTIONS)
define_option(ARROW_POSITION_INDEPENDENT_CODE
"Whether to create position-independent target" ON)
-
+
+ define_option(ARROW_USE_SCCACHE "Use sccache when compiling (if available), takes precdence\
+ over ccache" ON)
Review Comment:
I have added logic that only enables `sccache` if any of the storage back ends (including local) are explicitly specified, this way we can take advantage of default ON without worsening the DX.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r925058675
##########
cpp/cmake_modules/DefineOptions.cmake:
##########
@@ -111,7 +111,10 @@ if(ARROW_DEFINE_OPTIONS)
define_option(ARROW_POSITION_INDEPENDENT_CODE
"Whether to create position-independent target" ON)
-
+
+ define_option(ARROW_USE_SCCACHE "Use sccache when compiling (if available), takes precdence\
+ over ccache" ON)
Review Comment:
If there are no envvars set sccache will function in local mode similar to ccache https://github.com/mozilla/sccache#local so I would tend to leave it on as it allows us to just add the install script to the docker-compose command and the envvars to the github template to enable this on a lot of jobs, We could of course also switch it on in the build scripts but we would need to touch a bunch more files.
Alternatively we could check if a storage back end is configured and only use sccache over ccache in that case if both are installed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1189693396
Revision: d360bb141357ea153cc8ec68135969a450aa0113
Submitted crossbow builds: [ursacomputing/crossbow @ actions-5a2abfa2e4](https://github.com/ursacomputing/crossbow/branches/all?query=actions-5a2abfa2e4)
|Task|Status|
|----|------|
|conan-maximum|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-5a2abfa2e4-github-conan-maximum)|
|conan-minimum|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-5a2abfa2e4-github-conan-minimum)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r924256758
##########
dev/tasks/r/github.packages.yml:
##########
@@ -82,14 +82,24 @@ jobs:
{{ macros.github_checkout_arrow()|indent }}
{{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
{{ macros.github_install_archery()|indent }}
+
- name: Build libarrow
shell: bash
env:
- UBUNTU: {{ '${{ matrix.config.version}}' }}
+ SCCACHE_BUCKET: {{ '${{ secrets.SCCACHE_BUCKET }}' }}
+ SCCACHE_ID: {{ '${{ secrets.AWS_ACCESS_KEY_ID }}' }}
+ SCCACHE_KEY: {{' ${{ secrets.AWS_SECRET_ACCESS_KEY }}' }}
+ SCCACHE_PREFIX: sccache
Review Comment:
SCCACHE_KEY_PREFIX perhaps?
##########
dev/tasks/r/github.packages.yml:
##########
@@ -82,14 +82,24 @@ jobs:
{{ macros.github_checkout_arrow()|indent }}
{{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
{{ macros.github_install_archery()|indent }}
+
- name: Build libarrow
shell: bash
env:
- UBUNTU: {{ '${{ matrix.config.version}}' }}
+ SCCACHE_BUCKET: {{ '${{ secrets.SCCACHE_BUCKET }}' }}
+ SCCACHE_ID: {{ '${{ secrets.AWS_ACCESS_KEY_ID }}' }}
+ SCCACHE_KEY: {{' ${{ secrets.AWS_SECRET_ACCESS_KEY }}' }}
+ SCCACHE_PREFIX: sccache
Review Comment:
SCCACHE_S_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a diff in pull request #13556: ARROW-17021: [C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
kou commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r973663050
##########
ci/scripts/cpp_build.sh:
##########
@@ -181,6 +181,11 @@ if [ "${ARROW_USE_CCACHE}" == "ON" ]; then
ccache -s
fi
+if command -v sccache &> /dev/null; then
+ echo "=== sccache stats after the build ==="
+ sccache --show-stats
Review Comment:
@assignUser This causes an error on non-amd64 environment: https://app.travis-ci.com/github/apache/arrow/jobs/583028972#L3378
```text
+ sccache --show-stats
/arrow/ci/scripts/cpp_build.sh: line 186: /usr/local/bin/sccache: cannot execute binary file: Exec format error
ERROR: 126
```
Could you fix this as a follow-up task?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r969555094
##########
ci/scripts/PKGBUILD:
##########
@@ -73,6 +73,9 @@ build() {
# set the appropriate compiler definition.
export CPPFLAGS="-DUTF8PROC_STATIC"
+ # CMAKE_UNITY_BUILD is set to OFF as otherwise some compute functionality
Review Comment:
Added this comment to avoid a minor follow up PR from @paleolimbot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1245437487
Revision: 88066bcce1318201768fbfc5fce99cdfbfc0e880
Submitted crossbow builds: [ursacomputing/crossbow @ actions-bea2e1f5b6](https://github.com/ursacomputing/crossbow/branches/all?query=actions-bea2e1f5b6)
|Task|Status|
|----|------|
|conan-maximum|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-conan-maximum)|
|conan-minimum|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-conan-minimum)|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-test-conda-cpp)|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-test-conda-python-3.10)|
|test-conda-python-3.7|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-test-conda-python-3.7)|
|test-conda-python-3.7-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-test-conda-python-3.7-hdfs-2.9.2)|
|test-conda-python-3.7-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-test-conda-python-3.7-hdfs-3.2.1)|
|test-conda-python-3.7-kartothek-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-test-conda-python-3.7-kartothek-latest)|
|test-conda-python-3.7-kartothek-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-test-conda-python-3.7-kartothek-master)|
|test-conda-python-3.7-pandas-0.24|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-test-conda-python-3.7-pandas-0.24)|
|test-conda-python-3.7-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-test-conda-python-3.7-pandas-latest)|
|test-conda-python-3.7-spark-v3.1.2|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-test-conda-python-3.7-spark-v3.1.2)|
|test-conda-python-3.8|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-test-conda-python-3.8)|
|test-conda-python-3.8-hypothesis|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-test-conda-python-3.8-hypothesis)|
|test-conda-python-3.8-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-test-conda-python-3.8-pandas-latest)|
|test-conda-python-3.8-pandas-nightly|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-test-conda-python-3.8-pandas-nightly)|
|test-conda-python-3.8-spark-v3.2.0|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-test-conda-python-3.8-spark-v3.2.0)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-test-conda-python-3.9)|
|test-conda-python-3.9-dask-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-test-conda-python-3.9-dask-latest)|
|test-conda-python-3.9-dask-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-test-conda-python-3.9-dask-master)|
|test-conda-python-3.9-pandas-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-test-conda-python-3.9-pandas-master)|
|test-conda-python-3.9-spark-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-bea2e1f5b6-github-test-conda-python-3.9-spark-master)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1245732204
Revision: d408772a8e8cabad644dde5f321a6b28c5302672
Submitted crossbow builds: [ursacomputing/crossbow @ actions-c1c528ccc7](https://github.com/ursacomputing/crossbow/branches/all?query=actions-c1c528ccc7)
|Task|Status|
|----|------|
|conan-maximum|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-conan-maximum)|
|conan-minimum|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-conan-minimum)|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-test-conda-cpp)|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-test-conda-python-3.10)|
|test-conda-python-3.7|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-test-conda-python-3.7)|
|test-conda-python-3.7-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-test-conda-python-3.7-hdfs-2.9.2)|
|test-conda-python-3.7-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-test-conda-python-3.7-hdfs-3.2.1)|
|test-conda-python-3.7-kartothek-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-test-conda-python-3.7-kartothek-latest)|
|test-conda-python-3.7-kartothek-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-test-conda-python-3.7-kartothek-master)|
|test-conda-python-3.7-pandas-0.24|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-test-conda-python-3.7-pandas-0.24)|
|test-conda-python-3.7-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-test-conda-python-3.7-pandas-latest)|
|test-conda-python-3.7-spark-v3.1.2|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-test-conda-python-3.7-spark-v3.1.2)|
|test-conda-python-3.8|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-test-conda-python-3.8)|
|test-conda-python-3.8-hypothesis|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-test-conda-python-3.8-hypothesis)|
|test-conda-python-3.8-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-test-conda-python-3.8-pandas-latest)|
|test-conda-python-3.8-pandas-nightly|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-test-conda-python-3.8-pandas-nightly)|
|test-conda-python-3.8-spark-v3.2.0|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-test-conda-python-3.8-spark-v3.2.0)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-test-conda-python-3.9)|
|test-conda-python-3.9-dask-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-test-conda-python-3.9-dask-latest)|
|test-conda-python-3.9-dask-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-test-conda-python-3.9-dask-master)|
|test-conda-python-3.9-pandas-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-test-conda-python-3.9-pandas-master)|
|test-conda-python-3.9-spark-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-c1c528ccc7-github-test-conda-python-3.9-spark-master)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1179329368
@github-actions crossbow submit r-binary-packages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
kou commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r917139854
##########
cpp/cmake_modules/ThirdpartyToolchain.cmake:
##########
@@ -752,6 +752,14 @@ set(EP_COMMON_CMAKE_ARGS
-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=${CMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY}
-DCMAKE_VERBOSE_MAKEFILE=${CMAKE_VERBOSE_MAKEFILE})
+# Enable s/ccache if set by parent.
+if(CMAKE_CXX_COMPILER_LAUNCHER AND CMAKE_CXX_COMPILER_LAUNCHER)
+ set(EP_COMMON_CMAKE_ARGS
Review Comment:
Could you use `list(APPEND EP_COMMON_CMAKE_ARGS ...)`?
##########
dev/tasks/r/github.packages.yml:
##########
@@ -114,17 +124,38 @@ jobs:
- run: git config --global core.autocrlf false
{{ macros.github_checkout_arrow()|indent }}
{{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
-
- uses: r-lib/actions/setup-r@v2
with:
rtools-version: 40
r-version: "4.0"
Ncpus: 2
+ - name: Get sccache
+ shell: bash
+ run: |
+ SCCACHE_BUILD=pc-windows-msvc
+ SCCACHE_URL="https://github.com/mozilla/sccache/releases/download/v0.3.0/sccache-v0.3.0-x86_64-$SCCACHE_BUILD.tar.gz"
+
+ # Download archive and checksum
+ curl -L $SCCACHE_URL --output sccache.tar.gz
+ curl -L $SCCACHE_URL.sha256 --output sccache.tar.gz.sha256
+
+ SCCACHE_ARCHIVE=sccache.tar.gz
+ echo "$(cat $SCCACHE_ARCHIVE.sha256) $SCCACHE_ARCHIVE" | sha256sum --check --status
+
+ mkdir -p sccache
+ tar -xzvf $SCCACHE_ARCHIVE --strip-component=1 --directory sccache
+ echo "$(pwd)/sccache" >> $GITHUB_PATH
+ echo "SCCACHE_PATH=$(pwd)/sccache/sccache.exe" >> $GITHUB_ENV
- name: Build Arrow C++ with rtools40
shell: bash
env:
ARROW_HOME: "arrow"
+ AWS_SECRET_ACCESS_KEY: {{ '${{ secrets.AWS_SECRET_ACCESS_KEY }}' }}
+ AWS_ACCESS_KEY_ID: {{ '${{ secrets.AWS_ACCESS_KEY_ID }}' }}
+ SCCACHE_BUCKET: {{ '${{ secrets.SCCACHE_BUCKET }}' }}
+ SCCACHE_S3_KEY_PREFIX: 'sccache'
+ ARROW_USE_SCCACHE: ON
Review Comment:
Could you sort this list?
##########
cpp/cmake_modules/ThirdpartyToolchain.cmake:
##########
@@ -752,6 +752,14 @@ set(EP_COMMON_CMAKE_ARGS
-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=${CMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY}
-DCMAKE_VERBOSE_MAKEFILE=${CMAKE_VERBOSE_MAKEFILE})
+# Enable s/ccache if set by parent.
+if(CMAKE_CXX_COMPILER_LAUNCHER AND CMAKE_CXX_COMPILER_LAUNCHER)
+ set(EP_COMMON_CMAKE_ARGS
+ ${EP_COMMON_CMAKE_ARGS}
+ -DCMAKE_C_COMPILER_LAUNCHER=${CMAKE_C_COMPILER_LAUNCHER}
+ -DCMAKE_CXX_COMPILER_LAUNCHER=${CMAKE_CXX_COMPILER_LAUNCHER})
+endif()
Review Comment:
Wow! Can we split this as a separated pull request to merge only this without sscache change?
##########
cpp/CMakeLists.txt:
##########
@@ -215,18 +215,39 @@ else()
set(PYTHON_EXECUTABLE ${Python3_EXECUTABLE})
endif()
-if(ARROW_USE_CCACHE
- AND NOT CMAKE_C_COMPILER_LAUNCHER
- AND NOT CMAKE_CXX_COMPILER_LAUNCHER)
+if((ARROW_USE_SCCACHE)
+ AND NOT CMAKE_C_COMPILER_LAUNCHER
+ AND NOT CMAKE_CXX_COMPILER_LAUNCHER)
Review Comment:
```suggestion
AND NOT CMAKE_C_COMPILER_LAUNCHER
AND NOT CMAKE_CXX_COMPILER_LAUNCHER)
```
##########
cpp/CMakeLists.txt:
##########
@@ -215,18 +215,39 @@ else()
set(PYTHON_EXECUTABLE ${Python3_EXECUTABLE})
endif()
-if(ARROW_USE_CCACHE
- AND NOT CMAKE_C_COMPILER_LAUNCHER
- AND NOT CMAKE_CXX_COMPILER_LAUNCHER)
+if((ARROW_USE_SCCACHE)
+ AND NOT CMAKE_C_COMPILER_LAUNCHER
+ AND NOT CMAKE_CXX_COMPILER_LAUNCHER)
+
+ find_program(SCCACHE_FOUND sccache)
+
+ if(NOT SCCACHE_FOUND AND DEFINED ENV{SCCACHE_PATH})
+ # cmake has problems finding sccache from within mingw
Review Comment:
Can we remove this by adding `chmod +x sccache/sccache.exe`?
##########
ci/scripts/PKGBUILD:
##########
@@ -128,7 +129,7 @@ build() {
-DCMAKE_BUILD_TYPE="release" \
-DCMAKE_INSTALL_PREFIX=${MINGW_PREFIX} \
-DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD} \
- -DCMAKE_VERBOSE_MAKEFILE=ON
+ -DCMAKE_VERBOSE_MAKEFILE=ON
Review Comment:
```suggestion
-DCMAKE_VERBOSE_MAKEFILE=ON
```
##########
dev/tasks/r/github.packages.yml:
##########
@@ -82,14 +82,24 @@ jobs:
{{ macros.github_checkout_arrow()|indent }}
{{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
{{ macros.github_install_archery()|indent }}
+
- name: Build libarrow
shell: bash
env:
- UBUNTU: {{ '${{ matrix.config.version}}' }}
+ UBUNTU: {{ '${{ matrix.config.version }}' }}
+ SC_KEY: {{' ${{ secrets.AWS_SECRET_ACCESS_KEY }}' }}
+ SC_ID: {{ '${{ secrets.AWS_ACCESS_KEY_ID }}' }}
+ SC_BUCKET: {{ '${{ secrets.SCCACHE_BUCKET }}' }}
+ SC_PREFIX: sccache
Review Comment:
How about using `SCCACHE_` as prefix like others?
##########
dev/tasks/r/github.packages.yml:
##########
@@ -280,10 +319,33 @@ jobs:
with:
install-r: false
{{ macros.github_setup_local_r_repo(false, false)|indent }}
+ - name: Get sccache
+ shell: bash
+ run: |
+ SCCACHE_BUILD=unknown-linux-musl
+ SCCACHE_URL="https://github.com/mozilla/sccache/releases/download/v0.3.0/sccache-v0.3.0-x86_64-$SCCACHE_BUILD.tar.gz"
+
+ # Download archive and checksum
+ curl -L $SCCACHE_URL --output sccache.tar.gz
+ curl -L $SCCACHE_URL.sha256 --output sccache.tar.gz.sha256
+
+ SCCACHE_ARCHIVE=sccache.tar.gz
+ echo "$(cat $SCCACHE_ARCHIVE.sha256) $SCCACHE_ARCHIVE" | sha256sum --check --status
+
+ mkdir -p sccache
+ tar -xzvf $SCCACHE_ARCHIVE --strip-component=1 --directory sccache
Review Comment:
Do we need `chmod +x sscache/sscache`?
##########
r/inst/build_arrow_static.sh:
##########
@@ -81,10 +96,14 @@ ${CMAKE} -DARROW_BOOST_USE_SHARED=OFF \
-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON \
-DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD:-OFF} \
-Dxsimd_SOURCE=${xsimd_SOURCE:-} \
+ -DARROW_USE_SCCACHE=ON \
Review Comment:
Could you keep alphabetical order?
##########
dev/tasks/r/github.packages.yml:
##########
@@ -280,10 +319,33 @@ jobs:
with:
install-r: false
{{ macros.github_setup_local_r_repo(false, false)|indent }}
+ - name: Get sccache
+ shell: bash
+ run: |
+ SCCACHE_BUILD=unknown-linux-musl
+ SCCACHE_URL="https://github.com/mozilla/sccache/releases/download/v0.3.0/sccache-v0.3.0-x86_64-$SCCACHE_BUILD.tar.gz"
+
+ # Download archive and checksum
+ curl -L $SCCACHE_URL --output sccache.tar.gz
+ curl -L $SCCACHE_URL.sha256 --output sccache.tar.gz.sha256
+
+ SCCACHE_ARCHIVE=sccache.tar.gz
+ echo "$(cat $SCCACHE_ARCHIVE.sha256) $SCCACHE_ARCHIVE" | sha256sum --check --status
+
+ mkdir -p sccache
+ tar -xzvf $SCCACHE_ARCHIVE --strip-component=1 --directory sccache
+
+ echo "$(pwd)/sccache" >> $GITHUB_PATH
+ echo "SCCACHE_PATH=$(pwd)/sccache/sccache.exe" >> $GITHUB_ENV
Review Comment:
```suggestion
echo "SCCACHE_PATH=$(pwd)/sccache/sccache" >> $GITHUB_ENV
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1189673376
@github-actions crossbow submit r-binary-packages conan-*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on a diff in pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
assignUser commented on code in PR #13556:
URL: https://github.com/apache/arrow/pull/13556#discussion_r972223827
##########
dev/tasks/docker-tests/github.linux.yml:
##########
@@ -31,6 +31,8 @@ jobs:
- name: Execute Docker Build
shell: bash
+ env:
+ {{ macros.github_set_sccache_envvars()|indent(8) }}
Review Comment:
Indentation with jinja is a bit hard to determine wihtout actually rendering the template in my experience. There is some trimming happening by default and than this is a nested macro that adds another layer of whitespace, I tried stripping manually but that also removes the new line after `env:` and messes everything up completely....
This template is used e.g. in the conan builds and renders correctly with this setting:
```yml
- name: Execute Docker Build
shell: bash
env:
AWS_SECRET_ACCESS_KEY: "${{ secrets.AWS_SECRET_ACCESS_KEY }}"
AWS_ACCESS_KEY_ID: "${{ secrets.AWS_ACCESS_KEY_ID }}"
SCCACHE_BUCKET: "${{ secrets.SCCACHE_BUCKET }}"
SCCACHE_S3_KEY_PREFIX: "sccache"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1248391397
Thanks for the reviews @kou @pitrou
I have addressed your comments either via code or comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou merged pull request #13556: ARROW-17021: [C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
kou merged PR #13556:
URL: https://github.com/apache/arrow/pull/13556
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1246889750
Sorry for the review request spam, the github frontend seems to be broken for me at the moment. @kou @pitrou could you have a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1246533869
Revision: 64403dbe70431b32c3b7e78c13d2702d421858f6
Submitted crossbow builds: [ursacomputing/crossbow @ actions-aaf8cea625](https://github.com/ursacomputing/crossbow/branches/all?query=actions-aaf8cea625)
|Task|Status|
|----|------|
|conan-maximum|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-aaf8cea625-github-conan-maximum)|
|conan-minimum|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-aaf8cea625-github-conan-minimum)|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-aaf8cea625-github-r-binary-packages)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
assignUser commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1246531837
@github-actions crossbow submit conan-* r-binary-packages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13556: ARROW-17021:[C++][R][CI] Enable use of sccache in crossbow WIP
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13556:
URL: https://github.com/apache/arrow/pull/13556#issuecomment-1245350347
Revision: a32d88a570ff306f5512c066ffad0e4008bb5341
Submitted crossbow builds: [ursacomputing/crossbow @ actions-4ce21a1e6e](https://github.com/ursacomputing/crossbow/branches/all?query=actions-4ce21a1e6e)
|Task|Status|
|----|------|
|conan-maximum|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-conan-maximum)|
|conan-minimum|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-conan-minimum)|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-r-binary-packages)|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-test-conda-cpp)|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-test-conda-python-3.10)|
|test-conda-python-3.7|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-test-conda-python-3.7)|
|test-conda-python-3.7-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-test-conda-python-3.7-hdfs-2.9.2)|
|test-conda-python-3.7-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-test-conda-python-3.7-hdfs-3.2.1)|
|test-conda-python-3.7-kartothek-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-test-conda-python-3.7-kartothek-latest)|
|test-conda-python-3.7-kartothek-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-test-conda-python-3.7-kartothek-master)|
|test-conda-python-3.7-pandas-0.24|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-test-conda-python-3.7-pandas-0.24)|
|test-conda-python-3.7-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-test-conda-python-3.7-pandas-latest)|
|test-conda-python-3.7-spark-v3.1.2|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-test-conda-python-3.7-spark-v3.1.2)|
|test-conda-python-3.8|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-test-conda-python-3.8)|
|test-conda-python-3.8-hypothesis|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-test-conda-python-3.8-hypothesis)|
|test-conda-python-3.8-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-test-conda-python-3.8-pandas-latest)|
|test-conda-python-3.8-pandas-nightly|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-test-conda-python-3.8-pandas-nightly)|
|test-conda-python-3.8-spark-v3.2.0|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-test-conda-python-3.8-spark-v3.2.0)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-test-conda-python-3.9)|
|test-conda-python-3.9-dask-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-test-conda-python-3.9-dask-latest)|
|test-conda-python-3.9-dask-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-test-conda-python-3.9-dask-master)|
|test-conda-python-3.9-pandas-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-test-conda-python-3.9-pandas-master)|
|test-conda-python-3.9-spark-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-4ce21a1e6e-github-test-conda-python-3.9-spark-master)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org