You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/08/13 16:29:53 UTC
[GitHub] [arrow] kevingurney opened a new pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
kevingurney opened a new pull request #10932:
URL: https://github.com/apache/arrow/pull/10932
## Overview
This Pull Request:
1. Enables building of the MATLAB Interface C++ code, running of the C++ tests, and running of the MATLAB tests using GitHub Actions on an `ubuntu-latest` VM.
## Implementation
This implementation uses [`matlab-actions`](https://github.com/matlab-actions) to automatically install MATLAB into a GitHub Actions Linux VM.
We used the other `ci/scripts/<lang>_build.sh` scripts as inspiration for `matlab_build.sh`. Essentially, `matlab_build.sh` just invokes CMake, followed by CTest, to build and run the MATLAB Interface C++ source and tests. It also automatically builds the C++ Arrow libraries (i.e. `libarrow.so`) in the process. Support for automatically building `libarrow.so` as part of the MATLAB CMake build was added in https://github.com/apache/arrow/pull/10614.
We realize that many of the other Arrow language bindings use Docker for their CI workflows on Linux. However, [`matlab-actions/setup-matlab`](https://github.com/matlab-actions/setup-matlab) does not currently support installing MATLAB into Docker containers in GitHub Actions, so we used a "bare" Linux VM-based approach.
## Testing
1. The GitHub Actions workflows pass successfully with no errors.
2. All C++ test results and MATLAB test results can be viewed in the the workflow logs.
3. We made code changes that caused C++ and MATLAB test failures and were able to verify that the GitHub Actions workflow failed as a result.
## Future Directions
1. Enable support for macOS and Windows. [`matlab-actions/setup-matlab`](https://github.com/matlab-actions/setup-matlab) only supports Linux-based VMs in GitHub Actions right now. However, we have a strong desire to support all major platforms for CI in the future. Hopefully, Linux CI support should hold us over until we can add support for other platforms.
2. Enable Docker-based builds for greater control over dependencies and build environment. We also recognize the benefits of developers being able to run CI workflows in local Docker containers for pre-qualification. [`matlab-actions/setup-matlab`](https://github.com/matlab-actions/setup-matlab) does not currently support installing MATLAB into GitHub Actions hosted Docker containers.
3. Reduce overall build time by separating out building of the Arrow C++ libraries from building of the MATLAB interface. This would also enable us to avoid running all of the C++ library tests unnecessarily during the MATLAB build by building GoogleTest separately.
## Notes
This was a Team effort! Several MathWorkers played important roles in getting this working!
1. @sreeharihegden did the **vast majority** of the groundwork to get this working! He's away for a while, so @sgilmore10 and I took over in his absence. Thank you for all of your hard work, Sreehari! **I think @sreeharihegden's commits are not getting properly associated with his GitHub account. I would greatly appreciate it if whoever ends up integrating these changes after review can ensure that his name is properly added as a Co-author for this pull request. Thank you!**
2. @sgilmore10 worked directly with me to get this working! Thank you, Sarah!
3. @lafiona also helped me out a great deal to understand GitHub Actions and nail down the exact source of the build issues we were encountering. Thank you, Fiona!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #10932:
URL: https://github.com/apache/arrow/pull/10932#discussion_r690820325
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Fetch Submodules and Tags
+ shell: bash
+ run: ci/scripts/util_checkout.sh
+ - name: Install MATLAB
+ uses: matlab-actions/setup-matlab@v0
+ - name: Build MATLAB Interface
Review comment:
Do you want to work on `ccache` support in this pull request or a follow-up pull request?
If you choose a follow-up pull request, I'll merge this as-is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou closed pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
Posted by GitBox <gi...@apache.org>.
kou closed pull request #10932:
URL: https://github.com/apache/arrow/pull/10932
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] sgilmore10 commented on pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
Posted by GitBox <gi...@apache.org>.
sgilmore10 commented on pull request #10932:
URL: https://github.com/apache/arrow/pull/10932#issuecomment-899710519
@kou, Regarding your question about https://issues.apache.org/jira/browse/INFRA-22155, we do not believe this is a blocking issue. We are not publishing any artifacts in the ci workflow right now, so the lack of write access shouldn't cause any issues. However, if we want to publish test results in the future, then this issue will need to be addressed. To get ci working as soon as possible, we would rather split out the work of publishing test results in a future pull request.
Also, thank you for clearing up our confusion about why @sreeharihegden commits were not getting associated with his GitHub account. We'll let him know once he gets back.
Best,
Sarah
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kevingurney commented on pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
Posted by GitBox <gi...@apache.org>.
kevingurney commented on pull request #10932:
URL: https://github.com/apache/arrow/pull/10932#issuecomment-900448827
Thanks for all your feedback, @kou! We've updated the code based on your feedback. Let us know if you have any other questions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kevingurney commented on a change in pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
Posted by GitBox <gi...@apache.org>.
kevingurney commented on a change in pull request #10932:
URL: https://github.com/apache/arrow/pull/10932#discussion_r690600587
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Fetch Submodules and Tags
+ shell: bash
+ run: ci/scripts/util_checkout.sh
+ - name: Install MATLAB
+ uses: matlab-actions/setup-matlab@v0
+ - name: Build MATLAB Interface
Review comment:
My apologies!
I just realized I made a mistake when comparing the performance of the MATLAB CI build with and without `ccache`. I thought by simply omitting `-DARROW_USE_CCACHE=ON` that `ccache` wouldn't be used. However, I just realized that the default behavior is to use `ccache`, regardless of whether the flag is specified.
After comparing the build times again (this time explicitly setting `-DARROW_USE_CCACHE=OFF`), it seems like using `ccache` does give some fairly substantial performance gains when doing a clean build with a warm cache.
Sorry again for the confusion!
We'll look into properly integrating `ccache` in GitHub Actions by following the approach used by the other language binding [`.github/workflows/*.yml`](https://github.com/apache/arrow/blob/820e5061847c9d6d261c416e57d6013321175565/.github/workflows/cpp.yml#L301) files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10932:
URL: https://github.com/apache/arrow/pull/10932#issuecomment-898582560
https://issues.apache.org/jira/browse/ARROW-13202
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kevingurney commented on pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
Posted by GitBox <gi...@apache.org>.
kevingurney commented on pull request #10932:
URL: https://github.com/apache/arrow/pull/10932#issuecomment-900448827
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kevingurney commented on pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
Posted by GitBox <gi...@apache.org>.
kevingurney commented on pull request #10932:
URL: https://github.com/apache/arrow/pull/10932#issuecomment-901141943
The Travis CI failure appears unrelated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kevingurney commented on a change in pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
Posted by GitBox <gi...@apache.org>.
kevingurney commented on a change in pull request #10932:
URL: https://github.com/apache/arrow/pull/10932#discussion_r691260022
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Fetch Submodules and Tags
+ shell: bash
+ run: ci/scripts/util_checkout.sh
+ - name: Install MATLAB
+ uses: matlab-actions/setup-matlab@v0
+ - name: Build MATLAB Interface
Review comment:
In the spirit of incremental delivery, we think it makes sense to work on enabling `ccache` support in a follow-up pull request. We captured this work in: https://issues.apache.org/jira/browse/ARROW-13658. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kevingurney commented on pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
Posted by GitBox <gi...@apache.org>.
kevingurney commented on pull request #10932:
URL: https://github.com/apache/arrow/pull/10932#issuecomment-898640970
@kou - would you be able to review these changes? Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kevingurney commented on a change in pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
Posted by GitBox <gi...@apache.org>.
kevingurney commented on a change in pull request #10932:
URL: https://github.com/apache/arrow/pull/10932#discussion_r690412410
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+ pull_request:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
Review comment:
Thanks for the suggestion - this makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] sgilmore10 commented on a change in pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
Posted by GitBox <gi...@apache.org>.
sgilmore10 commented on a change in pull request #10932:
URL: https://github.com/apache/arrow/pull/10932#discussion_r690519323
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
Review comment:
done
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Fetch Submodules and Tags
+ shell: bash
Review comment:
fixed
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Fetch Submodules and Tags
+ shell: bash
+ run: ci/scripts/util_checkout.sh
+ - name: Install MATLAB
+ uses: matlab-actions/setup-matlab@v0
+ - name: Build MATLAB Interface
+ shell: bash
+ run: ci/scripts/matlab_build.sh $(pwd);
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kevingurney commented on a change in pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
Posted by GitBox <gi...@apache.org>.
kevingurney commented on a change in pull request #10932:
URL: https://github.com/apache/arrow/pull/10932#discussion_r690534937
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Fetch Submodules and Tags
+ shell: bash
+ run: ci/scripts/util_checkout.sh
+ - name: Install MATLAB
+ uses: matlab-actions/setup-matlab@v0
+ - name: Build MATLAB Interface
Review comment:
Thanks for suggesting that we use Ninja! This significantly speeds up local builds and has a noticeable impact on the CI time, as you've pointed out.
We investigated a number of other approaches to reducing the CI build time:
1. Using precompiled headers (`-D ARROW_USE_PRECOMPILED_HEADERS=ON`)
2. Using CMake Unity builds (`-D CMAKE_UNITY_BUILD=ON`)
3. Using ccache (`-D ARROW_USE_CCACHE=ON`)
4. Using `CMAKE_BUILD_PARALLEL_LEVEL`
5. Using the gold linker (`-D ARROW_USE_LD_GOLD=ON`)
Based on my ad-hoc testing, none of the options listed above appear to reduce the build times in any noticeable way. In fact, builds were slightly slower when using these options, in most cases.
Some of these approaches (in particular, using precompiled headers and using Unity/Jumbo builds) seem to be at odds with the parallel build strategies used by Ninja. In addition, Ninja seems to be automatically parallelizing over all available cores by default, so explicitly setting `CMAKE_BUILD_PARALLEL_LEVEL` doesn't seem to do much from my understanding. Using the gold linker had negligible impact from what we observed. Perhaps this is because there isn't a significant enough amount of linking going on here?
The `ccache` approach seems like it should theoretically be helpful when running clean builds with a warm cache. However, for some reason, we don't notice any reduction in build time when working with a warm cache (we observed a fairly high cache hit rate using `ccache -s`). It's possible we are misusing `ccache` in some way, but as far as we can tell, it doesn't appear to be giving any major performance gains for this use case.
I understand that the C++ CI build does use `ccache`, so if there is something we are missing here in terms of the performance impact, please let us know.
------------------------
Overall, using Ninja as the CMake generator was the most useful change.
As a side note - we realize now that we may be able to reduce the build time a bit more by building the Arrow C++ libraries separately. We could then `apt-get install` GoogleTest for running the MATLAB Interface C++ tests. Installing GoogleTest separately using the Ubuntu package repositories should prevent us from having to build and run all of the Arrow C++ tests as part of the MATLAB CI build. We've captured this work in [ARROW-13647](https://issues.apache.org/jira/browse/ARROW-13647).
If you have additional suggestions for reducing the CI time, we are happy to investigate them further. Thank you!
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Fetch Submodules and Tags
+ shell: bash
+ run: ci/scripts/util_checkout.sh
+ - name: Install MATLAB
+ uses: matlab-actions/setup-matlab@v0
+ - name: Build MATLAB Interface
Review comment:
My apologies!
I just realized I made a mistake when comparing the performance of the MATLAB CI build with and without `ccache`. I thought by simply omitting `-DARROW_USE_CCACHE=ON` that `ccache` wouldn't be used. However, I just realized that the default behavior is to use `ccache`, regardless of whether the flag is specified.
After comparing the build times again (this time explicitly setting `-DARROW_USE_CCACHE=OFF`), it seems like using `ccache` does give some fairly substantial performance gains when doing a clean build with a warm cache.
Sorry again for the confusion!
We'll look into properly integrating `ccache` in GitHub Actions by following the approach used by the other language binding [`.github/workflows/*.yml`](https://github.com/apache/arrow/blob/820e5061847c9d6d261c416e57d6013321175565/.github/workflows/cpp.yml#L301) files.
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Fetch Submodules and Tags
+ shell: bash
+ run: ci/scripts/util_checkout.sh
+ - name: Install MATLAB
+ uses: matlab-actions/setup-matlab@v0
+ - name: Build MATLAB Interface
Review comment:
In the spirit of incremental delivery, we think it makes sense to work on enabling `ccache` support in a follow-up pull request. We captured this work in: https://issues.apache.org/jira/browse/ARROW-13658. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] sgilmore10 commented on a change in pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
Posted by GitBox <gi...@apache.org>.
sgilmore10 commented on a change in pull request #10932:
URL: https://github.com/apache/arrow/pull/10932#discussion_r690519323
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
Review comment:
done
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Fetch Submodules and Tags
+ shell: bash
Review comment:
fixed
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Fetch Submodules and Tags
+ shell: bash
+ run: ci/scripts/util_checkout.sh
+ - name: Install MATLAB
+ uses: matlab-actions/setup-matlab@v0
+ - name: Build MATLAB Interface
+ shell: bash
+ run: ci/scripts/matlab_build.sh $(pwd);
Review comment:
done
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Fetch Submodules and Tags
+ shell: bash
+ run: ci/scripts/util_checkout.sh
+ - name: Install MATLAB
+ uses: matlab-actions/setup-matlab@v0
+ - name: Build MATLAB Interface
+ shell: bash
Review comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #10932:
URL: https://github.com/apache/arrow/pull/10932#discussion_r688734131
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Fetch Submodules and Tags
+ shell: bash
Review comment:
Could you remove redundant this line? (`bash` is the default on `ubuntu-latest`.)
Other `.yml`s doesn't specify this.
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Fetch Submodules and Tags
+ shell: bash
+ run: ci/scripts/util_checkout.sh
+ - name: Install MATLAB
+ uses: matlab-actions/setup-matlab@v0
+ - name: Build MATLAB Interface
+ shell: bash
+ run: ci/scripts/matlab_build.sh $(pwd);
Review comment:
```suggestion
run: ci/scripts/matlab_build.sh $(pwd)
```
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Fetch Submodules and Tags
+ shell: bash
+ run: ci/scripts/util_checkout.sh
+ - name: Install MATLAB
+ uses: matlab-actions/setup-matlab@v0
+ - name: Build MATLAB Interface
+ shell: bash
Review comment:
Ditto.
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
Review comment:
Could you also add `pull_request` entry to `on`?
So we can run this before merging a pull request.
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Fetch Submodules and Tags
+ shell: bash
+ run: ci/scripts/util_checkout.sh
+ - name: Install MATLAB
+ uses: matlab-actions/setup-matlab@v0
+ - name: Build MATLAB Interface
Review comment:
https://github.com/mathworks/arrow/runs/3323642808
It seems that this step takes 18m+ (mostly Apache Arrow C++ build).
Can we reduce the build time by using `ccache` and/or [`CMAKE_BUILD_PARALLEL_LEVEL`](https://cmake.org/cmake/help/latest/envvar/CMAKE_BUILD_PARALLEL_LEVEL.html) environment variable (or using `ninja` instead of `make`)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #10932:
URL: https://github.com/apache/arrow/pull/10932#discussion_r691572117
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Fetch Submodules and Tags
+ shell: bash
+ run: ci/scripts/util_checkout.sh
+ - name: Install MATLAB
+ uses: matlab-actions/setup-matlab@v0
+ - name: Build MATLAB Interface
Review comment:
OK.
I'll merge this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] sgilmore10 commented on a change in pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
Posted by GitBox <gi...@apache.org>.
sgilmore10 commented on a change in pull request #10932:
URL: https://github.com/apache/arrow/pull/10932#discussion_r690519706
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Fetch Submodules and Tags
+ shell: bash
+ run: ci/scripts/util_checkout.sh
+ - name: Install MATLAB
+ uses: matlab-actions/setup-matlab@v0
+ - name: Build MATLAB Interface
+ shell: bash
Review comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #10932:
URL: https://github.com/apache/arrow/pull/10932#discussion_r690820325
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Fetch Submodules and Tags
+ shell: bash
+ run: ci/scripts/util_checkout.sh
+ - name: Install MATLAB
+ uses: matlab-actions/setup-matlab@v0
+ - name: Build MATLAB Interface
Review comment:
Do you want to work on `ccache` support in this pull request or a follow-up pull request?
If you choose a follow-up pull request, I'll merge this as-is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kevingurney commented on a change in pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
Posted by GitBox <gi...@apache.org>.
kevingurney commented on a change in pull request #10932:
URL: https://github.com/apache/arrow/pull/10932#discussion_r690534937
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Fetch Submodules and Tags
+ shell: bash
+ run: ci/scripts/util_checkout.sh
+ - name: Install MATLAB
+ uses: matlab-actions/setup-matlab@v0
+ - name: Build MATLAB Interface
Review comment:
Thanks for suggesting that we use Ninja! This significantly speeds up local builds and has a noticeable impact on the CI time, as you've pointed out.
We investigated a number of other approaches to reducing the CI build time:
1. Using precompiled headers (`-D ARROW_USE_PRECOMPILED_HEADERS=ON`)
2. Using CMake Unity builds (`-D CMAKE_UNITY_BUILD=ON`)
3. Using ccache (`-D ARROW_USE_CCACHE=ON`)
4. Using `CMAKE_BUILD_PARALLEL_LEVEL`
5. Using the gold linker (`-D ARROW_USE_LD_GOLD=ON`)
Based on my ad-hoc testing, none of the options listed above appear to reduce the build times in any noticeable way. In fact, builds were slightly slower when using these options, in most cases.
Some of these approaches (in particular, using precompiled headers and using Unity/Jumbo builds) seem to be at odds with the parallel build strategies used by Ninja. In addition, Ninja seems to be automatically parallelizing over all available cores by default, so explicitly setting `CMAKE_BUILD_PARALLEL_LEVEL` doesn't seem to do much from my understanding. Using the gold linker had negligible impact from what we observed. Perhaps this is because there isn't a significant enough amount of linking going on here?
The `ccache` approach seems like it should theoretically be helpful when running clean builds with a warm cache. However, for some reason, we don't notice any reduction in build time when working with a warm cache (we observed a fairly high cache hit rate using `ccache -s`). It's possible we are misusing `ccache` in some way, but as far as we can tell, it doesn't appear to be giving any major performance gains for this use case.
I understand that the C++ CI build does use `ccache`, so if there is something we are missing here in terms of the performance impact, please let us know.
------------------------
Overall, using Ninja as the CMake generator was the most useful change.
As a side note - we realize now that we may be able to reduce the build time a bit more by building the Arrow C++ libraries separately. We could then `apt-get install` GoogleTest for running the MATLAB Interface C++ tests. Installing GoogleTest separately using the Ubuntu package repositories should prevent us from having to build and run all of the Arrow C++ tests as part of the MATLAB CI build. We've captured this work in [ARROW-13647](https://issues.apache.org/jira/browse/ARROW-13647).
If you have additional suggestions for reducing the CI time, we are happy to investigate them further. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #10932: ARROW-13202: [MATLAB] Enable GitHub Actions CI for MATLAB Interface on Linux
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #10932:
URL: https://github.com/apache/arrow/pull/10932#discussion_r689951809
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out repository
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Fetch Submodules and Tags
+ shell: bash
+ run: ci/scripts/util_checkout.sh
+ - name: Install MATLAB
+ uses: matlab-actions/setup-matlab@v0
+ - name: Build MATLAB Interface
Review comment:
18m+ -> 11m+ by Ninja: https://github.com/apache/arrow/pull/10932/checks?check_run_id=3343765575
##########
File path: .github/workflows/matlab.yml
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: MATLAB
+
+on:
+ push:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+ pull_request:
+ paths:
+ - '.github/workflows/matlab.yml'
+ - 'ci/scripts/matlab*.sh'
+ - 'matlab/**'
+ - 'cpp/src/arrow/**'
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+jobs:
+
+ matlab:
+ name: MATLAB
Review comment:
Could you use `${ARCHITECTURE} ${PLATFORM} ${LANGUAGE} ${NOTE}` format like other jobs? (See also other `.github/workflows/*.yml`.)
e.g.:
```suggestion
name: AMD64 Ubuntu 20.04 MATLAB
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org