You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by ra...@apache.org on 2022/12/21 11:09:10 UTC
[arrow-cookbook] branch main updated: [Dev] Run correct cookbook version based on branch and deploy development cookbooks and stable cookbooks (#284)
This is an automated email from the ASF dual-hosted git repository.
raulcd pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-cookbook.git
The following commit(s) were added to refs/heads/main by this push:
new e8ab575 [Dev] Run correct cookbook version based on branch and deploy development cookbooks and stable cookbooks (#284)
e8ab575 is described below
commit e8ab5755c5db1bd02ceda1310bca7a074579339d
Author: Raúl Cumplido <ra...@gmail.com>
AuthorDate: Wed Dec 21 12:09:05 2022 +0100
[Dev] Run correct cookbook version based on branch and deploy development cookbooks and stable cookbooks (#284)
* [Dev] Run correct cookbook version based on branch and deploy development cookbooks and stable cookbooks
* Update correct actions versions
* Renaming some jobs
* Update workflow file changes that trigger builds
* Update version for GitHub actions
* Fix needed job names
* Ignore if dev folder is not found when removing from git
* Use different index.html for development cookbooks
* Use Arrow nightlies to build the development version
* Update python version on conda dev environment to match stable
* Maintain dev cookbooks when deploying stable branch
* Fix deploying stable cookbooks to maintain dev cookbooks
* Try using a subfolder and backup dev
* Reduce workflows to use a single file both for main and stable
* Add new dependencies due to latest Arrow nightlies
* Update correct file for cache
* Simplify ARROW_NIGHTLY logic and add use-public-rspm as suggested on code review
* Add missing matomo on index_dev
* Add some documentation to the README around CI/Deployment
---
...kbooks.yml => deploy_development_cookbooks.yml} | 46 +++++++++-----
...y_cookbooks.yml => deploy_stable_cookbooks.yml} | 39 +++++++++---
.github/workflows/test_arrow_nightly_cookbook.yml | 15 ++++-
.github/workflows/test_cpp_cookbook.yml | 18 ++++--
...a_linux_cookbook.yml => test_java_cookbook.yml} | 41 +++++++++---
.github/workflows/test_java_osx_cookbook.yml | 55 -----------------
.github/workflows/test_python_cookbook.yml | 13 ++--
.github/workflows/test_r_cookbook.yml | 22 +++++--
CONTRIBUTING.md | 18 ++++++
README.rst | 5 +-
build/index_dev.html | 72 ++++++++++++++++++++++
cpp/dev.yml | 2 +-
12 files changed, 243 insertions(+), 103 deletions(-)
diff --git a/.github/workflows/deploy_cookbooks.yml b/.github/workflows/deploy_development_cookbooks.yml
similarity index 70%
copy from .github/workflows/deploy_cookbooks.yml
copy to .github/workflows/deploy_development_cookbooks.yml
index 952aade..7573809 100644
--- a/.github/workflows/deploy_cookbooks.yml
+++ b/.github/workflows/deploy_development_cookbooks.yml
@@ -3,35 +3,49 @@ on:
branches:
- main
-name: Deploy Cookbooks
+name: Deploy development Cookbooks
jobs:
- make_cookbooks:
+ make_dev_cookbooks:
name: build
runs-on: ubuntu-latest
+ env:
+ ARROW_NIGHTLY: 1
steps:
- uses: actions/checkout@v3
- name: Setup R
uses: r-lib/actions/setup-r@v2
+ with:
+ use-public-rspm: true
- name: Setup pandoc
uses: r-lib/actions/setup-pandoc@v2
- name: Install dependencies
run: |
sudo apt update
- sudo apt install libcurl4-openssl-dev libssl-dev python3-pip openjdk-11-jdk maven
+ sudo apt install libcurl4-openssl-dev \
+ libidn2-dev \
+ libkrb5-dev \
+ libldap-dev \
+ libnghttp2-dev \
+ libpsl-dev \
+ librtmp-dev \
+ libssh-dev \
+ libssh2-1-dev \
+ libssl-dev \
+ maven \
+ openjdk-11-jdk \
+ python3-pip
- name: Run tests
run: make test
- name: Build and render books
run: make all
- - name: Copy .asf.yaml
- run: cp .asf.yaml build/.asf.yaml
- name: Upload book artifact
uses: actions/upload-artifact@v3
with:
name: build_book
path: build/
- make_cpp:
+ make_dev_cpp:
name: build c++
runs-on: ubuntu-latest
defaults:
@@ -42,19 +56,19 @@ jobs:
- name: Cache conda
uses: actions/cache@v2
env:
- # Increase this value to reset cache if cpp/environment.yml has not changed
+ # Increase this value to reset cache if cpp/dev.yml has not changed
CACHE_NUMBER: 0
with:
path: ~/conda_pkgs_dir
key:
- ${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{ hashFiles('cpp/environment.yml') }}
+ ${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{ hashFiles('cpp/dev.yml') }}
- name: Setup environment
uses: conda-incubator/setup-miniconda@v2
with:
auto-update-conda: true
python-version: 3.9
- activate-environment: cookbook-cpp
- environment-file: cpp/environment.yml
+ activate-environment: cookbook-cpp-dev
+ environment-file: cpp/dev.yml
auto-activate-base: false
- name: Test
run:
@@ -68,10 +82,10 @@ jobs:
name: cpp_book
path: build/cpp
- deploy_cookbooks:
+ deploy_dev_cookbooks:
name: deploy
runs-on: ubuntu-latest
- needs: [make_cookbooks, make_cpp]
+ needs: [make_dev_cookbooks, make_dev_cpp]
steps:
- name: Checkout repo
uses: actions/checkout@v3
@@ -80,19 +94,21 @@ jobs:
- name: Prepare branch
run: |
git checkout --orphan gh-pages-tmp
- git rm -rf .
+ git rm -rf --ignore-unmatch ./dev
+ mkdir dev
- name: Download book artifact
uses: actions/download-artifact@v1.0.0
with:
name: build_book
- path: .
+ path: ./dev
- name: Download cpp book
uses: actions/download-artifact@v1.0.0
with:
name: cpp_book
- path: ./cpp
+ path: ./dev/cpp
- name: Push changes to gh-pages/asf-site branch
run: |
+ mv -f dev/index_dev.html dev/index.html
git config --global user.name 'GitHub Actions'
git config --global user.email 'actions@github.com'
git add --all
diff --git a/.github/workflows/deploy_cookbooks.yml b/.github/workflows/deploy_stable_cookbooks.yml
similarity index 75%
rename from .github/workflows/deploy_cookbooks.yml
rename to .github/workflows/deploy_stable_cookbooks.yml
index 952aade..be2c57f 100644
--- a/.github/workflows/deploy_cookbooks.yml
+++ b/.github/workflows/deploy_stable_cookbooks.yml
@@ -1,24 +1,38 @@
on:
push:
branches:
- - main
+ - stable
-name: Deploy Cookbooks
+name: Deploy Stable Cookbooks
jobs:
- make_cookbooks:
+ make_stable_cookbooks:
name: build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup R
uses: r-lib/actions/setup-r@v2
+ with:
+ use-public-rspm: true
- name: Setup pandoc
uses: r-lib/actions/setup-pandoc@v2
- name: Install dependencies
run: |
sudo apt update
- sudo apt install libcurl4-openssl-dev libssl-dev python3-pip openjdk-11-jdk maven
+ sudo apt install libcurl4-openssl-dev \
+ libidn2-dev \
+ libkrb5-dev \
+ libldap-dev \
+ libnghttp2-dev \
+ libpsl-dev \
+ librtmp-dev \
+ libssh-dev \
+ libssh2-1-dev \
+ libssl-dev \
+ maven \
+ openjdk-11-jdk \
+ python3-pip
- name: Run tests
run: make test
- name: Build and render books
@@ -31,7 +45,7 @@ jobs:
name: build_book
path: build/
- make_cpp:
+ make_stable_cpp:
name: build c++
runs-on: ubuntu-latest
defaults:
@@ -68,31 +82,38 @@ jobs:
name: cpp_book
path: build/cpp
- deploy_cookbooks:
+ deploy_stable_cookbooks:
name: deploy
runs-on: ubuntu-latest
- needs: [make_cookbooks, make_cpp]
+ needs: [make_stable_cookbooks, make_stable_cpp]
steps:
- name: Checkout repo
uses: actions/checkout@v3
with:
ref: gh-pages
+ path: cookbook
- name: Prepare branch
run: |
+ # Remove everything except dev cookbook
+ cp -R cookbook/dev .
+ cd cookbook
git checkout --orphan gh-pages-tmp
git rm -rf .
+ cp -R ../dev .
- name: Download book artifact
uses: actions/download-artifact@v1.0.0
with:
name: build_book
- path: .
+ path: ./cookbook
- name: Download cpp book
uses: actions/download-artifact@v1.0.0
with:
name: cpp_book
- path: ./cpp
+ path: ./cookbook/cpp
- name: Push changes to gh-pages/asf-site branch
run: |
+ cd cookbook
+ rm index_dev.html
git config --global user.name 'GitHub Actions'
git config --global user.email 'actions@github.com'
git add --all
diff --git a/.github/workflows/test_arrow_nightly_cookbook.yml b/.github/workflows/test_arrow_nightly_cookbook.yml
index ddb28b4..55ffe01 100644
--- a/.github/workflows/test_arrow_nightly_cookbook.yml
+++ b/.github/workflows/test_arrow_nightly_cookbook.yml
@@ -39,7 +39,16 @@ jobs:
- name: Install dependencies
run: |
sudo apt update
- sudo apt install libcurl4-openssl-dev libssl-dev
+ sudo apt install libcurl4-openssl-dev \
+ libidn2-dev \
+ libkrb5-dev \
+ libldap-dev \
+ libnghttp2-dev \
+ libpsl-dev \
+ librtmp-dev \
+ libssh-dev \
+ libssh2-1-dev \
+ libssl-dev
- name: Run tests
run: make rtest
- name: Build cookbooks
@@ -94,7 +103,7 @@ jobs:
strategy:
matrix:
python-version: [ '3.9' ]
- java-version: [ '18' ]
+ java-version: [ '11', '17', '18' ]
arrow_nightly: ['1']
env:
ARROW_NIGHTLY: ${{ matrix.arrow_nightly }}
@@ -118,7 +127,7 @@ jobs:
strategy:
matrix:
python-version: [ '3.9' ]
- java-version: [ '18' ]
+ java-version: [ '11', '17', '18' ]
arrow_nightly: ['1']
env:
ARROW_NIGHTLY: ${{ matrix.arrow_nightly }}
diff --git a/.github/workflows/test_cpp_cookbook.yml b/.github/workflows/test_cpp_cookbook.yml
index afcbe45..82f91e9 100644
--- a/.github/workflows/test_cpp_cookbook.yml
+++ b/.github/workflows/test_cpp_cookbook.yml
@@ -21,6 +21,7 @@ on:
pull_request:
branches:
- main
+ - stable
paths:
- "cpp/**"
- ".github/workflows/test_cpp_cookbook.yml"
@@ -30,8 +31,8 @@ concurrency:
cancel-in-progress: true
jobs:
- test_cpp_stable:
- name: "Test C++ Cookbook on Arrow stable"
+ test_cpp:
+ name: "Test C++ Cookbook on Arrow"
runs-on: ubuntu-latest
defaults:
run:
@@ -47,12 +48,21 @@ jobs:
path: ~/conda_pkgs_dir
key:
${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{ hashFiles('cpp/conda-linux-64.lock') }}
- - name: Setup environment
+ - name: Setup latest environment
+ if: github.event.pull_request.base.ref == 'main'
+ uses: conda-incubator/setup-miniconda@v2
+ with:
+ auto-update-conda: true
+ activate-environment: cookbook-cpp-dev
+ environment-file: cpp/dev.yml
+ auto-activate-base: false
+ - name: Setup stable environment
+ if: github.event.pull_request.base.ref == 'stable'
uses: conda-incubator/setup-miniconda@v2
with:
auto-update-conda: true
activate-environment: cookbook-cpp
- environment-file: cpp/conda-linux-64.lock
+ environment-file: cpp/environment.yml
auto-activate-base: false
- name: Test
run:
diff --git a/.github/workflows/test_java_linux_cookbook.yml b/.github/workflows/test_java_cookbook.yml
similarity index 64%
rename from .github/workflows/test_java_linux_cookbook.yml
rename to .github/workflows/test_java_cookbook.yml
index 4eb8eed..fb7f2b9 100644
--- a/.github/workflows/test_java_linux_cookbook.yml
+++ b/.github/workflows/test_java_cookbook.yml
@@ -15,15 +15,16 @@
# specific language governing permissions and limitations
# under the License.
-name: Test Java Cookbook On Linux
+name: Test Java Cookbook
on:
pull_request:
branches:
- main
+ - stable
paths:
- "java/**"
- - ".github/workflows/test_java_linux_cookbook.yml"
+ - ".github/workflows/test_java_cookbook.yml"
concurrency:
group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
@@ -33,13 +34,12 @@ jobs:
test_java_linux:
name: "Test Java Cookbook On Linux"
runs-on: ubuntu-latest
+ env:
+ ARROW_NIGHTLY: ${{ github.base_ref == 'main' && 1 || 0 }}
strategy:
matrix:
python-version: [ '3.9' ]
java-version: [ '11', '17', '18' ]
- arrow_nightly: ['0']
- env:
- ARROW_NIGHTLY: ${{ matrix.arrow_nightly }}
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v3
@@ -54,6 +54,33 @@ jobs:
sudo apt update
sudo apt install libcurl4-openssl-dev libssl-dev
- name: Run tests
- run: make javatest
+ run: |
+ make javatest
- name: Build cookbook
- run: make java
+ run: |
+ make java
+
+ test_java_osx:
+ name: "Test Java Cookbook on MacOS"
+ runs-on: macos-latest
+ env:
+ ARROW_NIGHTLY: ${{ github.base_ref == 'main' && 1 || 0 }}
+ strategy:
+ matrix:
+ python-version: [ '3.9' ]
+ java-version: [ '11', '17', '18' ]
+ steps:
+ - uses: actions/checkout@v3
+ - uses: actions/setup-python@v3
+ with:
+ python-version: ${{ matrix.python-version }}
+ - uses: actions/setup-java@v3
+ with:
+ distribution: 'temurin'
+ java-version: ${{ matrix.java-version }}
+ - name: Run tests
+ run: |
+ make javatest
+ - name: Build cookbook
+ run: |
+ make java
diff --git a/.github/workflows/test_java_osx_cookbook.yml b/.github/workflows/test_java_osx_cookbook.yml
deleted file mode 100644
index dc4b112..0000000
--- a/.github/workflows/test_java_osx_cookbook.yml
+++ /dev/null
@@ -1,55 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-name: Test Java Cookbook on MacOS
-
-on:
- pull_request:
- branches:
- - main
- paths:
- - "java/**"
- - ".github/workflows/test_java_osx_cookbook.yml"
-
-concurrency:
- group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }}
- cancel-in-progress: true
-
-jobs:
- test_java_osx:
- name: "Test Java Cookbook on MacOS"
- runs-on: macos-latest
- strategy:
- matrix:
- python-version: [ '3.9' ]
- java-version: [ '11', '17', '18' ]
- arrow_nightly: ['0']
- env:
- ARROW_NIGHTLY: ${{ matrix.arrow_nightly }}
- steps:
- - uses: actions/checkout@v3
- - uses: actions/setup-python@v3
- with:
- python-version: ${{ matrix.python-version }}
- - uses: actions/setup-java@v3
- with:
- distribution: 'temurin'
- java-version: ${{ matrix.java-version }}
- - name: Run tests
- run: make javatest
- - name: Build cookbook
- run: make java
diff --git a/.github/workflows/test_python_cookbook.yml b/.github/workflows/test_python_cookbook.yml
index 0b50734..fb8831b 100644
--- a/.github/workflows/test_python_cookbook.yml
+++ b/.github/workflows/test_python_cookbook.yml
@@ -21,6 +21,7 @@ on:
pull_request:
branches:
- main
+ - stable
paths:
- "python/**"
- ".github/workflows/test_python_cookbook.yml"
@@ -30,9 +31,11 @@ concurrency:
cancel-in-progress: true
jobs:
- test_py_stable:
- name: "Test Python Cookbook on Arrow stable"
+ test_py:
+ name: "Test Python Cookbook on Arrow"
runs-on: ubuntu-latest
+ env:
+ ARROW_NIGHTLY: ${{ github.base_ref == 'main' && 1 || 0 }}
steps:
- uses: actions/checkout@v3
- name: Install dependencies
@@ -40,6 +43,8 @@ jobs:
sudo apt update
sudo apt install libcurl4-openssl-dev libssl-dev python3-pip
- name: Run tests
- run: make pytest
+ run: |
+ make pytest
- name: Build cookbook
- run: make py
+ run: |
+ make py
diff --git a/.github/workflows/test_r_cookbook.yml b/.github/workflows/test_r_cookbook.yml
index ed975a2..2a210c8 100644
--- a/.github/workflows/test_r_cookbook.yml
+++ b/.github/workflows/test_r_cookbook.yml
@@ -21,6 +21,7 @@ on:
pull_request:
branches:
- main
+ - stable
paths:
- "r/**"
- ".github/workflows/test_r_cookbook.yml"
@@ -31,8 +32,10 @@ concurrency:
jobs:
test_r:
- name: "Test R Cookbook on Arrow stable"
+ name: "Test R Cookbook on Arrow"
runs-on: ubuntu-latest
+ env:
+ ARROW_NIGHTLY: ${{ github.base_ref == 'main' && 1 || 0 }}
steps:
- uses: actions/checkout@v3
- name: Setup R
@@ -44,8 +47,19 @@ jobs:
- name: Install dependencies
run: |
sudo apt update
- sudo apt install libcurl4-openssl-dev libssl-dev
+ sudo apt install libcurl4-openssl-dev \
+ libidn2-dev \
+ libkrb5-dev \
+ libldap-dev \
+ libnghttp2-dev \
+ libpsl-dev \
+ librtmp-dev \
+ libssh-dev \
+ libssh2-1-dev \
+ libssl-dev
- name: Run tests
- run: make rtest
+ run: |
+ make rtest
- name: Build cookbooks
- run: make r
+ run: |
+ make r
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index c9cbef7..84ac134 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -60,3 +60,21 @@ conduct](https://www.apache.org/foundation/policies/conduct.html).
We use the GitHub UI to merge Pull Requests instead of a script as we do on the
main Arrow repository. The repo is configured to use `Squash and merge`.
+
+## Deployment and CI
+
+We use GitHub actions for our Continuous Integration tests and to deploy the
+versions of the cookbooks.
+
+A Pull Request with base `main` will run the cookbooks using the nightly version
+of Apache Arrow.
+
+A Pull Request with base `stable` will run the cookbooks using the latest stable
+version released of Apache Arrow.
+
+Commits merged on `main` will deploy the development cookbooks using the latest
+Apache Arrow nightlies version to: https://arrow.apache.org/cookbook/dev
+
+Commits merge on `stable` will deploy the stable version of the cookbooks
+using the latest released version of Apache Arrow to build the cookbooks:
+https://arrow.apache.org/cookbook
diff --git a/README.rst b/README.rst
index 7c16b36..44db6dd 100644
--- a/README.rst
+++ b/README.rst
@@ -40,7 +40,10 @@ specific concerns and makes it possible to build/test all cookbooks
without any platform specific knowledge (as long as dependencies
are available on the target system).
-See https://arrow.apache.org/cookbook/ for the latest published version.
+See https://arrow.apache.org/cookbook/ for the latest published version using the
+latest stable version of Apache Arrow.
+See https://arrow.apache.org/cookbook/dev for the latest published version using
+the development version of Apache Arrow.
Building All Cookbooks
----------------------
diff --git a/build/index_dev.html b/build/index_dev.html
new file mode 100644
index 0000000..7b1c07c
--- /dev/null
+++ b/build/index_dev.html
@@ -0,0 +1,72 @@
+<!DOCTYPE html>
+
+<html>
+ <head>
+ <title>Apache Arrow Cookbook for Arrow Nightlies version</title>
+ <style>
+ body {
+ color: rgb(51, 51, 51);
+ font-family: sans-serif;
+ line-height: 1.65;
+ padding: 25px;
+ max-width: 900px;
+ margin-left: auto;
+ margin-right: auto;
+ }
+
+ a {
+ color: rgb(0, 91, 129);
+ }
+
+ #logo {
+ width: 50%;
+ margin-left: auto;
+ margin-right: auto;
+ }
+
+ #logo > img {
+ width: 100%;
+ }
+ </style>
+ <!-- Matomo -->
+ <script>
+ var _paq = window._paq = window._paq || [];
+ /* tracker methods like "setCustomDimension" should be called before "trackPageView" */
+ /* We explicitly disable cookie tracking to avoid privacy issues */
+ _paq.push(['disableCookies']);
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="https://analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '20']);
+ var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+ <!-- End Matomo Code -->
+ </head>
+ <body>
+ <div id="logo"><img src="arrow.png"/></div>
+ <h1>Apache Arrow Cookbook for Arrow Nightlies</h1>
+ <p>The cookbook is a collection of Apache Arrow recipes for
+ the languages and platforms supported by Arrow.<br/>
+ Most recipes will be common to all platforms,
+ but some are specific to the language and environment in use.<br/>
+ This uses the Apache Arrow nightlies version. Click on the link
+ if you want to use the latest
+ <a href="../index.html">stable Apache Arrow Cookbook</a>.
+ </p>
+ <ul>
+ <li><a href="cpp/index.html">C++ Cookbook</a></li>
+ <li><a href="java/index.html">Java Cookbook</a></li>
+ <li><a href="py/index.html">Python Cookbook</a></li>
+ <li><a href="r/index.html">R Cookbook</a></li>
+ <li><a href="https://github.com/apache/arrow-rs/tree/master/arrow/examples">Rust Examples</a></li>
+ </ul>
+ <p>If you are looking for the Apache Arrow Documentation itself
+ or the API reference, those are available at
+ <a href="https://arrow.apache.org/docs/">https://arrow.apache.org/docs/</a>
+ </p>
+ </body>
+</html>
diff --git a/cpp/dev.yml b/cpp/dev.yml
index e97fe1b..e3ca34d 100644
--- a/cpp/dev.yml
+++ b/cpp/dev.yml
@@ -19,7 +19,7 @@ channels:
- arrow-nightlies
- conda-forge
dependencies:
- - python=3.8
+ - python=3.9
- compilers
- arrow-nightlies::arrow-cpp
- sphinx