You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by ra...@apache.org on 2023/12/04 09:42:22 UTC
(arrow) branch maint-14.0.x updated (7f6a81317b -> 1fe7cd2b17)
This is an automated email from the ASF dual-hosted git repository.
raulcd pushed a change to branch maint-14.0.x
in repository https://gitbox.apache.org/repos/asf/arrow.git
from 7f6a81317b GH-38864: [R] Update NEWS.md for 14.0.0.1 (#38866)
new a9797184ca GH-38779: [R][CI] Use devtools on self-hosted machines and use macos-11 for intel package build (#38974)
new ef2f46b025 GH-38984: [Python][Packaging] Verification of wheels on AlmaLinux 8 are failing due to missing pip (#38985)
new 8f50fe52bd GH-38342: [Python] Update to_pandas to use non-deprecated DataFrame constructor (#38374)
new e825485085 GH-38902: [R] Handle failing library detection with pkg-config (#38970)
new 1fe7cd2b17 GH-38904:[R] Update news.md for 14.0.0.2 (#39022)
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
dev/release/setup-rhel-rebuilds.sh | 4 +---
dev/tasks/r/github.packages.yml | 27 ++++++++++++++------------
dev/tasks/tasks.yml | 6 +++---
python/pyarrow/pandas-shim.pxi | 11 ++++++++---
python/pyarrow/pandas_compat.py | 13 ++++++++++---
python/pyarrow/table.pxi | 6 +++---
r/NEWS.md | 19 +++++++++++++++++++
r/configure | 39 ++++++++++++++++++++++----------------
r/tools/nixlibs.R | 2 +-
r/tools/test-nixlibs.R | 4 ++--
10 files changed, 85 insertions(+), 46 deletions(-)
(arrow) 05/05: GH-38904:[R] Update news.md for 14.0.0.2 (#39022)
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
raulcd pushed a commit to branch maint-14.0.x
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit 1fe7cd2b172d82d8b5adb4021af93d87dded39a9
Author: Jacob Wujciak-Jens <ja...@wujciak.de>
AuthorDate: Fri Dec 1 19:29:34 2023 +0100
GH-38904:[R] Update news.md for 14.0.0.2 (#39022)
### Rationale for this change
Update news.md
### Are these changes tested?
no
* Closes: #38904
Authored-by: Jacob Wujciak-Jens <ja...@wujciak.de>
Signed-off-by: Jacob Wujciak-Jens <ja...@wujciak.de>
---
r/NEWS.md | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/r/NEWS.md b/r/NEWS.md
index 2e4b519b0d..63a861d54e 100644
--- a/r/NEWS.md
+++ b/r/NEWS.md
@@ -19,6 +19,25 @@
# arrow 14.0.1
# arrow 14.0.0.9000
+# arrow 14.0.1.9000
+
+# arrow 14.0.0.2
+
+## Minor improvements and fixes
+
+* Fixed the printf syntax to align with format checking (#38894)
+* Removed bashism in configure script (#38716).
+* Fixed a broken link in the README (#38657)
+* Properly escape the license header in the lintr config (#38639).
+* Removed spurious warnings from installation-script test suite (#38571).
+* Polished installation-script after refactor (#38534)
+
+## Installation
+
+* If pkg-config fails to detect the required libraries an additional search
+ without pkg-config is run (#38970).
+* Fetch the latest nightly Arrow C++ binary when installing a development
+ Version (#38236).
# arrow 14.0.0.1
(arrow) 04/05: GH-38902: [R] Handle failing library detection with pkg-config (#38970)
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
raulcd pushed a commit to branch maint-14.0.x
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit e825485085cbeef4b26baefe54096a47736a445b
Author: Jacob Wujciak-Jens <ja...@wujciak.de>
AuthorDate: Fri Dec 1 19:07:13 2023 +0100
GH-38902: [R] Handle failing library detection with pkg-config (#38970)
### Rationale for this change
We can get into a broken state with a working test compile in `nixlibs.R` but empty `PKG_LIBS` when pkg-config fails to find some libraries (e.g. libcurl on mac due to missing system stubs) in `configure`. This leads to a failed test compile in configure with pc errors silenced.
### What changes are included in this PR?
Catch this and rerun the pkg-config-less library detection that should fix this in most cases.
### Are these changes tested?
locally and on cran (where this error first surfaced)
* Closes: #38902
Lead-authored-by: Jacob Wujciak-Jens <ja...@wujciak.de>
Co-authored-by: Jonathan Keane <jk...@gmail.com>
Signed-off-by: Jacob Wujciak-Jens <ja...@wujciak.de>
---
r/configure | 39 +++++++++++++++++++++++----------------
r/tools/nixlibs.R | 2 +-
r/tools/test-nixlibs.R | 4 ++--
3 files changed, 26 insertions(+), 19 deletions(-)
diff --git a/r/configure b/r/configure
index e48bd2f010..96238f0b9a 100755
--- a/r/configure
+++ b/r/configure
@@ -79,9 +79,6 @@ VERSION=`grep '^Version' DESCRIPTION | sed s/Version:\ //`
UNAME=`uname -s`
: ${PKG_CONFIG:="pkg-config"}
-# These will only be set in the bundled build
-S3_LIBS=""
-GCS_LIBS=""
# If in development mode, run the codegen script to render arrowExports.*
if [ "$ARROW_R_DEV" = "true" ] && [ -f "data-raw/codegen.R" ]; then
@@ -116,7 +113,9 @@ fi
# Test if pkg-config is available to use
if ${PKG_CONFIG} --version >/dev/null 2>&1; then
PKG_CONFIG_AVAILABLE="true"
+ echo "*** pkg-config found."
else
+ echo "*** pkg-config not found."
PKG_CONFIG_AVAILABLE="false"
ARROW_USE_PKG_CONFIG="false"
fi
@@ -245,12 +244,6 @@ do_bundled_build () {
${LIB_DIR}/pkgconfig/*.pc
rm -f ${LIB_DIR}/pkgconfig/*.pc.bak
fi
- else
- # This case must be ARROW_DEPENDENCY_SOURCE=BUNDLED.
- # These would be identified by pkg-config, in Requires.private and Libs.private.
- # Rather than try to re-implement pkg-config, we can just hard-code them here.
- S3_LIBS="-lcurl -lssl -lcrypto"
- GCS_LIBS="-lcurl -lssl -lcrypto"
fi
else
# If the library directory does not exist, the script must not have been successful
@@ -293,15 +286,15 @@ set_pkg_vars () {
# If we have pkg-config, it will tell us what libarrow needs
set_lib_dir_with_pc () {
- LIB_DIR="`${PKG_CONFIG} --variable=libdir --silence-errors ${PKG_CONFIG_NAME}`"
+ LIB_DIR="`${PKG_CONFIG} --variable=libdir ${PKG_CONFIG_NAME}`"
}
set_pkg_vars_with_pc () {
pkg_config_names="${PKG_CONFIG_NAME} ${PKG_CONFIG_NAMES_FEATURES}"
- PKG_CFLAGS="`${PKG_CONFIG} --cflags --silence-errors ${pkg_config_names}` $PKG_CFLAGS"
+ PKG_CFLAGS="`${PKG_CONFIG} --cflags ${pkg_config_names}` $PKG_CFLAGS"
PKG_CFLAGS="$PKG_CFLAGS $PKG_CFLAGS_FEATURES"
- PKG_LIBS=`${PKG_CONFIG} --libs-only-l --libs-only-other --silence-errors ${pkg_config_names}`
+ PKG_LIBS=`${PKG_CONFIG} --libs-only-l --libs-only-other ${pkg_config_names}`
PKG_LIBS="$PKG_LIBS $PKG_LIBS_FEATURES"
- PKG_DIRS=`${PKG_CONFIG} --libs-only-L --silence-errors ${pkg_config_names}`
+ PKG_DIRS=`${PKG_CONFIG} --libs-only-L ${pkg_config_names}`
}
# If we don't have pkg-config, we can make some inferences
@@ -322,7 +315,7 @@ set_pkg_vars_without_pc () {
if [ -n "$(find "$LIB_DIR" -name 'libarrow_bundled_dependencies.*')" ]; then
PKG_LIBS="$PKG_LIBS -larrow_bundled_dependencies"
fi
- PKG_LIBS="$PKG_LIBS $PKG_LIBS_FEATURES"
+ PKG_LIBS="$PKG_LIBS $PKG_LIBS_FEATURES $SSL_LIBS_WITHOUT_PC"
# If on Raspberry Pi, need to manually link against latomic
# See: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81358 for similar example
@@ -380,11 +373,13 @@ add_feature_flags () {
fi
if arrow_built_with ARROW_S3; then
PKG_CFLAGS_FEATURES="$PKG_CFLAGS_FEATURES -DARROW_R_WITH_S3"
- PKG_LIBS_FEATURES="$PKG_LIBS_FEATURES $S3_LIBS"
fi
if arrow_built_with ARROW_GCS; then
PKG_CFLAGS_FEATURES="$PKG_CFLAGS_FEATURES -DARROW_R_WITH_GCS"
- PKG_LIBS_FEATURES="$PKG_LIBS_FEATURES $GCS_LIBS"
+ fi
+ if arrow_built_with ARROW_GCS || arrow_built_with ARROW_S3; then
+ # If pkg-config is available it will handle this for us automatically
+ SSL_LIBS_WITHOUT_PC="-lcurl -lssl -lcrypto"
fi
fi
}
@@ -405,6 +400,18 @@ find_or_build_libarrow
# Now set `PKG_LIBS`, `PKG_DIRS`, and `PKG_CFLAGS` based on that.
if [ "$_LIBARROW_FOUND" != "false" ] && [ "$_LIBARROW_FOUND" != "" ]; then
set_pkg_vars ${_LIBARROW_FOUND}
+
+ # If we didn't find any libraries with pkg-config, try again without pkg-config
+ FOUND_PKG_LIBS=`echo "$PKG_LIBS" | tr -d '[:space:]'`
+ if [ -z "$FOUND_PKG_LIBS" ] && [ "$PKG_CONFIG_AVAILABLE" = "true" ]; then
+ echo "*** pkg-config failed to find libraries. Running detection without pkg-config."
+ PKG_CONFIG_AVAILABLE="false"
+ set_pkg_vars ${_LIBARROW_FOUND}
+ fi
+else
+ # To make it easier to debug which code path was taken add a specific
+ # message to the log in addition to the 'NOTE'
+ echo "*** Failed to find Arrow C++ libraries."
fi
# Test that we can compile something with those flags
diff --git a/r/tools/nixlibs.R b/r/tools/nixlibs.R
index b003e7cea8..161cea3abd 100644
--- a/r/tools/nixlibs.R
+++ b/r/tools/nixlibs.R
@@ -72,7 +72,7 @@ find_latest_nightly <- function(description_version,
lg("Failed to find latest nightly for %s", description_version)
latest <- description_version
} else {
- lg("Found latest nightly for %s: %s", description_version, res)
+ lg("Latest available nightly for %s: %s", description_version, res)
latest <- res
}
latest
diff --git a/r/tools/test-nixlibs.R b/r/tools/test-nixlibs.R
index f97a80ccc2..ed5192d806 100644
--- a/r/tools/test-nixlibs.R
+++ b/r/tools/test-nixlibs.R
@@ -176,7 +176,7 @@ test_that("find_latest_nightly()", {
find_latest_nightly(package_version("13.0.1.9000"), list_uri = tf_uri),
package_version("13.0.0.100000335")
),
- "Found latest nightly"
+ "Latest available nightly"
)
expect_output(
@@ -184,7 +184,7 @@ test_that("find_latest_nightly()", {
find_latest_nightly(package_version("14.0.0.9000"), list_uri = tf_uri),
package_version("14.0.0.100000001")
),
- "Found latest nightly"
+ "Latest available nightly"
)
expect_output(
(arrow) 01/05: GH-38779: [R][CI] Use devtools on self-hosted machines and use macos-11 for intel package build (#38974)
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
raulcd pushed a commit to branch maint-14.0.x
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit a9797184ca9182cdbb2354cfb0a07bf990fb658a
Author: Jacob Wujciak-Jens <ja...@wujciak.de>
AuthorDate: Fri Dec 1 00:08:24 2023 +0100
GH-38779: [R][CI] Use devtools on self-hosted machines and use macos-11 for intel package build (#38974)
### Rationale for this change
The action does not work smoothly on the self-hosted runners.
### What changes are included in this PR?
Use devtools instead.
### Are these changes tested?
crossbow
* Closes: #38779
Authored-by: Jacob Wujciak-Jens <ja...@wujciak.de>
Signed-off-by: Jacob Wujciak-Jens <ja...@wujciak.de>
---
dev/tasks/r/github.packages.yml | 27 +++++++++++++++------------
dev/tasks/tasks.yml | 6 +++---
2 files changed, 18 insertions(+), 15 deletions(-)
diff --git a/dev/tasks/r/github.packages.yml b/dev/tasks/r/github.packages.yml
index 22dddabc10..2ddfd02e73 100644
--- a/dev/tasks/r/github.packages.yml
+++ b/dev/tasks/r/github.packages.yml
@@ -211,40 +211,43 @@ jobs:
matrix:
platform:
- { runs_on: 'windows-latest', name: "Windows"}
- - { runs_on: ["self-hosted", "macos-10.13"], name: "macOS High Sierra"}
- - { runs_on: ["self-hosted", "macOS", "arm64", "devops-managed"], name: "macOS Big Sur" }
- r_version:
- - { rtools: "{{ macros.r_release.rt }}", r: "{{ macros.r_release.ver }}" }
- - { rtools: "{{ macros.r_oldrel.rt }}", r: "{{ macros.r_oldrel.ver }}" }
+ - { runs_on: macos-11 , name: "macOS Big Sur"}
+ - { runs_on: ["self-hosted", "macOS", "arm64", "devops-managed"], name: "macOS Big Sur (M1)" }
+ r_version: [oldrel, release]
steps:
- uses: r-lib/actions/setup-r@v2
# expression marker prevents the ! being parsed as yaml tag
if: {{ "${{ !contains(matrix.platform.runs_on, 'self-hosted') }}" }}
with:
- r-version: {{ '${{ matrix.r_version.r }}' }}
- rtools-version: {{ '${{ matrix.r_version.rtools }}' }}
+ r-version: {{ '${{ matrix.r_version }}' }}
- name: Setup R Self-Hosted
if: contains(matrix.platform.runs_on, 'self-hosted')
run: |
- if [ "{{ "${{ contains(matrix.platform.runs_on, 'arm64') }}" }}" == "true" ]; then
- rig_arch="-arm64"
- fi
# rig is a system utility that allows for switching
# between pre-installed R version on the self-hosted runners
- rig default {{ '${{ matrix.r_version.r }}' }}$rig_arch
+ # rig add {{ '${{ matrix.r_version }}' }} #uncomment this to install latest release/oldrel
+ rig default {{ '${{ matrix.r_version }}' }}
rig system setup-user-lib
- rig system add-pak
{{ macros.github_setup_local_r_repo(false, true, true)|indent }}
- name: Prepare Dependency Installation
shell: bash
run: |
tar -xzf repo/src/contrib/arrow_*.tar.gz arrow/DESCRIPTION
- name: Install dependencies
+ if: {{ "${{ !contains(matrix.platform.runs_on, 'self-hosted') }}" }}
uses: r-lib/actions/setup-r-dependencies@v2
with:
working-directory: 'arrow'
extra-packages: cpp11
+ - name: Install dependencies self-hosted
+ if: {{ "${{ contains(matrix.platform.runs_on, 'self-hosted') }}" }}
+ shell: Rscript {0}
+ run: |
+ if (!requireNamespace("devtools", quietly = TRUE)) {
+ install.packages("devtools")
+ }
+ devtools::install_dev_deps('./arrow')
- name: Set CRAN like openssl
if: contains(matrix.platform.runs_on, 'arm64')
run: |
diff --git a/dev/tasks/tasks.yml b/dev/tasks/tasks.yml
index 326c94558a..a78dd9a16a 100644
--- a/dev/tasks/tasks.yml
+++ b/dev/tasks/tasks.yml
@@ -973,11 +973,11 @@ tasks:
- r-lib__libarrow__bin__darwin-arm64-openssl-3.0__arrow-{no_rc_r_version}\.zip
- r-lib__libarrow__bin__darwin-x86_64-openssl-1.1__arrow-{no_rc_r_version}\.zip
- r-lib__libarrow__bin__darwin-x86_64-openssl-3.0__arrow-{no_rc_r_version}\.zip
- - r-pkg__bin__windows__contrib__4.1__arrow_{no_rc_r_version}\.zip
+ - r-pkg__bin__windows__contrib__4.3__arrow_{no_rc_r_version}\.zip
- r-pkg__bin__windows__contrib__4.2__arrow_{no_rc_r_version}\.zip
- - r-pkg__bin__macosx__contrib__4.1__arrow_{no_rc_r_version}\.tgz
+ - r-pkg__bin__macosx__big-sur-x86_64__contrib__4.3__arrow_{no_rc_r_version}\.tgz
- r-pkg__bin__macosx__contrib__4.2__arrow_{no_rc_r_version}\.tgz
- - r-pkg__bin__macosx__big-sur-arm64__contrib__4.1__arrow_{no_rc_r_version}\.tgz
+ - r-pkg__bin__macosx__big-sur-arm64__contrib__4.3__arrow_{no_rc_r_version}\.tgz
- r-pkg__bin__macosx__big-sur-arm64__contrib__4.2__arrow_{no_rc_r_version}\.tgz
- r-pkg__src__contrib__arrow_{no_rc_r_version}\.tar\.gz
(arrow) 03/05: GH-38342: [Python] Update to_pandas to use non-deprecated DataFrame constructor (#38374)
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
raulcd pushed a commit to branch maint-14.0.x
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit 8f50fe52bddb88f1ce5822decc16e70b57363f00
Author: Joris Van den Bossche <jo...@gmail.com>
AuthorDate: Fri Dec 1 13:11:35 2023 +0100
GH-38342: [Python] Update to_pandas to use non-deprecated DataFrame constructor (#38374)
### Rationale for this change
Avoiding a deprecation warning from pandas
* Closes: #38342
Authored-by: Joris Van den Bossche <jo...@gmail.com>
Signed-off-by: Joris Van den Bossche <jo...@gmail.com>
---
python/pyarrow/pandas-shim.pxi | 11 ++++++++---
python/pyarrow/pandas_compat.py | 13 ++++++++++---
python/pyarrow/table.pxi | 6 +++---
3 files changed, 21 insertions(+), 9 deletions(-)
diff --git a/python/pyarrow/pandas-shim.pxi b/python/pyarrow/pandas-shim.pxi
index a0c0cabf6d..273575b779 100644
--- a/python/pyarrow/pandas-shim.pxi
+++ b/python/pyarrow/pandas-shim.pxi
@@ -37,7 +37,7 @@ cdef class _PandasAPIShim(object):
object _array_like_types, _is_extension_array_dtype
bint has_sparse
bint _pd024
- bint _is_v1
+ bint _is_v1, _is_ge_v21
def __init__(self):
self._tried_importing_pandas = False
@@ -74,8 +74,9 @@ cdef class _PandasAPIShim(object):
"installed. Therefore, pandas-specific integration is not "
"used.".format(self._version), stacklevel=2)
return
- elif self._loose_version < Version('2.0.0'):
- self._is_v1 = True
+
+ self._is_v1 = self._loose_version < Version('2.0.0')
+ self._is_ge_v21 = self._loose_version >= Version('2.1.0')
self._compat_module = pdcompat
self._data_frame = pd.DataFrame
@@ -158,6 +159,10 @@ cdef class _PandasAPIShim(object):
self._check_import()
return self._is_v1
+ def is_ge_v21(self):
+ self._check_import()
+ return self._is_ge_v21
+
@property
def categorical_type(self):
self._check_import()
diff --git a/python/pyarrow/pandas_compat.py b/python/pyarrow/pandas_compat.py
index e232603ba4..3f921285f9 100644
--- a/python/pyarrow/pandas_compat.py
+++ b/python/pyarrow/pandas_compat.py
@@ -748,9 +748,11 @@ def make_datetimetz(unit, tz):
return _pandas_api.datetimetz_type(unit, tz=tz)
-def table_to_blockmanager(options, table, categories=None,
- ignore_metadata=False, types_mapper=None):
+def table_to_dataframe(
+ options, table, categories=None, ignore_metadata=False, types_mapper=None
+):
from pandas.core.internals import BlockManager
+ from pandas import DataFrame
all_columns = []
column_indexes = []
@@ -774,7 +776,12 @@ def table_to_blockmanager(options, table, categories=None,
blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
axes = [columns, index]
- return BlockManager(blocks, axes)
+ mgr = BlockManager(blocks, axes)
+ if _pandas_api.is_ge_v21():
+ df = DataFrame._from_mgr(mgr, mgr.axes)
+ else:
+ df = DataFrame(mgr)
+ return df
# Set of the string repr of all numpy dtypes that can be stored in a pandas
diff --git a/python/pyarrow/table.pxi b/python/pyarrow/table.pxi
index bbf60416de..97b75d3e78 100644
--- a/python/pyarrow/table.pxi
+++ b/python/pyarrow/table.pxi
@@ -4188,12 +4188,12 @@ cdef class Table(_Tabular):
def _to_pandas(self, options, categories=None, ignore_metadata=False,
types_mapper=None):
- from pyarrow.pandas_compat import table_to_blockmanager
- mgr = table_to_blockmanager(
+ from pyarrow.pandas_compat import table_to_dataframe
+ df = table_to_dataframe(
options, self, categories,
ignore_metadata=ignore_metadata,
types_mapper=types_mapper)
- return pandas_api.data_frame(mgr)
+ return df
@property
def schema(self):
(arrow) 02/05: GH-38984: [Python][Packaging] Verification of wheels on AlmaLinux 8 are failing due to missing pip (#38985)
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
raulcd pushed a commit to branch maint-14.0.x
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit ef2f46b0251b1210a52d5dbc2acb5355637e0679
Author: Raúl Cumplido <ra...@gmail.com>
AuthorDate: Thu Nov 30 18:45:05 2023 -0600
GH-38984: [Python][Packaging] Verification of wheels on AlmaLinux 8 are failing due to missing pip (#38985)
### Rationale for this change
Almalinux 8 has been updated from 8.8 to 8.9. When using 8.9 python3 seems to be shipped without pip as the command `python3 -m pip install -U pip` fails to find pip.
### What changes are included in this PR?
Use the [ensurepip package](https://docs.python.org/3/library/ensurepip.html) which provides support for bootstrapping the pip installer into an existing Python installation.
### Are these changes tested?
Yes via archery.
### Are there any user-facing changes?
No
* Closes: #38984
Authored-by: Raúl Cumplido <ra...@gmail.com>
Signed-off-by: Sutou Kouhei <ko...@clear-code.com>
---
dev/release/setup-rhel-rebuilds.sh | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/dev/release/setup-rhel-rebuilds.sh b/dev/release/setup-rhel-rebuilds.sh
index 9cdc503215..702e6523a9 100755
--- a/dev/release/setup-rhel-rebuilds.sh
+++ b/dev/release/setup-rhel-rebuilds.sh
@@ -44,8 +44,6 @@ dnf -y install \
ninja-build \
nodejs \
openssl-devel \
- python38-devel \
- python38-pip \
ruby-devel \
sqlite-devel \
vala-devel \
@@ -54,5 +52,5 @@ dnf -y install \
npm install -g yarn
-python3 -m pip install -U pip
+python3 -m ensurepip --upgrade
alternatives --set python /usr/bin/python3