You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by yj...@apache.org on 2026/10/22 04:00:08 UTC
(cloudberry) branch main updated: datalake_fdw: build against the Arrow the Rocky images carry
This is an automated email from the ASF dual-hosted git repository.
yjhjstz pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git
The following commit(s) were added to refs/heads/main by this push:
new 707a2abdd70 datalake_fdw: build against the Arrow the Rocky images carry
707a2abdd70 is described below
commit 707a2abdd70c7733cb68d7591a7c586f3a872b55
Author: MisterRaindrop <27...@qq.com>
AuthorDate: Wed Oct 21 22:47:14 2026 +0800
datalake_fdw: build against the Arrow the Rocky images carry
The Rocky build images now ship Arrow and Parquet C++ themselves (#2155):
22.0.0 on Rocky 8 and 26.0.0 on Rocky 9 and 10, with the Arrow repository
left disabled. The ic-datalake-fdw step still installed its own copy, so
once the images were rebuilt it failed before building anything: on Rocky
9 and 10 EPEL's libarrow-devel conflicts with the image's arrow-devel over
the same headers, and on Rocky 8 the pinned 17.0.0 is no longer reachable.
The step now installs nothing and builds against the image's Arrow.
Arrow 23 moved its headers to C++20 (std::span, std::bit_width), so the
Makefile picks -std=c++20 when Arrow is 23 or newer and keeps C++17 below
that. C++20 deprecates ++ and -- on a volatile, which the server's own
HOLD_INTERRUPTS does, so -Wvolatile is off with it.
Arrow 26 removed the FileReader::GetRecordBatchReader overload that
returns its reader through an out parameter. The Result-returning form
exists from Arrow 19, so the reader uses it from there and keeps the old
spelling for older versions.
---
.github/workflows/build-cloudberry.yml | 29 +++-------------------
contrib/datalake_fdw/Makefile | 14 ++++++++++-
.../src/format/parquet/parquet_read.cpp | 17 ++++++++++---
.../src/format/parquet/parquet_write.cpp | 2 +-
4 files changed, 30 insertions(+), 32 deletions(-)
diff --git a/.github/workflows/build-cloudberry.yml b/.github/workflows/build-cloudberry.yml
index 83dcad822ad..bbaafe63b5f 100644
--- a/.github/workflows/build-cloudberry.yml
+++ b/.github/workflows/build-cloudberry.yml
@@ -1469,32 +1469,9 @@ jobs:
fi
# datalake_fdw needs the Arrow and Parquet C++ libraries, which the
- # build image does not carry -- nothing in the RPM uses them, so they
- # would be weight every other job paid for.
- if [[ "${PGXS_EXTENSION}" == "contrib/datalake_fdw" ]]; then
- . /etc/os-release
- if [[ "${VERSION_ID%%.*}" == "8" ]]; then
- # EPEL 8 has them, but its libarrow-devel needs a utf8proc-devel
- # that modular filtering keeps out of PowerTools, so it cannot be
- # installed. The Arrow project's own repository can. Pinned to
- # the version EPEL 10 carries, both because that is one version
- # fewer to have working and because the newest wants C++20, which
- # Rocky 8's gcc 8 does not have.
- # EPEL as well, and not only for Arrow itself: arrow-devel needs
- # re2-devel and parquet-devel needs thrift-devel, and on EL8 both
- # of those live in EPEL.
- dnf install -y \
- https://apache.jfrog.io/artifactory/arrow/almalinux/8/apache-arrow-release-latest.rpm
- dnf install -y --enablerepo=epel --enablerepo=powertools \
- arrow-devel-17.0.0-1.el8 parquet-devel-17.0.0-1.el8
- else
- # From EPEL, which the image has enrolled but left disabled,
- # exactly as it does for its own EPEL packages; CRB carries what
- # they depend on.
- dnf install -y --enablerepo=epel --enablerepo=crb \
- libarrow-devel parquet-libs-devel
- fi
- fi
+ # build images carry, pinned: 22.0.0 on Rocky 8, the last release
+ # whose headers are C++17, and 26.0.0 on Rocky 9 and 10. Nothing is
+ # installed here, so the version under test is the image's own.
# The RPM installs as root; the build runs as gpadmin, as everywhere
# else in this job. -H follows the command-line symlink.
diff --git a/contrib/datalake_fdw/Makefile b/contrib/datalake_fdw/Makefile
index a9ad29cd33e..ff259b1e60e 100644
--- a/contrib/datalake_fdw/Makefile
+++ b/contrib/datalake_fdw/Makefile
@@ -67,6 +67,18 @@ ARROW_CPPFLAGS := $(filter-out -std=%,\
$(shell $(PKG_CONFIG) --cflags $(ARROW_MODULES) 2>/dev/null))
ARROW_LIBS := $(shell $(PKG_CONFIG) --libs $(ARROW_MODULES) 2>/dev/null)
+# Arrow 23 moved its headers to C++20 (std::span, std::bit_width); older ones
+# are written for C++17, which every compiler this builds with has. C++20
+# deprecates ++ and -- on a volatile, and the server's own headers do that
+# (HOLD_INTERRUPTS in miscadmin.h), so that one warning is not ours to fix.
+ARROW_VERSION_MAJOR := $(firstword $(subst ., ,\
+ $(shell $(PKG_CONFIG) --modversion arrow 2>/dev/null)))
+ifeq ($(shell test "0$(ARROW_VERSION_MAJOR)" -ge 23 && echo yes),yes)
+DL_CXX_STD = -std=c++20 -Wno-volatile
+else
+DL_CXX_STD = -std=c++17
+endif
+
# This has to refuse at parse time. A recipe hung off `all` would run after
# pgxs.mk's own all-lib, so the compiler would fail on a missing arrow/api.h
# first and this would never be reached. Not for the clean targets, because
@@ -86,7 +98,7 @@ endif
# "override CFLAGS +=" would give CFLAGS override origin and silently discard
# Makefile.global's own "CFLAGS = @CFLAGS@" assignment.
PG_CFLAGS = -fvisibility=hidden
-PG_CXXFLAGS = -fvisibility=hidden -fvisibility-inlines-hidden -std=c++17
+PG_CXXFLAGS = -fvisibility=hidden -fvisibility-inlines-hidden $(DL_CXX_STD)
PG_CPPFLAGS = -I$(srcdir)/src $(ARROW_CPPFLAGS)
# The regression cases live with the rest of the test material rather than in a
diff --git a/contrib/datalake_fdw/src/format/parquet/parquet_read.cpp b/contrib/datalake_fdw/src/format/parquet/parquet_read.cpp
index c5432fd2292..421e975f8fd 100644
--- a/contrib/datalake_fdw/src/format/parquet/parquet_read.cpp
+++ b/contrib/datalake_fdw/src/format/parquet/parquet_read.cpp
@@ -404,13 +404,22 @@ parquet_open_reader(const Fragment *fragment, const ProjectionSet *projection,
}
/*
- * Arrow 21 deprecates this in favour of a Result-returning one that
- * Arrow 9 does not have, so whoever raises the floor past 21 gets a
- * warning here and a version guard to write -- the one around
- * FileWriter::Open in parquet_write.cpp is the shape of it.
+ * Arrow 19 added the form that returns its reader as a Result, 21
+ * deprecated the one that returns it through an out parameter, and 26
+ * removed that one. Arrow 9 has only the out parameter, so both
+ * spellings stay, as around FileWriter::Open in parquet_write.cpp.
*/
+#if ARROW_VERSION_MAJOR >= 19
+ arrow::Result<std::unique_ptr<arrow::RecordBatchReader>> batches =
+ impl->reader->GetRecordBatchReader(row_groups, columns);
+
+ status = batches.status();
+ if (status.ok())
+ impl->batches = std::move(*batches);
+#else
status = impl->reader->GetRecordBatchReader(row_groups, columns,
&impl->batches);
+#endif
if (!status.ok())
return DlArrowStatus(status, "open a Parquet batch reader");
diff --git a/contrib/datalake_fdw/src/format/parquet/parquet_write.cpp b/contrib/datalake_fdw/src/format/parquet/parquet_write.cpp
index ea60301df97..3dfad80906a 100644
--- a/contrib/datalake_fdw/src/format/parquet/parquet_write.cpp
+++ b/contrib/datalake_fdw/src/format/parquet/parquet_write.cpp
@@ -487,7 +487,7 @@ parquet_open_writer(const char *path, void *tupdesc_arg,
/*
* Arrow 11 deprecated the form that returns its writer through an out
* parameter. Both spellings have to be here because the versions this
- * builds against range from 9 to 17 depending on the distribution, and
+ * builds against range from 9 to 26 depending on the distribution, and
* the older one warns on the newer Arrow rather than failing -- which is
* the kind of warning that stops being read.
*/
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org