You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudberry.apache.org by "igor-suhorukov (via GitHub)" <gi...@apache.org> on 2026/09/30 08:37:37 UTC
[D] [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions [cloudberry]
GitHub user igor-suhorukov created a discussion: [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions
### Proposers
_No response_
### Proposal Status
Under Discussion
### Abstract
I would like to discuss running Apache Cloudberry on PostgreSQL 19 as a set of extensions on top of 22 dormant core hooks, instead of a forked kernel. A working prototype exists; it is my personal experiment, not an ASF release. Without the modules, the patched server is measurably vanilla PostgreSQL 19: the same PostgreSQL tests pass, the ABI is unchanged, catalogs are byte-identical, and the hooks add less than 0.2% to executed instructions. With them, it is a Greenplum-style MPP engine with plan slices, Motions, two-phase commit and distributed snapshots, running unmodified ORCA and stock PostGIS and pgvector. ORCA plans all 121 TPC-H and TPC-DS queries at SF1 with answers matching DuckDB's, and 691 of Cloudberry's 1,180 test files pass so far. Claude Code wrote the code over 12 days and 629 commits; I set the goals and signed off every change to the PostgreSQL core.
### Motivation
Greenplum and Cloudberry have always followed PostgreSQL a few releases behind: Greenplum 6 shipped on PostgreSQL 9.4, Greenplum 7 on 12, Cloudberry 2.x on 14, and `main` reached 16.9 in May 2026. The reason is structural. Cloudberry edits 882 PostgreSQL backend and header files, so every new PostgreSQL major is a large merge. #1095 estimated the 14 → 16 upgrade at 4–5 months, three of them for fixing regression tests, and noted that PostgreSQL 14, the base of Cloudberry 2.x, reaches end of life in 2026. Meanwhile PostgreSQL 17 and 18 have shipped, and 19 is close to release.
In #1877, the project named close compatibility with modern PostgreSQL as one of its goals. The prototype tests whether that goal can become cheap to sustain. If Cloudberry runs as extensions, the next PostgreSQL major means rebasing a small patch series instead of merging a fork. Users get a current PostgreSQL, including PostgreSQL 19 features such as eager aggregation and parallel autovacuum. They also get stock extensions: upstream PostGIS 3.7 instead of a PostGIS fork, and pgvector 0.8.6 built unpatched, which is relevant to #2043.
The prototype compiles Cloudberry's own sources in place, so it builds on the community's work rather than replacing it.
### Implementation
Details: [Greenplum Without the Fork](https://github.com/igor-suhorukov/cloudberry/blob/extension_postgresql_19/pg19/doc/greenplum-without-the-fork.md). Code: [core series](https://github.com/igor-suhorukov/postgres/tree/REL_19_STABLE_CLOUDBERRY), [port](https://github.com/igor-suhorukov/cloudberry/tree/extension_postgresql_19/pg19), [build instructions](https://github.com/igor-suhorukov/cloudberry/tree/extension_postgresql_19/pg19#building).
The ground rule: a server with the core patches but without Cloudberry loaded must be indistinguishable from vanilla PostgreSQL 19, and this is measured.
- **Core series:** 22 commits, 47 files, +1,578/−41 lines. Existing extension points come first: hooks, table AMs, CustomScan, custom WAL resource managers, background workers, security labels and FDWs. Where they are not enough, a new hook is a NULL-by-default function pointer, a new exported function, a registration list or an off-by-default flag. No existing signature changes, no struct gains a field, and no catalog, WAL, page or protocol format changes.
- **Vanilla checks** compare Docker images built from the same PostgreSQL commit with and without the series:
- the same 371 `meson test` results;
- `abidiff`: 32 symbols added, 0 changed;
- byte-identical catalogs, settings, keywords and stored parse trees;
- identical query IDs;
- interchangeable data directories and streaming replication;
- cachegrind: −0.005% to +0.155% executed instructions;
- a script that checks every statement added inside an existing function sits behind a hook, registry or flag, with one reviewed exception.
- **Cloudberry's tree is untouched.** The whole port is one new directory, `pg19/`. Cloudberry's files compile where they lie, so merges from Cloudberry keep applying, and a file that must change becomes a copy that says what changed. ORCA's 920 core files compile unmodified behind `planner_hook`, and PAX compiles in place with 65 of its 91 files unchanged.
- **Modules:**
- `gp_core`: dispatcher, Motions, interconnect, 2PC, distributed snapshots, GDD, FTS;
- `gp_orca`;
- `gp_sql`: Cloudberry syntax rewritten into PostgreSQL 19's grammar;
- storage: `gp_ao`, `pax`;
- external data: `gp_exttable`, `pxf_fdw`, `gpcloud`, `datalake_fdw`;
- `gp_resource`, `gp_security`, `gp_matview`, `gp_task`, `diskquota`;
- `gpfts` and gpMgmt's tools from `gpinitsystem` to `gpexpand`.
- **What changes for users:**
- migration is logical (dump and restore);
- settings are renamed `gp.*`, although `gpconfig` accepts the old names;
- DDL command tags become standard;
- the 20 shared catalogs are replaced by security labels, extension tables and a cluster file;
- TDE gives way to volume encryption.
### Rollout/Adoption Plan
Nothing here affects Cloudberry 2.x or the 3.0 plans: the prototype lives in my fork and changes no lines in Cloudberry's files. I would like to ask the community for four things:
1. **Feedback on the direction.** Is running Cloudberry as PostgreSQL extensions worth pursuing as a long-term path after 3.0?
2. **A home for `pg19/`.** It is one directory that tracks Cloudberry's sources without editing them. If the direction looks right, would the project consider hosting it, for example as an experimental branch or a separate repository?
3. **Help with the remaining tests.** The goal is to pass Cloudberry's original test suites with minimal changes to the tests. Every skipped test is listed with the statement that stops it, so tests are easy to pick up; `isolation2` and `greenplum_schedule` are next.
4. **A decision on Route B.** Should the port bring over Cloudberry's MPP variant of the PostgreSQL planner (30–45k lines), or keep shrinking ORCA's declines? On the regression suite, the port's own fallback reasons already fell from 563 to 52. A proposed `gp.mpp_planner` switch would allow side-by-side comparison, and I would value the view of the planner experts here.
For users, migration would be logical, similar to the cluster-to-cluster copy planned for 2.0 → 3.0 in #1095. Separately, four of the custom hooks (table AM registry, smgr file events, parser hook and OID hook) are general enough to propose to PostgreSQL upstream. Support from Cloudberry developers as their real-world user would strengthen that case.
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
GitHub link: https://github.com/apache/cloudberry/discussions/2065
----
This is an automatically sent email for dev@cloudberry.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@cloudberry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: dev-help@cloudberry.apache.org
Re: [D] [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
GitHub user yjhjstz added a comment to the discussion: [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions
I agree that an extension-based architecture is a promising long-term direction. A practical first step would be to modularize individual components incrementally in the current Cloudberry codebase, rather than adopting the entire PostgreSQL 19 prototype at once.
ORCA looks like a strong first high-value candidate. However, it would be useful to distinguish between:
- the ORCA optimizer engine, which is relatively independent of PostgreSQL; and
- the Cloudberry/PostgreSQL adapter, including Query-to-DXL translation, metadata access, DXL-to-PlannedStmt translation, Motion information, and fallback handling.
Cloudberry already has a `planner_hook`, but ORCA initialization and invocation are still embedded in `standard_planner()`. A concrete first milestone could therefore be:
1. Move ORCA initialization, GUC handling, planner selection, and fallback logic into a `gp_orca` module.
2. Keep the ORCA engine unchanged where possible.
3. Isolate PostgreSQL-version-dependent code in a relatively small adapter layer.
4. Allow Cloudberry to build and run normally without the ORCA module.
5. Build and test the module independently, with explicit API versioning between `gp_orca` and the Cloudberry core.
The acceptance criteria should include:
- PostgreSQL planner fallback remains fully functional.
- Loading or unloading ORCA does not change catalogs or storage formats.
- Existing ORCA regression, TPC-H, and TPC-DS results remain unchanged.
- The module can be upgraded independently within a documented compatibility range.
- No ORCA-specific control flow remains in `standard_planner()`.
After that, modules with well-defined PostgreSQL interfaces—such as AO/AOCS table access methods, external tables/FDWs, and the SQL compatibility layer—could be extracted incrementally. General-purpose hooks should also be proposed upstream whenever possible.
I would leave `gp_core`—the dispatcher, Motions, interconnect, distributed transactions, snapshots, GDD, and FTS—until later. That is the part that ultimately determines whether Cloudberry can truly operate as PostgreSQL extensions, and it requires complete `isolation2` coverage together with latency, concurrency, and failure-recovery benchmarks.
In short, ORCA would be an excellent first proof of the modularization approach, but its successful extraction would not by itself prove the feasibility of the entire extension-based architecture. The distributed execution and transaction layers remain the decisive test.
GitHub link: https://github.com/apache/cloudberry/discussions/2065#discussioncomment-18889981
----
This is an automatically sent email for dev@cloudberry.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@cloudberry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: dev-help@cloudberry.apache.org
Re: [D] [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions [cloudberry]
Posted by "igor-suhorukov (via GitHub)" <gi...@apache.org>.
GitHub user igor-suhorukov edited a discussion: [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions
### Proposers
Igor Suhorukov (@igor-suhorukov)
### Proposal Status
Under Discussion
### Abstract
I would like to discuss running Apache Cloudberry on PostgreSQL 19 as a set of extensions on top of 22 dormant core hooks, instead of a forked kernel. A working prototype exists; it is my personal experiment, not an ASF release. Without the modules, the patched server is measurably vanilla PostgreSQL 19: the same PostgreSQL tests pass, the ABI is unchanged, catalogs are byte-identical, and the hooks add less than 0.2% to executed instructions. With them, it is a Greenplum-style MPP engine with plan slices, Motions, two-phase commit and distributed snapshots, running unmodified ORCA and stock PostGIS and pgvector. ORCA plans all 121 TPC-H and TPC-DS queries at SF1 with answers matching DuckDB's, and 691 of Cloudberry's 1,180 test files pass so far. Claude Code wrote the code over 12 days and 629 commits; I set the goals and signed off every change to the PostgreSQL core.
### Motivation
Greenplum and Cloudberry have always followed PostgreSQL a few releases behind: Greenplum 6 shipped on PostgreSQL 9.4, Greenplum 7 on 12, Cloudberry 2.x on 14, and `main` reached 16.9 in May 2026. The reason is structural. Cloudberry edits 882 PostgreSQL backend and header files, so every new PostgreSQL major is a large merge. #1095 estimated the 14 → 16 upgrade at 4–5 months, three of them for fixing regression tests, and noted that PostgreSQL 14, the base of Cloudberry 2.x, reaches end of life in 2026. Meanwhile PostgreSQL 17 and 18 have shipped, and 19 is close to release.
In #1877, the project named close compatibility with modern PostgreSQL as one of its goals. The prototype tests whether that goal can become cheap to sustain. If Cloudberry runs as extensions, the next PostgreSQL major means rebasing a small patch series instead of merging a fork. Users get a current PostgreSQL, including PostgreSQL 19 features such as eager aggregation and parallel autovacuum. They also get stock extensions: upstream PostGIS 3.7 instead of a PostGIS fork, and pgvector 0.8.6 built unpatched, which is relevant to #2043.
The prototype compiles Cloudberry's own sources in place, so it builds on the community's work rather than replacing it.
### Implementation
Details: [Greenplum Without the Fork](https://github.com/igor-suhorukov/cloudberry/blob/extension_postgresql_19/pg19/doc/greenplum-without-the-fork.md). Code: [core series](https://github.com/igor-suhorukov/postgres/tree/REL_19_STABLE_CLOUDBERRY), [port](https://github.com/igor-suhorukov/cloudberry/tree/extension_postgresql_19/pg19), [build instructions](https://github.com/igor-suhorukov/cloudberry/tree/extension_postgresql_19/pg19#building).
The ground rule: a server with the core patches but without Cloudberry loaded must be indistinguishable from vanilla PostgreSQL 19, and this is measured.
- **Core series:** 22 commits, 47 files, +1,578/−41 lines. Existing extension points come first: hooks, table AMs, CustomScan, custom WAL resource managers, background workers, security labels and FDWs. Where they are not enough, a new hook is a NULL-by-default function pointer, a new exported function, a registration list or an off-by-default flag. No existing signature changes, no struct gains a field, and no catalog, WAL, page or protocol format changes.
- **Vanilla checks** compare Docker images built from the same PostgreSQL commit with and without the series:
- the same 371 `meson test` results;
- `abidiff`: 32 symbols added, 0 changed;
- byte-identical catalogs, settings, keywords and stored parse trees;
- identical query IDs;
- interchangeable data directories and streaming replication;
- cachegrind: −0.005% to +0.155% executed instructions;
- a script that checks every statement added inside an existing function sits behind a hook, registry or flag, with one reviewed exception.
- **Cloudberry's tree is untouched.** The whole port is one new directory, `pg19/`. Cloudberry's files compile where they lie, so merges from Cloudberry keep applying, and a file that must change becomes a copy that says what changed. ORCA's 920 core files compile unmodified behind `planner_hook`, and PAX compiles in place with 65 of its 91 files unchanged.
- **Modules:**
- `gp_core`: dispatcher, Motions, interconnect, 2PC, distributed snapshots, GDD, FTS;
- `gp_orca`;
- `gp_sql`: Cloudberry syntax rewritten into PostgreSQL 19's grammar;
- storage: `gp_ao`, `pax`;
- external data: `gp_exttable`, `pxf_fdw`, `gpcloud`, `datalake_fdw`;
- `gp_resource`, `gp_security`, `gp_matview`, `gp_task`, `diskquota`;
- `gpfts` and gpMgmt's tools from `gpinitsystem` to `gpexpand`.
- **What changes for users:**
- migration is logical (dump and restore);
- settings are renamed `gp.*`, although `gpconfig` accepts the old names;
- DDL command tags become standard;
- the 20 shared catalogs are replaced by security labels, extension tables and a cluster file;
- TDE gives way to volume encryption.
### Rollout/Adoption Plan
Nothing here affects Cloudberry 2.x or the 3.0 plans: the prototype lives in my fork and changes no lines in Cloudberry's files. I would like to ask the community for four things:
1. **Feedback on the direction.** Is running Cloudberry as PostgreSQL extensions worth pursuing as a long-term path after 3.0?
2. **A home for `pg19/`.** It is one directory that tracks Cloudberry's sources without editing them. If the direction looks right, would the project consider hosting it, for example as an experimental branch or a separate repository?
3. **Help with the remaining tests.** The goal is to pass Cloudberry's original test suites with minimal changes to the tests. Every skipped test is listed with the statement that stops it, so tests are easy to pick up; `isolation2` and `greenplum_schedule` are next.
4. **A decision on Route B.** Should the port bring over Cloudberry's MPP variant of the PostgreSQL planner (30–45k lines), or keep shrinking ORCA's declines? On the regression suite, the port's own fallback reasons already fell from 563 to 52. A proposed `gp.mpp_planner` switch would allow side-by-side comparison, and I would value the view of the planner experts here.
For users, migration would be logical, similar to the cluster-to-cluster copy planned for 2.0 → 3.0 in #1095. Separately, four of the custom hooks (table AM registry, smgr file events, parser hook and OID hook) are general enough to propose to PostgreSQL upstream. Support from Cloudberry developers as their real-world user would strengthen that case.
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
GitHub link: https://github.com/apache/cloudberry/discussions/2065
----
This is an automatically sent email for dev@cloudberry.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@cloudberry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: dev-help@cloudberry.apache.org
Re: [D] [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions [cloudberry]
Posted by "leborchuk (via GitHub)" <gi...@apache.org>.
GitHub user leborchuk added a comment to the discussion: [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions
Hi!
Thank you for the attention to our project )
The article https://github.com/igor-suhorukov/cloudberry/blob/extension_postgresql_19/pg19/doc/greenplum-without-the-fork.md looks more broader and has more interesting details, if you don't mind I'll answer mainly to ideas from you original post rather than technical description here. Because some aspects need discussion at first and only then implementation.
1. Kernel rebasing. It's the major reason why you need community. You cannot offload this task to LLM or whatever else. Of course, LLM will spend you tokens easily and you even could get some looks promising result. But without owning the code you won't get good basement for your further improvements. If you could generate whatever you want why one could need cloudberry at all? Or you just could ask LLM to rebase cloudberry to PG19 (it also complete task in a few days).
Ease rebasement - what we constantly bear in mind. And the story we usually share with postgres community. You could get postgres, add here PAX - and now you have analytical data storage format. And the same with other new components. There are not so much of them comparing with original greenplum innovations. But now it's just the matter of time. There will be more and more components, I believe )
It will be great to evolve old approaches and make them extension-based. For example, as you described - make ao and aoco tables as extensions. We could start with some specific functionality and then step-by-step goes to the situation where you have postgresql fork extended with various MPP functions. Not based on postgres our own MPP database. The difference of course in the number of changes in a core part (and conflicts need close attention) and code reusing.
The idealistic (I think it's too simple to be true) picture is you have a set of extensions and combining them could easily make your own specific database.
How to achieve this - if you are really interested in it and ready to participate, let's continue discussion (good example is https://github.com/apache/cloudberry/discussions/1683 ), volunteer project developers and gradually improve our codebase. LLM will help us, but not replace, we still need to clearly understand what we are doing.
2. Clickhouse benchmarks
Kernel rebasing is great but as you have written, we rebase kernel not to the sole kernel version but to achieve something - new functionality or better performance. Good example is clickbench - new kernels could get you better performance. But I need to say modern PG kernel is not enough. PG is just (not fully describe current situation but it's too hard to express it succinctly) not good enough to beat clickhouse in clickbench. Not because clickbench purposely was written by clickhouse developers to beat all other competitors (but because of that, too), but mainly because clickhouse constantly compact (sort) data in background and have many others good improvements. We also could improve our group by facilities and be comparable in some aspects with clickhouse. It's possible, but not easy. if you interested - feel free to create discussion. I believe we could do it.
Also we could create our own benchmark to beat all the competitors in it.
3. Not only Postgres
You mentioned other Postgres-related projects. We could not only compete with them but also get good approaches from them.
For example ORCA (and some other GP components) also is used by polardb - database with shared-everything architecture. We also could get some good approaches from polardb and use it in cloudberry. I'am not good at polardb to say which one - it was just a suggestion. But if you have good candidates - let us know.
GitHub link: https://github.com/apache/cloudberry/discussions/2065#discussioncomment-18697011
----
This is an automatically sent email for dev@cloudberry.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@cloudberry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: dev-help@cloudberry.apache.org
Re: [D] [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions [cloudberry]
Posted by "igor-suhorukov (via GitHub)" <gi...@apache.org>.
GitHub user igor-suhorukov edited a discussion: [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions
### Proposers
Igor Suhorukov ([@igor-suhorukov](https://github.com/igor-suhorukov))
### Proposal Status
Under Discussion
### Abstract
*Updated 2026-10-13: security updates at PostgreSQL's pace (Motivation), what has been built on the port since the first version, a vectorized executor and an Arrow Flight SQL endpoint (Motivation, Implementation), and current figures: 24 core commits, 968 test files, the progress of `isolation2` (Rollout, point 3). The first comments refer to the version of 2026-09-30.*
I would like to discuss running Apache Cloudberry on PostgreSQL 19 as a set of extensions on top of 24 dormant core hooks, instead of a forked kernel. A working prototype exists; it is my personal experiment, not an ASF release. Without the modules, the patched server is measurably vanilla PostgreSQL 19: the same PostgreSQL tests pass, the ABI is unchanged, catalogs are byte-identical, and the hooks add less than 0.2% to executed instructions. With them, it is a Greenplum-style MPP engine with plan slices, Motions, two-phase commit and distributed snapshots, running unmodified ORCA and stock PostGIS and pgvector. ORCA plans all 121 TPC-H and TPC-DS queries at SF1 with answers matching DuckDB's, and 968 of Cloudberry's 1,180 test files pass so far.
The port gives users more than a newer PostgreSQL. With Cloudberry's components as modules, new capabilities arrive as further extensions. [pg_vexec](https://github.com/igor-suhorukov/pg_vexec), built on the port without a single new core patch, adds vectorized execution that ORCA plans by cost, PAX and `ao_column` tables read and written by column, Arrow frames between segments, and an Arrow Flight SQL endpoint for BI and ML clients.
Claude Code wrote the code, the port over 12 days and 629 commits and pg_vexec over three days; I set the goals and signed off every change to the PostgreSQL core.
### Motivation
**Security fixes at PostgreSQL's pace.** The port leaves PostgreSQL's code as upstream wrote it: its 24 core commits sit on top of PostgreSQL's own `REL_19_STABLE` branch, and no Cloudberry code lives in PostgreSQL's tree. A PostgreSQL minor release, with its security fixes, is then a rebase of those 24 commits and a rebuild. This can run unattended as a scheduled GitHub Actions job: fetch the new release tag, rebase the series, run the comparison with vanilla PostgreSQL and the test suites, publish packages. The port's release workflow already builds and smoke-tests packages from any PostgreSQL ref. As a trial, on 2026-10-13 the series was rebased onto the head of `REL_19_STABLE`, 153 upstream commits past its base, and all 24 commits applied without a conflict. Users would get a PostgreSQL security release as soon as that job passes. In a fork, each fix waits for a manual backport: `main` is based on PostgreSQL 16.9 of May 2025, and the security fixes of the six PostgreSQL 16 minor
releases since reach it as individual cherry-picks, 25 commits for 16.15 alone.
Greenplum and Cloudberry have always followed PostgreSQL a few releases behind: Greenplum 6 shipped on PostgreSQL 9.4, Greenplum 7 on 12, Cloudberry 2.x on 14, and `main` reached 16.9 in May 2026. The reason is structural. Cloudberry edits 882 PostgreSQL backend and header files, so every new PostgreSQL major is a large merge. [#1095](https://github.com/apache/cloudberry/discussions/1095) estimated the 14 → 16 upgrade at 4–5 months, three of them for fixing regression tests, and noted that PostgreSQL 14, the base of Cloudberry 2.x, reaches end of life in 2026. Meanwhile PostgreSQL 17 and 18 have shipped, and 19 is close to release.
In [#1877](https://github.com/apache/cloudberry/discussions/1877), the project named close compatibility with modern PostgreSQL as one of its goals. The prototype tests whether that goal can become cheap to sustain. If Cloudberry runs as extensions, the next PostgreSQL major means rebasing a small patch series instead of merging a fork. Users get a current PostgreSQL, including PostgreSQL 19 features such as eager aggregation and parallel autovacuum. They also get stock extensions: upstream PostGIS 3.7 instead of a PostGIS fork, and pgvector 0.8.6 built unpatched, which is relevant to [#2043](https://github.com/apache/cloudberry/discussions/2043).
**More than a newer PostgreSQL.** A current base is the first gain, not the only one. As modules, Cloudberry's planner, storage and interconnect can be combined with other extensions, so new capabilities arrive as extensions rather than as more changes to a fork. Apache Cloudberry's code has no vectorized executor: only traces of a closed-source engine remain, a `create_vectorization_plan` flag the planner always passes as false, a `WindowHashAgg` node with no executor behind it, and a PAX adapter under `VEC_BUILD` that no longer compiles. [pg_vexec](https://github.com/igor-suhorukov/pg_vexec) was built on top of the port in three days of implementation ([write-up](https://igor-suhorukov.github.io/blog/2026/10/07/vectorized-postgresql-extension/)), and it brings Cloudberry users:
- **Vectorized execution, chosen by cost.** Batches of up to 1,024 rows by column replace the row-at-a-time loop on the coordinator and on every segment. ORCA weighs vector nodes in its search (`CCostModelVec`) and its translator builds them; PostgreSQL's planner gets them through its path hooks. No finished plan is rewritten, and a fallback to PostgreSQL's expression evaluator inside the same node keeps PostgreSQL's semantics.
- **Columnar storage without rows.** PAX and `ao_column` hand batches to the executor and take them back on insert. `count`, `min`, `max`, `sum` and `avg` without conditions come from PAX's statistics: `count(*)` over 10 million rows takes 0.33 ms instead of 265.
- **Arrow between segments.** Motions carry Arrow IPC frames, routed by a vectorized `cdbhash` that reproduces `GpHashSegment` bit for bit, and a new `shm` interconnect passes them through shared memory between the segments of one host.
- **Arrow Flight SQL.** `vexec_flight` serves results as Arrow record batches straight from the executor, and `adbc_ingest()` loads DataFrames into tables, by column into PAX and `ao_column`, with the standard ADBC Flight SQL and Flight SQL JDBC drivers. Logins are checked against `pg_hba.conf` and statements run through portals, so privileges, RLS and `pg_stat_statements` behave as they do over pgwire, which stays as it is.
- **PostGIS and pgvector over batches**, through kernel packs that declare which of their functions may be called on a batch's rows, without changing either extension.
These are first measurements, and partial ones; every answer was checked against DuckDB's. On one node, on ClickBench at 10 million rows, vexec in auto mode takes 19–24% off PostgreSQL's planner and 16–19% off ORCA by geometric mean, and the best queries run 8.8–9.5× faster, though a few got slower; Flight SQL delivers a million `lineitem` rows 4.7× faster than binary COPY and loads PAX 1.65× faster than COPY CSV. On four segments TPC-H Q1 runs 1.8–2.8× faster, but the first TPC run, with only scans and aggregates vectorized, averaged about 1×; the cluster's measurements with vectorized hash joins and Arrow frames through Motions are next.
None of it needed a new core patch: vexec also runs on vanilla PostgreSQL 19, and `gp_orca` now builds for it without `gp_core` (`-Dorca_single_node`), so ORCA plans vector nodes on a stock server too. It relies on what PostgreSQL added after 16: `heap_prepare_pagescan` (17), `explain_per_plan_hook` (18), and `planner_setup_hook`, `planner_shutdown_hook` and the batched visibility check `HeapTupleSatisfiesMVCCBatch` (19), none of which Cloudberry's `main` has. In a fork, each such capability would be one more set of kernel changes to carry across every PostgreSQL major.
The prototype compiles Cloudberry's own sources in place, so it builds on the community's work rather than replacing it.
### Implementation
Details: [Greenplum Without the Fork](https://github.com/igor-suhorukov/cloudberry/blob/extension_postgresql_19/pg19/doc/greenplum-without-the-fork.md). Code: [core series](https://github.com/igor-suhorukov/postgres/tree/REL_19_STABLE_CLOUDBERRY), [port](https://github.com/igor-suhorukov/cloudberry/tree/extension_postgresql_19/pg19), [build instructions](https://github.com/igor-suhorukov/cloudberry/tree/extension_postgresql_19/pg19#building).
The ground rule: a server with the core patches but without Cloudberry loaded must be indistinguishable from vanilla PostgreSQL 19, and this is measured.
- **Core series:** 24 commits, 49 files, +1,622/−41 lines. Existing extension points come first: hooks, table AMs, CustomScan, custom WAL resource managers, background workers, security labels and FDWs. Where they are not enough, a new hook is a NULL-by-default function pointer, a new exported function, a registration list or an off-by-default flag. No existing signature changes, no struct gains a field, and no catalog, WAL, page or protocol format changes.
- **Vanilla checks** compare Docker images built from the same PostgreSQL commit with and without the series:
- the same 371 `meson test` results;
- `abidiff`: 34 symbols added, 0 changed;
- byte-identical catalogs, settings, keywords and stored parse trees;
- identical query IDs;
- interchangeable data directories and streaming replication;
- cachegrind: −0.011% to +0.160% executed instructions;
- a script that checks every statement added inside an existing function sits behind a hook, registry or flag, with one reviewed exception.
- **Cloudberry's tree is untouched.** The whole port is one new directory, `pg19/`, plus a release workflow in `.github/workflows`. Cloudberry's files compile where they lie, so merges from Cloudberry keep applying, and a file that must change becomes a copy that says what changed. ORCA's 920 core files compile unmodified behind `planner_hook`, and PAX compiles in place with 57 of its 91 files unchanged.
- **Modules:**
- `gp_core`: dispatcher, Motions, interconnect, 2PC, distributed snapshots, GDD, FTS;
- `gp_orca`;
- `gp_sql`: Cloudberry syntax rewritten into PostgreSQL 19's grammar;
- storage: `gp_ao`, `pax`;
- external data: `gp_exttable`, `pxf_fdw`, `gpcloud`, `datalake_fdw`;
- `gp_resource`, `gp_security`, `gp_matview`, `gp_task`, `diskquota`;
- `gpfts` and gpMgmt's tools from `gpinitsystem` to `gpexpand`.
- **Built on the port since:** [pg_vexec](https://github.com/igor-suhorukov/pg_vexec), in its own repository: `vexec` (about 43,000 lines of C), `vexec_flight` (7,900) and the kernel packs for pgvector and PostGIS (1,000). vexec builds with PGXS both for vanilla PostgreSQL 19 and for the port. Its side of the port is module code in `pg19/`, about 13,600 lines with the `shm` transport and PAX's copies: gp_orca's API for vector nodes and `CCostModelVec`, gp_core's API 1.15 (a Motion rebuilt by another module, the distribution's hash functions, `squelch_subtree()`, figures in EXPLAIN ANALYZE), and PAX's and gp_ao's batch readers and sinks. vexec calls none of the core patches.
- **What changes for users:**
- migration is logical (dump and restore);
- settings are renamed `gp.*`, although `gpconfig` accepts the old names;
- DDL command tags become standard;
- the 20 shared catalogs are replaced by security labels, extension tables and a cluster file;
- TDE gives way to volume encryption.
### Rollout/Adoption Plan
Nothing here affects Cloudberry 2.x or the 3.0 plans: the prototype lives in my fork and changes no lines in Cloudberry's files. I would like to ask the community for four things:
1. **Feedback on the direction.** Is running Cloudberry as PostgreSQL extensions worth pursuing as a long-term path after 3.0?
2. **A home for `pg19/`.** It is one directory that tracks Cloudberry's sources without editing them. If the direction looks right, would the project consider hosting it, for example as an experimental branch or a separate repository?
3. **Help with the remaining tests.** The goal is to pass Cloudberry's original test suites with minimal changes to the tests. Every skipped test is listed with the statement that stops it, so tests are easy to pick up. Where the tests stand: 968 of Cloudberry's 1,180 test files pass, 82% (744 on 2026-10-08).
For users, migration would be logical, similar to the cluster-to-cluster copy planned for 2.0 → 3.0 in [#1095](https://github.com/apache/cloudberry/discussions/1095). Separately, four of the custom hooks (table AM registry, smgr file events, parser hook and OID hook) are general enough to propose to PostgreSQL upstream. Support from Cloudberry developers as their real-world user would strengthen that case.
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
GitHub link: https://github.com/apache/cloudberry/discussions/2065
----
This is an automatically sent email for dev@cloudberry.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@cloudberry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: dev-help@cloudberry.apache.org
Re: [D] [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions [cloudberry]
Posted by "igor-suhorukov (via GitHub)" <gi...@apache.org>.
GitHub user igor-suhorukov edited a discussion: [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions
### Proposers
Igor Suhorukov ([@igor-suhorukov](https://github.com/igor-suhorukov))
### Proposal Status
Under Discussion
### Abstract
*Updated 2026-10-13: security updates at PostgreSQL's pace (Motivation), what has been built on the port since the first version, a vectorized executor and an Arrow Flight SQL endpoint (Motivation, Implementation), and current figures: 24 core commits, 968 test files, the progress of `isolation2` (Rollout, point 3). The first comments refer to the version of 2026-09-30.*
I would like to discuss running Apache Cloudberry on PostgreSQL 19 as a set of extensions on top of 24 dormant core hooks, instead of a forked kernel. A working prototype exists; it is my personal experiment, not an ASF release. Without the modules, the patched server is measurably vanilla PostgreSQL 19: the same PostgreSQL tests pass, the ABI is unchanged, catalogs are byte-identical, and the hooks add less than 0.2% to executed instructions. With them, it is a Greenplum-style MPP engine with plan slices, Motions, two-phase commit and distributed snapshots, running unmodified ORCA and stock PostGIS and pgvector. ORCA plans all 121 TPC-H and TPC-DS queries at SF1 with answers matching DuckDB's, and 968 of Cloudberry's 1,180 test files pass so far.
The port gives users more than a newer PostgreSQL. With Cloudberry's components as modules, new capabilities arrive as further extensions. [pg_vexec](https://github.com/igor-suhorukov/pg_vexec), built on the port without a single new core patch, adds vectorized execution that ORCA plans by cost, PAX and `ao_column` tables read and written by column, Arrow frames between segments, and an Arrow Flight SQL endpoint for BI and ML clients.
Claude Code wrote the code, the port over 12 days and 629 commits and pg_vexec over three days; I set the goals and signed off every change to the PostgreSQL core.
### Motivation
**Security fixes at PostgreSQL's pace.** The port leaves PostgreSQL's code as upstream wrote it: its 24 core commits sit on top of PostgreSQL's own `REL_19_STABLE` branch, and no Cloudberry code lives in PostgreSQL's tree. A PostgreSQL minor release, with its security fixes, is then a rebase of those 24 commits and a rebuild. This can run unattended as a scheduled GitHub Actions job: fetch the new release tag, rebase the series, run the comparison with vanilla PostgreSQL and the test suites, publish packages. The port's release workflow already builds and smoke-tests packages from any PostgreSQL ref. As a trial, on 2026-10-13 the series was rebased onto the head of `REL_19_STABLE`, 153 upstream commits past its base, and all 24 commits applied without a conflict. Users would get a PostgreSQL security release as soon as that job passes. In a fork, each fix waits for a manual backport: `main` is based on PostgreSQL 16.9 of May 2025, and the security fixes of the six PostgreSQL 16 minor
releases since reach it as individual cherry-picks, 25 commits for 16.15 alone.
Greenplum and Cloudberry have always followed PostgreSQL a few releases behind: Greenplum 6 shipped on PostgreSQL 9.4, Greenplum 7 on 12, Cloudberry 2.x on 14, and `main` reached 16.9 in May 2026. The reason is structural. Cloudberry edits 882 PostgreSQL backend and header files, so every new PostgreSQL major is a large merge. [#1095](https://github.com/apache/cloudberry/discussions/1095) estimated the 14 → 16 upgrade at 4–5 months, three of them for fixing regression tests, and noted that PostgreSQL 14, the base of Cloudberry 2.x, reaches end of life in 2026. Meanwhile PostgreSQL 17 and 18 have shipped, and 19 is close to release.
In [#1877](https://github.com/apache/cloudberry/discussions/1877), the project named close compatibility with modern PostgreSQL as one of its goals. The prototype tests whether that goal can become cheap to sustain. If Cloudberry runs as extensions, the next PostgreSQL major means rebasing a small patch series instead of merging a fork. Users get a current PostgreSQL, including PostgreSQL 19 features such as eager aggregation and parallel autovacuum. They also get stock extensions: upstream PostGIS 3.7 instead of a PostGIS fork, and pgvector 0.8.6 built unpatched, which is relevant to [#2043](https://github.com/apache/cloudberry/discussions/2043).
**More than a newer PostgreSQL.** A current base is the first gain, not the only one. As modules, Cloudberry's planner, storage and interconnect can be combined with other extensions, so new capabilities arrive as extensions rather than as more changes to a fork. Apache Cloudberry's code has no vectorized executor: only traces of a closed-source engine remain, a `create_vectorization_plan` flag the planner always passes as false, a `WindowHashAgg` node with no executor behind it, and a PAX adapter under `VEC_BUILD` that no longer compiles. [pg_vexec](https://github.com/igor-suhorukov/pg_vexec) was built on top of the port in three days of implementation ([write-up](https://igor-suhorukov.github.io/blog/2026/10/07/vectorized-postgresql-extension/)), and it brings Cloudberry users:
- **Vectorized execution, chosen by cost.** Batches of up to 1,024 rows by column replace the row-at-a-time loop on the coordinator and on every segment. ORCA weighs vector nodes in its search (`CCostModelVec`) and its translator builds them; PostgreSQL's planner gets them through its path hooks. No finished plan is rewritten, and a fallback to PostgreSQL's expression evaluator inside the same node keeps PostgreSQL's semantics.
- **Columnar storage without rows.** PAX and `ao_column` hand batches to the executor and take them back on insert. `count`, `min`, `max`, `sum` and `avg` without conditions come from PAX's statistics: `count(*)` over 10 million rows takes 0.33 ms instead of 265.
- **Arrow between segments.** Motions carry Arrow IPC frames, routed by a vectorized `cdbhash` that reproduces `GpHashSegment` bit for bit, and a new `shm` interconnect passes them through shared memory between the segments of one host.
- **Arrow Flight SQL.** `vexec_flight` serves results as Arrow record batches straight from the executor, and `adbc_ingest()` loads DataFrames into tables, by column into PAX and `ao_column`, with the standard ADBC Flight SQL and Flight SQL JDBC drivers. Logins are checked against `pg_hba.conf` and statements run through portals, so privileges, RLS and `pg_stat_statements` behave as they do over pgwire, which stays as it is.
- **PostGIS and pgvector over batches**, through kernel packs that declare which of their functions may be called on a batch's rows, without changing either extension.
These are first measurements, and partial ones; every answer was checked against DuckDB's. On one node, on ClickBench at 10 million rows, vexec in auto mode takes 19–24% off PostgreSQL's planner and 16–19% off ORCA by geometric mean, and the best queries run 8.8–9.5× faster, though a few got slower; Flight SQL delivers a million `lineitem` rows 4.7× faster than binary COPY and loads PAX 1.65× faster than COPY CSV. On four segments TPC-H Q1 runs 1.8–2.8× faster, but the first TPC run, with only scans and aggregates vectorized, averaged about 1×; the cluster's measurements with vectorized hash joins and Arrow frames through Motions are next.
None of it needed a new core patch: vexec also runs on vanilla PostgreSQL 19, and `gp_orca` now builds for it without `gp_core` (`-Dorca_single_node`), so ORCA plans vector nodes on a stock server too. It relies on what PostgreSQL added after 16: `heap_prepare_pagescan` (17), `explain_per_plan_hook` (18), and `planner_setup_hook`, `planner_shutdown_hook` and the batched visibility check `HeapTupleSatisfiesMVCCBatch` (19), none of which Cloudberry's `main` has. In a fork, each such capability would be one more set of kernel changes to carry across every PostgreSQL major.
The prototype compiles Cloudberry's own sources in place, so it builds on the community's work rather than replacing it.
### Implementation
Details: [Greenplum Without the Fork](https://github.com/igor-suhorukov/cloudberry/blob/extension_postgresql_19/pg19/doc/greenplum-without-the-fork.md). Code: [core series](https://github.com/igor-suhorukov/postgres/tree/REL_19_STABLE_CLOUDBERRY), [port](https://github.com/igor-suhorukov/cloudberry/tree/extension_postgresql_19/pg19), [build instructions](https://github.com/igor-suhorukov/cloudberry/tree/extension_postgresql_19/pg19#building).
The ground rule: a server with the core patches but without Cloudberry loaded must be indistinguishable from vanilla PostgreSQL 19, and this is measured.
- **Core series:** 24 commits, 49 files, +1,622/−41 lines. Existing extension points come first: hooks, table AMs, CustomScan, custom WAL resource managers, background workers, security labels and FDWs. Where they are not enough, a new hook is a NULL-by-default function pointer, a new exported function, a registration list or an off-by-default flag. No existing signature changes, no struct gains a field, and no catalog, WAL, page or protocol format changes.
- **Vanilla checks** compare Docker images built from the same PostgreSQL commit with and without the series:
- the same 371 `meson test` results;
- `abidiff`: 34 symbols added, 0 changed;
- byte-identical catalogs, settings, keywords and stored parse trees;
- identical query IDs;
- interchangeable data directories and streaming replication;
- cachegrind: −0.011% to +0.160% executed instructions;
- a script that checks every statement added inside an existing function sits behind a hook, registry or flag, with one reviewed exception.
- **Cloudberry's tree is untouched.** The whole port is one new directory, `pg19/`, plus a release workflow in `.github/workflows`. Cloudberry's files compile where they lie, so merges from Cloudberry keep applying, and a file that must change becomes a copy that says what changed. ORCA's 920 core files compile unmodified behind `planner_hook`, and PAX compiles in place with 57 of its 91 files unchanged.
- **Modules:**
- `gp_core`: dispatcher, Motions, interconnect, 2PC, distributed snapshots, GDD, FTS;
- `gp_orca`;
- `gp_sql`: Cloudberry syntax rewritten into PostgreSQL 19's grammar;
- storage: `gp_ao`, `pax`;
- external data: `gp_exttable`, `pxf_fdw`, `gpcloud`, `datalake_fdw`;
- `gp_resource`, `gp_security`, `gp_matview`, `gp_task`, `diskquota`;
- `gpfts` and gpMgmt's tools from `gpinitsystem` to `gpexpand`.
- **Built on the port since:** [pg_vexec](https://github.com/igor-suhorukov/pg_vexec), in its own repository: `vexec` (about 43,000 lines of C), `vexec_flight` (7,900) and the kernel packs for pgvector and PostGIS (1,000). vexec builds with PGXS both for vanilla PostgreSQL 19 and for the port. Its side of the port is module code in `pg19/`, about 13,600 lines with the `shm` transport and PAX's copies: gp_orca's API for vector nodes and `CCostModelVec`, gp_core's API 1.15 (a Motion rebuilt by another module, the distribution's hash functions, `squelch_subtree()`, figures in EXPLAIN ANALYZE), and PAX's and gp_ao's batch readers and sinks. vexec calls none of the core patches.
- **What changes for users:**
- migration is logical (dump and restore);
- settings are renamed `gp.*`, although `gpconfig` accepts the old names;
- DDL command tags become standard;
- the 20 shared catalogs are replaced by security labels, extension tables and a cluster file;
- TDE gives way to volume encryption.
### Rollout/Adoption Plan
Nothing here affects Cloudberry 2.x or the 3.0 plans: the prototype lives in my fork and changes no lines in Cloudberry's files. I would like to ask the community for four things:
1. **Feedback on the direction.** Is running Cloudberry as PostgreSQL extensions worth pursuing as a long-term path after 3.0?
2. **A home for `pg19/`.** It is one directory that tracks Cloudberry's sources without editing them. If the direction looks right, would the project consider hosting it, for example as an experimental branch or a separate repository?
3. **Help with the remaining tests.** The goal is to pass Cloudberry's original test suites with minimal changes to the tests. Every skipped test is listed with the statement that stops it, so tests are easy to pick up. Where the tests stand at [40984df98e4](https://github.com/igor-suhorukov/cloudberry/commit/40984df98e4):
- 968 of Cloudberry's 1,180 test files pass, 82% (744 on 2026-10-08).
For users, migration would be logical, similar to the cluster-to-cluster copy planned for 2.0 → 3.0 in [#1095](https://github.com/apache/cloudberry/discussions/1095). Separately, four of the custom hooks (table AM registry, smgr file events, parser hook and OID hook) are general enough to propose to PostgreSQL upstream. Support from Cloudberry developers as their real-world user would strengthen that case.
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
GitHub link: https://github.com/apache/cloudberry/discussions/2065
----
This is an automatically sent email for dev@cloudberry.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@cloudberry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: dev-help@cloudberry.apache.org
Re: [D] [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions [cloudberry]
Posted by "igor-suhorukov (via GitHub)" <gi...@apache.org>.
GitHub user igor-suhorukov edited a discussion: [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions
### Proposers
Igor Suhorukov ([@igor-suhorukov](https://github.com/igor-suhorukov))
### Proposal Status
Under Discussion
### Abstract
*Updated 2026-10-08: what has been built on the port since the first version, a vectorized executor and an Arrow Flight SQL endpoint (Motivation, Implementation), and current figures: 24 core commits, 744 test files, the progress of `isolation2`. The first comments refer to the version of 2026-09-30.*
I would like to discuss running Apache Cloudberry on PostgreSQL 19 as a set of extensions on top of 24 dormant core hooks, instead of a forked kernel. A working prototype exists; it is my personal experiment, not an ASF release. Without the modules, the patched server is measurably vanilla PostgreSQL 19: the same PostgreSQL tests pass, the ABI is unchanged, catalogs are byte-identical, and the hooks add less than 0.2% to executed instructions. With them, it is a Greenplum-style MPP engine with plan slices, Motions, two-phase commit and distributed snapshots, running unmodified ORCA and stock PostGIS and pgvector. ORCA plans all 121 TPC-H and TPC-DS queries at SF1 with answers matching DuckDB's, and 744 of Cloudberry's 1,180 test files pass so far.
The port gives users more than a newer PostgreSQL. With Cloudberry's components as modules, new capabilities arrive as further extensions. [pg_vexec](https://github.com/igor-suhorukov/pg_vexec), built on the port without a single new core patch, adds vectorized execution that ORCA plans by cost, PAX and `ao_column` tables read and written by column, Arrow frames between segments, and an Arrow Flight SQL endpoint for BI and ML clients.
Claude Code wrote the code, the port over 12 days and 629 commits and pg_vexec over three days; I set the goals and signed off every change to the PostgreSQL core.
### Motivation
Greenplum and Cloudberry have always followed PostgreSQL a few releases behind: Greenplum 6 shipped on PostgreSQL 9.4, Greenplum 7 on 12, Cloudberry 2.x on 14, and `main` reached 16.9 in May 2026. The reason is structural. Cloudberry edits 882 PostgreSQL backend and header files, so every new PostgreSQL major is a large merge. [#1095](https://github.com/apache/cloudberry/discussions/1095) estimated the 14 → 16 upgrade at 4–5 months, three of them for fixing regression tests, and noted that PostgreSQL 14, the base of Cloudberry 2.x, reaches end of life in 2026. Meanwhile PostgreSQL 17 and 18 have shipped, and 19 is close to release.
In [#1877](https://github.com/apache/cloudberry/discussions/1877), the project named close compatibility with modern PostgreSQL as one of its goals. The prototype tests whether that goal can become cheap to sustain. If Cloudberry runs as extensions, the next PostgreSQL major means rebasing a small patch series instead of merging a fork. Users get a current PostgreSQL, including PostgreSQL 19 features such as eager aggregation and parallel autovacuum. They also get stock extensions: upstream PostGIS 3.7 instead of a PostGIS fork, and pgvector 0.8.6 built unpatched, which is relevant to [#2043](https://github.com/apache/cloudberry/discussions/2043).
**More than a newer PostgreSQL.** A current base is the first gain, not the only one. As modules, Cloudberry's planner, storage and interconnect can be combined with other extensions, so new capabilities arrive as extensions rather than as more changes to a fork. Apache Cloudberry's code has no vectorized executor: only traces of a closed-source engine remain, a `create_vectorization_plan` flag the planner always passes as false, a `WindowHashAgg` node with no executor behind it, and a PAX adapter under `VEC_BUILD` that no longer compiles. [pg_vexec](https://github.com/igor-suhorukov/pg_vexec) was built on top of the port in three days of implementation ([write-up](https://igor-suhorukov.github.io/blog/2026/10/07/vectorized-postgresql-extension/)), and it brings Cloudberry users:
- **Vectorized execution, chosen by cost.** Batches of up to 1,024 rows by column replace the row-at-a-time loop on the coordinator and on every segment. ORCA weighs vector nodes in its search (`CCostModelVec`) and its translator builds them; PostgreSQL's planner gets them through its path hooks. No finished plan is rewritten, and a fallback to PostgreSQL's expression evaluator inside the same node keeps PostgreSQL's semantics.
- **Columnar storage without rows.** PAX and `ao_column` hand batches to the executor and take them back on insert. `count`, `min`, `max`, `sum` and `avg` without conditions come from PAX's statistics: `count(*)` over 10 million rows takes 0.33 ms instead of 265.
- **Arrow between segments.** Motions carry Arrow IPC frames, routed by a vectorized `cdbhash` that reproduces `GpHashSegment` bit for bit, and a new `shm` interconnect passes them through shared memory between the segments of one host.
- **Arrow Flight SQL.** `vexec_flight` serves results as Arrow record batches straight from the executor, and `adbc_ingest()` loads DataFrames into tables, by column into PAX and `ao_column`, with the standard ADBC Flight SQL and Flight SQL JDBC drivers. Logins are checked against `pg_hba.conf` and statements run through portals, so privileges, RLS and `pg_stat_statements` behave as they do over pgwire, which stays as it is.
- **PostGIS and pgvector over batches**, through kernel packs that declare which of their functions may be called on a batch's rows, without changing either extension.
These are first measurements, and partial ones; every answer was checked against DuckDB's. On one node, on ClickBench at 10 million rows, vexec in auto mode takes 19–24% off PostgreSQL's planner and 16–19% off ORCA by geometric mean, and the best queries run 8.8–9.5× faster, though a few got slower; Flight SQL delivers a million `lineitem` rows 4.7× faster than binary COPY and loads PAX 1.65× faster than COPY CSV. On four segments TPC-H Q1 runs 1.8–2.8× faster, but the first TPC run, with only scans and aggregates vectorized, averaged about 1×; the cluster's measurements with vectorized hash joins and Arrow frames through Motions are next.
None of it needed a new core patch: vexec also runs on vanilla PostgreSQL 19, and `gp_orca` now builds for it without `gp_core` (`-Dorca_single_node`), so ORCA plans vector nodes on a stock server too. It relies on what PostgreSQL added after 16: `heap_prepare_pagescan` (17), `explain_per_plan_hook` (18), and `planner_setup_hook`, `planner_shutdown_hook` and the batched visibility check `HeapTupleSatisfiesMVCCBatch` (19), none of which Cloudberry's `main` has. In a fork, each such capability would be one more set of kernel changes to carry across every PostgreSQL major.
The prototype compiles Cloudberry's own sources in place, so it builds on the community's work rather than replacing it.
### Implementation
Details: [Greenplum Without the Fork](https://github.com/igor-suhorukov/cloudberry/blob/extension_postgresql_19/pg19/doc/greenplum-without-the-fork.md). Code: [core series](https://github.com/igor-suhorukov/postgres/tree/REL_19_STABLE_CLOUDBERRY), [port](https://github.com/igor-suhorukov/cloudberry/tree/extension_postgresql_19/pg19), [build instructions](https://github.com/igor-suhorukov/cloudberry/tree/extension_postgresql_19/pg19#building).
The ground rule: a server with the core patches but without Cloudberry loaded must be indistinguishable from vanilla PostgreSQL 19, and this is measured.
- **Core series:** 24 commits, 49 files, +1,622/−41 lines. Existing extension points come first: hooks, table AMs, CustomScan, custom WAL resource managers, background workers, security labels and FDWs. Where they are not enough, a new hook is a NULL-by-default function pointer, a new exported function, a registration list or an off-by-default flag. No existing signature changes, no struct gains a field, and no catalog, WAL, page or protocol format changes.
- **Vanilla checks** compare Docker images built from the same PostgreSQL commit with and without the series:
- the same 371 `meson test` results;
- `abidiff`: 34 symbols added, 0 changed;
- byte-identical catalogs, settings, keywords and stored parse trees;
- identical query IDs;
- interchangeable data directories and streaming replication;
- cachegrind: −0.011% to +0.160% executed instructions;
- a script that checks every statement added inside an existing function sits behind a hook, registry or flag, with one reviewed exception.
- **Cloudberry's tree is untouched.** The whole port is one new directory, `pg19/`, plus a release workflow in `.github/workflows`. Cloudberry's files compile where they lie, so merges from Cloudberry keep applying, and a file that must change becomes a copy that says what changed. ORCA's 920 core files compile unmodified behind `planner_hook`, and PAX compiles in place with 57 of its 91 files unchanged.
- **Modules:**
- `gp_core`: dispatcher, Motions, interconnect, 2PC, distributed snapshots, GDD, FTS;
- `gp_orca`;
- `gp_sql`: Cloudberry syntax rewritten into PostgreSQL 19's grammar;
- storage: `gp_ao`, `pax`;
- external data: `gp_exttable`, `pxf_fdw`, `gpcloud`, `datalake_fdw`;
- `gp_resource`, `gp_security`, `gp_matview`, `gp_task`, `diskquota`;
- `gpfts` and gpMgmt's tools from `gpinitsystem` to `gpexpand`.
- **Built on the port since:** [pg_vexec](https://github.com/igor-suhorukov/pg_vexec), in its own repository: `vexec` (about 43,000 lines of C), `vexec_flight` (7,900) and the kernel packs for pgvector and PostGIS (1,000). vexec builds with PGXS both for vanilla PostgreSQL 19 and for the port. Its side of the port is module code in `pg19/`, about 13,600 lines with the `shm` transport and PAX's copies: gp_orca's API for vector nodes and `CCostModelVec`, gp_core's API 1.15 (a Motion rebuilt by another module, the distribution's hash functions, `squelch_subtree()`, figures in EXPLAIN ANALYZE), and PAX's and gp_ao's batch readers and sinks. vexec calls none of the core patches.
- **What changes for users:**
- migration is logical (dump and restore);
- settings are renamed `gp.*`, although `gpconfig` accepts the old names;
- DDL command tags become standard;
- the 20 shared catalogs are replaced by security labels, extension tables and a cluster file;
- TDE gives way to volume encryption.
### Rollout/Adoption Plan
Nothing here affects Cloudberry 2.x or the 3.0 plans: the prototype lives in my fork and changes no lines in Cloudberry's files. I would like to ask the community for four things:
1. **Feedback on the direction.** Is running Cloudberry as PostgreSQL extensions worth pursuing as a long-term path after 3.0?
2. **A home for `pg19/`.** It is one directory that tracks Cloudberry's sources without editing them. If the direction looks right, would the project consider hosting it, for example as an experimental branch or a separate repository?
3. **Help with the remaining tests.** The goal is to pass Cloudberry's original test suites with minimal changes to the tests. Every skipped test is listed with the statement that stops it, so tests are easy to pick up. `isolation2_schedule` now runs 113 of its 262 tests in both passes (60 when this was first posted); its 133 append-optimized tests and the 95 skipped tests of `greenplum_schedule` are next.
4. **A decision on Route B.** Should the port bring over Cloudberry's MPP variant of the PostgreSQL planner (30–45k lines), or keep shrinking ORCA's declines? On the regression suite, the port's own fallback reasons already fell from 563 to 52. A proposed `gp.mpp_planner` switch would allow side-by-side comparison, and I would value the view of the planner experts here.
For users, migration would be logical, similar to the cluster-to-cluster copy planned for 2.0 → 3.0 in [#1095](https://github.com/apache/cloudberry/discussions/1095). Separately, four of the custom hooks (table AM registry, smgr file events, parser hook and OID hook) are general enough to propose to PostgreSQL upstream. Support from Cloudberry developers as their real-world user would strengthen that case.
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
GitHub link: https://github.com/apache/cloudberry/discussions/2065
----
This is an automatically sent email for dev@cloudberry.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@cloudberry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: dev-help@cloudberry.apache.org
Re: [D] [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions [cloudberry]
Posted by "igor-suhorukov (via GitHub)" <gi...@apache.org>.
GitHub user igor-suhorukov edited a discussion: [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions
### Proposers
Igor Suhorukov ([@igor-suhorukov](https://github.com/igor-suhorukov))
### Proposal Status
Under Discussion
### Abstract
*Updated 2026-10-12: what has been built on the port since the first version, a vectorized executor and an Arrow Flight SQL endpoint (Motivation, Implementation), and current figures: 24 core commits, 873 test files, the progress of `isolation2`. The first comments refer to the version of 2026-09-30.*
I would like to discuss running Apache Cloudberry on PostgreSQL 19 as a set of extensions on top of 24 dormant core hooks, instead of a forked kernel. A working prototype exists; it is my personal experiment, not an ASF release. Without the modules, the patched server is measurably vanilla PostgreSQL 19: the same PostgreSQL tests pass, the ABI is unchanged, catalogs are byte-identical, and the hooks add less than 0.2% to executed instructions. With them, it is a Greenplum-style MPP engine with plan slices, Motions, two-phase commit and distributed snapshots, running unmodified ORCA and stock PostGIS and pgvector. ORCA plans all 121 TPC-H and TPC-DS queries at SF1 with answers matching DuckDB's, and 744 of Cloudberry's 1,180 test files pass so far.
The port gives users more than a newer PostgreSQL. With Cloudberry's components as modules, new capabilities arrive as further extensions. [pg_vexec](https://github.com/igor-suhorukov/pg_vexec), built on the port without a single new core patch, adds vectorized execution that ORCA plans by cost, PAX and `ao_column` tables read and written by column, Arrow frames between segments, and an Arrow Flight SQL endpoint for BI and ML clients.
Claude Code wrote the code, the port over 12 days and 629 commits and pg_vexec over three days; I set the goals and signed off every change to the PostgreSQL core.
### Motivation
Greenplum and Cloudberry have always followed PostgreSQL a few releases behind: Greenplum 6 shipped on PostgreSQL 9.4, Greenplum 7 on 12, Cloudberry 2.x on 14, and `main` reached 16.9 in May 2026. The reason is structural. Cloudberry edits 882 PostgreSQL backend and header files, so every new PostgreSQL major is a large merge. [#1095](https://github.com/apache/cloudberry/discussions/1095) estimated the 14 → 16 upgrade at 4–5 months, three of them for fixing regression tests, and noted that PostgreSQL 14, the base of Cloudberry 2.x, reaches end of life in 2026. Meanwhile PostgreSQL 17 and 18 have shipped, and 19 is close to release.
In [#1877](https://github.com/apache/cloudberry/discussions/1877), the project named close compatibility with modern PostgreSQL as one of its goals. The prototype tests whether that goal can become cheap to sustain. If Cloudberry runs as extensions, the next PostgreSQL major means rebasing a small patch series instead of merging a fork. Users get a current PostgreSQL, including PostgreSQL 19 features such as eager aggregation and parallel autovacuum. They also get stock extensions: upstream PostGIS 3.7 instead of a PostGIS fork, and pgvector 0.8.6 built unpatched, which is relevant to [#2043](https://github.com/apache/cloudberry/discussions/2043).
**More than a newer PostgreSQL.** A current base is the first gain, not the only one. As modules, Cloudberry's planner, storage and interconnect can be combined with other extensions, so new capabilities arrive as extensions rather than as more changes to a fork. Apache Cloudberry's code has no vectorized executor: only traces of a closed-source engine remain, a `create_vectorization_plan` flag the planner always passes as false, a `WindowHashAgg` node with no executor behind it, and a PAX adapter under `VEC_BUILD` that no longer compiles. [pg_vexec](https://github.com/igor-suhorukov/pg_vexec) was built on top of the port in three days of implementation ([write-up](https://igor-suhorukov.github.io/blog/2026/10/07/vectorized-postgresql-extension/)), and it brings Cloudberry users:
- **Vectorized execution, chosen by cost.** Batches of up to 1,024 rows by column replace the row-at-a-time loop on the coordinator and on every segment. ORCA weighs vector nodes in its search (`CCostModelVec`) and its translator builds them; PostgreSQL's planner gets them through its path hooks. No finished plan is rewritten, and a fallback to PostgreSQL's expression evaluator inside the same node keeps PostgreSQL's semantics.
- **Columnar storage without rows.** PAX and `ao_column` hand batches to the executor and take them back on insert. `count`, `min`, `max`, `sum` and `avg` without conditions come from PAX's statistics: `count(*)` over 10 million rows takes 0.33 ms instead of 265.
- **Arrow between segments.** Motions carry Arrow IPC frames, routed by a vectorized `cdbhash` that reproduces `GpHashSegment` bit for bit, and a new `shm` interconnect passes them through shared memory between the segments of one host.
- **Arrow Flight SQL.** `vexec_flight` serves results as Arrow record batches straight from the executor, and `adbc_ingest()` loads DataFrames into tables, by column into PAX and `ao_column`, with the standard ADBC Flight SQL and Flight SQL JDBC drivers. Logins are checked against `pg_hba.conf` and statements run through portals, so privileges, RLS and `pg_stat_statements` behave as they do over pgwire, which stays as it is.
- **PostGIS and pgvector over batches**, through kernel packs that declare which of their functions may be called on a batch's rows, without changing either extension.
These are first measurements, and partial ones; every answer was checked against DuckDB's. On one node, on ClickBench at 10 million rows, vexec in auto mode takes 19–24% off PostgreSQL's planner and 16–19% off ORCA by geometric mean, and the best queries run 8.8–9.5× faster, though a few got slower; Flight SQL delivers a million `lineitem` rows 4.7× faster than binary COPY and loads PAX 1.65× faster than COPY CSV. On four segments TPC-H Q1 runs 1.8–2.8× faster, but the first TPC run, with only scans and aggregates vectorized, averaged about 1×; the cluster's measurements with vectorized hash joins and Arrow frames through Motions are next.
None of it needed a new core patch: vexec also runs on vanilla PostgreSQL 19, and `gp_orca` now builds for it without `gp_core` (`-Dorca_single_node`), so ORCA plans vector nodes on a stock server too. It relies on what PostgreSQL added after 16: `heap_prepare_pagescan` (17), `explain_per_plan_hook` (18), and `planner_setup_hook`, `planner_shutdown_hook` and the batched visibility check `HeapTupleSatisfiesMVCCBatch` (19), none of which Cloudberry's `main` has. In a fork, each such capability would be one more set of kernel changes to carry across every PostgreSQL major.
The prototype compiles Cloudberry's own sources in place, so it builds on the community's work rather than replacing it.
### Implementation
Details: [Greenplum Without the Fork](https://github.com/igor-suhorukov/cloudberry/blob/extension_postgresql_19/pg19/doc/greenplum-without-the-fork.md). Code: [core series](https://github.com/igor-suhorukov/postgres/tree/REL_19_STABLE_CLOUDBERRY), [port](https://github.com/igor-suhorukov/cloudberry/tree/extension_postgresql_19/pg19), [build instructions](https://github.com/igor-suhorukov/cloudberry/tree/extension_postgresql_19/pg19#building).
The ground rule: a server with the core patches but without Cloudberry loaded must be indistinguishable from vanilla PostgreSQL 19, and this is measured.
- **Core series:** 24 commits, 49 files, +1,622/−41 lines. Existing extension points come first: hooks, table AMs, CustomScan, custom WAL resource managers, background workers, security labels and FDWs. Where they are not enough, a new hook is a NULL-by-default function pointer, a new exported function, a registration list or an off-by-default flag. No existing signature changes, no struct gains a field, and no catalog, WAL, page or protocol format changes.
- **Vanilla checks** compare Docker images built from the same PostgreSQL commit with and without the series:
- the same 371 `meson test` results;
- `abidiff`: 34 symbols added, 0 changed;
- byte-identical catalogs, settings, keywords and stored parse trees;
- identical query IDs;
- interchangeable data directories and streaming replication;
- cachegrind: −0.011% to +0.160% executed instructions;
- a script that checks every statement added inside an existing function sits behind a hook, registry or flag, with one reviewed exception.
- **Cloudberry's tree is untouched.** The whole port is one new directory, `pg19/`, plus a release workflow in `.github/workflows`. Cloudberry's files compile where they lie, so merges from Cloudberry keep applying, and a file that must change becomes a copy that says what changed. ORCA's 920 core files compile unmodified behind `planner_hook`, and PAX compiles in place with 57 of its 91 files unchanged.
- **Modules:**
- `gp_core`: dispatcher, Motions, interconnect, 2PC, distributed snapshots, GDD, FTS;
- `gp_orca`;
- `gp_sql`: Cloudberry syntax rewritten into PostgreSQL 19's grammar;
- storage: `gp_ao`, `pax`;
- external data: `gp_exttable`, `pxf_fdw`, `gpcloud`, `datalake_fdw`;
- `gp_resource`, `gp_security`, `gp_matview`, `gp_task`, `diskquota`;
- `gpfts` and gpMgmt's tools from `gpinitsystem` to `gpexpand`.
- **Built on the port since:** [pg_vexec](https://github.com/igor-suhorukov/pg_vexec), in its own repository: `vexec` (about 43,000 lines of C), `vexec_flight` (7,900) and the kernel packs for pgvector and PostGIS (1,000). vexec builds with PGXS both for vanilla PostgreSQL 19 and for the port. Its side of the port is module code in `pg19/`, about 13,600 lines with the `shm` transport and PAX's copies: gp_orca's API for vector nodes and `CCostModelVec`, gp_core's API 1.15 (a Motion rebuilt by another module, the distribution's hash functions, `squelch_subtree()`, figures in EXPLAIN ANALYZE), and PAX's and gp_ao's batch readers and sinks. vexec calls none of the core patches.
- **What changes for users:**
- migration is logical (dump and restore);
- settings are renamed `gp.*`, although `gpconfig` accepts the old names;
- DDL command tags become standard;
- the 20 shared catalogs are replaced by security labels, extension tables and a cluster file;
- TDE gives way to volume encryption.
### Rollout/Adoption Plan
Nothing here affects Cloudberry 2.x or the 3.0 plans: the prototype lives in my fork and changes no lines in Cloudberry's files. I would like to ask the community for four things:
1. **Feedback on the direction.** Is running Cloudberry as PostgreSQL extensions worth pursuing as a long-term path after 3.0?
2. **A home for `pg19/`.** It is one directory that tracks Cloudberry's sources without editing them. If the direction looks right, would the project consider hosting it, for example as an experimental branch or a separate repository?
3. **Help with the remaining tests.** The goal is to pass Cloudberry's original test suites with minimal changes to the tests. Every skipped test is listed with the statement that stops it, so tests are easy to pick up. `isolation2_schedule` now runs 113 of its 262 tests in both passes (60 when this was first posted); its 133 append-optimized tests and the 95 skipped tests of `greenplum_schedule` are next.
4. **A decision on Route B.** Should the port bring over Cloudberry's MPP variant of the PostgreSQL planner (30–45k lines), or keep shrinking ORCA's declines? On the regression suite, the port's own fallback reasons already fell from 563 to 52. A proposed `gp.mpp_planner` switch would allow side-by-side comparison, and I would value the view of the planner experts here.
For users, migration would be logical, similar to the cluster-to-cluster copy planned for 2.0 → 3.0 in [#1095](https://github.com/apache/cloudberry/discussions/1095). Separately, four of the custom hooks (table AM registry, smgr file events, parser hook and OID hook) are general enough to propose to PostgreSQL upstream. Support from Cloudberry developers as their real-world user would strengthen that case.
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
GitHub link: https://github.com/apache/cloudberry/discussions/2065
----
This is an automatically sent email for dev@cloudberry.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@cloudberry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: dev-help@cloudberry.apache.org
Re: [D] [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions [cloudberry]
Posted by "igor-suhorukov (via GitHub)" <gi...@apache.org>.
GitHub user igor-suhorukov added a comment to the discussion: [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions
@yjhjstz @leborchuk thank you for sharing your expert opinion!
@yjhjstz thanks, point 2 is fair, so I took it as a test plan.
Today the port runs 60 of the 262 tests in isolation2_schedule, plus 14 of the 15 parallel retrieve cursor tests. These are the ones on distributed transactions, snapshots, locks, GDD, FTS
and mirrors, and each runs under the planner and under ORCA.
Among them, reader_waits_for_lock reaches XactAdoptTransactionState(), under ORCA. distributed_snapshot, distributedlog-bug and gdd/concurrent_update reach the hidden-commit path.
vacuum_recently_dead_tuple_due_to_distributed_snapshot reaches the slot.
None of Cloudberry's tests reaches a segment waiting on a prepared part. Cloudberry's coordinator PANICs rather than end a transaction with a part still prepared, so its tests never get
there. I added tests for it.
Added after your comment:
1. igor-suhorukov/cloudberry@3f2b6df967a: fix. A lost wake-up of the recovery process could leave a prepared part, and every reader on its segment, waiting 5 minutes.
2. igor-suhorukov/cloudberry@d054391a681: isolation2 port/dtx_wait_prepared. A reader waits on a segment for a prepared part (336 s before the fix, 35 s after).
3. igor-suhorukov/cloudberry@3dbbc77a1b0: isolation2 port/dtx_horizon. The VACUUM test runs with and without the slot (38/37/25 rows vs 0/37/25), as a negative control.
4. igor-suhorukov/cloudberry@76bb3ae76ef: combocid_readers. combocid_gp's checks, counted, with ORCA's readers asserted.
5. igor-suhorukov/cloudberry@ac1eb409d76: dtm_retry's skip reason corrected.
Full run on that commit: isolation2 80/80 in both passes, greenplum_schedule 392/392 and 394/394.
Still to do: the other 200 isolation2 tests (133 of them append-optimized) and latency and concurrency benchmarks (your point 3).
GitHub link: https://github.com/apache/cloudberry/discussions/2065#discussioncomment-18708218
----
This is an automatically sent email for dev@cloudberry.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@cloudberry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: dev-help@cloudberry.apache.org
Re: [D] [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
GitHub user yjhjstz added a comment to the discussion: [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions
Hi @igor-suhorukov,
Thanks for this impressive experiment. It shows something many of us assumed was impossible: distributed snapshots, 2PC and shared reader/writer transactions running on a small set of dormant hooks, with ORCA compiled unmodified behind `planner_hook`. I agree that an extension-based Cloudberry is the right long-term direction. Following PostgreSQL a few majors behind is a structural problem, and this is the most credible answer to it I have seen.
That said, I don't think the community should adopt the prototype as it is, for five reasons:
1. **It is a different product, not the next version.** Upgrading requires dump/restore, GUCs are renamed to `gp.*`, the shared catalogs and TDE are gone, and DDL command tags change. For existing 2.x users that is a migration to a new system, which makes it hard to position as Cloudberry's successor.
2. **The riskiest parts are not yet verified.** Transaction semantics rest on new mechanisms: `XactAdoptTransactionState()`, segments waiting on prepared transactions, and a replication slot holding back VACUUM. `isolation2`, the suite that exercises exactly these paths, has not passed yet, and overall coverage is 691 of 1,180 test files. Until those pass under concurrency, correctness is an open question.
3. **The performance cost is not measured.** Dispatching each slice as `SELECT gp_internal.exec_fragment(...)` over libpq, and passing the snapshot as `SET LOCAL`, adds overhead on every query. The TPC results (SF1, one container, parallel off) show correct answers, not performance. We would need short-query latency, high-concurrency throughput and larger-scale numbers before comparing it with the native dispatcher.
4. **It is still a fork, only a smaller one.** Of the 22 hooks, four look upstreamable. The other 18 mean maintaining a patched PostgreSQL indefinitely, so users still can't run on stock or managed PostgreSQL. The maintenance burden shrinks but does not go away.
5. **Code ownership.** About 600 commits written in 12 days is a remarkable result, but the community would inherit a ~33k-line translator and a new transaction layer that nobody here knows deeply yet. As @leborchuk said, rebasing and long-term evolution depend on people who own and understand the code.
My suggestions:
1. **Evolve from `main` incrementally instead of replacing it.** Start with modules that have clear boundaries and need no deep kernel changes, such as AO/AOCS as table AMs, external tables as FDWs, and the `gp_sql` syntax layer. Use `pg19/` as the reference implementation and requirements list, not as a drop-in replacement.
2. **Upstream the general-purpose hooks first.** The table AM registry, smgr file events, parser hook and OID hook would benefit PostgreSQL broadly, and Cloudberry can back them as a real-world user. Every hook accepted upstream lowers the cost of the extension approach for good.
3. **Validate the transaction layer before going further.** Get `isolation2` passing and publish short-query latency and concurrency benchmarks against the native dispatcher. That data should decide whether the MPP core can move to extensions, not the other way round.
4. **On Route B: don't port it; keep reducing ORCA fallbacks.** Porting 30–45k lines of the MPP planner would recreate the very merge burden this effort is trying to remove. Fallback reductions (multi-level partitions, aggregate `FILTER`, `LATERAL`, non-default collations) benefit both `main` and the extension port. ORCA is also the natural first shared component, since its core does not depend on PostgreSQL headers and only the translator tracks the PostgreSQL version.
5. **Host it as a separate experimental repository.** Something like `cloudberry-pg-ext` keeps the work visible and open to contributors without implying it is the official roadmap. That separation can be revisited once items 2 and 3 have results.
GitHub link: https://github.com/apache/cloudberry/discussions/2065#discussioncomment-18701062
----
This is an automatically sent email for dev@cloudberry.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@cloudberry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: dev-help@cloudberry.apache.org
Re: [D] [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions [cloudberry]
Posted by "roseduan (via GitHub)" <gi...@apache.org>.
GitHub user roseduan added a comment to the discussion: [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions
This would be an enormous undertaking. Cloudberry is built by embedding lot of code directly into the PostgreSQL kernel. Some components, such as ORCA, AO/AOCS, and the interconnect, could potentially be decoupled, but much of the remaining code is deeply intertwined and would be very difficult to separate.
AI tools might help with this effort, but it would still require a long-term commitment and significant ongoing resources to maintain. I’m also concerned that a codebase of this scale, with millions of lines of code, may exceed what current AI coding tools can handle effectively.
GitHub link: https://github.com/apache/cloudberry/discussions/2065#discussioncomment-18895239
----
This is an automatically sent email for dev@cloudberry.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@cloudberry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: dev-help@cloudberry.apache.org
Re: [D] [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions [cloudberry]
Posted by "igor-suhorukov (via GitHub)" <gi...@apache.org>.
GitHub user igor-suhorukov edited a comment on the discussion: [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions
@MisterRaindrop thank you for initiating a substantive discussion regarding the proposal.
The three interfaces vexec uses are not specific to it, and a DataFusion engine could be their second user: [`vexec_source.h`](https://github.com/igor-suhorukov/cloudberry/blob/extension_postgresql_19/pg19/include/vexec_source.h) (PAX and `ao_column` hand over column batches in Arrow layouts), [`gp_orca_vec.h`](https://github.com/igor-suhorukov/cloudberry/blob/extension_postgresql_19/pg19/include/gp_orca_vec.h) (ORCA prices and places an engine's nodes) and [`gp_core_api.h`](https://github.com/igor-suhorukov/cloudberry/blob/extension_postgresql_19/pg19/include/gp_core_api.h) (routing batches across Motions).
1. The 24 upstream core commits fall into three groups.
- General: [`new_oid_hook`](https://github.com/igor-suhorukov/postgres/commit/7e2a0be6218), [`raw_parser_hook`](https://github.com/igor-suhorukov/postgres/commit/480f3128f87), [`smgr_file_event_hook`](https://github.com/igor-suhorukov/postgres/commit/a91cbb5aaf0), and a [registry](https://github.com/igor-suhorukov/postgres/commit/21f45bdde60) of extra table access method routines with five uses: [reloptions](https://github.com/igor-suhorukov/postgres/commit/112462d155c), [the columns a scan reads](https://github.com/igor-suhorukov/postgres/commit/3a2d4fcc1c2), [a unique index's probe](https://github.com/igor-suhorukov/postgres/commit/150343312fe), [ADD COLUMN without a rewrite](https://github.com/igor-suhorukov/postgres/commit/93eb6501b97) and [BRIN over sparse block numbers](https://github.com/igor-suhorukov/postgres/commit/5ac8dbfc9ab).
- A general problem, but another shape upstream: [`analyze_sample_rows_hook`](https://github.com/igor-suhorukov/postgres/commit/6f8d26d16c2), [`memory_block_alloc_hook`](https://github.com/igor-suhorukov/postgres/commit/41107a43583), [materialized view maintenance](https://github.com/igor-suhorukov/postgres/commit/a903759f70d) and [an injection point](https://github.com/igor-suhorukov/postgres/commit/4026e083a14).
- Specific to an MPP engine: a transaction handed to reader processes and the order of its commit ([`XactAdoptTransactionState()`](https://github.com/igor-suhorukov/postgres/commit/60e82c61d7c), [combo command ID hooks](https://github.com/igor-suhorukov/postgres/commit/791fd7171a0), [`xact_commit_recorded_hook`](https://github.com/igor-suhorukov/postgres/commit/1853d9244f0), [`XactAbortAgainAfterEnd`](https://github.com/igor-suhorukov/postgres/commit/871f0c5102e)); [`query_lockmode_hook`](https://github.com/igor-suhorukov/postgres/commit/8c908a24f4c); [`physical_slot_restart_lsn_hook`](https://github.com/igor-suhorukov/postgres/commit/bea8a637438); [tablespace location hooks](https://github.com/igor-suhorukov/postgres/commit/9b351c13e72); [pg_checksums' marks](https://github.com/igor-suhorukov/postgres/commit/b213d194740); and Cloudberry's syntax ([`gp_segment_id`](https://github.com/igor-suhorukov/postgres/commit/f26d74cc863), [`gp_dist_random()`](https://github.com/igor-suhorukov/p
ostgres/commit/aeae8a306d2), [columns hidden from `*`](https://github.com/igor-suhorukov/postgres/commit/19b27fd7687)).
2. Following `main`: The port edits no Cloudberry file, so `git merge main` cannot conflict. For the port's 244 modified copies, a new script (not pushed yet) does a three-way merge with the original at the last branch or merge point and the original in `main`. A clean merge is applied automatically; a conflict goes to an LLM and then to a reviewer. On today's `main`, 54 commits ahead, 13 copies were affected: 7 merged automatically, 1 already had the change, and 5 had conflicts, which the LLM resolved.
3. Yes, of course I plan to. A benchmark that others can reproduce has to run in a public cloud, and I have no prepaid AWS access. Given access to an AWS account, I will run all of it: Motion-heavy joins, concurrency, spill and failure recovery. I would do it after the port passes as many of Cloudberry's tests as it can and vexec gets a little more work, so that the measurements include it.
4. I would prefer an experimental branch in the project's main repository. `pg19/` compiles Cloudberry's files in place, so it has to sit inside Cloudberry's tree, and on a branch, following `main` is a plain `git merge main`. And a repository for Postgres kernel 24 hooks is also needed.
GitHub link: https://github.com/apache/cloudberry/discussions/2065#discussioncomment-18906162
----
This is an automatically sent email for dev@cloudberry.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@cloudberry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: dev-help@cloudberry.apache.org
Re: [D] [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions [cloudberry]
Posted by "igor-suhorukov (via GitHub)" <gi...@apache.org>.
GitHub user igor-suhorukov edited a discussion: [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions
### Proposers
Igor Suhorukov ([@igor-suhorukov](https://github.com/igor-suhorukov))
### Proposal Status
Under Discussion
### Abstract
*Updated 2026-10-14: security updates at PostgreSQL's pace (Motivation), what has been built on the port since the first version, a vectorized executor and an Arrow Flight SQL endpoint (Motivation, Implementation), and current figures: 24 core commits, 969 test files, the progress of `isolation2` (Rollout, point 3). The first comments refer to the version of 2026-09-30.*
I would like to discuss running Apache Cloudberry on PostgreSQL 19 as a set of extensions on top of 24 dormant core hooks, instead of a forked kernel. A working prototype exists; it is my personal experiment, not an ASF release. Without the modules, the patched server is measurably vanilla PostgreSQL 19: the same PostgreSQL tests pass, the ABI is unchanged, catalogs are byte-identical, and the hooks add less than 0.2% to executed instructions. With them, it is a Greenplum-style MPP engine with plan slices, Motions, two-phase commit and distributed snapshots, running unmodified ORCA and stock PostGIS and pgvector. ORCA plans all 121 TPC-H and TPC-DS queries at SF1 with answers matching DuckDB's, and 969 of Cloudberry's 1,180 test files pass so far.
The port gives users more than a newer PostgreSQL. With Cloudberry's components as modules, new capabilities arrive as further extensions. [pg_vexec](https://github.com/igor-suhorukov/pg_vexec), built on the port without a single new core patch, adds vectorized execution that ORCA plans by cost, PAX and `ao_column` tables read and written by column, Arrow frames between segments, and an Arrow Flight SQL endpoint for BI and ML clients.
Claude Code wrote the code, the port over 12 days and 629 commits and pg_vexec over three days; I set the goals and signed off every change to the PostgreSQL core.
### Motivation
**Security fixes at PostgreSQL's pace.** The port leaves PostgreSQL's code as upstream wrote it: its 24 core commits sit on top of PostgreSQL's own `REL_19_STABLE` branch, and no Cloudberry code lives in PostgreSQL's tree. A PostgreSQL minor release, with its security fixes, is then a rebase of those 24 commits and a rebuild. This can run unattended as a scheduled GitHub Actions job: fetch the new release tag, rebase the series, run the comparison with vanilla PostgreSQL and the test suites, publish packages. The port's release workflow already builds and smoke-tests packages from any PostgreSQL ref. As a trial, on 2026-10-13 the series was rebased onto the head of `REL_19_STABLE`, 153 upstream commits past its base, and all 24 commits applied without a conflict. Users would get a PostgreSQL security release as soon as that job passes. In a fork, each fix waits for a manual backport: `main` is based on PostgreSQL 16.9 of May 2025, and the security fixes of the six PostgreSQL 16 minor
releases since reach it as individual cherry-picks, 25 commits for 16.15 alone.
Greenplum and Cloudberry have always followed PostgreSQL a few releases behind: Greenplum 6 shipped on PostgreSQL 9.4, Greenplum 7 on 12, Cloudberry 2.x on 14, and `main` reached 16.9 in May 2026. The reason is structural. Cloudberry edits 882 PostgreSQL backend and header files, so every new PostgreSQL major is a large merge. [#1095](https://github.com/apache/cloudberry/discussions/1095) estimated the 14 → 16 upgrade at 4–5 months, three of them for fixing regression tests, and noted that PostgreSQL 14, the base of Cloudberry 2.x, reaches end of life in 2026. Meanwhile PostgreSQL 17 and 18 have shipped, and 19 is close to release.
In [#1877](https://github.com/apache/cloudberry/discussions/1877), the project named close compatibility with modern PostgreSQL as one of its goals. The prototype tests whether that goal can become cheap to sustain. If Cloudberry runs as extensions, the next PostgreSQL major means rebasing a small patch series instead of merging a fork. Users get a current PostgreSQL, including PostgreSQL 19 features such as eager aggregation and parallel autovacuum. They also get stock extensions: upstream PostGIS 3.7 instead of a PostGIS fork, and pgvector 0.8.6 built unpatched, which is relevant to [#2043](https://github.com/apache/cloudberry/discussions/2043).
**More than a newer PostgreSQL.** A current base is the first gain, not the only one. As modules, Cloudberry's planner, storage and interconnect can be combined with other extensions, so new capabilities arrive as extensions rather than as more changes to a fork. Apache Cloudberry's code has no vectorized executor: only traces of a closed-source engine remain, a `create_vectorization_plan` flag the planner always passes as false, a `WindowHashAgg` node with no executor behind it, and a PAX adapter under `VEC_BUILD` that no longer compiles. [pg_vexec](https://github.com/igor-suhorukov/pg_vexec) was built on top of the port in three days of implementation ([write-up](https://igor-suhorukov.github.io/blog/2026/10/07/vectorized-postgresql-extension/)), and it brings Cloudberry users:
- **Vectorized execution, chosen by cost.** Batches of up to 1,024 rows by column replace the row-at-a-time loop on the coordinator and on every segment. ORCA weighs vector nodes in its search (`CCostModelVec`) and its translator builds them; PostgreSQL's planner gets them through its path hooks. No finished plan is rewritten, and a fallback to PostgreSQL's expression evaluator inside the same node keeps PostgreSQL's semantics.
- **Columnar storage without rows.** PAX and `ao_column` hand batches to the executor and take them back on insert. `count`, `min`, `max`, `sum` and `avg` without conditions come from PAX's statistics: `count(*)` over 10 million rows takes 0.33 ms instead of 265.
- **Arrow between segments.** Motions carry Arrow IPC frames, routed by a vectorized `cdbhash` that reproduces `GpHashSegment` bit for bit, and a new `shm` interconnect passes them through shared memory between the segments of one host.
- **Arrow Flight SQL.** `vexec_flight` serves results as Arrow record batches straight from the executor, and `adbc_ingest()` loads DataFrames into tables, by column into PAX and `ao_column`, with the standard ADBC Flight SQL and Flight SQL JDBC drivers. Logins are checked against `pg_hba.conf` and statements run through portals, so privileges, RLS and `pg_stat_statements` behave as they do over pgwire, which stays as it is.
- **PostGIS and pgvector over batches**, through kernel packs that declare which of their functions may be called on a batch's rows, without changing either extension.
These are first measurements, and partial ones; every answer was checked against DuckDB's. On one node, on ClickBench at 10 million rows, vexec in auto mode takes 19–24% off PostgreSQL's planner and 16–19% off ORCA by geometric mean, and the best queries run 8.8–9.5× faster, though a few got slower; Flight SQL delivers a million `lineitem` rows 4.7× faster than binary COPY and loads PAX 1.65× faster than COPY CSV. On four segments TPC-H Q1 runs 1.8–2.8× faster, but the first TPC run, with only scans and aggregates vectorized, averaged about 1×; the cluster's measurements with vectorized hash joins and Arrow frames through Motions are next.
None of it needed a new core patch: vexec also runs on vanilla PostgreSQL 19, and `gp_orca` now builds for it without `gp_core` (`-Dorca_single_node`), so ORCA plans vector nodes on a stock server too. It relies on what PostgreSQL added after 16: `heap_prepare_pagescan` (17), `explain_per_plan_hook` (18), and `planner_setup_hook`, `planner_shutdown_hook` and the batched visibility check `HeapTupleSatisfiesMVCCBatch` (19), none of which Cloudberry's `main` has. In a fork, each such capability would be one more set of kernel changes to carry across every PostgreSQL major.
The prototype compiles Cloudberry's own sources in place, so it builds on the community's work rather than replacing it.
### Implementation
Details: [Greenplum Without the Fork](https://github.com/igor-suhorukov/cloudberry/blob/extension_postgresql_19/pg19/doc/greenplum-without-the-fork.md). Code: [core series](https://github.com/igor-suhorukov/postgres/tree/REL_19_STABLE_CLOUDBERRY), [port](https://github.com/igor-suhorukov/cloudberry/tree/extension_postgresql_19/pg19), [build instructions](https://github.com/igor-suhorukov/cloudberry/tree/extension_postgresql_19/pg19#building).
The ground rule: a server with the core patches but without Cloudberry loaded must be indistinguishable from vanilla PostgreSQL 19, and this is measured.
- **Core series:** 24 commits, 49 files, +1,622/−41 lines. Existing extension points come first: hooks, table AMs, CustomScan, custom WAL resource managers, background workers, security labels and FDWs. Where they are not enough, a new hook is a NULL-by-default function pointer, a new exported function, a registration list or an off-by-default flag. No existing signature changes, no struct gains a field, and no catalog, WAL, page or protocol format changes.
- **Vanilla checks** compare Docker images built from the same PostgreSQL commit with and without the series:
- the same 371 `meson test` results;
- `abidiff`: 34 symbols added, 0 changed;
- byte-identical catalogs, settings, keywords and stored parse trees;
- identical query IDs;
- interchangeable data directories and streaming replication;
- cachegrind: −0.011% to +0.160% executed instructions;
- a script that checks every statement added inside an existing function sits behind a hook, registry or flag, with one reviewed exception.
- **Cloudberry's tree is untouched.** The whole port is one new directory, `pg19/`, plus a release workflow in `.github/workflows`. Cloudberry's files compile where they lie, so merges from Cloudberry keep applying, and a file that must change becomes a copy that says what changed. ORCA's 920 core files compile unmodified behind `planner_hook`, and PAX compiles in place with 57 of its 91 files unchanged.
- **Modules:**
- `gp_core`: dispatcher, Motions, interconnect, 2PC, distributed snapshots, GDD, FTS;
- `gp_orca`;
- `gp_sql`: Cloudberry syntax rewritten into PostgreSQL 19's grammar;
- storage: `gp_ao`, `pax`;
- external data: `gp_exttable`, `pxf_fdw`, `gpcloud`, `datalake_fdw`;
- `gp_resource`, `gp_security`, `gp_matview`, `gp_task`, `diskquota`;
- `gpfts` and gpMgmt's tools from `gpinitsystem` to `gpexpand`.
- **Built on the port since:** [pg_vexec](https://github.com/igor-suhorukov/pg_vexec), in its own repository: `vexec` (about 43,000 lines of C), `vexec_flight` (7,900) and the kernel packs for pgvector and PostGIS (1,000). vexec builds with PGXS both for vanilla PostgreSQL 19 and for the port. Its side of the port is module code in `pg19/`, about 13,600 lines with the `shm` transport and PAX's copies: gp_orca's API for vector nodes and `CCostModelVec`, gp_core's API 1.15 (a Motion rebuilt by another module, the distribution's hash functions, `squelch_subtree()`, figures in EXPLAIN ANALYZE), and PAX's and gp_ao's batch readers and sinks. vexec calls none of the core patches.
- **What changes for users:**
- migration is logical (dump and restore);
- settings are renamed `gp.*`, although `gpconfig` accepts the old names;
- DDL command tags become standard;
- the 20 shared catalogs are replaced by security labels, extension tables and a cluster file;
- TDE gives way to volume encryption.
### Rollout/Adoption Plan
Nothing here affects Cloudberry 2.x or the 3.0 plans: the prototype lives in my fork and changes no lines in Cloudberry's files. I would like to ask the community for four things:
1. **Feedback on the direction.** Is running Cloudberry as PostgreSQL extensions worth pursuing as a long-term path after 3.0?
2. **A home for `pg19/`.** It is one directory that tracks Cloudberry's sources without editing them. If the direction looks right, would the project consider hosting it, for example as an experimental branch or a separate repository?
3. **Help with the remaining tests.** The goal is to pass Cloudberry's original test suites with minimal changes to the tests. Every skipped test is listed with the statement that stops it, so tests are easy to pick up. Where the tests stand: 969 of Cloudberry's 1,180 test files pass, 82% (744 on 2026-10-08).
For users, migration would be logical, similar to the cluster-to-cluster copy planned for 2.0 → 3.0 in [#1095](https://github.com/apache/cloudberry/discussions/1095). Separately, four of the custom hooks (table AM registry, smgr file events, parser hook and OID hook) are general enough to propose to PostgreSQL upstream. Support from Cloudberry developers as their real-world user would strengthen that case.
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
GitHub link: https://github.com/apache/cloudberry/discussions/2065
----
This is an automatically sent email for dev@cloudberry.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@cloudberry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: dev-help@cloudberry.apache.org
Re: [D] [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions [cloudberry]
Posted by "igor-suhorukov (via GitHub)" <gi...@apache.org>.
GitHub user igor-suhorukov added a comment to the discussion: [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions
@leborchuk thanks! I'm coming back to the ClickBench part now that there are numbers.
> PG is not good enough to beat clickhouse in clickbench … mainly because clickhouse constantly compact (sort) data in background
Agreed. So instead of competing with MergeTree, I put MergeTree inside PostgreSQL:
- [ch_table_engines](https://github.com/igor-suhorukov/pg_ch_table_engines) runs ClickHouse's own MergeTree family (v26.8 LTS, linked from its sources and not copied) as PostgreSQL 19 table access methods. Background merge workers sort and merge the parts. PostgreSQL keeps MVCC, WAL, indexes, replicas and pg_dump.
- [pg_vexec](https://github.com/igor-suhorukov/pg_vexec) reads those columns in 1,024-row batches. This is where the GROUP BY work you mention lives: vectorized aggregation partitioned in shared memory across parallel workers, and grouping by dictionary on LowCardinality columns.
- Neither needs a core patch, so there is no kernel to rebase for them. The same library also runs on the pg19 port of Cloudberry, on one node and on a cluster. Each segment keeps and merges its own parts.
ClickBench, 10M rows, same laptop, geometric mean of hot runs ([write-up](https://igor-suhorukov.github.io/blog/2026/10/21/my-postgresql-is-faster-than-yours/)):
| | geomean | vs ClickHouse |
| ------------------------------------------- | ------: | ------------: |
| PostgreSQL 19, heap | 715 ms | ~39× |
| PostgreSQL 19 + ch_table_engines + pg_vexec | 49.9 ms | ~3× |
| ClickHouse 26.9 | 16.8 ms | 1× |
That isn't beating ClickHouse yet. Most of the remaining gap is fixed per-query cost: backend planning and starting parallel workers. The light queries are 4–18× slower (Q1, Q19), but the heavy GROUP BY queries are within 1.4–1.9× (Q20, Q28, Q33). This is one host and 10M rows. Multi-host and 100M-row runs are still to do.
I'd prefer to keep the standard ClickBench, with answers checked against DuckDB, over writing our own benchmark. I'm happy to open a separate discussion on bringing this to Cloudberry.
GitHub link: https://github.com/apache/cloudberry/discussions/2065#discussioncomment-19092936
----
This is an automatically sent email for dev@cloudberry.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@cloudberry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: dev-help@cloudberry.apache.org
Re: [D] [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
GitHub user MisterRaindrop added a comment to the discussion: [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions
Thank you for sharing this work. I find this a very interesting and potentially important direction for Cloudberry.
The most valuable part, in my view, is not only moving Cloudberry to PostgreSQL 19. It is making the boundary between PostgreSQL and Cloudberry’s MPP capabilities much more explicit. If the dispatcher, Motions, distributed transactions and snapshots, ORCA, and the storage engines can live behind a relatively small and well-defined set of hooks, this could significantly reduce the cost of following future PostgreSQL releases.
I am currently exploring DataFusion-based vectorized execution in Cloudberry, so this modular boundary is especially interesting to me. In our current direction, Cloudberry still owns planning, slices, Motions, and distributed execution, while supported fragments are executed by DataFusion. Your work suggests a broader possibility: execution engines such as vexec, DataFusion, Velox, or future alternatives could evolve behind clearer interfaces, instead of requiring more changes throughout the PostgreSQL fork.
I have a few questions that I think would be valuable for evaluating the long-term direction:
1. Which of the 24 PostgreSQL core hooks are general enough to propose upstream, and which are inherently Cloudberry-specific?
2. How do you plan to keep the adapted or copied files under `pg19/` synchronized with Cloudberry `main` over time?
3. Do you plan to publish an apples-to-apples comparison between the current Cloudberry implementation and the PostgreSQL 19 port on the same multi-host environment, especially for Motion-heavy joins, concurrent workloads, spill, and failure recovery?
4. Would an experimental branch or separate repository be the preferred first step for community collaboration?
The current vectorized-execution results are promising, although most of the published measurements appear to be from one physical host. A real multi-host comparison would make it much easier to understand both the performance potential and the remaining distributed-execution overhead.
Overall, I think this experiment demonstrates that Cloudberry’s relationship with PostgreSQL does not necessarily have to remain a deeply coupled fork forever. Even if some core hooks are still required, turning the majority of the MPP system into modules looks like a direction worth exploring further.
GitHub link: https://github.com/apache/cloudberry/discussions/2065#discussioncomment-18901577
----
This is an automatically sent email for dev@cloudberry.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@cloudberry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: dev-help@cloudberry.apache.org
Re: [D] [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions [cloudberry]
Posted by "igor-suhorukov (via GitHub)" <gi...@apache.org>.
GitHub user igor-suhorukov added a comment to the discussion: [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions
@MisterRaindrop thank you for initiating a substantive discussion regarding the proposal.
The three interfaces vexec uses are not specific to it, and a DataFusion engine could be their second user: [`vexec_source.h`](https://github.com/igor-suhorukov/cloudberry/blob/extension_postgresql_19/pg19/include/vexec_source.h) (PAX and `ao_column` hand over column batches in Arrow layouts), [`gp_orca_vec.h`](https://github.com/igor-suhorukov/cloudberry/blob/extension_postgresql_19/pg19/include/gp_orca_vec.h) (ORCA prices and places an engine's nodes) and [`gp_core_api.h`](https://github.com/igor-suhorukov/cloudberry/blob/extension_postgresql_19/pg19/include/gp_core_api.h) (routing batches across Motions).
1. The 24 upstream core commits fall into three groups.
- General: [`new_oid_hook`](https://github.com/igor-suhorukov/postgres/commit/7e2a0be6218), [`raw_parser_hook`](https://github.com/igor-suhorukov/postgres/commit/480f3128f87), [`smgr_file_event_hook`](https://github.com/igor-suhorukov/postgres/commit/a91cbb5aaf0), and a [registry](https://github.com/igor-suhorukov/postgres/commit/21f45bdde60) of extra table access method routines with five uses: [reloptions](https://github.com/igor-suhorukov/postgres/commit/112462d155c), [the columns a scan reads](https://github.com/igor-suhorukov/postgres/commit/3a2d4fcc1c2), [a unique index's probe](https://github.com/igor-suhorukov/postgres/commit/150343312fe), [ADD COLUMN without a rewrite](https://github.com/igor-suhorukov/postgres/commit/93eb6501b97) and [BRIN over sparse block numbers](https://github.com/igor-suhorukov/postgres/commit/5ac8dbfc9ab).
- A general problem, but another shape upstream: [`analyze_sample_rows_hook`](https://github.com/igor-suhorukov/postgres/commit/6f8d26d16c2), [`memory_block_alloc_hook`](https://github.com/igor-suhorukov/postgres/commit/41107a43583), [materialized view maintenance](https://github.com/igor-suhorukov/postgres/commit/a903759f70d) and [an injection point](https://github.com/igor-suhorukov/postgres/commit/4026e083a14).
- Specific to an MPP engine: a transaction handed to reader processes and the order of its commit ([`XactAdoptTransactionState()`](https://github.com/igor-suhorukov/postgres/commit/60e82c61d7c), [combo command ID hooks](https://github.com/igor-suhorukov/postgres/commit/791fd7171a0), [`xact_commit_recorded_hook`](https://github.com/igor-suhorukov/postgres/commit/1853d9244f0), [`XactAbortAgainAfterEnd`](https://github.com/igor-suhorukov/postgres/commit/871f0c5102e)); [`query_lockmode_hook`](https://github.com/igor-suhorukov/postgres/commit/8c908a24f4c); [`physical_slot_restart_lsn_hook`](https://github.com/igor-suhorukov/postgres/commit/bea8a637438); [tablespace location hooks](https://github.com/igor-suhorukov/postgres/commit/9b351c13e72); [pg_checksums' marks](https://github.com/igor-suhorukov/postgres/commit/b213d194740); and Cloudberry's syntax ([`gp_segment_id`](https://github.com/igor-suhorukov/postgres/commit/f26d74cc863), [`gp_dist_random()`](https://github.com/igor-suhorukov/p
ostgres/commit/aeae8a306d2), [columns hidden from `*`](https://github.com/igor-suhorukov/postgres/commit/19b27fd7687)).
2. Following `main`. The port edits no Cloudberry file, so `git merge main` cannot conflict. For the port's 244 modified copies, a new script (not pushed yet) does a three-way merge with the original at the last branch or merge point and the original in `main`. A clean merge is applied automatically; a conflict goes to an LLM and then to a reviewer. On today's `main`, 54 commits ahead, 13 copies were affected: 7 merged automatically, 1 already had the change, and 5 had conflicts, which the LLM resolved.
3. Comparison. Yes, of course I plan to. A benchmark that others can reproduce has to run in a public cloud, and I have no prepaid AWS access. Given access to an AWS account, I will run all of it: Motion-heavy joins, concurrency, spill and failure recovery. I would do it after the port passes as many of Cloudberry's tests as it can and vexec gets a little more work, so that the measurements include it.
4. Branch or repository. I would prefer an experimental branch in the project's main repository. `pg19/` compiles Cloudberry's files in place, so it has to sit inside Cloudberry's tree, and on a branch, following `main` is a plain `git merge main`. And a repository for Postgres kernel 24 hooks is also needed.
GitHub link: https://github.com/apache/cloudberry/discussions/2065#discussioncomment-18906162
----
This is an automatically sent email for dev@cloudberry.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@cloudberry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: dev-help@cloudberry.apache.org