You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by "jiaqizho (via GitHub)" <gi...@apache.org> on 2025/03/21 06:37:57 UTC
[PR] Feature: introduce a high-performance hybrid row-column storage engine [cloudberry]
jiaqizho opened a new pull request, #1002:
URL: https://github.com/apache/cloudberry/pull/1002
<!-- Thank you for your contribution to Apache Cloudberry (Incubating)! -->
Fixes #ISSUE_Number
### What does this PR do?
<!-- Brief overview of the changes, including any major features or fixes -->
### Type of Change
- [ ] Bug fix (non-breaking change)
- [ ] New feature (non-breaking change)
- [ ] Breaking change (fix or feature with breaking changes)
- [ ] Documentation update
### Breaking Changes
<!-- Remove if not applicable. If yes, explain impact and migration path -->
### Test Plan
<!-- How did you test these changes? -->
- [ ] Unit tests added/updated
- [ ] Integration tests added/updated
- [ ] Passed `make installcheck`
- [ ] Passed `make -C src/test installcheck-cbdb-parallel`
### Impact
<!-- Remove sections that don't apply -->
**Performance:**
<!-- Any performance implications? -->
**User-facing changes:**
<!-- Any changes visible to users? -->
**Dependencies:**
<!-- New dependencies or version changes? -->
### Checklist
- [ ] Followed [contribution guide](https://cloudberry.apache.org/contribute/code)
- [ ] Added/updated documentation
- [ ] Reviewed code for security implications
- [ ] Requested review from [cloudberry committers](https://github.com/orgs/apache/teams/cloudberry-committers)
### Additional Context
<!-- Any other information that would help reviewers? Remove if none -->
### CI Skip Instructions
<!--
To skip CI builds, add the appropriate CI skip identifier to your PR title.
The identifier must:
- Be in square brackets []
- Include the word "ci" and either "skip" or "no"
- Only use for documentation-only changes or when absolutely necessary
-->
---
<!-- Join our community:
- Mailing list: [dev@cloudberry.apache.org](https://lists.apache.org/list.html?dev@cloudberry.apache.org) (subscribe: dev-subscribe@cloudberry.apache.org)
- Discussions: https://github.com/apache/cloudberry/discussions -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2025129347
##########
contrib/pax_storage/FindDependencies.cmake:
##########
@@ -0,0 +1,22 @@
+## protobuf
+include(FindProtobuf)
+find_package(Protobuf 3.6.1 REQUIRED)
+
+# ztsd
+# in our image snapshot, zstd is managed using pkg-config, so so the pkg-config method is used first here
+find_package(PkgConfig QUIET)
+if(PKGCONFIG_FOUND)
Review Comment:
current cbdb use zstd 1.5.1, I found pax use 1.5.5, can we union version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2030365073
##########
contrib/pax_storage/FindDependencies.cmake:
##########
@@ -0,0 +1,22 @@
+## protobuf
+include(FindProtobuf)
+find_package(Protobuf 3.6.1 REQUIRED)
+
+# ztsd
+# in our image snapshot, zstd is managed using pkg-config, so so the pkg-config method is used first here
+find_package(PkgConfig QUIET)
+if(PKGCONFIG_FOUND)
Review Comment:
if not found zstd, we can fallback to support only rle, delta ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "jiaqizho (via GitHub)" <gi...@apache.org>.
jiaqizho commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2025929157
##########
contrib/pax_storage/src/cpp/manifest/manifest.c:
##########
@@ -0,0 +1,793 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * manifest.c
+ *
+ * IDENTIFICATION
+ * contrib/pax_storage/src/cpp/manifest/manifest.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include <errno.h>
+#include <sys/param.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include <string.h>
+
+#include "postgres.h"
+#include "nodes/pg_list.h"
+#include "storage/relfilenode.h"
+
+#include "comm/pax_rel.h"
+#include "manifest.h"
+#include "tuple.h"
+#include "yyjson.h"
+
+#include "access/relation.h"
+#include "access/genam.h"
+#include "access/heapam.h"
+#include "access/table.h"
+#include "access/tableam.h"
+#include "access/tupmacs.h"
+#include "access/xact.h"
+#include "catalog/indexing.h"
+#include "catalog/pax_catalog_columns.h"
+#include "catalog/pg_namespace_d.h"
+#include "catalog/heap.h"
+#include "catalog/index.h"
+#include "catalog/pg_opclass_d.h"
+#include "catalog/dependency.h"
+#include "catalog/storage.h"
+#include "fmgr.h"
+#include "storage/lmgr.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/hsearch.h"
+#include "utils/memutils.h"
+
+#include "utils/bytea.h"
+#include "utils/relfilenodemap.h"
+#include "utils/lsyscache.h"
+
+#include "cdb/cdbvars.h"
+
+// headers from pax
+#include "catalog/pax_fastsequence.h"
+#include "manifest_wrapper.h"
+
+
+#define PAX_MICROPARTITION_DIR_POSTFIX "_pax"
+
+/*
+ * issues:
+ * there is a issue for our cache memory, we setup two cache, one is for heap
+ * and the other is for relation, the HTAB global variable initialization should
+ * happend after the null check, but when the transaction exit with exception,
+ * we have no chance to clear the HTAB global variables, so that we have no way
+ * to do the right initialization.
+ *
+ * extend descriptor fields actually should only appended on init, we'd better
+ * not provide a method to extend the fields at any time.
+ */
+
+MemoryContext manifest_mcontext = NULL;
Review Comment:
export USE_MANIFEST_API=ON
export USE_PAX_CATALOG=OFF
rebuild the cloudberry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#issuecomment-2771592382
+Comments:
3. Need to remove the original HashData copyright license header from source
like: https://github.com/jiaqizho/cloudberrydb/blob/287abbdc714e410b2f5c02927c365186899c5d7c/contrib/pax_storage/src/cpp/catalog/pax_fastsequence.h#L28
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2025669776
##########
contrib/pax_storage/src/cpp/access/pax_access_handle.cc:
##########
@@ -0,0 +1,1217 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * pax_access_handle.cc
+ *
+ * IDENTIFICATION
+ * contrib/pax_storage/src/cpp/access/pax_access_handle.cc
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "access/pax_access_handle.h"
+
+#include "comm/cbdb_api.h"
+
+#include "access/pax_dml_state.h"
+#include "access/pax_scanner.h"
+#include "access/pax_table_cluster.h"
+#include "access/pax_updater.h"
+#include "access/paxc_rel_options.h"
+#include "catalog/pax_catalog.h"
+#include "clustering/zorder_utils.h"
+#include "comm/guc.h"
+#include "comm/pax_memory.h"
+#include "comm/pax_resource.h"
+#include "comm/paxc_wrappers.h"
+#include "comm/vec_numeric.h"
+#include "exceptions/CException.h"
+#include "storage/local_file_system.h"
+#ifdef VEC_BUILD
+#include "storage/vec_parallel_pax.h"
+#endif
+#include "storage/paxc_smgr.h"
+#include "storage/wal/pax_wal.h"
+#include "storage/wal/paxc_wal.h"
+
+#define NOT_IMPLEMENTED_YET \
+ ereport(ERROR, \
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \
+ errmsg("not implemented yet on pax relations: %s", __func__)))
+
+#define NOT_SUPPORTED_YET \
+ ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \
+ errmsg("not supported on pax relations: %s", __func__)))
+
+#define RELATION_IS_PAX(rel) \
+ (OidIsValid((rel)->rd_rel->relam) && RelationIsPAX(rel))
+
+// access methods that are implemented in C++
+namespace pax {
+
+TableScanDesc CCPaxAccessMethod::ScanBegin(Relation relation, Snapshot snapshot,
+ int nkeys, struct ScanKeyData *key,
+ ParallelTableScanDesc pscan,
+ uint32 flags) {
+ CBDB_TRY();
+ {
+ auto desc = PAX_NEW<PaxScanDesc>();
+ pax::common::RememberResourceCallback(pax::ReleaseTopObject<PaxScanDesc>,
+ cbdb::PointerToDatum(desc));
+
+ return desc->BeginScan(relation, snapshot, nkeys, key, pscan, flags,
+ nullptr, true);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_END_TRY();
+
+ pg_unreachable();
+}
+
+void CCPaxAccessMethod::ScanEnd(TableScanDesc scan) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ desc->EndScan();
+
+ pax::common::ForgetResourceCallback(pax::ReleaseTopObject<PaxScanDesc>,
+ cbdb::PointerToDatum(desc));
+ PAX_DELETE<PaxScanDesc>(desc);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+TableScanDesc CCPaxAccessMethod::ScanExtractColumns(
+ Relation rel, Snapshot snapshot, int nkeys, struct ScanKeyData *key,
+ ParallelTableScanDesc parallel_scan, struct PlanState *ps, uint32 flags) {
+ CBDB_TRY();
+ {
+ auto desc = PAX_NEW<PaxScanDesc>();
+ pax::common::RememberResourceCallback(pax::ReleaseTopObject<PaxScanDesc>,
+ cbdb::PointerToDatum(desc));
+
+ return desc->BeginScanExtractColumns(rel, snapshot, nkeys, key,
+ parallel_scan, ps, flags);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::IndexUniqueCheck(Relation rel, ItemPointer tid,
+ Snapshot snapshot, bool *all_dead) {
+ CBDB_TRY();
+ {
+ auto dsl = pax::CPaxDmlStateLocal::Instance();
+ if (dsl->IsInitialized()) {
+ auto del = dsl->GetDeleter(rel, snapshot, true);
+
+ if (del && del->IsMarked(*tid)) return false;
+ }
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+
+ return paxc::IndexUniqueCheck(rel, tid, snapshot, all_dead);
+}
+
+struct IndexFetchTableData *CCPaxAccessMethod::IndexFetchBegin(Relation rel) {
+ CBDB_TRY();
+ {
+ Assert(RELATION_IS_PAX(rel));
+ auto desc = PAX_NEW<PaxIndexScanDesc>(rel);
+ pax::common::RememberResourceCallback(
+ pax::ReleaseTopObject<PaxIndexScanDesc>, PointerGetDatum(desc));
+ return desc->ToBase();
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ return nullptr; // keep compiler quiet
+}
+
+void CCPaxAccessMethod::IndexFetchEnd(IndexFetchTableData *scan) {
+ CBDB_TRY();
+ {
+ auto desc = PaxIndexScanDesc::FromBase(scan);
+ desc->Release();
+
+ pax::common::ForgetResourceCallback(pax::ReleaseTopObject<PaxIndexScanDesc>,
+ PointerGetDatum(desc));
+ PAX_DELETE(desc);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+bool CCPaxAccessMethod::IndexFetchTuple(struct IndexFetchTableData *scan,
+ ItemPointer tid, Snapshot snapshot,
+ TupleTableSlot *slot, bool *call_again,
+ bool *all_dead) {
+ CBDB_TRY();
+ {
+ auto desc = PaxIndexScanDesc::FromBase(scan);
+ return desc->FetchTuple(tid, snapshot, slot, call_again, all_dead);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ return false; // keep compiler quiet
+}
+
+void CCPaxAccessMethod::IndexFetchReset(IndexFetchTableData * /*scan*/) {}
+
+void CCPaxAccessMethod::ScanRescan(TableScanDesc scan, ScanKey key,
+ bool set_params, bool allow_strat,
+ bool allow_sync, bool allow_pagemode) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ desc->ReScan(key, set_params, allow_strat, allow_sync, allow_pagemode);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+bool CCPaxAccessMethod::ScanGetNextSlot(TableScanDesc scan,
+ ScanDirection /*direction*/,
+ TupleTableSlot *slot) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ bool result;
+
+ result = desc->GetNextSlot(slot);
+ if (result) {
+ pgstat_count_heap_getnext(desc->GetRelation());
+ }
+ return result;
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+
+ pg_unreachable();
+}
+
+void CCPaxAccessMethod::TupleInsert(Relation relation, TupleTableSlot *slot,
+ CommandId cid, int options,
+ BulkInsertState bistate) {
+ CBDB_TRY();
+ {
+ MemoryContext old_ctx;
+ Assert(cbdb::pax_memory_context);
+
+#ifdef FAULT_INJECTOR
+ FaultInjector_InjectFaultIfSet("pax_insert", DDLNotSpecified, "",
+ RelationGetRelationName(relation));
+#endif
+
+ old_ctx = MemoryContextSwitchTo(cbdb::pax_memory_context);
+ CPaxInserter::TupleInsert(relation, slot, cid, options, bistate);
+ MemoryContextSwitchTo(old_ctx);
+
+ pgstat_count_heap_insert(relation, 1);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({
+ // FIXME: destroy CPaxInserter?
+ });
+ CBDB_END_TRY();
+}
+
+TM_Result CCPaxAccessMethod::TupleDelete(Relation relation, ItemPointer tid,
+ CommandId cid, Snapshot snapshot,
+ Snapshot /*crosscheck*/, bool /*wait*/,
+ TM_FailureData *tmfd,
+ bool /*changing_part*/) {
+ CBDB_TRY();
+ {
+ auto result = CPaxDeleter::DeleteTuple(relation, tid, cid, snapshot, tmfd);
+ if (result == TM_Ok) pgstat_count_heap_delete(relation);
+ return result;
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+TM_Result CCPaxAccessMethod::TupleUpdate(Relation relation, ItemPointer otid,
+ TupleTableSlot *slot, CommandId cid,
+ Snapshot snapshot, Snapshot crosscheck,
+ bool wait, TM_FailureData *tmfd,
+ LockTupleMode *lockmode,
+ bool *update_indexes) {
+ CBDB_TRY();
+ {
+ MemoryContext old_ctx;
+ TM_Result result;
+
+ Assert(cbdb::pax_memory_context);
+ old_ctx = MemoryContextSwitchTo(cbdb::pax_memory_context);
+ result = CPaxUpdater::UpdateTuple(relation, otid, slot, cid, snapshot,
+ crosscheck, wait, tmfd, lockmode,
+ update_indexes);
+ MemoryContextSwitchTo(old_ctx);
+ if (result == TM_Ok) pgstat_count_heap_update(relation, false);
+ return result;
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanAnalyzeNextBlock(TableScanDesc scan,
+ BlockNumber blockno,
+ BufferAccessStrategy bstrategy) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->ScanAnalyzeNextBlock(blockno, bstrategy);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanAnalyzeNextTuple(TableScanDesc scan,
+ TransactionId oldest_xmin,
+ double *liverows, double *deadrows,
+ TupleTableSlot *slot) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->ScanAnalyzeNextTuple(oldest_xmin, liverows, deadrows, slot);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanBitmapNextBlock(TableScanDesc scan,
+ TBMIterateResult *tbmres) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->BitmapNextBlock(tbmres);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanBitmapNextTuple(TableScanDesc scan,
+ TBMIterateResult *tbmres,
+ TupleTableSlot *slot) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ bool result;
+ result = desc->BitmapNextTuple(tbmres, slot);
+ if (result) {
+ pgstat_count_heap_fetch(desc->GetRelation());
+ }
+ return result;
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanSampleNextBlock(TableScanDesc scan,
+ SampleScanState *scanstate) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->ScanSampleNextBlock(scanstate);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanSampleNextTuple(TableScanDesc scan,
+ SampleScanState *scanstate,
+ TupleTableSlot *slot) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->ScanSampleNextTuple(scanstate, slot);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+void CCPaxAccessMethod::MultiInsert(Relation relation, TupleTableSlot **slots,
+ int ntuples, CommandId cid, int options,
+ BulkInsertState bistate) {
+ CBDB_TRY();
+ {
+ MemoryContext old_ctx;
+ Assert(cbdb::pax_memory_context);
+
+ old_ctx = MemoryContextSwitchTo(cbdb::pax_memory_context);
+ CPaxInserter::MultiInsert(relation, slots, ntuples, cid, options, bistate);
+ MemoryContextSwitchTo(old_ctx);
+
+ pgstat_count_heap_insert(relation, ntuples);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({
+ // FIXME: destroy CPaxInserter?
+ });
+ CBDB_END_TRY();
+}
+
+void CCPaxAccessMethod::FinishBulkInsert(Relation relation, int options) {
+ // Implement Pax dml cleanup for case "create table xxx1 as select * from
+ // xxx2", which would not call ExtDmlFini callback function and relies on
+ // FinishBulkInsert callback function to cleanup its dml state.
+ CBDB_TRY();
+ {
+ // no need switch memory context
+ // cause it just call dml finish
+ pax::CPaxInserter::FinishBulkInsert(relation, options);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({
+ // FIXME: destroy CPaxInserter?
+ });
+ CBDB_END_TRY();
+}
+
+void CCPaxAccessMethod::ExtDmlInit(Relation rel, CmdType operation) {
+ if (!RELATION_IS_PAX(rel)) return;
+
+ CBDB_TRY();
+ { pax::CPaxDmlStateLocal::Instance()->InitDmlState(rel, operation); }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+void CCPaxAccessMethod::ExtDmlFini(Relation rel, CmdType operation) {
+ if (!RELATION_IS_PAX(rel)) return;
+
+ CBDB_TRY();
+ { pax::CPaxDmlStateLocal::Instance()->FinishDmlState(rel, operation); }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+} // namespace pax
+// END of C++ implementation
+
+// access methods that are implemented in C
+namespace paxc {
+const TupleTableSlotOps *PaxAccessMethod::SlotCallbacks(
+ Relation /*rel*/) noexcept {
+ return &TTSOpsVirtual;
+}
+
+uint32 PaxAccessMethod::ScanFlags(Relation relation) {
+ uint32 flags = 0;
+#ifdef VEC_BUILD
+ flags |= SCAN_SUPPORT_VECTORIZATION | SCAN_SUPPORT_COLUMN_ORIENTED_SCAN;
+#else
+ flags |= SCAN_SUPPORT_COLUMN_ORIENTED_SCAN;
+#endif
+
+#if defined(USE_MANIFEST_API) && !defined(USE_PAX_CATALOG)
+ flags |= SCAN_FORCE_BIG_WRITE_LOCK;
+#endif
+
+ return flags;
+}
+
+Size PaxAccessMethod::ParallelscanEstimate(Relation /*rel*/) {
+ return sizeof(ParallelBlockTableScanDescData);
+}
+
+Size PaxAccessMethod::ParallelscanInitialize(Relation rel,
+ ParallelTableScanDesc pscan) {
+ ParallelBlockTableScanDesc bpscan = (ParallelBlockTableScanDesc)pscan;
+ bpscan->base.phs_relid = RelationGetRelid(rel);
+ pg_atomic_init_u64(&bpscan->phs_nallocated, 0);
+ // Like ao/aocs, we don't need phs_mutex and phs_startblock, though, init
+ // them.
+ SpinLockInit(&bpscan->phs_mutex);
+ bpscan->phs_startblock = InvalidBlockNumber;
+ return sizeof(ParallelBlockTableScanDescData);
+}
+
+void PaxAccessMethod::ParallelscanReinitialize(Relation rel,
+ ParallelTableScanDesc pscan) {
+ ParallelBlockTableScanDesc bpscan = (ParallelBlockTableScanDesc)pscan;
+
+ pg_atomic_write_u64(&bpscan->phs_nallocated, 0);
+}
+
+void PaxAccessMethod::TupleInsertSpeculative(Relation /*relation*/,
+ TupleTableSlot * /*slot*/,
+ CommandId /*cid*/, int /*options*/,
+ BulkInsertState /*bistate*/,
+ uint32 /*spec_token*/) {
+ NOT_IMPLEMENTED_YET;
+}
+
+void PaxAccessMethod::TupleCompleteSpeculative(Relation /*relation*/,
+ TupleTableSlot * /*slot*/,
+ uint32 /*spec_token*/,
+ bool /*succeeded*/) {
+ NOT_IMPLEMENTED_YET;
+}
+
+TM_Result PaxAccessMethod::TupleLock(Relation /*relation*/, ItemPointer /*tid*/,
+ Snapshot /*snapshot*/,
+ TupleTableSlot * /*slot*/,
+ CommandId /*cid*/, LockTupleMode /*mode*/,
+ LockWaitPolicy /*wait_policy*/,
+ uint8 /*flags*/,
+ TM_FailureData * /*tmfd*/) {
+ NOT_IMPLEMENTED_YET;
+ return TM_Ok;
+}
+
+bool PaxAccessMethod::TupleFetchRowVersion(Relation /*relation*/,
+ ItemPointer /*tid*/,
+ Snapshot /*snapshot*/,
+ TupleTableSlot * /*slot*/) {
+ NOT_IMPLEMENTED_YET;
+ return false;
+}
+
+bool PaxAccessMethod::TupleTidValid(TableScanDesc /*scan*/,
+ ItemPointer /*tid*/) {
+ NOT_IMPLEMENTED_YET;
+ return false;
+}
+
+void PaxAccessMethod::TupleGetLatestTid(TableScanDesc /*sscan*/,
+ ItemPointer /*tid*/) {
+ NOT_SUPPORTED_YET;
+}
+
+bool PaxAccessMethod::TupleSatisfiesSnapshot(Relation /*rel*/,
+ TupleTableSlot * /*slot*/,
+ Snapshot /*snapshot*/) {
+ NOT_IMPLEMENTED_YET;
+ return true;
+}
+
+TransactionId PaxAccessMethod::IndexDeleteTuples(
+ Relation /*rel*/, TM_IndexDeleteOp * /*delstate*/) {
+ NOT_SUPPORTED_YET;
+ return 0;
+}
+
+void PaxAccessMethod::RelationVacuum(Relation /*onerel*/,
+ VacuumParams * /*params*/,
+ BufferAccessStrategy /*bstrategy*/) {
+ /* PAX: micro-partitions have no dead tuples, so vacuum is empty */
+}
+
+BlockSequence *PaxAccessMethod::RelationGetBlockSequences(Relation rel,
+ int *numSequences) {
+ // PAX not support brin index yet
+ NOT_IMPLEMENTED_YET;
+ return nullptr;
+}
+
+void PaxAccessMethod::RelationGetBlockSequence(Relation rel, BlockNumber blkNum,
+ BlockSequence *sequence) {
+ // PAX not support brin index yet
+ NOT_IMPLEMENTED_YET;
+}
+
+bool PaxAccessMethod::RelationNeedsToastTable(Relation /*rel*/) {
+ // PAX never used the toasting, don't create the toast table from Cloudberry 7
+
+ return false;
+}
+
+double PaxAccessMethod::IndexBuildRangeScan(
+ Relation heap_relation, Relation index_relation, IndexInfo *index_info,
+ bool /*allow_sync*/, bool anyvisible, bool progress,
+ BlockNumber start_blockno, BlockNumber numblocks,
+ IndexBuildCallback callback, void *callback_state, TableScanDesc scan) {
+ Datum values[INDEX_MAX_KEYS];
+ bool isnull[INDEX_MAX_KEYS];
+ double reltuples = 0;
+ ExprState *predicate;
+ TupleTableSlot *slot;
+ EState *estate;
+ ExprContext *econtext;
+ Snapshot snapshot;
+ TransactionId OldestXmin;
+
+ bool checking_uniqueness pg_attribute_unused();
+ bool need_unregister_snapshot = false;
+ BlockNumber previous_blkno = InvalidBlockNumber;
+
+ Assert(OidIsValid(index_relation->rd_rel->relam));
+ Assert(!IsSystemRelation(heap_relation));
+
+ if (index_relation->rd_rel->relam != BTREE_AM_OID &&
+ index_relation->rd_rel->relam != HASH_AM_OID &&
+ index_relation->rd_rel->relam != GIN_AM_OID &&
+ index_relation->rd_rel->relam != BITMAP_AM_OID)
+ elog(ERROR, "pax only support btree/hash/gin/bitmap indexes");
+
+ checking_uniqueness =
+ (index_info->ii_Unique || index_info->ii_ExclusionOps != NULL);
+ // "Any visible" mode is not compatible with uniqueness checks; make sure
+ // only one of those is requested.
+ (void)anyvisible; // keep compiler quiet for release version
+ Assert(!(anyvisible && checking_uniqueness));
+
+ slot = table_slot_create(heap_relation, NULL);
+ estate = CreateExecutorState();
+ econtext = GetPerTupleExprContext(estate);
+ econtext->ecxt_scantuple = slot;
+ predicate = ExecPrepareQual(index_info->ii_Predicate, estate);
+
+ /*
+ * Prepare for scan of the base relation. In a normal index build, we use
+ * SnapshotAny because we must retrieve all tuples and do our own time
+ * qual checks (because we have to index RECENTLY_DEAD tuples). In a
+ * concurrent build, or during bootstrap, we take a regular MVCC snapshot
+ * and index whatever's live according to that.
+ */
+ OldestXmin = InvalidTransactionId;
+
+ /* okay to ignore lazy VACUUMs here */
+ if (!IsBootstrapProcessingMode() && !index_info->ii_Concurrent)
+ OldestXmin = GetOldestNonRemovableTransactionId(heap_relation);
+
+ if (!scan) {
+ if (!TransactionIdIsValid(OldestXmin)) {
+ snapshot = RegisterSnapshot(GetTransactionSnapshot());
+ need_unregister_snapshot = true;
+ } else {
+ need_unregister_snapshot = false;
+ snapshot = SnapshotAny;
+ }
+ scan = table_beginscan(heap_relation, snapshot, 0, NULL);
+ } else {
+ snapshot = scan->rs_snapshot;
+ need_unregister_snapshot = false;
+ }
+
+ // FIXME: Only brin index uses partial index now. setup start_blockno
+ // and numblocks is too late after beginscan is called now, because
+ // the current micro partition is opened. The workaround is ugly to
+ // check and close the current micro partition and open another one.
+ if (start_blockno != 0 || numblocks != InvalidBlockNumber)
+ elog(ERROR, "PAX doesn't support partial index scan now");
+
+ while (table_scan_getnextslot(scan, ForwardScanDirection, slot)) {
+ CHECK_FOR_INTERRUPTS();
+
+ if (progress) {
+ BlockNumber blkno = pax::GetBlockNumber(slot->tts_tid);
+ if (previous_blkno == InvalidBlockNumber)
+ previous_blkno = blkno;
+ else if (previous_blkno != blkno) {
+ pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+ blkno - start_blockno);
+ previous_blkno = blkno;
+ }
+ }
+ reltuples += 1;
+
+ MemoryContextReset(econtext->ecxt_per_tuple_memory);
+
+ /*
+ * In a partial index, discard tuples that don't satisfy the
+ * predicate.
+ */
+ if (predicate && !ExecQual(predicate, econtext)) continue;
+
+ /*
+ * For the current heap tuple, extract all the attributes we use in
+ * this index, and note which are null. This also performs evaluation
+ * of any expressions needed.
+ */
+ FormIndexDatum(index_info, slot, estate, values, isnull);
+
+ /*
+ * You'd think we should go ahead and build the index tuple here, but
+ * some index AMs want to do further processing on the data first. So
+ * pass the values[] and isnull[] arrays, instead.
+ */
+ callback(index_relation, &slot->tts_tid, values, isnull, true,
+ callback_state);
+ }
+
+ /* Report scan progress one last time. */
+ if (progress && previous_blkno != InvalidBlockNumber)
+ pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+ previous_blkno + 1 - start_blockno);
+
+ table_endscan(scan);
+ if (need_unregister_snapshot) UnregisterSnapshot(snapshot);
+
+ ExecDropSingleTupleTableSlot(slot);
+ FreeExecutorState(estate);
+
+ /* These may have been pointing to the now-gone estate */
+ index_info->ii_ExpressionsState = NIL;
+ index_info->ii_PredicateState = NULL;
+
+ return reltuples;
+}
+
+void PaxAccessMethod::IndexValidateScan(Relation /*heap_relation*/,
+ Relation /*index_relation*/,
+ IndexInfo * /*index_info*/,
+ Snapshot /*snapshot*/,
+ ValidateIndexState * /*state*/) {
+ NOT_IMPLEMENTED_YET;
+}
+
+bytea *PaxAccessMethod::AmOptions(Datum reloptions, char relkind,
+ bool validate) {
+ return paxc_default_rel_options(reloptions, relkind, validate);
+}
+
+void PaxAccessMethod::ValidateColumnEncodingClauses(List *encoding_opts) {
+ paxc_validate_column_encoding_clauses(encoding_opts);
+}
+
+List *PaxAccessMethod::TransformColumnEncodingClauses(Relation /*rel*/,
+ List *encoding_opts,
+ bool validate,
+ bool from_type) {
+ return paxc_transform_column_encoding_clauses(encoding_opts, validate,
+ from_type);
+}
+
+} // namespace paxc
+// END of C implementation
+
+#define SET_LOCKTAG_INT64(tag, key64) \
+ SET_LOCKTAG_ADVISORY(tag, MyDatabaseId, (uint32)((key64) >> 32), \
+ (uint32)(key64), 1)
+
+extern "C" {
+
+static const TableAmRoutine kPaxColumnMethods = {
+ .type = T_TableAmRoutine,
+ .slot_callbacks = paxc::PaxAccessMethod::SlotCallbacks,
+ .scan_begin = pax::CCPaxAccessMethod::ScanBegin,
+ .scan_begin_extractcolumns = pax::CCPaxAccessMethod::ScanExtractColumns,
+ .scan_end = pax::CCPaxAccessMethod::ScanEnd,
+ .scan_rescan = pax::CCPaxAccessMethod::ScanRescan,
+ .scan_getnextslot = pax::CCPaxAccessMethod::ScanGetNextSlot,
+ .scan_flags = paxc::PaxAccessMethod::ScanFlags,
+
+ .parallelscan_estimate = paxc::PaxAccessMethod::ParallelscanEstimate,
+ .parallelscan_initialize = paxc::PaxAccessMethod::ParallelscanInitialize,
+ .parallelscan_reinitialize =
+ paxc::PaxAccessMethod::ParallelscanReinitialize,
+
+ .index_fetch_begin = pax::CCPaxAccessMethod::IndexFetchBegin,
+ .index_fetch_reset = pax::CCPaxAccessMethod::IndexFetchReset,
+ .index_fetch_end = pax::CCPaxAccessMethod::IndexFetchEnd,
+ .index_fetch_tuple = pax::CCPaxAccessMethod::IndexFetchTuple,
+ .index_unique_check = pax::CCPaxAccessMethod::IndexUniqueCheck,
+
+ .tuple_fetch_row_version = paxc::PaxAccessMethod::TupleFetchRowVersion,
+ .tuple_tid_valid = paxc::PaxAccessMethod::TupleTidValid,
+ .tuple_get_latest_tid = paxc::PaxAccessMethod::TupleGetLatestTid,
+ .tuple_satisfies_snapshot = paxc::PaxAccessMethod::TupleSatisfiesSnapshot,
+ .index_delete_tuples = paxc::PaxAccessMethod::IndexDeleteTuples,
+
+ .tuple_insert = pax::CCPaxAccessMethod::TupleInsert,
+ .tuple_insert_speculative = paxc::PaxAccessMethod::TupleInsertSpeculative,
+ .tuple_complete_speculative =
+ paxc::PaxAccessMethod::TupleCompleteSpeculative,
+ .multi_insert = pax::CCPaxAccessMethod::MultiInsert,
+ .tuple_delete = pax::CCPaxAccessMethod::TupleDelete,
+ .tuple_update = pax::CCPaxAccessMethod::TupleUpdate,
+ .tuple_lock = paxc::PaxAccessMethod::TupleLock,
+ .finish_bulk_insert = pax::CCPaxAccessMethod::FinishBulkInsert,
+
+ .relation_set_new_filenode = pax::CCPaxAccessMethod::RelationSetNewFilenode,
+ .relation_nontransactional_truncate =
+ pax::CCPaxAccessMethod::RelationNontransactionalTruncate,
+ .relation_copy_data = pax::CCPaxAccessMethod::RelationCopyData,
+ .relation_copy_for_cluster = pax::CCPaxAccessMethod::RelationCopyForCluster,
+ .relation_vacuum = paxc::PaxAccessMethod::RelationVacuum,
+ .scan_analyze_next_block = pax::CCPaxAccessMethod::ScanAnalyzeNextBlock,
+ .scan_analyze_next_tuple = pax::CCPaxAccessMethod::ScanAnalyzeNextTuple,
+ .index_build_range_scan = paxc::PaxAccessMethod::IndexBuildRangeScan,
+ .index_validate_scan = paxc::PaxAccessMethod::IndexValidateScan,
+
+ .relation_size = paxc::PaxAccessMethod::RelationSize,
+ .relation_get_block_sequences =
+ paxc::PaxAccessMethod::RelationGetBlockSequences,
+ .relation_get_block_sequence =
+ paxc::PaxAccessMethod::RelationGetBlockSequence,
+ .relation_needs_toast_table =
+ paxc::PaxAccessMethod::RelationNeedsToastTable,
+
+ .relation_estimate_size = paxc::PaxAccessMethod::EstimateRelSize,
+ .scan_bitmap_next_block = pax::CCPaxAccessMethod::ScanBitmapNextBlock,
+ .scan_bitmap_next_tuple = pax::CCPaxAccessMethod::ScanBitmapNextTuple,
+ .scan_sample_next_block = pax::CCPaxAccessMethod::ScanSampleNextBlock,
+ .scan_sample_next_tuple = pax::CCPaxAccessMethod::ScanSampleNextTuple,
+
+ .amoptions = paxc::PaxAccessMethod::AmOptions,
+ .swap_relation_files = paxc::PaxAccessMethod::SwapRelationFiles,
+ .validate_column_encoding_clauses =
+ paxc::PaxAccessMethod::ValidateColumnEncodingClauses,
+ .transform_column_encoding_clauses =
+ paxc::PaxAccessMethod::TransformColumnEncodingClauses,
+};
+
+PG_MODULE_MAGIC;
+PG_FUNCTION_INFO_V1(pax_tableam_handler);
+Datum pax_tableam_handler(PG_FUNCTION_ARGS) { // NOLINT
+ PG_RETURN_POINTER(&kPaxColumnMethods);
+}
+
+static object_access_hook_type prev_object_access_hook = NULL;
+
+static ProcessUtility_hook_type prev_ProcessUtilit_hook = NULL;
+
+static bool relation_has_cluster_columns_options(Relation rel) {
+ auto *options = (paxc::PaxOptions *)(rel->rd_options);
+
+ if (options == nullptr) {
+ return false;
+ }
+
+ // if not zorder and lexical cluster type, return false
+ if (strcasecmp(options->cluster_type, PAX_LEXICAL_CLUSTER_TYPE) != 0 &&
+ strcasecmp(options->cluster_type, PAX_ZORDER_CLUSTER_TYPE) != 0) {
+ return false;
+ }
+
+ // no need to check zorder cluster type, because it has been checked in
+ // PaxObjectAccessHook
+ Bitmapset *bms = paxc::paxc_get_columns_index_by_options(
+ rel, options->cluster_columns(), nullptr, false);
+
+ if (bms) {
+ bms_free(bms);
+ }
+ return bms != nullptr;
+}
+
+static bool relation_has_clustered_index(Relation rel) {
+ // We need to find the index that has indisclustered set.
+ Oid indexOid = InvalidOid;
+ ListCell *index;
+ foreach (index, RelationGetIndexList(rel)) {
+ indexOid = lfirst_oid(index);
+ if (get_index_isclustered(indexOid)) break;
+ indexOid = InvalidOid;
+ }
+ return OidIsValid(indexOid);
+}
+
+static bool table_can_be_clustered(Relation rel) {
+ return relation_has_cluster_columns_options(rel) ||
+ relation_has_clustered_index(rel);
+}
+
+static void cluster_pax_rel(ClusterStmt *stmt, Relation rel,
+ Snapshot snapshot) {
+ CBDB_TRY();
+ { pax::Cluster(rel, snapshot, false); }
+ CBDB_CATCH_DEFAULT();
+ CBDB_END_TRY();
+}
+
+static void validateAlterTableStmt(AlterTableStmt *stmt) {
+ ListCell *lcmd = NULL;
+
+ if (stmt->objtype != OBJECT_TABLE) return;
+
+ foreach (lcmd, stmt->cmds) {
+ AlterTableCmd *cmd = (AlterTableCmd *)lfirst(lcmd);
+ /*only check set tablespace command*/
+ if (cmd->subtype == AT_SetTableSpace) {
+ Oid tablespaceId;
+
+ /* Check that the tablespace exists */
+ tablespaceId = get_tablespace_oid(cmd->name, false);
+
+ /* alter table stmt does not support setting tablespace of dfs tablespace
+ * type */
+ if (OidIsValid(tablespaceId) && paxc::IsDfsTablespaceById(tablespaceId)) {
+ elog(ERROR,
+ "changing the dfs tablespace type in alter table stmt is not "
+ "supported");
+ }
+ }
+ }
+}
+
+static void validateAlterDatabaseStmt(AlterDatabaseStmt *stmt) {
+ ListCell *lcmd = NULL;
+
+ foreach (lcmd, stmt->options) {
+ DefElem *defel = (DefElem *)lfirst(lcmd);
+ if (pg_strcasecmp(defel->defname, "tablespace") == 0) {
+ Oid tablespaceId;
+
+ /* Check that the tablespace exists */
+ tablespaceId = get_tablespace_oid(defGetString(defel), false);
+
+ /* alter database stmt does not support setting tablespace of dfs
+ * tablespace type */
+ if (OidIsValid(tablespaceId) && paxc::IsDfsTablespaceById(tablespaceId)) {
+ elog(ERROR,
+ "changing dfs tablespace in alter database stmt is not supported");
+ }
+ }
+ }
+}
+
+static void checkUnsupportDfsTableSpaceStmt(Node *stmt) {
+ switch (nodeTag(stmt)) {
+ case T_AlterTableStmt:
+ validateAlterTableStmt((AlterTableStmt *)stmt);
+ break;
+ case T_AlterDatabaseStmt:
+ validateAlterDatabaseStmt((AlterDatabaseStmt *)stmt);
+ break;
+ default:
+ return;
+ }
+}
+
+static void paxProcessUtility(PlannedStmt *pstmt, const char *queryString,
+ bool readOnlyTree, ProcessUtilityContext context,
+ ParamListInfo params, QueryEnvironment *queryEnv,
+ DestReceiver *dest,
+ QueryCompletion *completionTag) {
+ bool isTopLevel = (context == PROCESS_UTILITY_TOPLEVEL);
+ // if is pax table, do something
+ switch (nodeTag(pstmt->utilityStmt)) {
+ case T_ClusterStmt: {
+ ClusterStmt *stmt = (ClusterStmt *)pstmt->utilityStmt;
+ if (stmt->relation) {
+ Relation rel = table_openrv(stmt->relation, AccessShareLock);
+ if (RelationIsPAX(rel)) {
+ bool has_cluster_columns;
+ if (stmt->indexname == NULL && !table_can_be_clustered(rel)) {
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("there is no previously clustered index or "
+ "cluster_columns reloptions for table \"%s\"",
+ stmt->relation->relname)));
+ }
+
+ has_cluster_columns = relation_has_cluster_columns_options(rel);
+
+ // cluster table using indexname,we should check if
+ // relation_has_zorder_clusted
+ if (stmt->indexname && has_cluster_columns) {
+ elog(ERROR,
+ "cannot using index to cluster table which has "
+ "cluster-columns ");
+ }
+
+ if (has_cluster_columns) {
+ if (Gp_role == GP_ROLE_DISPATCH) {
+ CdbDispatchUtilityStatement(
+ (Node *)stmt,
+ DF_CANCEL_ON_ERROR | DF_WITH_SNAPSHOT | DF_NEED_TWO_PHASE,
+ GetAssignedOidsForDispatch(), NULL);
+ } else if (Gp_role == GP_ROLE_EXECUTE) {
+ cluster_pax_rel(stmt, rel, GetActiveSnapshot());
+ }
+
+ table_close(rel, NoLock);
+ return;
+ }
+ }
+
+ table_close(rel, NoLock);
+ } else {
+ if (Gp_role == GP_ROLE_DISPATCH) {
+ // only cluster pax zorder clustered table
+ List *relids = NULL;
+ SysScanDesc scan;
+ HeapTuple tuple;
+
+ // We cannot run this form of CLUSTER inside a user transaction block;
+ // we'd be holding locks way too long.
+ PreventInTransactionBlock(isTopLevel, "CLUSTER");
+
+#if defined(USE_MANIFEST_API)
+ auto pax_rel = table_open(RelationRelationId, AccessShareLock);
+ scan = systable_beginscan(pax_rel, InvalidOid, false,
+ GetActiveSnapshot(), 0, nullptr);
+ while (HeapTupleIsValid(tuple = systable_getnext(scan))) {
+ Datum datum;
+ bool isnull;
+
+ datum = heap_getattr(tuple, Anum_pg_class_relam, pax_rel->rd_att,
+ &isnull);
+ if (isnull || DatumGetObjectId(datum) != PAX_TABLE_AM_OID) continue;
+
+ datum = heap_getattr(tuple, Anum_pg_class_oid, pax_rel->rd_att,
+ &isnull);
+ Assert(!isnull);
+ Oid relid = DatumGetObjectId(datum);
+ relids = lappend_oid(relids, relid);
+ }
+ systable_endscan(scan);
+ table_close(pax_rel, AccessShareLock);
+#else
+ auto pax_aux_rel =
+ table_open(PAX_TABLES_RELATION_ID, AccessShareLock);
+ scan = systable_beginscan(pax_aux_rel, InvalidOid, false,
+ GetActiveSnapshot(), 0, NULL);
+
+ while ((tuple = systable_getnext(scan)) != NULL) {
+ Datum datum;
+ bool isnull;
+ datum = heap_getattr(tuple, ANUM_PG_PAX_TABLES_RELID,
+ pax_aux_rel->rd_att, &isnull);
+ Assert(!isnull);
+ Oid relid = DatumGetObjectId(datum);
+ relids = lappend_oid(relids, relid);
+ }
+ systable_endscan(scan);
+ table_close(pax_aux_rel, AccessShareLock);
+#endif
+
+ ListCell *lc = NULL;
+ foreach (lc, relids) {
+ Oid pax_rel_id = Oid(lfirst_oid(lc));
+
+ Relation rel = table_open(pax_rel_id, RowExclusiveLock);
+ if (relation_has_cluster_columns_options(rel)) {
+ stmt->relation = makeNode(RangeVar);
+ stmt->relation->schemaname =
+ get_namespace_name(rel->rd_rel->relnamespace);
+ stmt->relation->relname = pstrdup(rel->rd_rel->relname.data);
+ CdbDispatchUtilityStatement((Node *)stmt,
+ DF_CANCEL_ON_ERROR | DF_WITH_SNAPSHOT,
+ GetAssignedOidsForDispatch(), NULL);
+ pfree(stmt->relation);
+ stmt->relation = NULL;
+ }
+ table_close(rel, RowExclusiveLock);
+ }
+
+ list_free(relids);
+ }
+ }
+ break;
+ }
+ default:
+ if (Gp_role == GP_ROLE_DISPATCH)
+ checkUnsupportDfsTableSpaceStmt(pstmt->utilityStmt);
+ break;
+ }
+
+ if (prev_ProcessUtilit_hook)
+ prev_ProcessUtilit_hook(pstmt, queryString, readOnlyTree, context, params,
+ queryEnv, dest, completionTag);
+ else
+ standard_ProcessUtility(pstmt, queryString, readOnlyTree, context, params,
+ queryEnv, dest, completionTag);
+}
+
+static void PaxCheckMinMaxColumns(Relation rel, const char *minmax_columns) {
Review Comment:
how to config all columns use with ?
for example : create table t_index_cluster(c1 int, c2 int) with (minmax_columns='all') ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "jiaqizho (via GitHub)" <gi...@apache.org>.
jiaqizho commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2023882177
##########
contrib/pax_storage/doc/README.md:
##########
@@ -0,0 +1,140 @@
+# PAX
+
+## Overview
+
+PAX(Partition Attributes Across) is an access method in database that combines the advantages of row storage (NSM, N-ary Storage Model) and column storage (DSM, Decomposition Storage Model), aiming to improve database query performance, especially in terms of cache efficiency.
+
+In OLAP scenarios, PAX has batch write performance comparable to row storage and read performance comparable to column storage. PAX can adapt to both OSS storage models in cloud environments and traditional offline physical file-based storage methods.
+
+PAX has the following features:
+
+- **CRUD Operations**: Supports basic Create, Read, Update, and Delete capabilities
+- **Indexes Support**: Supports multiple types of indexes to accelerate query operations, particularly enhancing data retrieval speed when handling large datasets.
+- **Concurrency Control & Isolation**: Utilizes Multiversion concurrency control (MVCC) to achieve efficient concurrency management and read-write isolation, operating at the granularity of individual data files for improved security and performance.
+- **Data Encoding & Compression**: Offers multiple encoding schemes (e.g., run-length encoding) and compression methods (e.g., zstd, zlib) with configurable levels, effectively reducing storage requirements while optimizing read performance.
+- **Statistics**: Data files contain comprehensive statistical information used for rapid filtering and query optimization, minimizing unnecessary data scanning and accelerating query processing.
+- **Sparsing Filter & Row Filter**: Enables efficient sparse filtering and row filtering through statistical metadata to reduce redundant data transfers.
+- **Data Clustering**: Supports data sorting via specified algorithms on single or multiple columns to enhance data orderliness and query efficiency.
+- **Vectorized Engine**: Includes an experimental vectorized execution engine designed to boost data processing capabilities and query performance, particularly for analytical workloads and report generation scenarios.
+
+## Build
+
+### Build PAX
+
+PAX will be built with `--enable-pax` when users building the Cloudberry.
+
+### Build debug version
+
+When using the configure script in the Cloudberry, add `--enable-cassert`.
+
+`--enable-cassert` will cause PAX to be compiled in **DEBUG** mode. And the `GTEST` in PAX will be built.
+
+Run GTEST:
+
+```
+cd contrib/pax_storage/build
+./src/cpp/test_main
+```
+
+## Usage
+
+### Create PAX table
+
+To create a table in PAX format, you need to set the table access method to PAX. You can do this by following one of the following methods:
+
+- Use the `USING PAX` clause when creating the table
+```
+CREATE TABLE t1(a int, b int, c text) USING PAX;
+```
+
+- Set the default access mode for the table before create the table
+```
+-- set default table access method to PAX, All newly created tables will use the PAX.
+SET default_table_access_method = pax;
+
+-- Will create the PAX table
+CREATE TABLE t1(a int, b int, c text);
+```
+
+### Create PAX table with reloptions
+
+Users can specify reloptions (such as `compresstype`, `storage_format`...) when creating a PAX table.
+
+```
+--- use the 'WITH' to specify reloptions
+CREATE TABLE p2(a INT, b INT, c INT) USING pax WITH(compresstype='zstd');
+
+--- use ',' to split multi reloptions
+CREATE TABLE p2(a INT, b INT, c INT) USING pax WITH(compresstype='zstd', compresslevel=5);
Review Comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "jiaqizho (via GitHub)" <gi...@apache.org>.
jiaqizho commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2025924951
##########
contrib/pax_storage/src/cpp/access/pax_access_handle.cc:
##########
@@ -0,0 +1,1217 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * pax_access_handle.cc
+ *
+ * IDENTIFICATION
+ * contrib/pax_storage/src/cpp/access/pax_access_handle.cc
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "access/pax_access_handle.h"
+
+#include "comm/cbdb_api.h"
+
+#include "access/pax_dml_state.h"
+#include "access/pax_scanner.h"
+#include "access/pax_table_cluster.h"
+#include "access/pax_updater.h"
+#include "access/paxc_rel_options.h"
+#include "catalog/pax_catalog.h"
+#include "clustering/zorder_utils.h"
+#include "comm/guc.h"
+#include "comm/pax_memory.h"
+#include "comm/pax_resource.h"
+#include "comm/paxc_wrappers.h"
+#include "comm/vec_numeric.h"
+#include "exceptions/CException.h"
+#include "storage/local_file_system.h"
+#ifdef VEC_BUILD
+#include "storage/vec_parallel_pax.h"
+#endif
+#include "storage/paxc_smgr.h"
+#include "storage/wal/pax_wal.h"
+#include "storage/wal/paxc_wal.h"
+
+#define NOT_IMPLEMENTED_YET \
+ ereport(ERROR, \
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \
+ errmsg("not implemented yet on pax relations: %s", __func__)))
+
+#define NOT_SUPPORTED_YET \
+ ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \
+ errmsg("not supported on pax relations: %s", __func__)))
+
+#define RELATION_IS_PAX(rel) \
+ (OidIsValid((rel)->rd_rel->relam) && RelationIsPAX(rel))
+
+// access methods that are implemented in C++
+namespace pax {
+
+TableScanDesc CCPaxAccessMethod::ScanBegin(Relation relation, Snapshot snapshot,
+ int nkeys, struct ScanKeyData *key,
+ ParallelTableScanDesc pscan,
+ uint32 flags) {
+ CBDB_TRY();
+ {
+ auto desc = PAX_NEW<PaxScanDesc>();
+ pax::common::RememberResourceCallback(pax::ReleaseTopObject<PaxScanDesc>,
+ cbdb::PointerToDatum(desc));
+
+ return desc->BeginScan(relation, snapshot, nkeys, key, pscan, flags,
+ nullptr, true);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_END_TRY();
+
+ pg_unreachable();
+}
+
+void CCPaxAccessMethod::ScanEnd(TableScanDesc scan) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ desc->EndScan();
+
+ pax::common::ForgetResourceCallback(pax::ReleaseTopObject<PaxScanDesc>,
+ cbdb::PointerToDatum(desc));
+ PAX_DELETE<PaxScanDesc>(desc);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+TableScanDesc CCPaxAccessMethod::ScanExtractColumns(
+ Relation rel, Snapshot snapshot, int nkeys, struct ScanKeyData *key,
+ ParallelTableScanDesc parallel_scan, struct PlanState *ps, uint32 flags) {
+ CBDB_TRY();
+ {
+ auto desc = PAX_NEW<PaxScanDesc>();
+ pax::common::RememberResourceCallback(pax::ReleaseTopObject<PaxScanDesc>,
+ cbdb::PointerToDatum(desc));
+
+ return desc->BeginScanExtractColumns(rel, snapshot, nkeys, key,
+ parallel_scan, ps, flags);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::IndexUniqueCheck(Relation rel, ItemPointer tid,
+ Snapshot snapshot, bool *all_dead) {
+ CBDB_TRY();
+ {
+ auto dsl = pax::CPaxDmlStateLocal::Instance();
+ if (dsl->IsInitialized()) {
+ auto del = dsl->GetDeleter(rel, snapshot, true);
+
+ if (del && del->IsMarked(*tid)) return false;
+ }
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+
+ return paxc::IndexUniqueCheck(rel, tid, snapshot, all_dead);
+}
+
+struct IndexFetchTableData *CCPaxAccessMethod::IndexFetchBegin(Relation rel) {
+ CBDB_TRY();
+ {
+ Assert(RELATION_IS_PAX(rel));
+ auto desc = PAX_NEW<PaxIndexScanDesc>(rel);
+ pax::common::RememberResourceCallback(
+ pax::ReleaseTopObject<PaxIndexScanDesc>, PointerGetDatum(desc));
+ return desc->ToBase();
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ return nullptr; // keep compiler quiet
+}
+
+void CCPaxAccessMethod::IndexFetchEnd(IndexFetchTableData *scan) {
+ CBDB_TRY();
+ {
+ auto desc = PaxIndexScanDesc::FromBase(scan);
+ desc->Release();
+
+ pax::common::ForgetResourceCallback(pax::ReleaseTopObject<PaxIndexScanDesc>,
+ PointerGetDatum(desc));
+ PAX_DELETE(desc);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+bool CCPaxAccessMethod::IndexFetchTuple(struct IndexFetchTableData *scan,
+ ItemPointer tid, Snapshot snapshot,
+ TupleTableSlot *slot, bool *call_again,
+ bool *all_dead) {
+ CBDB_TRY();
+ {
+ auto desc = PaxIndexScanDesc::FromBase(scan);
+ return desc->FetchTuple(tid, snapshot, slot, call_again, all_dead);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ return false; // keep compiler quiet
+}
+
+void CCPaxAccessMethod::IndexFetchReset(IndexFetchTableData * /*scan*/) {}
+
+void CCPaxAccessMethod::ScanRescan(TableScanDesc scan, ScanKey key,
+ bool set_params, bool allow_strat,
+ bool allow_sync, bool allow_pagemode) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ desc->ReScan(key, set_params, allow_strat, allow_sync, allow_pagemode);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+bool CCPaxAccessMethod::ScanGetNextSlot(TableScanDesc scan,
+ ScanDirection /*direction*/,
+ TupleTableSlot *slot) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ bool result;
+
+ result = desc->GetNextSlot(slot);
+ if (result) {
+ pgstat_count_heap_getnext(desc->GetRelation());
+ }
+ return result;
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+
+ pg_unreachable();
+}
+
+void CCPaxAccessMethod::TupleInsert(Relation relation, TupleTableSlot *slot,
+ CommandId cid, int options,
+ BulkInsertState bistate) {
+ CBDB_TRY();
+ {
+ MemoryContext old_ctx;
+ Assert(cbdb::pax_memory_context);
+
+#ifdef FAULT_INJECTOR
+ FaultInjector_InjectFaultIfSet("pax_insert", DDLNotSpecified, "",
+ RelationGetRelationName(relation));
+#endif
+
+ old_ctx = MemoryContextSwitchTo(cbdb::pax_memory_context);
+ CPaxInserter::TupleInsert(relation, slot, cid, options, bistate);
+ MemoryContextSwitchTo(old_ctx);
+
+ pgstat_count_heap_insert(relation, 1);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({
+ // FIXME: destroy CPaxInserter?
+ });
+ CBDB_END_TRY();
+}
+
+TM_Result CCPaxAccessMethod::TupleDelete(Relation relation, ItemPointer tid,
+ CommandId cid, Snapshot snapshot,
+ Snapshot /*crosscheck*/, bool /*wait*/,
+ TM_FailureData *tmfd,
+ bool /*changing_part*/) {
+ CBDB_TRY();
+ {
+ auto result = CPaxDeleter::DeleteTuple(relation, tid, cid, snapshot, tmfd);
+ if (result == TM_Ok) pgstat_count_heap_delete(relation);
+ return result;
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+TM_Result CCPaxAccessMethod::TupleUpdate(Relation relation, ItemPointer otid,
+ TupleTableSlot *slot, CommandId cid,
+ Snapshot snapshot, Snapshot crosscheck,
+ bool wait, TM_FailureData *tmfd,
+ LockTupleMode *lockmode,
+ bool *update_indexes) {
+ CBDB_TRY();
+ {
+ MemoryContext old_ctx;
+ TM_Result result;
+
+ Assert(cbdb::pax_memory_context);
+ old_ctx = MemoryContextSwitchTo(cbdb::pax_memory_context);
+ result = CPaxUpdater::UpdateTuple(relation, otid, slot, cid, snapshot,
+ crosscheck, wait, tmfd, lockmode,
+ update_indexes);
+ MemoryContextSwitchTo(old_ctx);
+ if (result == TM_Ok) pgstat_count_heap_update(relation, false);
+ return result;
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanAnalyzeNextBlock(TableScanDesc scan,
+ BlockNumber blockno,
+ BufferAccessStrategy bstrategy) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->ScanAnalyzeNextBlock(blockno, bstrategy);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanAnalyzeNextTuple(TableScanDesc scan,
+ TransactionId oldest_xmin,
+ double *liverows, double *deadrows,
+ TupleTableSlot *slot) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->ScanAnalyzeNextTuple(oldest_xmin, liverows, deadrows, slot);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanBitmapNextBlock(TableScanDesc scan,
+ TBMIterateResult *tbmres) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->BitmapNextBlock(tbmres);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanBitmapNextTuple(TableScanDesc scan,
+ TBMIterateResult *tbmres,
+ TupleTableSlot *slot) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ bool result;
+ result = desc->BitmapNextTuple(tbmres, slot);
+ if (result) {
+ pgstat_count_heap_fetch(desc->GetRelation());
+ }
+ return result;
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanSampleNextBlock(TableScanDesc scan,
+ SampleScanState *scanstate) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->ScanSampleNextBlock(scanstate);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanSampleNextTuple(TableScanDesc scan,
+ SampleScanState *scanstate,
+ TupleTableSlot *slot) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->ScanSampleNextTuple(scanstate, slot);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+void CCPaxAccessMethod::MultiInsert(Relation relation, TupleTableSlot **slots,
+ int ntuples, CommandId cid, int options,
+ BulkInsertState bistate) {
+ CBDB_TRY();
+ {
+ MemoryContext old_ctx;
+ Assert(cbdb::pax_memory_context);
+
+ old_ctx = MemoryContextSwitchTo(cbdb::pax_memory_context);
+ CPaxInserter::MultiInsert(relation, slots, ntuples, cid, options, bistate);
+ MemoryContextSwitchTo(old_ctx);
+
+ pgstat_count_heap_insert(relation, ntuples);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({
+ // FIXME: destroy CPaxInserter?
+ });
+ CBDB_END_TRY();
+}
+
+void CCPaxAccessMethod::FinishBulkInsert(Relation relation, int options) {
+ // Implement Pax dml cleanup for case "create table xxx1 as select * from
+ // xxx2", which would not call ExtDmlFini callback function and relies on
+ // FinishBulkInsert callback function to cleanup its dml state.
+ CBDB_TRY();
+ {
+ // no need switch memory context
+ // cause it just call dml finish
+ pax::CPaxInserter::FinishBulkInsert(relation, options);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({
+ // FIXME: destroy CPaxInserter?
+ });
+ CBDB_END_TRY();
+}
+
+void CCPaxAccessMethod::ExtDmlInit(Relation rel, CmdType operation) {
+ if (!RELATION_IS_PAX(rel)) return;
+
+ CBDB_TRY();
+ { pax::CPaxDmlStateLocal::Instance()->InitDmlState(rel, operation); }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+void CCPaxAccessMethod::ExtDmlFini(Relation rel, CmdType operation) {
+ if (!RELATION_IS_PAX(rel)) return;
+
+ CBDB_TRY();
+ { pax::CPaxDmlStateLocal::Instance()->FinishDmlState(rel, operation); }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+} // namespace pax
+// END of C++ implementation
+
+// access methods that are implemented in C
+namespace paxc {
+const TupleTableSlotOps *PaxAccessMethod::SlotCallbacks(
+ Relation /*rel*/) noexcept {
+ return &TTSOpsVirtual;
+}
+
+uint32 PaxAccessMethod::ScanFlags(Relation relation) {
+ uint32 flags = 0;
+#ifdef VEC_BUILD
+ flags |= SCAN_SUPPORT_VECTORIZATION | SCAN_SUPPORT_COLUMN_ORIENTED_SCAN;
+#else
+ flags |= SCAN_SUPPORT_COLUMN_ORIENTED_SCAN;
+#endif
+
+#if defined(USE_MANIFEST_API) && !defined(USE_PAX_CATALOG)
+ flags |= SCAN_FORCE_BIG_WRITE_LOCK;
+#endif
+
+ return flags;
+}
+
+Size PaxAccessMethod::ParallelscanEstimate(Relation /*rel*/) {
+ return sizeof(ParallelBlockTableScanDescData);
+}
+
+Size PaxAccessMethod::ParallelscanInitialize(Relation rel,
+ ParallelTableScanDesc pscan) {
+ ParallelBlockTableScanDesc bpscan = (ParallelBlockTableScanDesc)pscan;
+ bpscan->base.phs_relid = RelationGetRelid(rel);
+ pg_atomic_init_u64(&bpscan->phs_nallocated, 0);
+ // Like ao/aocs, we don't need phs_mutex and phs_startblock, though, init
+ // them.
+ SpinLockInit(&bpscan->phs_mutex);
+ bpscan->phs_startblock = InvalidBlockNumber;
+ return sizeof(ParallelBlockTableScanDescData);
+}
+
+void PaxAccessMethod::ParallelscanReinitialize(Relation rel,
+ ParallelTableScanDesc pscan) {
+ ParallelBlockTableScanDesc bpscan = (ParallelBlockTableScanDesc)pscan;
+
+ pg_atomic_write_u64(&bpscan->phs_nallocated, 0);
+}
+
+void PaxAccessMethod::TupleInsertSpeculative(Relation /*relation*/,
+ TupleTableSlot * /*slot*/,
+ CommandId /*cid*/, int /*options*/,
+ BulkInsertState /*bistate*/,
+ uint32 /*spec_token*/) {
+ NOT_IMPLEMENTED_YET;
+}
+
+void PaxAccessMethod::TupleCompleteSpeculative(Relation /*relation*/,
+ TupleTableSlot * /*slot*/,
+ uint32 /*spec_token*/,
+ bool /*succeeded*/) {
+ NOT_IMPLEMENTED_YET;
+}
+
+TM_Result PaxAccessMethod::TupleLock(Relation /*relation*/, ItemPointer /*tid*/,
+ Snapshot /*snapshot*/,
+ TupleTableSlot * /*slot*/,
+ CommandId /*cid*/, LockTupleMode /*mode*/,
+ LockWaitPolicy /*wait_policy*/,
+ uint8 /*flags*/,
+ TM_FailureData * /*tmfd*/) {
+ NOT_IMPLEMENTED_YET;
+ return TM_Ok;
+}
+
+bool PaxAccessMethod::TupleFetchRowVersion(Relation /*relation*/,
+ ItemPointer /*tid*/,
+ Snapshot /*snapshot*/,
+ TupleTableSlot * /*slot*/) {
+ NOT_IMPLEMENTED_YET;
+ return false;
+}
+
+bool PaxAccessMethod::TupleTidValid(TableScanDesc /*scan*/,
+ ItemPointer /*tid*/) {
+ NOT_IMPLEMENTED_YET;
+ return false;
+}
+
+void PaxAccessMethod::TupleGetLatestTid(TableScanDesc /*sscan*/,
+ ItemPointer /*tid*/) {
+ NOT_SUPPORTED_YET;
+}
+
+bool PaxAccessMethod::TupleSatisfiesSnapshot(Relation /*rel*/,
+ TupleTableSlot * /*slot*/,
+ Snapshot /*snapshot*/) {
+ NOT_IMPLEMENTED_YET;
+ return true;
+}
+
+TransactionId PaxAccessMethod::IndexDeleteTuples(
+ Relation /*rel*/, TM_IndexDeleteOp * /*delstate*/) {
+ NOT_SUPPORTED_YET;
+ return 0;
+}
+
+void PaxAccessMethod::RelationVacuum(Relation /*onerel*/,
+ VacuumParams * /*params*/,
+ BufferAccessStrategy /*bstrategy*/) {
+ /* PAX: micro-partitions have no dead tuples, so vacuum is empty */
+}
+
+BlockSequence *PaxAccessMethod::RelationGetBlockSequences(Relation rel,
+ int *numSequences) {
+ // PAX not support brin index yet
+ NOT_IMPLEMENTED_YET;
+ return nullptr;
+}
+
+void PaxAccessMethod::RelationGetBlockSequence(Relation rel, BlockNumber blkNum,
+ BlockSequence *sequence) {
+ // PAX not support brin index yet
+ NOT_IMPLEMENTED_YET;
+}
+
+bool PaxAccessMethod::RelationNeedsToastTable(Relation /*rel*/) {
+ // PAX never used the toasting, don't create the toast table from Cloudberry 7
+
+ return false;
+}
+
+double PaxAccessMethod::IndexBuildRangeScan(
+ Relation heap_relation, Relation index_relation, IndexInfo *index_info,
+ bool /*allow_sync*/, bool anyvisible, bool progress,
+ BlockNumber start_blockno, BlockNumber numblocks,
+ IndexBuildCallback callback, void *callback_state, TableScanDesc scan) {
+ Datum values[INDEX_MAX_KEYS];
+ bool isnull[INDEX_MAX_KEYS];
+ double reltuples = 0;
+ ExprState *predicate;
+ TupleTableSlot *slot;
+ EState *estate;
+ ExprContext *econtext;
+ Snapshot snapshot;
+ TransactionId OldestXmin;
+
+ bool checking_uniqueness pg_attribute_unused();
+ bool need_unregister_snapshot = false;
+ BlockNumber previous_blkno = InvalidBlockNumber;
+
+ Assert(OidIsValid(index_relation->rd_rel->relam));
+ Assert(!IsSystemRelation(heap_relation));
+
+ if (index_relation->rd_rel->relam != BTREE_AM_OID &&
+ index_relation->rd_rel->relam != HASH_AM_OID &&
+ index_relation->rd_rel->relam != GIN_AM_OID &&
+ index_relation->rd_rel->relam != BITMAP_AM_OID)
+ elog(ERROR, "pax only support btree/hash/gin/bitmap indexes");
+
+ checking_uniqueness =
+ (index_info->ii_Unique || index_info->ii_ExclusionOps != NULL);
+ // "Any visible" mode is not compatible with uniqueness checks; make sure
+ // only one of those is requested.
+ (void)anyvisible; // keep compiler quiet for release version
+ Assert(!(anyvisible && checking_uniqueness));
+
+ slot = table_slot_create(heap_relation, NULL);
+ estate = CreateExecutorState();
+ econtext = GetPerTupleExprContext(estate);
+ econtext->ecxt_scantuple = slot;
+ predicate = ExecPrepareQual(index_info->ii_Predicate, estate);
+
+ /*
+ * Prepare for scan of the base relation. In a normal index build, we use
+ * SnapshotAny because we must retrieve all tuples and do our own time
+ * qual checks (because we have to index RECENTLY_DEAD tuples). In a
+ * concurrent build, or during bootstrap, we take a regular MVCC snapshot
+ * and index whatever's live according to that.
+ */
+ OldestXmin = InvalidTransactionId;
+
+ /* okay to ignore lazy VACUUMs here */
+ if (!IsBootstrapProcessingMode() && !index_info->ii_Concurrent)
+ OldestXmin = GetOldestNonRemovableTransactionId(heap_relation);
+
+ if (!scan) {
+ if (!TransactionIdIsValid(OldestXmin)) {
+ snapshot = RegisterSnapshot(GetTransactionSnapshot());
+ need_unregister_snapshot = true;
+ } else {
+ need_unregister_snapshot = false;
+ snapshot = SnapshotAny;
+ }
+ scan = table_beginscan(heap_relation, snapshot, 0, NULL);
+ } else {
+ snapshot = scan->rs_snapshot;
+ need_unregister_snapshot = false;
+ }
+
+ // FIXME: Only brin index uses partial index now. setup start_blockno
+ // and numblocks is too late after beginscan is called now, because
+ // the current micro partition is opened. The workaround is ugly to
+ // check and close the current micro partition and open another one.
+ if (start_blockno != 0 || numblocks != InvalidBlockNumber)
+ elog(ERROR, "PAX doesn't support partial index scan now");
+
+ while (table_scan_getnextslot(scan, ForwardScanDirection, slot)) {
+ CHECK_FOR_INTERRUPTS();
+
+ if (progress) {
+ BlockNumber blkno = pax::GetBlockNumber(slot->tts_tid);
+ if (previous_blkno == InvalidBlockNumber)
+ previous_blkno = blkno;
+ else if (previous_blkno != blkno) {
+ pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+ blkno - start_blockno);
+ previous_blkno = blkno;
+ }
+ }
+ reltuples += 1;
+
+ MemoryContextReset(econtext->ecxt_per_tuple_memory);
+
+ /*
+ * In a partial index, discard tuples that don't satisfy the
+ * predicate.
+ */
+ if (predicate && !ExecQual(predicate, econtext)) continue;
+
+ /*
+ * For the current heap tuple, extract all the attributes we use in
+ * this index, and note which are null. This also performs evaluation
+ * of any expressions needed.
+ */
+ FormIndexDatum(index_info, slot, estate, values, isnull);
+
+ /*
+ * You'd think we should go ahead and build the index tuple here, but
+ * some index AMs want to do further processing on the data first. So
+ * pass the values[] and isnull[] arrays, instead.
+ */
+ callback(index_relation, &slot->tts_tid, values, isnull, true,
+ callback_state);
+ }
+
+ /* Report scan progress one last time. */
+ if (progress && previous_blkno != InvalidBlockNumber)
+ pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
+ previous_blkno + 1 - start_blockno);
+
+ table_endscan(scan);
+ if (need_unregister_snapshot) UnregisterSnapshot(snapshot);
+
+ ExecDropSingleTupleTableSlot(slot);
+ FreeExecutorState(estate);
+
+ /* These may have been pointing to the now-gone estate */
+ index_info->ii_ExpressionsState = NIL;
+ index_info->ii_PredicateState = NULL;
+
+ return reltuples;
+}
+
+void PaxAccessMethod::IndexValidateScan(Relation /*heap_relation*/,
+ Relation /*index_relation*/,
+ IndexInfo * /*index_info*/,
+ Snapshot /*snapshot*/,
+ ValidateIndexState * /*state*/) {
+ NOT_IMPLEMENTED_YET;
+}
+
+bytea *PaxAccessMethod::AmOptions(Datum reloptions, char relkind,
+ bool validate) {
+ return paxc_default_rel_options(reloptions, relkind, validate);
+}
+
+void PaxAccessMethod::ValidateColumnEncodingClauses(List *encoding_opts) {
+ paxc_validate_column_encoding_clauses(encoding_opts);
+}
+
+List *PaxAccessMethod::TransformColumnEncodingClauses(Relation /*rel*/,
+ List *encoding_opts,
+ bool validate,
+ bool from_type) {
+ return paxc_transform_column_encoding_clauses(encoding_opts, validate,
+ from_type);
+}
+
+} // namespace paxc
+// END of C implementation
+
+#define SET_LOCKTAG_INT64(tag, key64) \
+ SET_LOCKTAG_ADVISORY(tag, MyDatabaseId, (uint32)((key64) >> 32), \
+ (uint32)(key64), 1)
+
+extern "C" {
+
+static const TableAmRoutine kPaxColumnMethods = {
+ .type = T_TableAmRoutine,
+ .slot_callbacks = paxc::PaxAccessMethod::SlotCallbacks,
+ .scan_begin = pax::CCPaxAccessMethod::ScanBegin,
+ .scan_begin_extractcolumns = pax::CCPaxAccessMethod::ScanExtractColumns,
+ .scan_end = pax::CCPaxAccessMethod::ScanEnd,
+ .scan_rescan = pax::CCPaxAccessMethod::ScanRescan,
+ .scan_getnextslot = pax::CCPaxAccessMethod::ScanGetNextSlot,
+ .scan_flags = paxc::PaxAccessMethod::ScanFlags,
+
+ .parallelscan_estimate = paxc::PaxAccessMethod::ParallelscanEstimate,
+ .parallelscan_initialize = paxc::PaxAccessMethod::ParallelscanInitialize,
+ .parallelscan_reinitialize =
+ paxc::PaxAccessMethod::ParallelscanReinitialize,
+
+ .index_fetch_begin = pax::CCPaxAccessMethod::IndexFetchBegin,
+ .index_fetch_reset = pax::CCPaxAccessMethod::IndexFetchReset,
+ .index_fetch_end = pax::CCPaxAccessMethod::IndexFetchEnd,
+ .index_fetch_tuple = pax::CCPaxAccessMethod::IndexFetchTuple,
+ .index_unique_check = pax::CCPaxAccessMethod::IndexUniqueCheck,
+
+ .tuple_fetch_row_version = paxc::PaxAccessMethod::TupleFetchRowVersion,
+ .tuple_tid_valid = paxc::PaxAccessMethod::TupleTidValid,
+ .tuple_get_latest_tid = paxc::PaxAccessMethod::TupleGetLatestTid,
+ .tuple_satisfies_snapshot = paxc::PaxAccessMethod::TupleSatisfiesSnapshot,
+ .index_delete_tuples = paxc::PaxAccessMethod::IndexDeleteTuples,
+
+ .tuple_insert = pax::CCPaxAccessMethod::TupleInsert,
+ .tuple_insert_speculative = paxc::PaxAccessMethod::TupleInsertSpeculative,
+ .tuple_complete_speculative =
+ paxc::PaxAccessMethod::TupleCompleteSpeculative,
+ .multi_insert = pax::CCPaxAccessMethod::MultiInsert,
+ .tuple_delete = pax::CCPaxAccessMethod::TupleDelete,
+ .tuple_update = pax::CCPaxAccessMethod::TupleUpdate,
+ .tuple_lock = paxc::PaxAccessMethod::TupleLock,
+ .finish_bulk_insert = pax::CCPaxAccessMethod::FinishBulkInsert,
+
+ .relation_set_new_filenode = pax::CCPaxAccessMethod::RelationSetNewFilenode,
+ .relation_nontransactional_truncate =
+ pax::CCPaxAccessMethod::RelationNontransactionalTruncate,
+ .relation_copy_data = pax::CCPaxAccessMethod::RelationCopyData,
+ .relation_copy_for_cluster = pax::CCPaxAccessMethod::RelationCopyForCluster,
+ .relation_vacuum = paxc::PaxAccessMethod::RelationVacuum,
+ .scan_analyze_next_block = pax::CCPaxAccessMethod::ScanAnalyzeNextBlock,
+ .scan_analyze_next_tuple = pax::CCPaxAccessMethod::ScanAnalyzeNextTuple,
+ .index_build_range_scan = paxc::PaxAccessMethod::IndexBuildRangeScan,
+ .index_validate_scan = paxc::PaxAccessMethod::IndexValidateScan,
+
+ .relation_size = paxc::PaxAccessMethod::RelationSize,
+ .relation_get_block_sequences =
+ paxc::PaxAccessMethod::RelationGetBlockSequences,
+ .relation_get_block_sequence =
+ paxc::PaxAccessMethod::RelationGetBlockSequence,
+ .relation_needs_toast_table =
+ paxc::PaxAccessMethod::RelationNeedsToastTable,
+
+ .relation_estimate_size = paxc::PaxAccessMethod::EstimateRelSize,
+ .scan_bitmap_next_block = pax::CCPaxAccessMethod::ScanBitmapNextBlock,
+ .scan_bitmap_next_tuple = pax::CCPaxAccessMethod::ScanBitmapNextTuple,
+ .scan_sample_next_block = pax::CCPaxAccessMethod::ScanSampleNextBlock,
+ .scan_sample_next_tuple = pax::CCPaxAccessMethod::ScanSampleNextTuple,
+
+ .amoptions = paxc::PaxAccessMethod::AmOptions,
+ .swap_relation_files = paxc::PaxAccessMethod::SwapRelationFiles,
+ .validate_column_encoding_clauses =
+ paxc::PaxAccessMethod::ValidateColumnEncodingClauses,
+ .transform_column_encoding_clauses =
+ paxc::PaxAccessMethod::TransformColumnEncodingClauses,
+};
+
+PG_MODULE_MAGIC;
+PG_FUNCTION_INFO_V1(pax_tableam_handler);
+Datum pax_tableam_handler(PG_FUNCTION_ARGS) { // NOLINT
+ PG_RETURN_POINTER(&kPaxColumnMethods);
+}
+
+static object_access_hook_type prev_object_access_hook = NULL;
+
+static ProcessUtility_hook_type prev_ProcessUtilit_hook = NULL;
+
+static bool relation_has_cluster_columns_options(Relation rel) {
+ auto *options = (paxc::PaxOptions *)(rel->rd_options);
+
+ if (options == nullptr) {
+ return false;
+ }
+
+ // if not zorder and lexical cluster type, return false
+ if (strcasecmp(options->cluster_type, PAX_LEXICAL_CLUSTER_TYPE) != 0 &&
+ strcasecmp(options->cluster_type, PAX_ZORDER_CLUSTER_TYPE) != 0) {
+ return false;
+ }
+
+ // no need to check zorder cluster type, because it has been checked in
+ // PaxObjectAccessHook
+ Bitmapset *bms = paxc::paxc_get_columns_index_by_options(
+ rel, options->cluster_columns(), nullptr, false);
+
+ if (bms) {
+ bms_free(bms);
+ }
+ return bms != nullptr;
+}
+
+static bool relation_has_clustered_index(Relation rel) {
+ // We need to find the index that has indisclustered set.
+ Oid indexOid = InvalidOid;
+ ListCell *index;
+ foreach (index, RelationGetIndexList(rel)) {
+ indexOid = lfirst_oid(index);
+ if (get_index_isclustered(indexOid)) break;
+ indexOid = InvalidOid;
+ }
+ return OidIsValid(indexOid);
+}
+
+static bool table_can_be_clustered(Relation rel) {
+ return relation_has_cluster_columns_options(rel) ||
+ relation_has_clustered_index(rel);
+}
+
+static void cluster_pax_rel(ClusterStmt *stmt, Relation rel,
+ Snapshot snapshot) {
+ CBDB_TRY();
+ { pax::Cluster(rel, snapshot, false); }
+ CBDB_CATCH_DEFAULT();
+ CBDB_END_TRY();
+}
+
+static void validateAlterTableStmt(AlterTableStmt *stmt) {
+ ListCell *lcmd = NULL;
+
+ if (stmt->objtype != OBJECT_TABLE) return;
+
+ foreach (lcmd, stmt->cmds) {
+ AlterTableCmd *cmd = (AlterTableCmd *)lfirst(lcmd);
+ /*only check set tablespace command*/
+ if (cmd->subtype == AT_SetTableSpace) {
+ Oid tablespaceId;
+
+ /* Check that the tablespace exists */
+ tablespaceId = get_tablespace_oid(cmd->name, false);
+
+ /* alter table stmt does not support setting tablespace of dfs tablespace
+ * type */
+ if (OidIsValid(tablespaceId) && paxc::IsDfsTablespaceById(tablespaceId)) {
+ elog(ERROR,
+ "changing the dfs tablespace type in alter table stmt is not "
+ "supported");
+ }
+ }
+ }
+}
+
+static void validateAlterDatabaseStmt(AlterDatabaseStmt *stmt) {
+ ListCell *lcmd = NULL;
+
+ foreach (lcmd, stmt->options) {
+ DefElem *defel = (DefElem *)lfirst(lcmd);
+ if (pg_strcasecmp(defel->defname, "tablespace") == 0) {
+ Oid tablespaceId;
+
+ /* Check that the tablespace exists */
+ tablespaceId = get_tablespace_oid(defGetString(defel), false);
+
+ /* alter database stmt does not support setting tablespace of dfs
+ * tablespace type */
+ if (OidIsValid(tablespaceId) && paxc::IsDfsTablespaceById(tablespaceId)) {
+ elog(ERROR,
+ "changing dfs tablespace in alter database stmt is not supported");
+ }
+ }
+ }
+}
+
+static void checkUnsupportDfsTableSpaceStmt(Node *stmt) {
+ switch (nodeTag(stmt)) {
+ case T_AlterTableStmt:
+ validateAlterTableStmt((AlterTableStmt *)stmt);
+ break;
+ case T_AlterDatabaseStmt:
+ validateAlterDatabaseStmt((AlterDatabaseStmt *)stmt);
+ break;
+ default:
+ return;
+ }
+}
+
+static void paxProcessUtility(PlannedStmt *pstmt, const char *queryString,
+ bool readOnlyTree, ProcessUtilityContext context,
+ ParamListInfo params, QueryEnvironment *queryEnv,
+ DestReceiver *dest,
+ QueryCompletion *completionTag) {
+ bool isTopLevel = (context == PROCESS_UTILITY_TOPLEVEL);
+ // if is pax table, do something
+ switch (nodeTag(pstmt->utilityStmt)) {
+ case T_ClusterStmt: {
+ ClusterStmt *stmt = (ClusterStmt *)pstmt->utilityStmt;
+ if (stmt->relation) {
+ Relation rel = table_openrv(stmt->relation, AccessShareLock);
+ if (RelationIsPAX(rel)) {
+ bool has_cluster_columns;
+ if (stmt->indexname == NULL && !table_can_be_clustered(rel)) {
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("there is no previously clustered index or "
+ "cluster_columns reloptions for table \"%s\"",
+ stmt->relation->relname)));
+ }
+
+ has_cluster_columns = relation_has_cluster_columns_options(rel);
+
+ // cluster table using indexname,we should check if
+ // relation_has_zorder_clusted
+ if (stmt->indexname && has_cluster_columns) {
+ elog(ERROR,
+ "cannot using index to cluster table which has "
+ "cluster-columns ");
+ }
+
+ if (has_cluster_columns) {
+ if (Gp_role == GP_ROLE_DISPATCH) {
+ CdbDispatchUtilityStatement(
+ (Node *)stmt,
+ DF_CANCEL_ON_ERROR | DF_WITH_SNAPSHOT | DF_NEED_TWO_PHASE,
+ GetAssignedOidsForDispatch(), NULL);
+ } else if (Gp_role == GP_ROLE_EXECUTE) {
+ cluster_pax_rel(stmt, rel, GetActiveSnapshot());
+ }
+
+ table_close(rel, NoLock);
+ return;
+ }
+ }
+
+ table_close(rel, NoLock);
+ } else {
+ if (Gp_role == GP_ROLE_DISPATCH) {
+ // only cluster pax zorder clustered table
+ List *relids = NULL;
+ SysScanDesc scan;
+ HeapTuple tuple;
+
+ // We cannot run this form of CLUSTER inside a user transaction block;
+ // we'd be holding locks way too long.
+ PreventInTransactionBlock(isTopLevel, "CLUSTER");
+
+#if defined(USE_MANIFEST_API)
+ auto pax_rel = table_open(RelationRelationId, AccessShareLock);
+ scan = systable_beginscan(pax_rel, InvalidOid, false,
+ GetActiveSnapshot(), 0, nullptr);
+ while (HeapTupleIsValid(tuple = systable_getnext(scan))) {
+ Datum datum;
+ bool isnull;
+
+ datum = heap_getattr(tuple, Anum_pg_class_relam, pax_rel->rd_att,
+ &isnull);
+ if (isnull || DatumGetObjectId(datum) != PAX_TABLE_AM_OID) continue;
+
+ datum = heap_getattr(tuple, Anum_pg_class_oid, pax_rel->rd_att,
+ &isnull);
+ Assert(!isnull);
+ Oid relid = DatumGetObjectId(datum);
+ relids = lappend_oid(relids, relid);
+ }
+ systable_endscan(scan);
+ table_close(pax_rel, AccessShareLock);
+#else
+ auto pax_aux_rel =
+ table_open(PAX_TABLES_RELATION_ID, AccessShareLock);
+ scan = systable_beginscan(pax_aux_rel, InvalidOid, false,
+ GetActiveSnapshot(), 0, NULL);
+
+ while ((tuple = systable_getnext(scan)) != NULL) {
+ Datum datum;
+ bool isnull;
+ datum = heap_getattr(tuple, ANUM_PG_PAX_TABLES_RELID,
+ pax_aux_rel->rd_att, &isnull);
+ Assert(!isnull);
+ Oid relid = DatumGetObjectId(datum);
+ relids = lappend_oid(relids, relid);
+ }
+ systable_endscan(scan);
+ table_close(pax_aux_rel, AccessShareLock);
+#endif
+
+ ListCell *lc = NULL;
+ foreach (lc, relids) {
+ Oid pax_rel_id = Oid(lfirst_oid(lc));
+
+ Relation rel = table_open(pax_rel_id, RowExclusiveLock);
+ if (relation_has_cluster_columns_options(rel)) {
+ stmt->relation = makeNode(RangeVar);
+ stmt->relation->schemaname =
+ get_namespace_name(rel->rd_rel->relnamespace);
+ stmt->relation->relname = pstrdup(rel->rd_rel->relname.data);
+ CdbDispatchUtilityStatement((Node *)stmt,
+ DF_CANCEL_ON_ERROR | DF_WITH_SNAPSHOT,
+ GetAssignedOidsForDispatch(), NULL);
+ pfree(stmt->relation);
+ stmt->relation = NULL;
+ }
+ table_close(rel, RowExclusiveLock);
+ }
+
+ list_free(relids);
+ }
+ }
+ break;
+ }
+ default:
+ if (Gp_role == GP_ROLE_DISPATCH)
+ checkUnsupportDfsTableSpaceStmt(pstmt->utilityStmt);
+ break;
+ }
+
+ if (prev_ProcessUtilit_hook)
+ prev_ProcessUtilit_hook(pstmt, queryString, readOnlyTree, context, params,
+ queryEnv, dest, completionTag);
+ else
+ standard_ProcessUtility(pstmt, queryString, readOnlyTree, context, params,
+ queryEnv, dest, completionTag);
+}
+
+static void PaxCheckMinMaxColumns(Relation rel, const char *minmax_columns) {
Review Comment:
PAX not support `minmax_columns='all'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2024983923
##########
contrib/pax_storage/Makefile:
##########
@@ -0,0 +1,132 @@
+# contrib/pax_storage/Makefile
+
+MODULE_big = pax
+OBJS = \
+ $(WIN32RES)
+PG_CPPFLAGS = -I/usr/local/include
+PG_CXXFLAGS = -std=c++14
+
+PGFILEDESC = "pax - PAX table access method"
+SHLIB_LINK += -luuid
+
+PAX_REGRESS_DIR = src/test/regress
+PAX_VEC_REGRESS_DIR = src/test/vec
+PAX_ISOLATION2_DIR = src/test/isolation2
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+PGXS_DIR := $(dir $(PGXS))
+PG_REGRESS = $(PGXS_DIR)/../../src/test/regress/pg_regress
+else
+subdir = contrib/pax_storage
+top_builddir = ../../
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+PG_REGRESS = $(top_builddir)/src/test/regress/pg_regress
+endif
+
+.PHONY: all
+all: build
+
+DIRS = $(shell find src/cpp -maxdepth 2 -type d)
+SRC = $(foreach dir, $(DIRS), $(wildcard $(dir)/*.cc))
+CSRC = $(foreach dir, $(DIRS), $(wildcard $(dir)/*.c))
+
+# read options from ENV
+USE_MANIFEST_API := $(shell echo $$USE_MANIFEST_API)
+USE_PAX_CATALOG := $(shell echo $$USE_PAX_CATALOG)
+
+# if the ENVs are not set, set default value
+ifeq ($(USE_MANIFEST_API),)
+ USE_MANIFEST_API := OFF
+endif
+
+ifeq ($(USE_PAX_CATALOG),)
+ USE_PAX_CATALOG := ON
+endif
+
+.PHONY: install-data build
+build: $(SRC) $(CSRC)
+ @echo "build pax, USE_MANIFEST_API=$(USE_MANIFEST_API) USE_PAX_CATALOG=$(USE_PAX_CATALOG)"
+ @if [ ! -f build/Makefile ]; then \
+ mkdir -p build && \
+ cd build && \
+ cmake -DCMAKE_INSTALL_PREFIX=$(DESTDIR)$(prefix) \
+ -DUSE_MANIFEST_API=$(USE_MANIFEST_API) \
+ -DUSE_PAX_CATALOG=$(USE_PAX_CATALOG) .. ; \
+ fi
+ cd build && make -j8
Review Comment:
suggest not use nproc, in docker env, nproc maybe large than 8, but in fact virtual cpu cores is 4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "jiaqizho (via GitHub)" <gi...@apache.org>.
jiaqizho commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2034444114
##########
contrib/pax_storage/FindDependencies.cmake:
##########
@@ -0,0 +1,22 @@
+## protobuf
+include(FindProtobuf)
+find_package(Protobuf 3.6.1 REQUIRED)
+
+# ztsd
+# in our image snapshot, zstd is managed using pkg-config, so so the pkg-config method is used first here
+find_package(PkgConfig QUIET)
+if(PKGCONFIG_FOUND)
Review Comment:
nope, can't build if zstd not found
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2024153810
##########
contrib/pax_storage/Makefile:
##########
@@ -0,0 +1,132 @@
+# contrib/pax_storage/Makefile
+
+MODULE_big = pax
+OBJS = \
+ $(WIN32RES)
+PG_CPPFLAGS = -I/usr/local/include
+PG_CXXFLAGS = -std=c++14
+
+PGFILEDESC = "pax - PAX table access method"
+SHLIB_LINK += -luuid
+
+PAX_REGRESS_DIR = src/test/regress
+PAX_VEC_REGRESS_DIR = src/test/vec
+PAX_ISOLATION2_DIR = src/test/isolation2
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+PGXS_DIR := $(dir $(PGXS))
+PG_REGRESS = $(PGXS_DIR)/../../src/test/regress/pg_regress
+else
+subdir = contrib/pax_storage
+top_builddir = ../../
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+PG_REGRESS = $(top_builddir)/src/test/regress/pg_regress
+endif
+
+.PHONY: all
+all: build
+
+DIRS = $(shell find src/cpp -maxdepth 2 -type d)
+SRC = $(foreach dir, $(DIRS), $(wildcard $(dir)/*.cc))
+CSRC = $(foreach dir, $(DIRS), $(wildcard $(dir)/*.c))
+
+# read options from ENV
+USE_MANIFEST_API := $(shell echo $$USE_MANIFEST_API)
+USE_PAX_CATALOG := $(shell echo $$USE_PAX_CATALOG)
+
+# if the ENVs are not set, set default value
+ifeq ($(USE_MANIFEST_API),)
+ USE_MANIFEST_API := OFF
+endif
+
+ifeq ($(USE_PAX_CATALOG),)
+ USE_PAX_CATALOG := ON
+endif
+
+.PHONY: install-data build
+build: $(SRC) $(CSRC)
+ @echo "build pax, USE_MANIFEST_API=$(USE_MANIFEST_API) USE_PAX_CATALOG=$(USE_PAX_CATALOG)"
+ @if [ ! -f build/Makefile ]; then \
+ mkdir -p build && \
+ cd build && \
+ cmake -DCMAKE_INSTALL_PREFIX=$(DESTDIR)$(prefix) \
+ -DUSE_MANIFEST_API=$(USE_MANIFEST_API) \
+ -DUSE_PAX_CATALOG=$(USE_PAX_CATALOG) .. ; \
+ fi
+ cd build && make -j8
Review Comment:
```suggestion
cd build && make -j$(nproc)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#issuecomment-2772793693
Add support `.scan_flags` to add `SCAN_SUPPORT_RUNTIME_FILTER` capacity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2025232610
##########
contrib/pax_storage/src/cpp/access/pax_access_handle.cc:
##########
@@ -0,0 +1,1217 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * pax_access_handle.cc
+ *
+ * IDENTIFICATION
+ * contrib/pax_storage/src/cpp/access/pax_access_handle.cc
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "access/pax_access_handle.h"
+
+#include "comm/cbdb_api.h"
+
+#include "access/pax_dml_state.h"
+#include "access/pax_scanner.h"
+#include "access/pax_table_cluster.h"
+#include "access/pax_updater.h"
+#include "access/paxc_rel_options.h"
+#include "catalog/pax_catalog.h"
+#include "clustering/zorder_utils.h"
+#include "comm/guc.h"
+#include "comm/pax_memory.h"
+#include "comm/pax_resource.h"
+#include "comm/paxc_wrappers.h"
+#include "comm/vec_numeric.h"
+#include "exceptions/CException.h"
+#include "storage/local_file_system.h"
+#ifdef VEC_BUILD
+#include "storage/vec_parallel_pax.h"
+#endif
+#include "storage/paxc_smgr.h"
+#include "storage/wal/pax_wal.h"
+#include "storage/wal/paxc_wal.h"
+
+#define NOT_IMPLEMENTED_YET \
+ ereport(ERROR, \
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \
+ errmsg("not implemented yet on pax relations: %s", __func__)))
+
+#define NOT_SUPPORTED_YET \
+ ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \
+ errmsg("not supported on pax relations: %s", __func__)))
+
+#define RELATION_IS_PAX(rel) \
+ (OidIsValid((rel)->rd_rel->relam) && RelationIsPAX(rel))
+
+// access methods that are implemented in C++
+namespace pax {
+
+TableScanDesc CCPaxAccessMethod::ScanBegin(Relation relation, Snapshot snapshot,
+ int nkeys, struct ScanKeyData *key,
+ ParallelTableScanDesc pscan,
+ uint32 flags) {
+ CBDB_TRY();
+ {
+ auto desc = PAX_NEW<PaxScanDesc>();
+ pax::common::RememberResourceCallback(pax::ReleaseTopObject<PaxScanDesc>,
+ cbdb::PointerToDatum(desc));
+
+ return desc->BeginScan(relation, snapshot, nkeys, key, pscan, flags,
+ nullptr, true);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_END_TRY();
+
+ pg_unreachable();
+}
+
+void CCPaxAccessMethod::ScanEnd(TableScanDesc scan) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ desc->EndScan();
+
+ pax::common::ForgetResourceCallback(pax::ReleaseTopObject<PaxScanDesc>,
+ cbdb::PointerToDatum(desc));
+ PAX_DELETE<PaxScanDesc>(desc);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+TableScanDesc CCPaxAccessMethod::ScanExtractColumns(
+ Relation rel, Snapshot snapshot, int nkeys, struct ScanKeyData *key,
+ ParallelTableScanDesc parallel_scan, struct PlanState *ps, uint32 flags) {
+ CBDB_TRY();
+ {
+ auto desc = PAX_NEW<PaxScanDesc>();
+ pax::common::RememberResourceCallback(pax::ReleaseTopObject<PaxScanDesc>,
+ cbdb::PointerToDatum(desc));
+
+ return desc->BeginScanExtractColumns(rel, snapshot, nkeys, key,
+ parallel_scan, ps, flags);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::IndexUniqueCheck(Relation rel, ItemPointer tid,
+ Snapshot snapshot, bool *all_dead) {
+ CBDB_TRY();
+ {
+ auto dsl = pax::CPaxDmlStateLocal::Instance();
+ if (dsl->IsInitialized()) {
+ auto del = dsl->GetDeleter(rel, snapshot, true);
+
+ if (del && del->IsMarked(*tid)) return false;
+ }
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+
+ return paxc::IndexUniqueCheck(rel, tid, snapshot, all_dead);
+}
+
+struct IndexFetchTableData *CCPaxAccessMethod::IndexFetchBegin(Relation rel) {
+ CBDB_TRY();
+ {
+ Assert(RELATION_IS_PAX(rel));
+ auto desc = PAX_NEW<PaxIndexScanDesc>(rel);
+ pax::common::RememberResourceCallback(
+ pax::ReleaseTopObject<PaxIndexScanDesc>, PointerGetDatum(desc));
+ return desc->ToBase();
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ return nullptr; // keep compiler quiet
+}
+
+void CCPaxAccessMethod::IndexFetchEnd(IndexFetchTableData *scan) {
+ CBDB_TRY();
+ {
+ auto desc = PaxIndexScanDesc::FromBase(scan);
+ desc->Release();
+
+ pax::common::ForgetResourceCallback(pax::ReleaseTopObject<PaxIndexScanDesc>,
+ PointerGetDatum(desc));
+ PAX_DELETE(desc);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+bool CCPaxAccessMethod::IndexFetchTuple(struct IndexFetchTableData *scan,
+ ItemPointer tid, Snapshot snapshot,
+ TupleTableSlot *slot, bool *call_again,
+ bool *all_dead) {
+ CBDB_TRY();
+ {
+ auto desc = PaxIndexScanDesc::FromBase(scan);
+ return desc->FetchTuple(tid, snapshot, slot, call_again, all_dead);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ return false; // keep compiler quiet
+}
+
+void CCPaxAccessMethod::IndexFetchReset(IndexFetchTableData * /*scan*/) {}
+
+void CCPaxAccessMethod::ScanRescan(TableScanDesc scan, ScanKey key,
+ bool set_params, bool allow_strat,
+ bool allow_sync, bool allow_pagemode) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ desc->ReScan(key, set_params, allow_strat, allow_sync, allow_pagemode);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+bool CCPaxAccessMethod::ScanGetNextSlot(TableScanDesc scan,
+ ScanDirection /*direction*/,
+ TupleTableSlot *slot) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ bool result;
+
+ result = desc->GetNextSlot(slot);
+ if (result) {
+ pgstat_count_heap_getnext(desc->GetRelation());
+ }
+ return result;
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+
+ pg_unreachable();
+}
+
+void CCPaxAccessMethod::TupleInsert(Relation relation, TupleTableSlot *slot,
+ CommandId cid, int options,
+ BulkInsertState bistate) {
+ CBDB_TRY();
+ {
+ MemoryContext old_ctx;
+ Assert(cbdb::pax_memory_context);
+
+#ifdef FAULT_INJECTOR
+ FaultInjector_InjectFaultIfSet("pax_insert", DDLNotSpecified, "",
+ RelationGetRelationName(relation));
+#endif
+
+ old_ctx = MemoryContextSwitchTo(cbdb::pax_memory_context);
+ CPaxInserter::TupleInsert(relation, slot, cid, options, bistate);
+ MemoryContextSwitchTo(old_ctx);
+
+ pgstat_count_heap_insert(relation, 1);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({
+ // FIXME: destroy CPaxInserter?
+ });
+ CBDB_END_TRY();
+}
+
+TM_Result CCPaxAccessMethod::TupleDelete(Relation relation, ItemPointer tid,
+ CommandId cid, Snapshot snapshot,
+ Snapshot /*crosscheck*/, bool /*wait*/,
+ TM_FailureData *tmfd,
+ bool /*changing_part*/) {
+ CBDB_TRY();
+ {
+ auto result = CPaxDeleter::DeleteTuple(relation, tid, cid, snapshot, tmfd);
+ if (result == TM_Ok) pgstat_count_heap_delete(relation);
+ return result;
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+TM_Result CCPaxAccessMethod::TupleUpdate(Relation relation, ItemPointer otid,
+ TupleTableSlot *slot, CommandId cid,
+ Snapshot snapshot, Snapshot crosscheck,
+ bool wait, TM_FailureData *tmfd,
+ LockTupleMode *lockmode,
+ bool *update_indexes) {
+ CBDB_TRY();
+ {
+ MemoryContext old_ctx;
+ TM_Result result;
+
+ Assert(cbdb::pax_memory_context);
+ old_ctx = MemoryContextSwitchTo(cbdb::pax_memory_context);
+ result = CPaxUpdater::UpdateTuple(relation, otid, slot, cid, snapshot,
+ crosscheck, wait, tmfd, lockmode,
+ update_indexes);
+ MemoryContextSwitchTo(old_ctx);
+ if (result == TM_Ok) pgstat_count_heap_update(relation, false);
+ return result;
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanAnalyzeNextBlock(TableScanDesc scan,
+ BlockNumber blockno,
+ BufferAccessStrategy bstrategy) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->ScanAnalyzeNextBlock(blockno, bstrategy);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanAnalyzeNextTuple(TableScanDesc scan,
+ TransactionId oldest_xmin,
+ double *liverows, double *deadrows,
+ TupleTableSlot *slot) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->ScanAnalyzeNextTuple(oldest_xmin, liverows, deadrows, slot);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanBitmapNextBlock(TableScanDesc scan,
+ TBMIterateResult *tbmres) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->BitmapNextBlock(tbmres);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanBitmapNextTuple(TableScanDesc scan,
+ TBMIterateResult *tbmres,
+ TupleTableSlot *slot) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ bool result;
+ result = desc->BitmapNextTuple(tbmres, slot);
+ if (result) {
+ pgstat_count_heap_fetch(desc->GetRelation());
+ }
+ return result;
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanSampleNextBlock(TableScanDesc scan,
+ SampleScanState *scanstate) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->ScanSampleNextBlock(scanstate);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanSampleNextTuple(TableScanDesc scan,
+ SampleScanState *scanstate,
+ TupleTableSlot *slot) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->ScanSampleNextTuple(scanstate, slot);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+void CCPaxAccessMethod::MultiInsert(Relation relation, TupleTableSlot **slots,
+ int ntuples, CommandId cid, int options,
+ BulkInsertState bistate) {
+ CBDB_TRY();
+ {
+ MemoryContext old_ctx;
+ Assert(cbdb::pax_memory_context);
+
+ old_ctx = MemoryContextSwitchTo(cbdb::pax_memory_context);
+ CPaxInserter::MultiInsert(relation, slots, ntuples, cid, options, bistate);
+ MemoryContextSwitchTo(old_ctx);
+
+ pgstat_count_heap_insert(relation, ntuples);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({
+ // FIXME: destroy CPaxInserter?
+ });
+ CBDB_END_TRY();
+}
+
+void CCPaxAccessMethod::FinishBulkInsert(Relation relation, int options) {
+ // Implement Pax dml cleanup for case "create table xxx1 as select * from
+ // xxx2", which would not call ExtDmlFini callback function and relies on
+ // FinishBulkInsert callback function to cleanup its dml state.
+ CBDB_TRY();
+ {
+ // no need switch memory context
+ // cause it just call dml finish
+ pax::CPaxInserter::FinishBulkInsert(relation, options);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({
+ // FIXME: destroy CPaxInserter?
+ });
+ CBDB_END_TRY();
+}
+
+void CCPaxAccessMethod::ExtDmlInit(Relation rel, CmdType operation) {
+ if (!RELATION_IS_PAX(rel)) return;
+
+ CBDB_TRY();
+ { pax::CPaxDmlStateLocal::Instance()->InitDmlState(rel, operation); }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+void CCPaxAccessMethod::ExtDmlFini(Relation rel, CmdType operation) {
+ if (!RELATION_IS_PAX(rel)) return;
+
+ CBDB_TRY();
+ { pax::CPaxDmlStateLocal::Instance()->FinishDmlState(rel, operation); }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+} // namespace pax
+// END of C++ implementation
+
+// access methods that are implemented in C
+namespace paxc {
+const TupleTableSlotOps *PaxAccessMethod::SlotCallbacks(
+ Relation /*rel*/) noexcept {
+ return &TTSOpsVirtual;
+}
+
+uint32 PaxAccessMethod::ScanFlags(Relation relation) {
+ uint32 flags = 0;
+#ifdef VEC_BUILD
+ flags |= SCAN_SUPPORT_VECTORIZATION | SCAN_SUPPORT_COLUMN_ORIENTED_SCAN;
+#else
+ flags |= SCAN_SUPPORT_COLUMN_ORIENTED_SCAN;
Review Comment:
add support `SCAN_SUPPORT_RUNTIME_FILTER`, I will push qual to pax as we done in cloud.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "gfphoenix78 (via GitHub)" <gi...@apache.org>.
gfphoenix78 commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2036380818
##########
src/include/access/tableam.h:
##########
@@ -53,6 +53,18 @@ struct ValidateIndexState;
#define SCAN_SUPPORT_COLUMN_ORIENTED_SCAN (1 << 0) /* support column-oriented scanning*/
#define SCAN_SUPPORT_RUNTIME_FILTER (1 << 1) /* support runtime filter scan */
+/*
+ * Flags represented the supported features of scan.
+ *
+ * Reserve first 8 bits for public.
+ *
+ * NOTE: If you add a new flag, make sure the flag's bit is consecutive with
+ * the previous one.
+ */
+#define SCAN_SUPPORT_VECTORIZATION (1 << 8) /* support vectorization scan */
+
+#define SCAN_FORCE_BIG_WRITE_LOCK (1 << 9) /* force big write lock */
+
Review Comment:
The flags should be compact. The lower bits are used by kernel, higher bits are used by other modules or future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2025357288
##########
contrib/pax_storage/src/cpp/manifest/manifest.c:
##########
@@ -0,0 +1,793 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * manifest.c
+ *
+ * IDENTIFICATION
+ * contrib/pax_storage/src/cpp/manifest/manifest.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include <errno.h>
+#include <sys/param.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include <string.h>
+
+#include "postgres.h"
+#include "nodes/pg_list.h"
+#include "storage/relfilenode.h"
+
+#include "comm/pax_rel.h"
+#include "manifest.h"
+#include "tuple.h"
+#include "yyjson.h"
+
+#include "access/relation.h"
+#include "access/genam.h"
+#include "access/heapam.h"
+#include "access/table.h"
+#include "access/tableam.h"
+#include "access/tupmacs.h"
+#include "access/xact.h"
+#include "catalog/indexing.h"
+#include "catalog/pax_catalog_columns.h"
+#include "catalog/pg_namespace_d.h"
+#include "catalog/heap.h"
+#include "catalog/index.h"
+#include "catalog/pg_opclass_d.h"
+#include "catalog/dependency.h"
+#include "catalog/storage.h"
+#include "fmgr.h"
+#include "storage/lmgr.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/hsearch.h"
+#include "utils/memutils.h"
+
+#include "utils/bytea.h"
+#include "utils/relfilenodemap.h"
+#include "utils/lsyscache.h"
+
+#include "cdb/cdbvars.h"
+
+// headers from pax
+#include "catalog/pax_fastsequence.h"
+#include "manifest_wrapper.h"
+
+
+#define PAX_MICROPARTITION_DIR_POSTFIX "_pax"
+
+/*
+ * issues:
+ * there is a issue for our cache memory, we setup two cache, one is for heap
+ * and the other is for relation, the HTAB global variable initialization should
+ * happend after the null check, but when the transaction exit with exception,
+ * we have no chance to clear the HTAB global variables, so that we have no way
+ * to do the right initialization.
+ *
+ * extend descriptor fields actually should only appended on init, we'd better
+ * not provide a method to extend the fields at any time.
+ */
+
+MemoryContext manifest_mcontext = NULL;
Review Comment:
where used manifest_mcontext ?
##########
contrib/pax_storage/Makefile:
##########
@@ -0,0 +1,111 @@
+# contrib/pax_storage/Makefile
+
+MODULE_big = pax
+OBJS = \
+ $(WIN32RES)
+PG_CPPFLAGS = -I/usr/local/include
+PG_CXXFLAGS = -std=c++14
Review Comment:
why here is c++14 , not c++17 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#issuecomment-2771496684
Hey, this URL is also not available to the public.
https://github.com/jiaqizho/cloudberrydb/blob/287abbdc714e410b2f5c02927c365186899c5d7c/contrib/pax_storage/hd-ci/compile_pax.bash#L19
If the dir `hd-ci` is not used to the public, can remove them. FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2023542929
##########
contrib/pax_storage/.gitmodules:
##########
@@ -0,0 +1,10 @@
+[submodule "src/cpp/contrib/googletest"]
+ path = src/cpp/contrib/googletest
+ url = https://code.hashdata.xyz/cloudberry/googletest
+[submodule "src/cpp/contrib/zstd"]
+ path = src/cpp/contrib/zstd
+ url = https://code.hashdata.xyz/cloudberry/lib_zstd.git
+ branch = v1.5.5
Review Comment:
don't use `code.hashdata.xyz`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2023545173
##########
contrib/pax_storage/CMakeLists.txt:
##########
@@ -0,0 +1,96 @@
+project(Pax)
+cmake_minimum_required (VERSION 3.11.0)
+set(CMAKE_CXX_STANDARD 17)
Review Comment:
why c++17?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "jiaqizho (via GitHub)" <gi...@apache.org>.
jiaqizho commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2023882177
##########
contrib/pax_storage/doc/README.md:
##########
@@ -0,0 +1,140 @@
+# PAX
+
+## Overview
+
+PAX(Partition Attributes Across) is an access method in database that combines the advantages of row storage (NSM, N-ary Storage Model) and column storage (DSM, Decomposition Storage Model), aiming to improve database query performance, especially in terms of cache efficiency.
+
+In OLAP scenarios, PAX has batch write performance comparable to row storage and read performance comparable to column storage. PAX can adapt to both OSS storage models in cloud environments and traditional offline physical file-based storage methods.
+
+PAX has the following features:
+
+- **CRUD Operations**: Supports basic Create, Read, Update, and Delete capabilities
+- **Indexes Support**: Supports multiple types of indexes to accelerate query operations, particularly enhancing data retrieval speed when handling large datasets.
+- **Concurrency Control & Isolation**: Utilizes Multiversion concurrency control (MVCC) to achieve efficient concurrency management and read-write isolation, operating at the granularity of individual data files for improved security and performance.
+- **Data Encoding & Compression**: Offers multiple encoding schemes (e.g., run-length encoding) and compression methods (e.g., zstd, zlib) with configurable levels, effectively reducing storage requirements while optimizing read performance.
+- **Statistics**: Data files contain comprehensive statistical information used for rapid filtering and query optimization, minimizing unnecessary data scanning and accelerating query processing.
+- **Sparsing Filter & Row Filter**: Enables efficient sparse filtering and row filtering through statistical metadata to reduce redundant data transfers.
+- **Data Clustering**: Supports data sorting via specified algorithms on single or multiple columns to enhance data orderliness and query efficiency.
+- **Vectorized Engine**: Includes an experimental vectorized execution engine designed to boost data processing capabilities and query performance, particularly for analytical workloads and report generation scenarios.
+
+## Build
+
+### Build PAX
+
+PAX will be built with `--enable-pax` when users building the Cloudberry.
+
+### Build debug version
+
+When using the configure script in the Cloudberry, add `--enable-cassert`.
+
+`--enable-cassert` will cause PAX to be compiled in **DEBUG** mode. And the `GTEST` in PAX will be built.
+
+Run GTEST:
+
+```
+cd contrib/pax_storage/build
+./src/cpp/test_main
+```
+
+## Usage
+
+### Create PAX table
+
+To create a table in PAX format, you need to set the table access method to PAX. You can do this by following one of the following methods:
+
+- Use the `USING PAX` clause when creating the table
+```
+CREATE TABLE t1(a int, b int, c text) USING PAX;
+```
+
+- Set the default access mode for the table before create the table
+```
+-- set default table access method to PAX, All newly created tables will use the PAX.
+SET default_table_access_method = pax;
+
+-- Will create the PAX table
+CREATE TABLE t1(a int, b int, c text);
+```
+
+### Create PAX table with reloptions
+
+Users can specify reloptions (such as `compresstype`, `storage_format`...) when creating a PAX table.
+
+```
+--- use the 'WITH' to specify reloptions
+CREATE TABLE p2(a INT, b INT, c INT) USING pax WITH(compresstype='zstd');
+
+--- use ',' to split multi reloptions
+CREATE TABLE p2(a INT, b INT, c INT) USING pax WITH(compresstype='zstd', compresslevel=5);
Review Comment:
sure, no commit missing in current version of PAX
##########
contrib/pax_storage/CMakeLists.txt:
##########
@@ -0,0 +1,96 @@
+project(Pax)
+cmake_minimum_required (VERSION 3.11.0)
+set(CMAKE_CXX_STANDARD 17)
Review Comment:
used the `std::optional`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "jiaqizho (via GitHub)" <gi...@apache.org>.
jiaqizho commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2025931415
##########
contrib/pax_storage/src/cpp/access/pax_access_handle.cc:
##########
@@ -0,0 +1,1217 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * pax_access_handle.cc
+ *
+ * IDENTIFICATION
+ * contrib/pax_storage/src/cpp/access/pax_access_handle.cc
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "access/pax_access_handle.h"
+
+#include "comm/cbdb_api.h"
+
+#include "access/pax_dml_state.h"
+#include "access/pax_scanner.h"
+#include "access/pax_table_cluster.h"
+#include "access/pax_updater.h"
+#include "access/paxc_rel_options.h"
+#include "catalog/pax_catalog.h"
+#include "clustering/zorder_utils.h"
+#include "comm/guc.h"
+#include "comm/pax_memory.h"
+#include "comm/pax_resource.h"
+#include "comm/paxc_wrappers.h"
+#include "comm/vec_numeric.h"
+#include "exceptions/CException.h"
+#include "storage/local_file_system.h"
+#ifdef VEC_BUILD
+#include "storage/vec_parallel_pax.h"
+#endif
+#include "storage/paxc_smgr.h"
+#include "storage/wal/pax_wal.h"
+#include "storage/wal/paxc_wal.h"
+
+#define NOT_IMPLEMENTED_YET \
+ ereport(ERROR, \
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \
+ errmsg("not implemented yet on pax relations: %s", __func__)))
+
+#define NOT_SUPPORTED_YET \
+ ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \
+ errmsg("not supported on pax relations: %s", __func__)))
+
+#define RELATION_IS_PAX(rel) \
+ (OidIsValid((rel)->rd_rel->relam) && RelationIsPAX(rel))
+
+// access methods that are implemented in C++
+namespace pax {
+
+TableScanDesc CCPaxAccessMethod::ScanBegin(Relation relation, Snapshot snapshot,
+ int nkeys, struct ScanKeyData *key,
+ ParallelTableScanDesc pscan,
+ uint32 flags) {
+ CBDB_TRY();
+ {
+ auto desc = PAX_NEW<PaxScanDesc>();
+ pax::common::RememberResourceCallback(pax::ReleaseTopObject<PaxScanDesc>,
+ cbdb::PointerToDatum(desc));
+
+ return desc->BeginScan(relation, snapshot, nkeys, key, pscan, flags,
+ nullptr, true);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_END_TRY();
+
+ pg_unreachable();
+}
+
+void CCPaxAccessMethod::ScanEnd(TableScanDesc scan) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ desc->EndScan();
+
+ pax::common::ForgetResourceCallback(pax::ReleaseTopObject<PaxScanDesc>,
+ cbdb::PointerToDatum(desc));
+ PAX_DELETE<PaxScanDesc>(desc);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+TableScanDesc CCPaxAccessMethod::ScanExtractColumns(
+ Relation rel, Snapshot snapshot, int nkeys, struct ScanKeyData *key,
+ ParallelTableScanDesc parallel_scan, struct PlanState *ps, uint32 flags) {
+ CBDB_TRY();
+ {
+ auto desc = PAX_NEW<PaxScanDesc>();
+ pax::common::RememberResourceCallback(pax::ReleaseTopObject<PaxScanDesc>,
+ cbdb::PointerToDatum(desc));
+
+ return desc->BeginScanExtractColumns(rel, snapshot, nkeys, key,
+ parallel_scan, ps, flags);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::IndexUniqueCheck(Relation rel, ItemPointer tid,
+ Snapshot snapshot, bool *all_dead) {
+ CBDB_TRY();
+ {
+ auto dsl = pax::CPaxDmlStateLocal::Instance();
+ if (dsl->IsInitialized()) {
+ auto del = dsl->GetDeleter(rel, snapshot, true);
+
+ if (del && del->IsMarked(*tid)) return false;
+ }
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+
+ return paxc::IndexUniqueCheck(rel, tid, snapshot, all_dead);
+}
+
+struct IndexFetchTableData *CCPaxAccessMethod::IndexFetchBegin(Relation rel) {
+ CBDB_TRY();
+ {
+ Assert(RELATION_IS_PAX(rel));
+ auto desc = PAX_NEW<PaxIndexScanDesc>(rel);
+ pax::common::RememberResourceCallback(
+ pax::ReleaseTopObject<PaxIndexScanDesc>, PointerGetDatum(desc));
+ return desc->ToBase();
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ return nullptr; // keep compiler quiet
+}
+
+void CCPaxAccessMethod::IndexFetchEnd(IndexFetchTableData *scan) {
+ CBDB_TRY();
+ {
+ auto desc = PaxIndexScanDesc::FromBase(scan);
+ desc->Release();
+
+ pax::common::ForgetResourceCallback(pax::ReleaseTopObject<PaxIndexScanDesc>,
+ PointerGetDatum(desc));
+ PAX_DELETE(desc);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+bool CCPaxAccessMethod::IndexFetchTuple(struct IndexFetchTableData *scan,
+ ItemPointer tid, Snapshot snapshot,
+ TupleTableSlot *slot, bool *call_again,
+ bool *all_dead) {
+ CBDB_TRY();
+ {
+ auto desc = PaxIndexScanDesc::FromBase(scan);
+ return desc->FetchTuple(tid, snapshot, slot, call_again, all_dead);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ return false; // keep compiler quiet
+}
+
+void CCPaxAccessMethod::IndexFetchReset(IndexFetchTableData * /*scan*/) {}
+
+void CCPaxAccessMethod::ScanRescan(TableScanDesc scan, ScanKey key,
+ bool set_params, bool allow_strat,
+ bool allow_sync, bool allow_pagemode) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ desc->ReScan(key, set_params, allow_strat, allow_sync, allow_pagemode);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+bool CCPaxAccessMethod::ScanGetNextSlot(TableScanDesc scan,
+ ScanDirection /*direction*/,
+ TupleTableSlot *slot) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ bool result;
+
+ result = desc->GetNextSlot(slot);
+ if (result) {
+ pgstat_count_heap_getnext(desc->GetRelation());
+ }
+ return result;
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+
+ pg_unreachable();
+}
+
+void CCPaxAccessMethod::TupleInsert(Relation relation, TupleTableSlot *slot,
+ CommandId cid, int options,
+ BulkInsertState bistate) {
+ CBDB_TRY();
+ {
+ MemoryContext old_ctx;
+ Assert(cbdb::pax_memory_context);
+
+#ifdef FAULT_INJECTOR
+ FaultInjector_InjectFaultIfSet("pax_insert", DDLNotSpecified, "",
+ RelationGetRelationName(relation));
+#endif
+
+ old_ctx = MemoryContextSwitchTo(cbdb::pax_memory_context);
+ CPaxInserter::TupleInsert(relation, slot, cid, options, bistate);
+ MemoryContextSwitchTo(old_ctx);
+
+ pgstat_count_heap_insert(relation, 1);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({
+ // FIXME: destroy CPaxInserter?
+ });
+ CBDB_END_TRY();
+}
+
+TM_Result CCPaxAccessMethod::TupleDelete(Relation relation, ItemPointer tid,
+ CommandId cid, Snapshot snapshot,
+ Snapshot /*crosscheck*/, bool /*wait*/,
+ TM_FailureData *tmfd,
+ bool /*changing_part*/) {
+ CBDB_TRY();
+ {
+ auto result = CPaxDeleter::DeleteTuple(relation, tid, cid, snapshot, tmfd);
+ if (result == TM_Ok) pgstat_count_heap_delete(relation);
+ return result;
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+TM_Result CCPaxAccessMethod::TupleUpdate(Relation relation, ItemPointer otid,
+ TupleTableSlot *slot, CommandId cid,
+ Snapshot snapshot, Snapshot crosscheck,
+ bool wait, TM_FailureData *tmfd,
+ LockTupleMode *lockmode,
+ bool *update_indexes) {
+ CBDB_TRY();
+ {
+ MemoryContext old_ctx;
+ TM_Result result;
+
+ Assert(cbdb::pax_memory_context);
+ old_ctx = MemoryContextSwitchTo(cbdb::pax_memory_context);
+ result = CPaxUpdater::UpdateTuple(relation, otid, slot, cid, snapshot,
+ crosscheck, wait, tmfd, lockmode,
+ update_indexes);
+ MemoryContextSwitchTo(old_ctx);
+ if (result == TM_Ok) pgstat_count_heap_update(relation, false);
+ return result;
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanAnalyzeNextBlock(TableScanDesc scan,
+ BlockNumber blockno,
+ BufferAccessStrategy bstrategy) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->ScanAnalyzeNextBlock(blockno, bstrategy);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanAnalyzeNextTuple(TableScanDesc scan,
+ TransactionId oldest_xmin,
+ double *liverows, double *deadrows,
+ TupleTableSlot *slot) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->ScanAnalyzeNextTuple(oldest_xmin, liverows, deadrows, slot);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanBitmapNextBlock(TableScanDesc scan,
+ TBMIterateResult *tbmres) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->BitmapNextBlock(tbmres);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanBitmapNextTuple(TableScanDesc scan,
+ TBMIterateResult *tbmres,
+ TupleTableSlot *slot) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ bool result;
+ result = desc->BitmapNextTuple(tbmres, slot);
+ if (result) {
+ pgstat_count_heap_fetch(desc->GetRelation());
+ }
+ return result;
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanSampleNextBlock(TableScanDesc scan,
+ SampleScanState *scanstate) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->ScanSampleNextBlock(scanstate);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanSampleNextTuple(TableScanDesc scan,
+ SampleScanState *scanstate,
+ TupleTableSlot *slot) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->ScanSampleNextTuple(scanstate, slot);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+void CCPaxAccessMethod::MultiInsert(Relation relation, TupleTableSlot **slots,
+ int ntuples, CommandId cid, int options,
+ BulkInsertState bistate) {
+ CBDB_TRY();
+ {
+ MemoryContext old_ctx;
+ Assert(cbdb::pax_memory_context);
+
+ old_ctx = MemoryContextSwitchTo(cbdb::pax_memory_context);
+ CPaxInserter::MultiInsert(relation, slots, ntuples, cid, options, bistate);
+ MemoryContextSwitchTo(old_ctx);
+
+ pgstat_count_heap_insert(relation, ntuples);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({
+ // FIXME: destroy CPaxInserter?
+ });
+ CBDB_END_TRY();
+}
+
+void CCPaxAccessMethod::FinishBulkInsert(Relation relation, int options) {
+ // Implement Pax dml cleanup for case "create table xxx1 as select * from
+ // xxx2", which would not call ExtDmlFini callback function and relies on
+ // FinishBulkInsert callback function to cleanup its dml state.
+ CBDB_TRY();
+ {
+ // no need switch memory context
+ // cause it just call dml finish
+ pax::CPaxInserter::FinishBulkInsert(relation, options);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({
+ // FIXME: destroy CPaxInserter?
+ });
+ CBDB_END_TRY();
+}
+
+void CCPaxAccessMethod::ExtDmlInit(Relation rel, CmdType operation) {
+ if (!RELATION_IS_PAX(rel)) return;
+
+ CBDB_TRY();
+ { pax::CPaxDmlStateLocal::Instance()->InitDmlState(rel, operation); }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+void CCPaxAccessMethod::ExtDmlFini(Relation rel, CmdType operation) {
+ if (!RELATION_IS_PAX(rel)) return;
+
+ CBDB_TRY();
+ { pax::CPaxDmlStateLocal::Instance()->FinishDmlState(rel, operation); }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+} // namespace pax
+// END of C++ implementation
+
+// access methods that are implemented in C
+namespace paxc {
+const TupleTableSlotOps *PaxAccessMethod::SlotCallbacks(
+ Relation /*rel*/) noexcept {
+ return &TTSOpsVirtual;
+}
+
+uint32 PaxAccessMethod::ScanFlags(Relation relation) {
+ uint32 flags = 0;
+#ifdef VEC_BUILD
+ flags |= SCAN_SUPPORT_VECTORIZATION | SCAN_SUPPORT_COLUMN_ORIENTED_SCAN;
+#else
+ flags |= SCAN_SUPPORT_COLUMN_ORIENTED_SCAN;
Review Comment:
I don't think that is a good idea to make PAX support runtime filter in current PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2023597884
##########
contrib/pax_storage/doc/README.md:
##########
@@ -0,0 +1,140 @@
+# PAX
+
+## Overview
+
+PAX(Partition Attributes Across) is an access method in database that combines the advantages of row storage (NSM, N-ary Storage Model) and column storage (DSM, Decomposition Storage Model), aiming to improve database query performance, especially in terms of cache efficiency.
+
+In OLAP scenarios, PAX has batch write performance comparable to row storage and read performance comparable to column storage. PAX can adapt to both OSS storage models in cloud environments and traditional offline physical file-based storage methods.
+
+PAX has the following features:
+
+- **CRUD Operations**: Supports basic Create, Read, Update, and Delete capabilities
+- **Indexes Support**: Supports multiple types of indexes to accelerate query operations, particularly enhancing data retrieval speed when handling large datasets.
+- **Concurrency Control & Isolation**: Utilizes Multiversion concurrency control (MVCC) to achieve efficient concurrency management and read-write isolation, operating at the granularity of individual data files for improved security and performance.
+- **Data Encoding & Compression**: Offers multiple encoding schemes (e.g., run-length encoding) and compression methods (e.g., zstd, zlib) with configurable levels, effectively reducing storage requirements while optimizing read performance.
+- **Statistics**: Data files contain comprehensive statistical information used for rapid filtering and query optimization, minimizing unnecessary data scanning and accelerating query processing.
+- **Sparsing Filter & Row Filter**: Enables efficient sparse filtering and row filtering through statistical metadata to reduce redundant data transfers.
+- **Data Clustering**: Supports data sorting via specified algorithms on single or multiple columns to enhance data orderliness and query efficiency.
+- **Vectorized Engine**: Includes an experimental vectorized execution engine designed to boost data processing capabilities and query performance, particularly for analytical workloads and report generation scenarios.
+
+## Build
+
+### Build PAX
+
+PAX will be built with `--enable-pax` when users building the Cloudberry.
+
+### Build debug version
+
+When using the configure script in the Cloudberry, add `--enable-cassert`.
+
+`--enable-cassert` will cause PAX to be compiled in **DEBUG** mode. And the `GTEST` in PAX will be built.
+
+Run GTEST:
+
+```
+cd contrib/pax_storage/build
+./src/cpp/test_main
+```
+
+## Usage
+
+### Create PAX table
+
+To create a table in PAX format, you need to set the table access method to PAX. You can do this by following one of the following methods:
+
+- Use the `USING PAX` clause when creating the table
+```
+CREATE TABLE t1(a int, b int, c text) USING PAX;
+```
+
+- Set the default access mode for the table before create the table
+```
+-- set default table access method to PAX, All newly created tables will use the PAX.
+SET default_table_access_method = pax;
+
+-- Will create the PAX table
+CREATE TABLE t1(a int, b int, c text);
+```
+
+### Create PAX table with reloptions
+
+Users can specify reloptions (such as `compresstype`, `storage_format`...) when creating a PAX table.
+
+```
+--- use the 'WITH' to specify reloptions
+CREATE TABLE p2(a INT, b INT, c INT) USING pax WITH(compresstype='zstd');
+
+--- use ',' to split multi reloptions
+CREATE TABLE p2(a INT, b INT, c INT) USING pax WITH(compresstype='zstd', compresslevel=5);
Review Comment:
suuport minmax_columns this version ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "jiaqizho (via GitHub)" <gi...@apache.org>.
jiaqizho commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2023954933
##########
contrib/pax_storage/.gitmodules:
##########
@@ -0,0 +1,10 @@
+[submodule "src/cpp/contrib/googletest"]
+ path = src/cpp/contrib/googletest
+ url = https://code.hashdata.xyz/cloudberry/googletest
+[submodule "src/cpp/contrib/zstd"]
+ path = src/cpp/contrib/zstd
+ url = https://code.hashdata.xyz/cloudberry/lib_zstd.git
+ branch = v1.5.5
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "gfphoenix78 (via GitHub)" <gi...@apache.org>.
gfphoenix78 commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2036370390
##########
contrib/pax_storage/src/test/isolation2/isolation2_unsupported_schedule:
##########
@@ -0,0 +1,7 @@
+# FIXME: MANIFEST: can't write the same table at the same time.
+# held a big write lock to avoid multiple write concurrently.
+
+test: crash_recovery
+test: resource_queue_deadlock
+test: segwalrep/commit_blocking
+test: segwalrep/fts_unblock_primary
Review Comment:
The following tests involves parallel write on pax tables, that will hang if manifest implementation is on.
test: pax/insert_policy
test: pax/max_concurrency2
test: pax/max_concurrency
test: pax/parallel_delete
test: pax/parallel_update_readcommitted
test: pax/parallel_update
test: pax/vacuum_while_insert
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2025943415
##########
contrib/pax_storage/src/cpp/access/pax_access_handle.cc:
##########
@@ -0,0 +1,1217 @@
+/*-------------------------------------------------------------------------
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * pax_access_handle.cc
+ *
+ * IDENTIFICATION
+ * contrib/pax_storage/src/cpp/access/pax_access_handle.cc
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "access/pax_access_handle.h"
+
+#include "comm/cbdb_api.h"
+
+#include "access/pax_dml_state.h"
+#include "access/pax_scanner.h"
+#include "access/pax_table_cluster.h"
+#include "access/pax_updater.h"
+#include "access/paxc_rel_options.h"
+#include "catalog/pax_catalog.h"
+#include "clustering/zorder_utils.h"
+#include "comm/guc.h"
+#include "comm/pax_memory.h"
+#include "comm/pax_resource.h"
+#include "comm/paxc_wrappers.h"
+#include "comm/vec_numeric.h"
+#include "exceptions/CException.h"
+#include "storage/local_file_system.h"
+#ifdef VEC_BUILD
+#include "storage/vec_parallel_pax.h"
+#endif
+#include "storage/paxc_smgr.h"
+#include "storage/wal/pax_wal.h"
+#include "storage/wal/paxc_wal.h"
+
+#define NOT_IMPLEMENTED_YET \
+ ereport(ERROR, \
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \
+ errmsg("not implemented yet on pax relations: %s", __func__)))
+
+#define NOT_SUPPORTED_YET \
+ ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \
+ errmsg("not supported on pax relations: %s", __func__)))
+
+#define RELATION_IS_PAX(rel) \
+ (OidIsValid((rel)->rd_rel->relam) && RelationIsPAX(rel))
+
+// access methods that are implemented in C++
+namespace pax {
+
+TableScanDesc CCPaxAccessMethod::ScanBegin(Relation relation, Snapshot snapshot,
+ int nkeys, struct ScanKeyData *key,
+ ParallelTableScanDesc pscan,
+ uint32 flags) {
+ CBDB_TRY();
+ {
+ auto desc = PAX_NEW<PaxScanDesc>();
+ pax::common::RememberResourceCallback(pax::ReleaseTopObject<PaxScanDesc>,
+ cbdb::PointerToDatum(desc));
+
+ return desc->BeginScan(relation, snapshot, nkeys, key, pscan, flags,
+ nullptr, true);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_END_TRY();
+
+ pg_unreachable();
+}
+
+void CCPaxAccessMethod::ScanEnd(TableScanDesc scan) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ desc->EndScan();
+
+ pax::common::ForgetResourceCallback(pax::ReleaseTopObject<PaxScanDesc>,
+ cbdb::PointerToDatum(desc));
+ PAX_DELETE<PaxScanDesc>(desc);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+TableScanDesc CCPaxAccessMethod::ScanExtractColumns(
+ Relation rel, Snapshot snapshot, int nkeys, struct ScanKeyData *key,
+ ParallelTableScanDesc parallel_scan, struct PlanState *ps, uint32 flags) {
+ CBDB_TRY();
+ {
+ auto desc = PAX_NEW<PaxScanDesc>();
+ pax::common::RememberResourceCallback(pax::ReleaseTopObject<PaxScanDesc>,
+ cbdb::PointerToDatum(desc));
+
+ return desc->BeginScanExtractColumns(rel, snapshot, nkeys, key,
+ parallel_scan, ps, flags);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::IndexUniqueCheck(Relation rel, ItemPointer tid,
+ Snapshot snapshot, bool *all_dead) {
+ CBDB_TRY();
+ {
+ auto dsl = pax::CPaxDmlStateLocal::Instance();
+ if (dsl->IsInitialized()) {
+ auto del = dsl->GetDeleter(rel, snapshot, true);
+
+ if (del && del->IsMarked(*tid)) return false;
+ }
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+
+ return paxc::IndexUniqueCheck(rel, tid, snapshot, all_dead);
+}
+
+struct IndexFetchTableData *CCPaxAccessMethod::IndexFetchBegin(Relation rel) {
+ CBDB_TRY();
+ {
+ Assert(RELATION_IS_PAX(rel));
+ auto desc = PAX_NEW<PaxIndexScanDesc>(rel);
+ pax::common::RememberResourceCallback(
+ pax::ReleaseTopObject<PaxIndexScanDesc>, PointerGetDatum(desc));
+ return desc->ToBase();
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ return nullptr; // keep compiler quiet
+}
+
+void CCPaxAccessMethod::IndexFetchEnd(IndexFetchTableData *scan) {
+ CBDB_TRY();
+ {
+ auto desc = PaxIndexScanDesc::FromBase(scan);
+ desc->Release();
+
+ pax::common::ForgetResourceCallback(pax::ReleaseTopObject<PaxIndexScanDesc>,
+ PointerGetDatum(desc));
+ PAX_DELETE(desc);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+bool CCPaxAccessMethod::IndexFetchTuple(struct IndexFetchTableData *scan,
+ ItemPointer tid, Snapshot snapshot,
+ TupleTableSlot *slot, bool *call_again,
+ bool *all_dead) {
+ CBDB_TRY();
+ {
+ auto desc = PaxIndexScanDesc::FromBase(scan);
+ return desc->FetchTuple(tid, snapshot, slot, call_again, all_dead);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ return false; // keep compiler quiet
+}
+
+void CCPaxAccessMethod::IndexFetchReset(IndexFetchTableData * /*scan*/) {}
+
+void CCPaxAccessMethod::ScanRescan(TableScanDesc scan, ScanKey key,
+ bool set_params, bool allow_strat,
+ bool allow_sync, bool allow_pagemode) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ desc->ReScan(key, set_params, allow_strat, allow_sync, allow_pagemode);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+bool CCPaxAccessMethod::ScanGetNextSlot(TableScanDesc scan,
+ ScanDirection /*direction*/,
+ TupleTableSlot *slot) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ bool result;
+
+ result = desc->GetNextSlot(slot);
+ if (result) {
+ pgstat_count_heap_getnext(desc->GetRelation());
+ }
+ return result;
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+
+ pg_unreachable();
+}
+
+void CCPaxAccessMethod::TupleInsert(Relation relation, TupleTableSlot *slot,
+ CommandId cid, int options,
+ BulkInsertState bistate) {
+ CBDB_TRY();
+ {
+ MemoryContext old_ctx;
+ Assert(cbdb::pax_memory_context);
+
+#ifdef FAULT_INJECTOR
+ FaultInjector_InjectFaultIfSet("pax_insert", DDLNotSpecified, "",
+ RelationGetRelationName(relation));
+#endif
+
+ old_ctx = MemoryContextSwitchTo(cbdb::pax_memory_context);
+ CPaxInserter::TupleInsert(relation, slot, cid, options, bistate);
+ MemoryContextSwitchTo(old_ctx);
+
+ pgstat_count_heap_insert(relation, 1);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({
+ // FIXME: destroy CPaxInserter?
+ });
+ CBDB_END_TRY();
+}
+
+TM_Result CCPaxAccessMethod::TupleDelete(Relation relation, ItemPointer tid,
+ CommandId cid, Snapshot snapshot,
+ Snapshot /*crosscheck*/, bool /*wait*/,
+ TM_FailureData *tmfd,
+ bool /*changing_part*/) {
+ CBDB_TRY();
+ {
+ auto result = CPaxDeleter::DeleteTuple(relation, tid, cid, snapshot, tmfd);
+ if (result == TM_Ok) pgstat_count_heap_delete(relation);
+ return result;
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+TM_Result CCPaxAccessMethod::TupleUpdate(Relation relation, ItemPointer otid,
+ TupleTableSlot *slot, CommandId cid,
+ Snapshot snapshot, Snapshot crosscheck,
+ bool wait, TM_FailureData *tmfd,
+ LockTupleMode *lockmode,
+ bool *update_indexes) {
+ CBDB_TRY();
+ {
+ MemoryContext old_ctx;
+ TM_Result result;
+
+ Assert(cbdb::pax_memory_context);
+ old_ctx = MemoryContextSwitchTo(cbdb::pax_memory_context);
+ result = CPaxUpdater::UpdateTuple(relation, otid, slot, cid, snapshot,
+ crosscheck, wait, tmfd, lockmode,
+ update_indexes);
+ MemoryContextSwitchTo(old_ctx);
+ if (result == TM_Ok) pgstat_count_heap_update(relation, false);
+ return result;
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanAnalyzeNextBlock(TableScanDesc scan,
+ BlockNumber blockno,
+ BufferAccessStrategy bstrategy) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->ScanAnalyzeNextBlock(blockno, bstrategy);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanAnalyzeNextTuple(TableScanDesc scan,
+ TransactionId oldest_xmin,
+ double *liverows, double *deadrows,
+ TupleTableSlot *slot) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->ScanAnalyzeNextTuple(oldest_xmin, liverows, deadrows, slot);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanBitmapNextBlock(TableScanDesc scan,
+ TBMIterateResult *tbmres) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->BitmapNextBlock(tbmres);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanBitmapNextTuple(TableScanDesc scan,
+ TBMIterateResult *tbmres,
+ TupleTableSlot *slot) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ bool result;
+ result = desc->BitmapNextTuple(tbmres, slot);
+ if (result) {
+ pgstat_count_heap_fetch(desc->GetRelation());
+ }
+ return result;
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanSampleNextBlock(TableScanDesc scan,
+ SampleScanState *scanstate) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->ScanSampleNextBlock(scanstate);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+bool CCPaxAccessMethod::ScanSampleNextTuple(TableScanDesc scan,
+ SampleScanState *scanstate,
+ TupleTableSlot *slot) {
+ CBDB_TRY();
+ {
+ auto desc = PaxScanDesc::ToDesc(scan);
+ return desc->ScanSampleNextTuple(scanstate, slot);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+ pg_unreachable();
+}
+
+void CCPaxAccessMethod::MultiInsert(Relation relation, TupleTableSlot **slots,
+ int ntuples, CommandId cid, int options,
+ BulkInsertState bistate) {
+ CBDB_TRY();
+ {
+ MemoryContext old_ctx;
+ Assert(cbdb::pax_memory_context);
+
+ old_ctx = MemoryContextSwitchTo(cbdb::pax_memory_context);
+ CPaxInserter::MultiInsert(relation, slots, ntuples, cid, options, bistate);
+ MemoryContextSwitchTo(old_ctx);
+
+ pgstat_count_heap_insert(relation, ntuples);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({
+ // FIXME: destroy CPaxInserter?
+ });
+ CBDB_END_TRY();
+}
+
+void CCPaxAccessMethod::FinishBulkInsert(Relation relation, int options) {
+ // Implement Pax dml cleanup for case "create table xxx1 as select * from
+ // xxx2", which would not call ExtDmlFini callback function and relies on
+ // FinishBulkInsert callback function to cleanup its dml state.
+ CBDB_TRY();
+ {
+ // no need switch memory context
+ // cause it just call dml finish
+ pax::CPaxInserter::FinishBulkInsert(relation, options);
+ }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({
+ // FIXME: destroy CPaxInserter?
+ });
+ CBDB_END_TRY();
+}
+
+void CCPaxAccessMethod::ExtDmlInit(Relation rel, CmdType operation) {
+ if (!RELATION_IS_PAX(rel)) return;
+
+ CBDB_TRY();
+ { pax::CPaxDmlStateLocal::Instance()->InitDmlState(rel, operation); }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+void CCPaxAccessMethod::ExtDmlFini(Relation rel, CmdType operation) {
+ if (!RELATION_IS_PAX(rel)) return;
+
+ CBDB_TRY();
+ { pax::CPaxDmlStateLocal::Instance()->FinishDmlState(rel, operation); }
+ CBDB_CATCH_DEFAULT();
+ CBDB_FINALLY({});
+ CBDB_END_TRY();
+}
+
+} // namespace pax
+// END of C++ implementation
+
+// access methods that are implemented in C
+namespace paxc {
+const TupleTableSlotOps *PaxAccessMethod::SlotCallbacks(
+ Relation /*rel*/) noexcept {
+ return &TTSOpsVirtual;
+}
+
+uint32 PaxAccessMethod::ScanFlags(Relation relation) {
+ uint32 flags = 0;
+#ifdef VEC_BUILD
+ flags |= SCAN_SUPPORT_VECTORIZATION | SCAN_SUPPORT_COLUMN_ORIENTED_SCAN;
+#else
+ flags |= SCAN_SUPPORT_COLUMN_ORIENTED_SCAN;
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "jiaqizho (via GitHub)" <gi...@apache.org>.
jiaqizho commented on code in PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#discussion_r2025938290
##########
contrib/pax_storage/FindDependencies.cmake:
##########
@@ -0,0 +1,22 @@
+## protobuf
+include(FindProtobuf)
+find_package(Protobuf 3.6.1 REQUIRED)
+
+# ztsd
+# in our image snapshot, zstd is managed using pkg-config, so so the pkg-config method is used first here
+find_package(PkgConfig QUIET)
+if(PKGCONFIG_FOUND)
Review Comment:
PAX won't check the version of zstd. Only use the `find_package` to check zstd exist. And direct link the zstd.
So no matter 1.5.1 or 1.5.5, PAX will link it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "jiaqizho (via GitHub)" <gi...@apache.org>.
jiaqizho commented on PR #1002:
URL: https://github.com/apache/cloudberry/pull/1002#issuecomment-2788342541
TPCC
env
```
cbdb: release build
segments: 3
warehouses=100
terminals=4
runMins=10
```
--enable-pax
```
13:25:03,442 [Thread-1] INFO jTPCC : Term-00,
13:25:03,442 [Thread-1] INFO jTPCC : Term-00,
13:25:03,443 [Thread-1] INFO jTPCC : Term-00, Measured tpmC (NewOrders) = 2.98
13:25:03,443 [Thread-1] INFO jTPCC : Term-00, Measured tpmTOTAL = 6.86
13:25:03,443 [Thread-1] INFO jTPCC : Term-00, Session Start = 2025-04-09 13:13:59
13:25:03,443 [Thread-1] INFO jTPCC : Term-00, Session End = 2025-04-09 13:25:03
13:25:03,443 [Thread-1] INFO jTPCC : Term-00, Transaction Count = 75
13:25:03,443 [Thread-1] INFO jTPCC : Term-00, Session finished!
```
--disable-pax
```
13:40:39,075 [Thread-4] INFO jTPCC : Term-00,
13:40:39,075 [Thread-4] INFO jTPCC : Term-00,
13:40:39,075 [Thread-4] INFO jTPCC : Term-00, Measured tpmC (NewOrders) = 3.29
13:40:39,076 [Thread-4] INFO jTPCC : Term-00, Measured tpmTOTAL = 6.78
13:40:39,076 [Thread-4] INFO jTPCC : Term-00, Session Start = 2025-04-09 13:30:02
13:40:39,076 [Thread-4] INFO jTPCC : Term-00, Session End = 2025-04-09 13:40:39
13:40:39,076 [Thread-4] INFO jTPCC : Term-00, Transaction Count = 71
13:40:39,077 [Thread-4] INFO jTPCC : Term-00, Session finished!
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Feature: introduce a high-performance hybrid row-columnar storage engine [cloudberry]
Posted by "jiaqizho (via GitHub)" <gi...@apache.org>.
jiaqizho closed pull request #1002: Feature: introduce a high-performance hybrid row-columnar storage engine
URL: https://github.com/apache/cloudberry/pull/1002
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org