[I] [Feature] Add query-scoped planning limits and cooperative cancellation [paimon-cpp]

["realAaronWu (via GitHub)" <gi...@apache.org>]

realAaronWu opened a new issue, #435:
URL: https://github.com/apache/paimon-cpp/issues/435

   Parent integration: sirius-db/sirius#2181. Consumer bridge tracking: polardb/duckdb-paimon#139.
   
   ## Need
   
   A database embedding paimon-cpp needs to admit planning work before materializing an arbitrarily large native plan, and stop submitting work after cooperative cancellation. This is a feature request, not a claim that existing standalone planning is incorrect.
   
   The integration pins `53f9c86d45aabb0a6f1a379271da07d7a9f27a3d`. The following API gap also remains in inspected main `242decffdfdefef01dcd0e52e48c9a1e69792479`:
   
   - `ScanContextBuilder::WithMemoryPool` routes pool-backed allocations, but does not bound ordinary STL planning containers or their transitive metadata allocations.
   - `TableScan::CreatePlan()` returns a complete plan; it exposes no planning cancellation/budget contract. `SetLimit` is a row-result pushdown, not a planning-memory quota: `DataTableBatchScan::CreatePlan` calls `starting_scanner_->Scan` before `ApplyPushDownLimit`.
   - `FileStoreScan::ReadFileEntries` builds a vector of futures for all supplied manifest metadata, then collects complete per-manifest entry vectors. A custom serial executor can prevent concurrent task execution, but does not itself bound those retained containers/results.
   
   Source references at the inspected main:
   - [Scan context API](https://github.com/apache/paimon-cpp/blob/242decffdfdefef01dcd0e52e48c9a1e69792479/include/paimon/scan_context.h)
   - [TableScan API](https://github.com/apache/paimon-cpp/blob/242decffdfdefef01dcd0e52e48c9a1e69792479/include/paimon/table/source/table_scan.h)
   - [Batch scan / limit ordering](https://github.com/apache/paimon-cpp/blob/242decffdfdefef01dcd0e52e48c9a1e69792479/src/paimon/core/table/source/data_table_batch_scan.cpp)
   - [Manifest task/results materialization](https://github.com/apache/paimon-cpp/blob/242decffdfdefef01dcd0e52e48c9a1e69792479/src/paimon/core/operation/file_store_scan.cpp)
   
   ## Proposed Contract To Discuss
   
   Provide optional query-scoped planning controls, preserving existing unlimited behavior by default. The API shape should be agreed with maintainers; a post-plan split-count check is not sufficient.
   
   - Check conservative manifest/entry/split/retained-metadata quotas before the corresponding allocation or task submission; identify which allocations are covered and what remains outside the guarantee.
   - Bound outstanding manifest work and retained results, including ordinary containers, rather than only pool-backed buffers.
   - Check a cooperative cancellation token before I/O/submission and inside substantial decode/merge loops. Do not promise interruption of an arbitrary blocking filesystem call unless the filesystem supports it.
   - Return a distinguishable resource-refusal/cancellation status, never partial success or EOF. Stop new submissions and drain accepted work before returning, including exceptions and submission failure.
   - Avoid changing process-global executors, Arrow thread capacity, or allocation resources.
   
   Acceptance tests should use many small manifests and injected allocation/I/O/submission failures. Assert the limit is enforced before excess work is accepted, accepted jobs are drained on every terminal path, and cancellation cannot return a partial successful plan. Include serial/nested executor configurations to guard against deadlock. Keep row LIMIT semantics separate.
   
   This issue does not request another Arrow batch-lifetime implementation; that work already exists in #259/#260. No hard whole-process-memory bound or prompt cancellation of uninterruptible I/O is claimed by the current integration.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Add query-scoped planning limits and cooperative cancellation [paimon-cpp]

["realAaronWu (via GitHub)" <gi...@apache.org>]

realAaronWu commented on issue #435:
URL: https://github.com/apache/paimon-cpp/issues/435#issuecomment-6203744474

   The source-first ownership audit found a narrow executor prerequisite, now tracked separately in #436 with an actual main-source CPU reproduction. Work is isolated and undergoing internal review. This does not implement or close query cancellation, task draining, or pre-materialization planning budgets; those remain in this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Add query-scoped planning limits and cooperative cancellation [paimon-cpp]

["realAaronWu (via GitHub)" <gi...@apache.org>]

realAaronWu commented on issue #435:
URL: https://github.com/apache/paimon-cpp/issues/435#issuecomment-6294611295

   We are discontinuing the expanded implementation in #440; this issue remains open. Existing whole-call waiting can suppress a late result, but does not provide in-call cancellation or pre-materialization planning-memory admission.
   
   Two narrower native safety requirements also remain, independently of a cancellation API. The examples below were source-reviewed at `dddfda5ed35792035c3cd14fd19663dc3d298253`; runtime fault injection has **not** been performed:
   
   - **Drain accepted work on failure.** [`CollectAll`](https://github.com/apache/paimon-cpp/blob/dddfda5ed35792035c3cd14fd19663dc3d298253/src/paimon/common/utils/future.h#L100) can unwind on allocation or a throwing `get()` before later accepted work is collected. Planner submissions borrow scan/snapshot state. [`SnapshotFileCollector::ReadManifests`](https://github.com/apache/paimon-cpp/blob/dddfda5ed35792035c3cd14fd19663dc3d298253/src/paimon/core/operation/snapshot_file_scan.cpp#L179) also has a later-manifest Status return after earlier work may have been submitted. The accepted work must not outlive borrowed planner/collector state. Test the actual planning/ListFiles entry points with one accepted task gated, a later Status/exception/submission failure, and release/drain assertions before return or unwinding. Preserve the first failure; do not return a partial successful plan/file list. Include nested single-worker progress separately from startup acceptance fixed in #436.
   - **Keep exported Arrow roots owned if lifetime-holder allocation throws.** [`AddArrowArrayLifetime`](https://github.com/apache/paimon-cpp/blob/dddfda5ed35792035c3cd14fd19663dc3d298253/src/paimon/common/utils/arrow/mem_utils.cpp#L151) allocates its holder after export. Callers such as [`ManifestMetaReader::NextBatch`](https://github.com/apache/paimon-cpp/blob/dddfda5ed35792035c3cd14fd19663dc3d298253/src/paimon/manifest/manifest_meta_reader.cpp#L60) hold ordinary C-struct shells, whose deletion does not invoke the C release callbacks. Inject that exact post-export allocation failure and verify both callbacks exactly once, original error identity and actual affected reader paths. This is distinct from the retained-batch lifetime work in #259/#260 and from rejecting an empty lifetime argument; it does not require a new cancellation framework or converting every OOM into an allocating Status.
   
   These are source-confirmed failure-path gaps, not claims of reproduced crashes or completed fixes. Planning admission/bounded retained containers, accepted-work draining/progress and any future agreed cancellation responsiveness remain separate acceptance requirements. Closing #440 must not mark them complete.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org