You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org> on 2026/06/28 17:30:01 UTC
[PR] feat(dao): add operator_port_cache table [texera]
Xiao-zhen-Liu opened a new pull request, #5967:
URL: https://github.com/apache/texera/pull/5967
### What changes were proposed in this PR?
Adds the `operator_port_cache` table that records a materialized output port
result so it can be reused across executions. It is keyed by
`(workflow_id, global_port_id, cache_key)` and stores the JSON the cache key was
computed from, the result location, an optional tuple count and source execution
id, and a database-managed `updated_at`. The foreign key to `workflow(wid)` is
`ON DELETE CASCADE`. The stored JSON (`cache_key_json`) lets a lookup confirm a
hash match by comparing the full JSON, so a hash collision never reuses the wrong
result.
The change is additive: a new table in `sql/texera_ddl.sql` (fresh installs) plus
a Liquibase migration `sql/updates/26.sql` registered in `sql/changelog.xml`
(existing deployments). No code reads or writes the table yet; the cache read/write
logic and its tests land with the cache service that uses it, following the
convention of testing a table through its consumer (as `feedback` is tested via
`FeedbackResourceSpec`).
### Any related issues, documentation, discussions?
Part of #5882 (umbrella #5881). Design discussion: #5880.
### How was this PR tested?
Verified the schema directly against Postgres: the migration applies cleanly, the
columns and primary key `(workflow_id, global_port_id, cache_key)` are correct,
the foreign key's delete rule is `CASCADE`, the schema file and the migration
define identical columns/keys, and `changelog.xml` is well-formed and registers
`26.sql`. The generated jOOQ classes build from the table. The table's runtime
behavior is exercised by the cache service tests in the follow-up PR.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8 (Claude Code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(dao): add operator_port_cache table [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #5967:
URL: https://github.com/apache/texera/pull/5967#issuecomment-4826863639
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/5967?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 56.28%. Comparing base ([`a24d1d1`](https://app.codecov.io/gh/apache/texera/commit/a24d1d1701dfffc46279ecda1db34268e2dca44c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f1d85fb`](https://app.codecov.io/gh/apache/texera/commit/f1d85fbbf6f300e51f21299a70bc86750edc1329?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #5967 +/- ##
=========================================
Coverage 56.28% 56.28%
Complexity 2992 2992
=========================================
Files 1120 1120
Lines 43217 43217
Branches 4662 4662
=========================================
Hits 24326 24326
Misses 17472 17472
Partials 1419 1419
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/5967/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/5967/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.00% <ø> (ø)` | | Carriedforward from [a24d1d1](https://app.codecov.io/gh/apache/texera/commit/a24d1d1701dfffc46279ecda1db34268e2dca44c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/5967/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.95% <ø> (ø)` | | Carriedforward from [a24d1d1](https://app.codecov.io/gh/apache/texera/commit/a24d1d1701dfffc46279ecda1db34268e2dca44c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [amber](https://app.codecov.io/gh/apache/texera/pull/5967/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.80% <ø> (ø)` | | Carriedforward from [a24d1d1](https://app.codecov.io/gh/apache/texera/commit/a24d1d1701dfffc46279ecda1db34268e2dca44c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/5967/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | | Carriedforward from [a24d1d1](https://app.codecov.io/gh/apache/texera/commit/a24d1d1701dfffc46279ecda1db34268e2dca44c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/5967/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `51.56% <ø> (ø)` | | |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/5967/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `59.02% <ø> (ø)` | | Carriedforward from [a24d1d1](https://app.codecov.io/gh/apache/texera/commit/a24d1d1701dfffc46279ecda1db34268e2dca44c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/5967/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.33% <ø> (ø)` | | Carriedforward from [a24d1d1](https://app.codecov.io/gh/apache/texera/commit/a24d1d1701dfffc46279ecda1db34268e2dca44c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [notebook-migration-service](https://app.codecov.io/gh/apache/texera/pull/5967/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.57% <ø> (ø)` | | Carriedforward from [a24d1d1](https://app.codecov.io/gh/apache/texera/commit/a24d1d1701dfffc46279ecda1db34268e2dca44c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/5967/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.20% <ø> (ø)` | | Carriedforward from [a24d1d1](https://app.codecov.io/gh/apache/texera/commit/a24d1d1701dfffc46279ecda1db34268e2dca44c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [python](https://app.codecov.io/gh/apache/texera/pull/5967/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.76% <ø> (ø)` | | Carriedforward from [a24d1d1](https://app.codecov.io/gh/apache/texera/commit/a24d1d1701dfffc46279ecda1db34268e2dca44c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/5967/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.14% <ø> (ø)` | | Carriedforward from [a24d1d1](https://app.codecov.io/gh/apache/texera/commit/a24d1d1701dfffc46279ecda1db34268e2dca44c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/5967?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(dao): add operator_port_cache table [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
Xiao-zhen-Liu commented on code in PR #5967:
URL: https://github.com/apache/texera/pull/5967#discussion_r3493440286
##########
sql/updates/26.sql:
##########
@@ -0,0 +1,47 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- ============================================
+-- 1. Connect to the texera_db database
+-- ============================================
+\c texera_db
+
+SET search_path TO texera_db;
+
+-- ============================================
+-- 2. Add the operator_port_cache table
+-- ============================================
+BEGIN;
+
+-- Caches a materialized output port result across executions, keyed by the cache
+-- key (a SHA-256 hash of the upstream sub-DAG). cache_key_json holds the JSON
+-- the cache key is computed from.
+CREATE TABLE IF NOT EXISTS operator_port_cache
+(
+ workflow_id INT NOT NULL,
+ global_port_id VARCHAR(200) NOT NULL,
+ cache_key CHAR(64) NOT NULL,
+ cache_key_json TEXT NOT NULL,
Review Comment:
Two columns on purpose: `cache_key` is the SHA-256 hash we look up by (and part of the PK), and `cache_key_json` is the JSON it was computed from. On a hash hit we compare the stored JSON to confirm the match, so a hash collision can never reuse the wrong result.
##########
sql/updates/26.sql:
##########
@@ -0,0 +1,47 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- ============================================
+-- 1. Connect to the texera_db database
+-- ============================================
+\c texera_db
+
+SET search_path TO texera_db;
+
+-- ============================================
+-- 2. Add the operator_port_cache table
+-- ============================================
+BEGIN;
+
+-- Caches a materialized output port result across executions, keyed by the cache
+-- key (a SHA-256 hash of the upstream sub-DAG). cache_key_json holds the JSON
+-- the cache key is computed from.
+CREATE TABLE IF NOT EXISTS operator_port_cache
+(
+ workflow_id INT NOT NULL,
+ global_port_id VARCHAR(200) NOT NULL,
+ cache_key CHAR(64) NOT NULL,
+ cache_key_json TEXT NOT NULL,
+ result_uri TEXT NOT NULL,
Review Comment:
Done, renamed to `storage_uri`.
##########
sql/updates/26.sql:
##########
@@ -0,0 +1,47 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- ============================================
+-- 1. Connect to the texera_db database
+-- ============================================
+\c texera_db
+
+SET search_path TO texera_db;
+
+-- ============================================
+-- 2. Add the operator_port_cache table
+-- ============================================
+BEGIN;
+
+-- Caches a materialized output port result across executions, keyed by the cache
+-- key (a SHA-256 hash of the upstream sub-DAG). cache_key_json holds the JSON
+-- the cache key is computed from.
+CREATE TABLE IF NOT EXISTS operator_port_cache
+(
+ workflow_id INT NOT NULL,
+ global_port_id VARCHAR(200) NOT NULL,
+ cache_key CHAR(64) NOT NULL,
+ cache_key_json TEXT NOT NULL,
+ result_uri TEXT NOT NULL,
+ tuple_count BIGINT,
Review Comment:
I'd keep `tuple_count`. A cache row is immutable (a changed computation is a new row, never an update), so it can't drift from the Iceberg result. It's populated at materialization and read by the coordinator during execution — the coordinator already queries this table for the cache lookup, so a cached region's output stats come from the same row with no extra read. Reading it from `getCount()` instead is a catalog/storage round-trip per cached port on the coordinator, which is the work the cache is meant to avoid.
##########
sql/texera_ddl.sql:
##########
@@ -371,6 +372,24 @@ CREATE TABLE operator_port_executions
FOREIGN KEY (workflow_execution_id) REFERENCES workflow_executions(eid) ON DELETE CASCADE
);
+-- operator_port_cache
+-- Caches a materialized output port result across executions, keyed by the cache
+-- key (a SHA-256 hash of the upstream sub-DAG). cache_key_json holds the JSON
+-- the cache key is computed from.
+CREATE TABLE operator_port_cache
+(
+ workflow_id INT NOT NULL,
+ global_port_id VARCHAR(200) NOT NULL,
+ cache_key CHAR(64) NOT NULL,
+ cache_key_json TEXT NOT NULL,
+ result_uri TEXT NOT NULL,
+ tuple_count BIGINT,
+ source_execution_id BIGINT,
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+ PRIMARY KEY (workflow_id, global_port_id, cache_key),
Review Comment:
Intentional — the cache is reused across executions, keyed by `cache_key` (the upstream computation hash). Putting `execution_id` in the PK would create a row per execution and prevent reuse. `source_execution_id` is provenance metadata. The per-execution tables key by execution because they store per-execution results — a different concern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(dao): add operator_port_cache table [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
carloea2 commented on code in PR #5967:
URL: https://github.com/apache/texera/pull/5967#discussion_r3496774082
##########
sql/updates/26.sql:
##########
@@ -0,0 +1,55 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- ============================================
+-- 1. Connect to the texera_db database
+-- ============================================
+\c texera_db
+
+SET search_path TO texera_db;
+
+-- ============================================
+-- 2. Add the operator_port_cache table
+-- ============================================
+BEGIN;
+
+-- Caches a materialized output port result so it can be reused across executions.
+-- A row is identified by (workflow_id, global_port_id, cache_key), where cache_key is a
+-- SHA-256 hash of the upstream sub-DAG that produces the port (its operators, their
+-- parameters and exec info, schemas, and wiring). cache_key is the lookup key;
+-- cache_key_json is the JSON the hash was computed from, kept so a hash match can be
+-- confirmed against the full content (collision safety). A different upstream computation
+-- (for example an operator parameter or version change) produces a different cache_key and
+-- therefore a new row, so existing entries are never overwritten: each row is the result of
+-- one specific computation of one port. tuple_count is the result's row count, kept so the
+-- coordinator can report a reused region's output stats without a second query to the
+-- Iceberg catalog.
+CREATE TABLE IF NOT EXISTS operator_port_cache
+(
+ workflow_id INT NOT NULL,
+ global_port_id VARCHAR(200) NOT NULL,
+ cache_key CHAR(64) NOT NULL,
Review Comment:
May be a dumb question, can we name it cache_key_hash?
##########
sql/updates/26.sql:
##########
@@ -0,0 +1,55 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- ============================================
+-- 1. Connect to the texera_db database
+-- ============================================
+\c texera_db
+
+SET search_path TO texera_db;
+
+-- ============================================
+-- 2. Add the operator_port_cache table
+-- ============================================
+BEGIN;
+
+-- Caches a materialized output port result so it can be reused across executions.
+-- A row is identified by (workflow_id, global_port_id, cache_key), where cache_key is a
+-- SHA-256 hash of the upstream sub-DAG that produces the port (its operators, their
+-- parameters and exec info, schemas, and wiring). cache_key is the lookup key;
+-- cache_key_json is the JSON the hash was computed from, kept so a hash match can be
+-- confirmed against the full content (collision safety). A different upstream computation
+-- (for example an operator parameter or version change) produces a different cache_key and
+-- therefore a new row, so existing entries are never overwritten: each row is the result of
+-- one specific computation of one port. tuple_count is the result's row count, kept so the
+-- coordinator can report a reused region's output stats without a second query to the
+-- Iceberg catalog.
+CREATE TABLE IF NOT EXISTS operator_port_cache
+(
+ workflow_id INT NOT NULL,
+ global_port_id VARCHAR(200) NOT NULL,
+ cache_key CHAR(64) NOT NULL,
Review Comment:
May be a dumb question, can we name it cache_key_hash?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(dao): add operator_port_cache table [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
carloea2 commented on code in PR #5967:
URL: https://github.com/apache/texera/pull/5967#discussion_r3496732553
##########
sql/updates/26.sql:
##########
@@ -0,0 +1,47 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- ============================================
+-- 1. Connect to the texera_db database
+-- ============================================
+\c texera_db
+
+SET search_path TO texera_db;
+
+-- ============================================
+-- 2. Add the operator_port_cache table
+-- ============================================
+BEGIN;
+
+-- Caches a materialized output port result across executions, keyed by the cache
+-- key (a SHA-256 hash of the upstream sub-DAG). cache_key_json holds the JSON
+-- the cache key is computed from.
+CREATE TABLE IF NOT EXISTS operator_port_cache
+(
+ workflow_id INT NOT NULL,
+ global_port_id VARCHAR(200) NOT NULL,
+ cache_key CHAR(64) NOT NULL,
+ cache_key_json TEXT NOT NULL,
Review Comment:
May be dumb question, can we name it cache_key_hash?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(dao): add operator_port_cache table [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #5967:
URL: https://github.com/apache/texera/pull/5967#issuecomment-4828013486
@Xiao-zhen-Liu please link issue properly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(dao): add operator_port_cache table [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5967:
URL: https://github.com/apache/texera/pull/5967#issuecomment-4826857826
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- **Contributors with relevant context:** `@aicam`
You can notify them by mentioning `@aicam` in a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(dao): add operator_port_cache table [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
Xiao-zhen-Liu merged PR #5967:
URL: https://github.com/apache/texera/pull/5967
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(dao): add operator_port_cache table [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
Xiao-zhen-Liu commented on PR #5967:
URL: https://github.com/apache/texera/pull/5967#issuecomment-4835104146
Thanks @Yicong-Huang — replies inline. Renamed `result_uri` -> `storage_uri`. Two I kept, with reasoning inline: `tuple_count` (a cache row is immutable so it can't drift, and the coordinator reads it alongside the cache lookup so cached-region stats need no extra storage round-trip) and the PK without `execution_id` (the cache is reused across executions, keyed by `cache_key`). Re-requesting your review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(dao): add operator_port_cache table [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #5967:
URL: https://github.com/apache/texera/pull/5967#discussion_r3488362169
##########
sql/updates/26.sql:
##########
@@ -0,0 +1,47 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- ============================================
+-- 1. Connect to the texera_db database
+-- ============================================
+\c texera_db
+
+SET search_path TO texera_db;
+
+-- ============================================
+-- 2. Add the operator_port_cache table
+-- ============================================
+BEGIN;
+
+-- Caches a materialized output port result across executions, keyed by the cache
+-- key (a SHA-256 hash of the upstream sub-DAG). cache_key_json holds the JSON
+-- the cache key is computed from.
+CREATE TABLE IF NOT EXISTS operator_port_cache
+(
+ workflow_id INT NOT NULL,
+ global_port_id VARCHAR(200) NOT NULL,
+ cache_key CHAR(64) NOT NULL,
+ cache_key_json TEXT NOT NULL,
+ result_uri TEXT NOT NULL,
Review Comment:
please change to `storage_uri`. `result` indicates direction.
##########
sql/updates/26.sql:
##########
@@ -0,0 +1,47 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- ============================================
+-- 1. Connect to the texera_db database
+-- ============================================
+\c texera_db
+
+SET search_path TO texera_db;
+
+-- ============================================
+-- 2. Add the operator_port_cache table
+-- ============================================
+BEGIN;
+
+-- Caches a materialized output port result across executions, keyed by the cache
+-- key (a SHA-256 hash of the upstream sub-DAG). cache_key_json holds the JSON
+-- the cache key is computed from.
+CREATE TABLE IF NOT EXISTS operator_port_cache
+(
+ workflow_id INT NOT NULL,
+ global_port_id VARCHAR(200) NOT NULL,
+ cache_key CHAR(64) NOT NULL,
+ cache_key_json TEXT NOT NULL,
Review Comment:
why do we have two parts?
##########
sql/texera_ddl.sql:
##########
@@ -371,6 +372,24 @@ CREATE TABLE operator_port_executions
FOREIGN KEY (workflow_execution_id) REFERENCES workflow_executions(eid) ON DELETE CASCADE
);
+-- operator_port_cache
+-- Caches a materialized output port result across executions, keyed by the cache
+-- key (a SHA-256 hash of the upstream sub-DAG). cache_key_json holds the JSON
+-- the cache key is computed from.
+CREATE TABLE operator_port_cache
+(
+ workflow_id INT NOT NULL,
+ global_port_id VARCHAR(200) NOT NULL,
+ cache_key CHAR(64) NOT NULL,
+ cache_key_json TEXT NOT NULL,
+ result_uri TEXT NOT NULL,
+ tuple_count BIGINT,
+ source_execution_id BIGINT,
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+ PRIMARY KEY (workflow_id, global_port_id, cache_key),
Review Comment:
how come execution id is not part of primary key?
##########
sql/updates/26.sql:
##########
@@ -0,0 +1,47 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- ============================================
+-- 1. Connect to the texera_db database
+-- ============================================
+\c texera_db
+
+SET search_path TO texera_db;
+
+-- ============================================
+-- 2. Add the operator_port_cache table
+-- ============================================
+BEGIN;
+
+-- Caches a materialized output port result across executions, keyed by the cache
+-- key (a SHA-256 hash of the upstream sub-DAG). cache_key_json holds the JSON
+-- the cache key is computed from.
+CREATE TABLE IF NOT EXISTS operator_port_cache
+(
+ workflow_id INT NOT NULL,
+ global_port_id VARCHAR(200) NOT NULL,
+ cache_key CHAR(64) NOT NULL,
+ cache_key_json TEXT NOT NULL,
+ result_uri TEXT NOT NULL,
+ tuple_count BIGINT,
Review Comment:
do we want to store statistics separately? I think iceberg catalog has this info already, is this duplicate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(dao): add operator_port_cache table [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
Xiao-zhen-Liu commented on code in PR #5967:
URL: https://github.com/apache/texera/pull/5967#discussion_r3500244523
##########
sql/updates/26.sql:
##########
@@ -0,0 +1,55 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- ============================================
+-- 1. Connect to the texera_db database
+-- ============================================
+\c texera_db
+
+SET search_path TO texera_db;
+
+-- ============================================
+-- 2. Add the operator_port_cache table
+-- ============================================
+BEGIN;
+
+-- Caches a materialized output port result so it can be reused across executions.
+-- A row is identified by (workflow_id, global_port_id, cache_key), where cache_key is a
+-- SHA-256 hash of the upstream sub-DAG that produces the port (its operators, their
+-- parameters and exec info, schemas, and wiring). cache_key is the lookup key;
+-- cache_key_json is the JSON the hash was computed from, kept so a hash match can be
+-- confirmed against the full content (collision safety). A different upstream computation
+-- (for example an operator parameter or version change) produces a different cache_key and
+-- therefore a new row, so existing entries are never overwritten: each row is the result of
+-- one specific computation of one port. tuple_count is the result's row count, kept so the
+-- coordinator can report a reused region's output stats without a second query to the
+-- Iceberg catalog.
+CREATE TABLE IF NOT EXISTS operator_port_cache
+(
+ workflow_id INT NOT NULL,
+ global_port_id VARCHAR(200) NOT NULL,
+ cache_key CHAR(64) NOT NULL,
Review Comment:
Done — renamed `cache_key` to `cache_key_hash` (column, primary key, and the comments). `cache_key_json` stays as the JSON it was computed from.
##########
sql/updates/26.sql:
##########
@@ -0,0 +1,55 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- ============================================
+-- 1. Connect to the texera_db database
+-- ============================================
+\c texera_db
+
+SET search_path TO texera_db;
+
+-- ============================================
+-- 2. Add the operator_port_cache table
+-- ============================================
+BEGIN;
+
+-- Caches a materialized output port result so it can be reused across executions.
+-- A row is identified by (workflow_id, global_port_id, cache_key), where cache_key is a
+-- SHA-256 hash of the upstream sub-DAG that produces the port (its operators, their
+-- parameters and exec info, schemas, and wiring). cache_key is the lookup key;
+-- cache_key_json is the JSON the hash was computed from, kept so a hash match can be
+-- confirmed against the full content (collision safety). A different upstream computation
+-- (for example an operator parameter or version change) produces a different cache_key and
+-- therefore a new row, so existing entries are never overwritten: each row is the result of
+-- one specific computation of one port. tuple_count is the result's row count, kept so the
+-- coordinator can report a reused region's output stats without a second query to the
+-- Iceberg catalog.
+CREATE TABLE IF NOT EXISTS operator_port_cache
+(
+ workflow_id INT NOT NULL,
+ global_port_id VARCHAR(200) NOT NULL,
+ cache_key CHAR(64) NOT NULL,
Review Comment:
Done — renamed `cache_key` to `cache_key_hash` (column, primary key, and the comments). `cache_key_json` stays as the JSON it was computed from.
##########
sql/updates/26.sql:
##########
@@ -0,0 +1,47 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- ============================================
+-- 1. Connect to the texera_db database
+-- ============================================
+\c texera_db
+
+SET search_path TO texera_db;
+
+-- ============================================
+-- 2. Add the operator_port_cache table
+-- ============================================
+BEGIN;
+
+-- Caches a materialized output port result across executions, keyed by the cache
+-- key (a SHA-256 hash of the upstream sub-DAG). cache_key_json holds the JSON
+-- the cache key is computed from.
+CREATE TABLE IF NOT EXISTS operator_port_cache
+(
+ workflow_id INT NOT NULL,
+ global_port_id VARCHAR(200) NOT NULL,
+ cache_key CHAR(64) NOT NULL,
+ cache_key_json TEXT NOT NULL,
Review Comment:
Renamed `cache_key` to `cache_key_hash` as suggested, so the hash column and the `cache_key_json` column read clearly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(dao): add operator_port_cache table [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5967:
URL: https://github.com/apache/texera/pull/5967#issuecomment-4826871928
<!-- texera-benchmarks-comment -->
## ⚠️ Benchmark changes need a look
🟢 0 better · 🔴 9 worse · ⚪ 6 noise (<±5%) · 0 without baseline
> Compared against main `a24d1d1` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
[Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/28330275998)
| | config | throughput | MB/s | latency | max Δ latest / 7d |
|---|---|---:|---:|---:|---:|
| 🔴 | bs=10 sw=10 sl=64 | 352 | 0.215 | 28,313/39,520/39,520 us | 🔴 +27.6% / 🔴 +157.7% |
| 🔴 | bs=100 sw=10 sl=64 | 760 | 0.464 | 126,902/163,869/163,869 us | 🔴 +18.8% / 🔴 +48.2% |
| ⚪ | bs=1000 sw=10 sl=64 | 895 | 0.546 | 1,119,618/1,193,232/1,193,232 us | ⚪ within ±5% / 🔴 +12.2% |
<details><summary>Baseline details</summary>
Latest main `a24d1d1` from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---:|---:|---:|---:|---:|
| bs=10 sw=10 sl=64 | throughput | 352 tuples/sec | 408 tuples/sec | 757.16 tuples/sec | -13.7% | -53.5% |
| bs=10 sw=10 sl=64 | MB/s | 0.215 MB/s | 0.249 MB/s | 0.462 MB/s | -13.7% | -53.5% |
| bs=10 sw=10 sl=64 | p50 | 28,313 us | 22,196 us | 12,971 us | +27.6% | +118.3% |
| bs=10 sw=10 sl=64 | p95 | 39,520 us | 32,808 us | 15,333 us | +20.5% | +157.7% |
| bs=10 sw=10 sl=64 | p99 | 39,520 us | 32,808 us | 18,732 us | +20.5% | +111.0% |
| bs=100 sw=10 sl=64 | throughput | 760 tuples/sec | 803 tuples/sec | 957.66 tuples/sec | -5.4% | -20.6% |
| bs=100 sw=10 sl=64 | MB/s | 0.464 MB/s | 0.49 MB/s | 0.585 MB/s | -5.3% | -20.6% |
| bs=100 sw=10 sl=64 | p50 | 126,902 us | 125,306 us | 103,982 us | +1.3% | +22.0% |
| bs=100 sw=10 sl=64 | p95 | 163,869 us | 137,909 us | 110,583 us | +18.8% | +48.2% |
| bs=100 sw=10 sl=64 | p99 | 163,869 us | 137,909 us | 118,369 us | +18.8% | +38.4% |
| bs=1000 sw=10 sl=64 | throughput | 895 tuples/sec | 891 tuples/sec | 979.6 tuples/sec | +0.4% | -8.6% |
| bs=1000 sw=10 sl=64 | MB/s | 0.546 MB/s | 0.544 MB/s | 0.598 MB/s | +0.4% | -8.7% |
| bs=1000 sw=10 sl=64 | p50 | 1,119,618 us | 1,117,125 us | 1,024,553 us | +0.2% | +9.3% |
| bs=1000 sw=10 sl=64 | p95 | 1,193,232 us | 1,182,699 us | 1,063,789 us | +0.9% | +12.2% |
| bs=1000 sw=10 sl=64 | p99 | 1,193,232 us | 1,182,699 us | 1,096,239 us | +0.9% | +8.8% |
</details>
<details><summary>Raw CSV</summary>
```csv
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,568.90,200,128000,352,0.215,28312.93,39520.26,39520.26
1,100,10,64,20,2630.17,2000,1280000,760,0.464,126901.60,163868.52,163868.52
2,1000,10,64,20,22341.67,20000,12800000,895,0.546,1119618.05,1193232.22,1193232.22
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(dao): add operator_port_cache table [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
Xiao-zhen-Liu commented on PR #5967:
URL: https://github.com/apache/texera/pull/5967#issuecomment-4826870471
@carloea2 could you help review this one when you get a chance? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org