You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2026/10/21 06:01:25 UTC
(doris) branch master updated: [refactor](changelog) Rename row binlog to changelog and split it from CCR binlog (#69311)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 73a987e6059 [refactor](changelog) Rename row binlog to changelog and split it from CCR binlog (#69311)
73a987e6059 is described below
commit 73a987e6059910ec82c3ea14014aa4742a85cb17
Author: Mingyu Chen (Rayner) <mo...@gmail.com>
AuthorDate: Wed Oct 21 07:01:14 2026 +0100
[refactor](changelog) Rename row binlog to changelog and split it from CCR binlog (#69311)
### What problem does this PR solve?
Issue Number: None
Related PR: #69238, #69240, #69242
Problem Summary:
**In short.** Row binlog (`binlog.format = 'ROW'`), the change stream
that Table Stream and incremental materialized views (IVM) read, is
renamed to **changelog**, and it gets its own configuration instead of
riding on CCR's `BinlogConfig`. "Binlog" names a change stream in one
fixed format; changelog is the general notion, and a binlog can later be
built on top of it. The feature exists only on master and is documented
as experimental, so this is a hard cut: there is no code that reads data
written under the old names. Instead, guardrails make leftovers from an
earlier master build fail with a clear message instead of crashing a
process or being silently misread. CCR binlog and MySQL binlog CDC do
not change.
**Background.**
- A row binlog table writes every change of a base tablet into a
companion tablet. The companion schema carries the key, the after
values, three hidden columns `__DORIS_BINLOG_TSO__`,
`__DORIS_BINLOG_LSN__`, `__DORIS_BINLOG_OP__`, and, with
`binlog.need_historical_value`, a before-image column
`__BEFORE__<col>__` per value column. The changes are read with the
`binlog(...)` table function, `@incr`, Table Stream and IVM refresh.
- It was switched on with CCR's properties: `binlog.enable = true` plus
`binlog.format = ROW`. It shared `binlog.enable` and
`binlog.ttl_seconds` with CCR, and `enable_feature_binlog` gated CCR,
row binlog table creation, TSO allocation, Table Stream INSERT and TTL
cleanup alike.
- None of this is on a release branch (3.1 / 4.0 / 4.1 / 4.2). The
documentation marks it experimental, and `mtmv/ivm/AGENTS.md` allows
incompatible changes before 2027-01. Master already made incompatible
changes to it in #64133, #65810 and #66889.
**The problem, and what it cost.**
1. **One word for two features.** CCR binlog (cross-cluster replication)
and row binlog share property names, a switch, a configuration object
and most code names, so every reader has to work out which one a
`binlog` means.
2. **Row binlog leaks into every table.** Every OLAP table created on
master persists `binlog.format` and `binlog.need_historical_value` and
prints them in SHOW CREATE TABLE, CCR tables included; a 4.x CCR
downstream rejects that DDL as unknown properties.
3. **The shared keys interact badly.**
- `ALTER DATABASE ... SET ("binlog.enable" = "true")` on a database
whose default is ROW validates the tables with the CCR rules, but what
takes effect is the ROW default.
- A table with `binlog.format = ROW` and `binlog.enable = false` can
never be enabled: the error says "can't disable binlog when format is
[Row]".
4. **Before-image columns clash with user columns** (reproduced on
master before this change). `__BEFORE__<col>__` is not under the
reserved `__DORIS_` prefix, and schema change matched it with
`contains("__BEFORE__")`:
| User column | What happened |
|---|---|
| `__BEFORE__v1__` next to `v1` | the companion index had two columns
named `__BEFORE__v1__`; `@incr` DETAIL returned after values in the
UPDATE_BEFORE rows; the column could not be named in `binlog()`
("ambiguous"); as a key column every `@incr` read failed; after `DROP
COLUMN v1` every load failed |
| `x__BEFORE__y` as the last value column | after `ADD COLUMN v2` the
companion index put `v2` before it, so the two columns swapped values in
every row written afterwards, without any error; with different types
every later load failed with a bad cast |
5. FE and BE had two families of change codes with the same names and
different values (`DELETE` is 1 in FE and 2 in BE).
**How this PR fixes it.**
1. **New names.** User-visible ones:
| Old | New |
|---|---|
| `"binlog.enable" = "true"` + `"binlog.format" = "ROW"` |
`"changelog.enable" = "true"` |
| `binlog.ttl_seconds` (on a ROW table or database),
`binlog.need_historical_value` | `changelog.ttl_seconds`,
`changelog.need_historical_value` |
| internal `__binlog_config_version` | `__changelog_config_version` (no
longer shown in `information_schema.table_properties`) |
| `binlog(...)` table function | `changelog(...)` |
| `__DORIS_BINLOG_{TSO,LSN,OP}__` | `__DORIS_CHANGELOG_{TSO,LSN,OP}__` |
| `__BEFORE__<col>__` | `__DORIS_CHANGELOG_BEFORE_<col>__` |
| companion index `binlog<Row>(t)` | `changelog(t)` |
| SHOW DATA `BinlogSize` / `LocalBinlogSize`, SHOW PARTITIONS
`BinlogSize`, SHOW TABLET(S) / PROC `IsRowBinlog`,
`RowBinlog{Mismatch,Redundant}{Num,Tablets}`, `RowBinlogRepairReason` |
`ChangelogSize`, `LocalChangelogSize`, `IsChangelog`, `Changelog...` |
| IVM fallback reasons `BINLOG_BROKEN`, `BINLOG_NOT_ENABLED` |
`CHANGELOG_BROKEN`, `CHANGELOG_NOT_ENABLED` |
| FE `tablet_binlog_missing_timeout_second`,
`tablet_binlog_missing_max_times` | `tablet_changelog_missing_*` |
| BE `binlog_compaction_*` (6), `max_binlog_compaction_threads`, cloud
`binlog_compaction_thread_num_factor` | `changelog_compaction_*`,
`max_changelog_compaction_threads`,
`changelog_compaction_thread_num_factor` |
| metrics `compaction_task_state_total{type="binlog"}`,
`tablet_binlog_max_compaction_score`; bvars `row_binlog_ttl_*`; thread
pools | `type="changelog"`, `tablet_changelog_max_compaction_score`,
`changelog_ttl_*`, `Changelog...` pools |
| HTTP `compact_type=row_binlog_ttl`; compaction policy `"binlog"`; load
profile `RowsetBuilder {id} binlog<row>` | `changelog_ttl`;
`"changelog"`; `RowsetBuilder {id} changelog` |
CCR keeps every name: `binlog.enable / ttl_seconds / max_bytes /
max_history_nums`, `SHOW PROC '/binlog'`, getBinlog / ingest_binlog,
`BinlogManager`.
2. **Separate configuration.**
- New `ChangelogConfig` (`enable`, `ttlSeconds`, `needHistoricalValue`,
`configVersion`), stored as `changelog.*` keys in the table / database
property map. `BinlogConfig` is back to the four CCR fields, so plain
and CCR tables persist and print only the four `binlog.*` keys, as on
4.x.
- The wire format does not change. `TabletBinlogConfig.of` is now the
only place that builds the `TBinlogConfig` / `BinlogConfigPB` sent to BE
and the meta service: a changelog table still sends `enable = true`,
`binlog_format = 1` (now named `CHANGELOG`), its TTL,
`need_historical_value` and `config_version`; plain and CCR tables send
exactly what they sent before. The enum mapping is explicit instead of
`valueOf(name())`.
- The TTL generation protocol (short TTL first, published after every
tablet accepted it) moves unchanged to the changelog configuration.
- `changelog.enable` and `changelog.need_historical_value` can only be
set at CREATE; `changelog.ttl_seconds` can be altered. `binlog.format`
and `binlog.need_historical_value` are rejected in every DDL, naming
their replacement.
- In a database whose default is `"changelog.enable" = "true"`, SHOW
CREATE TABLE of a table without changelog prints `"changelog.enable" =
"false"` (output only), so CREATE TABLE LIKE and re-running the DDL keep
it out of the default. Temporary tables never have changelog: an
explicit `"changelog.enable" = "true"` fails, a database default is
ignored.
- CCR and changelog stay mutually exclusive, now by explicit checks in
CREATE / ALTER DATABASE, CREATE / ALTER TABLE and RECOVER TABLE: the CCR
full sync relies on backup / restore, which changelog tables do not
support, and one tablet carries only one `BinlogConfigPB`.
- New switch `enable_feature_changelog` (FE and BE) gates changelog
tables, TSO allocation, Table Stream INSERT, TTL cleanup and the
changelog part of SHOW CREATE. `enable_feature_binlog` is back to CCR
only.
3. **IDL names** of the row-only elements are renamed; ids, enum values
and types are unchanged, and shared names (`TBinlogConfig`,
`BinlogConfigPB`, `binlog_config`, `binlog_format`) stay:
| File | Old (id) | New |
|---|---|---|
| AgentService.thrift | `TTabletRole.TABLET_ROLE_ROW_BINLOG` (1),
`TBinlogFormat.ROW` (1) | `TABLET_ROLE_CHANGELOG`, `CHANGELOG` |
| Descriptors.thrift | `TOlapTableIndexSchema.row_binlog_id` (7),
`TOlapTableSchemaParam.row_binlog_index_schemas` (17) |
`changelog_index_id`, `changelog_index_schemas` |
| HeartbeatService.thrift | `TMasterInfo.row_binlog_ttl_reference_tso`
(15) | `changelog_ttl_reference_tso` |
| PlanNodes.thrift | `TBinlogScanType`,
`TPaloScanRange.binlog_scan_type` (12), `TOlapScanNode.read_row_binlog`
(26) | `TChangelogScanType`, `changelog_scan_type`, `read_changelog` |
| olap_file.proto | `RowsetMeta(Cloud)PB.is_row_binlog` (58),
`TabletMeta(Cloud)PB.binlog_tablet_id` (37 / 42),
`TabletRolePB.TABLET_ROLE_ROW_BINLOG` (1), `BinlogFormatPB.ROW` (1) |
`is_changelog`, `changelog_tablet_id`, `TABLET_ROLE_CHANGELOG`,
`CHANGELOG` |
| descriptors.proto | `POlapTableIndexSchema.row_binlog_id` (6),
`POlapTableSchemaParam.row_binlog_index_schemas` (18) |
`changelog_index_id`, `changelog_index_schemas` |
| internal_service.proto | `PTabletWithPartition.binlog_tablet_id` (3),
`PSyncTabletMetaRequest.discover_row_binlog_ttl` (2) |
`changelog_tablet_id`, `discover_changelog_ttl` |
| cloud.proto | `TabletCompactionJobPB.row_binlog_ttl_seconds` (31) |
`changelog_ttl_seconds` |
4. **Guardrails for leftovers of an earlier master build.**
- FE, on image load, edit log replay and backup restore: plain and CCR
tables and databases drop the three old keys and behave exactly as
before. A table that still says `binlog.format = ROW` with
`binlog.enable = true` cannot be used except by DROP; queries (including
`@incr`, `changelog()` and streams), every load path and DDL fail with
`Table <db>.<t> was created with "binlog.format" = "ROW", which is no
longer supported; drop the table and create it again with
"changelog.enable" = "true"`. Background daemons skip it instead of
throwing or rebuilding its companion tablets. A database whose default
was ROW gets `binlog.enable = false` back, with a warning, so it does
not turn into a CCR database.
- BE: publishing a load that an older BE committed on a changelog rowset
without the new hidden columns fails with an error instead of reading
column -1 (which crashed the BE on every publish retry). A changelog
tablet uses the changelog compaction policy by its role, whatever policy
string its meta carries; an unknown policy string falls back to
size_based with a warning (looking it up used to throw and abort the BE
on every restart); the derive stage returns an error instead of a
`CHECK` when the hidden columns are missing; a changelog read or
compaction without the TSO column fails instead of writing NULL TSO.
5. **Before-image columns** are `__DORIS_CHANGELOG_BEFORE_<col>__`. A
user cannot create a column under `__DORIS_`, every check uses this
prefix, and `IvmUtil.isCommonHiddenSlot` excludes it (a user column
ending in `_COL` would otherwise look like a common hidden slot).
6. **Change codes** have two name families: FE `CHANGE_TYPE_*` (folded
change types) and BE `CHANGELOG_OP_*` (persisted op values). Values do
not change.
7. Every persisted or externally hard-coded name has an `//
ATTN(changelog-rename): formerly "<old>"` line at its definition, so a
later compatibility layer knows what to map.
**Results.**
| Scenario | Before | After |
|---|---|---|
| SHOW CREATE TABLE of a plain or CCR table | prints `binlog.format`,
and `binlog.need_historical_value` on ROW tables | prints the four CCR
`binlog.*` keys only (plus `"changelog.enable" = "false"` in a database
whose default enables changelog) |
| `ALTER DATABASE ... SET ("binlog.enable"="true")` on a database whose
tables use changelog | CCR checks, ROW default applied | rejected: `Can
not enable binlog in database <db> because table <t> enables changelog`
|
| user column `__BEFORE__v1__` / `x__BEFORE__y` with before images |
wrong or swapped values, failing loads | ordinary user columns; values
correct (`test_changelog_before_column_name`) |
| FE and BE of this PR started on metadata written by an earlier master
build (9 ROW tables) | — | both start; one WARN per old table; SHOW /
DESC work; SELECT, INSERT, UPDATE, DELETE, ALTER, TRUNCATE, `@incr`,
`changelog()`, CREATE TABLE LIKE and CREATE STREAM fail with the message
above; DROP works and removes base and companion tablets; no repeating
errors afterwards |
| BE restarted on old changelog tablets with policy `"binlog"` | looking
the policy up throws `std::out_of_range` (local mode) | the tablets load
with role changelog and policy `"changelog"` |
**Upgrade from an earlier master build** (none of this applies to a
release):
- Before upgrading, drop every table created with `"binlog.format" =
"ROW"` together with its Table Streams and incremental MVs, and set
`"binlog.enable" = "false"` on databases whose default was ROW. Recreate
them with `"changelog.enable" = "true"` afterwards. Do not create or
alter changelog tables while FEs of both versions run; downgrade is not
supported.
- Add `enable_feature_changelog = true` to fe.conf and be.conf;
`enable_feature_binlog` no longer enables changelog or TSO.
- Renamed FE / BE configs are not aliased: old names in conf files are
ignored. Update dashboards that use the renamed metrics, bvars,
thread-pool labels and SHOW columns.
- SQL and views that reference `binlog(...)`, the hidden columns or the
before columns must use the new names. Before-image columns now carry
the reserved `__DORIS_` prefix, so `CREATE VIEW` / CTAS / MTMV `AS
SELECT * FROM changelog(...)` on a table with
`changelog.need_historical_value` has to list its columns (as it already
had to for the TSO / LSN / OP columns). Flink Doris Connector
incremental reads need a connector release that knows the new hidden
column names.
---
be/src/agent/heartbeat_server.cpp | 10 +-
be/src/agent/task_worker_pool.cpp | 6 +-
be/src/cloud/cloud_compaction_action.cpp | 8 +-
be/src/cloud/cloud_cumulative_compaction.cpp | 26 +-
...oud_cumulative_compaction_changelog_policy.cpp} | 57 +-
...cloud_cumulative_compaction_changelog_policy.h} | 18 +-
be/src/cloud/cloud_delta_writer.cpp | 6 +-
be/src/cloud/cloud_delta_writer.h | 2 +-
.../cloud/cloud_engine_calc_delete_bitmap_task.cpp | 10 +-
be/src/cloud/cloud_internal_service.cpp | 6 +-
be/src/cloud/cloud_meta_mgr.cpp | 30 +-
be/src/cloud/cloud_meta_mgr.h | 6 +-
be/src/cloud/cloud_rowset_builder.cpp | 57 +-
be/src/cloud/cloud_rowset_builder.h | 8 +-
be/src/cloud/cloud_rowset_writer.cpp | 8 +-
be/src/cloud/cloud_schema_change_job.cpp | 2 +-
be/src/cloud/cloud_storage_engine.cpp | 172 +++---
be/src/cloud/cloud_storage_engine.h | 10 +-
be/src/cloud/cloud_tablet.cpp | 50 +-
be/src/cloud/cloud_tablet_mgr.cpp | 10 +-
be/src/cloud/cloud_tablets_channel.cpp | 38 +-
be/src/cloud/cloud_txn_delete_bitmap_cache.cpp | 10 +-
be/src/cloud/cloud_txn_delete_bitmap_cache.h | 10 +-
be/src/cloud/config.cpp | 3 +-
be/src/cloud/config.h | 2 +-
be/src/cloud/pb_convert.cpp | 32 +-
be/src/common/config.cpp | 26 +-
be/src/common/config.h | 18 +-
be/src/common/metrics/doris_metrics.cpp | 21 +-
be/src/common/metrics/doris_metrics.h | 6 +-
be/src/exec/operator/olap_scan_operator.cpp | 49 +-
be/src/exec/operator/olap_scan_operator.h | 6 +-
be/src/exec/pipeline/pipeline_fragment_context.cpp | 10 +-
be/src/exec/scan/olap_scanner.cpp | 42 +-
be/src/exec/scan/olap_scanner.h | 4 +-
be/src/exec/scan/parallel_scanner_builder.cpp | 4 +-
be/src/exec/sink/writer/vtablet_writer.cpp | 33 +-
be/src/load/channel/tablets_channel.cpp | 52 +-
be/src/load/delta_writer/delta_writer.cpp | 6 +-
be/src/load/delta_writer/delta_writer.h | 4 +-
be/src/load/delta_writer/delta_writer_context.h | 10 +-
be/src/load/memtable/memtable_flush_executor.cpp | 20 +-
be/src/load/memtable/memtable_flush_executor.h | 4 +-
be/src/load/memtable/memtable_writer.cpp | 6 +-
be/src/runtime/cluster_info.h | 12 +-
be/src/runtime/query_cache/query_cache.cpp | 4 +-
be/src/runtime/query_cache/query_cache.h | 6 +-
be/src/service/http/action/compaction_action.cpp | 8 +-
be/src/service/http/action/compaction_action.h | 5 +-
be/src/storage/binlog.h | 145 +----
be/src/storage/binlog_config.cpp | 14 +-
be/src/storage/binlog_config.h | 8 +-
be/src/storage/{binlog.h => changelog.h} | 104 +---
...g_ttl_scanner.cpp => changelog_ttl_scanner.cpp} | 95 +--
be/src/storage/compaction/compaction.cpp | 111 ++--
be/src/storage/compaction/compaction.h | 16 +-
.../storage/compaction/cumulative_compaction.cpp | 14 +-
... => cumulative_compaction_changelog_policy.cpp} | 88 +--
....h => cumulative_compaction_changelog_policy.h} | 35 +-
.../compaction/cumulative_compaction_policy.cpp | 30 +-
.../compaction/cumulative_compaction_policy.h | 11 +
be/src/storage/compaction_task_tracker.h | 2 +-
be/src/storage/data_dir.cpp | 104 ++--
be/src/storage/iterator/block_reader.cpp | 105 ++--
be/src/storage/iterator/block_reader.h | 4 +-
...ader_utils.h => changelog_block_reader_utils.h} | 25 +-
be/src/storage/iterator/olap_data_convertor.h | 2 +-
be/src/storage/iterator/vcollect_iterator.cpp | 6 +-
be/src/storage/iterator/vcollect_iterator.h | 2 +-
be/src/storage/iterator/vertical_block_reader.cpp | 2 +-
be/src/storage/iterator/vgeneric_iterators.h | 2 +-
be/src/storage/iterators.h | 4 +-
be/src/storage/merger.cpp | 12 +-
be/src/storage/mow/historical_row_fetcher.h | 4 +-
be/src/storage/mow/key_probe.h | 12 +-
be/src/storage/olap_common.h | 4 +-
be/src/storage/olap_server.cpp | 143 ++---
be/src/storage/partial_update_info.cpp | 2 +-
be/src/storage/partial_update_info.h | 2 +-
be/src/storage/predicate/null_predicate.cpp | 2 -
be/src/storage/rowset/beta_rowset_reader.cpp | 14 +-
be/src/storage/rowset/beta_rowset_writer.cpp | 20 +-
be/src/storage/rowset/beta_rowset_writer.h | 4 +-
be/src/storage/rowset/beta_rowset_writer_v2.cpp | 4 +-
be/src/storage/rowset/group_rowset_writer.cpp | 37 +-
be/src/storage/rowset/group_rowset_writer.h | 10 +-
be/src/storage/rowset/rowset_factory.cpp | 6 +-
be/src/storage/rowset/rowset_meta.h | 10 +-
be/src/storage/rowset/rowset_meta_manager.cpp | 40 +-
be/src/storage/rowset/rowset_meta_manager.h | 10 +-
be/src/storage/rowset/rowset_reader_context.h | 2 +-
be/src/storage/rowset/rowset_writer_context.h | 24 +-
.../storage/rowset/vertical_beta_rowset_writer.cpp | 2 +-
be/src/storage/rowset_builder.cpp | 119 ++--
be/src/storage/rowset_builder.h | 30 +-
be/src/storage/schema.cpp | 43 +-
be/src/storage/schema.h | 40 +-
be/src/storage/schema_change/schema_change.cpp | 2 +-
.../storage/segment/historical_row_retriever.cpp | 34 +-
be/src/storage/segment/historical_row_retriever.h | 4 +-
be/src/storage/segment/segment.cpp | 4 +-
be/src/storage/segment/segment_iterator.cpp | 10 +-
be/src/storage/snapshot/snapshot_manager.cpp | 4 +-
be/src/storage/storage_engine.cpp | 8 +-
be/src/storage/storage_engine.h | 28 +-
be/src/storage/tablet/base_tablet.cpp | 156 ++---
be/src/storage/tablet/base_tablet.h | 8 +-
be/src/storage/tablet/tablet.cpp | 90 +--
be/src/storage/tablet/tablet.h | 14 +-
be/src/storage/tablet/tablet_manager.cpp | 50 +-
be/src/storage/tablet/tablet_manager.h | 2 +-
be/src/storage/tablet/tablet_meta.cpp | 25 +-
be/src/storage/tablet/tablet_meta.h | 12 +-
be/src/storage/tablet/tablet_reader.cpp | 15 +-
be/src/storage/tablet/tablet_reader.h | 6 +-
be/src/storage/tablet/tablet_schema.cpp | 50 +-
be/src/storage/tablet/tablet_schema.h | 16 +-
be/src/storage/tablet_info.cpp | 38 +-
be/src/storage/tablet_info.h | 38 +-
.../storage/task/engine_publish_version_task.cpp | 60 +-
be/src/storage/task/engine_publish_version_task.h | 6 +-
be/src/storage/transform/block_transform.cpp | 16 +-
be/src/storage/transform/block_transform.h | 4 +-
...{row_binlog_derive.cpp => changelog_derive.cpp} | 164 +++---
.../{row_binlog_derive.h => changelog_derive.h} | 36 +-
be/src/storage/txn/txn_manager.cpp | 164 +++---
be/src/storage/txn/txn_manager.h | 24 +-
be/src/storage/utils.h | 9 +-
be/src/util/thrift_util.cpp | 4 +-
be/src/util/thrift_util.h | 2 +-
be/test/cloud/cloud_compaction_test.cpp | 131 ++---
.../cloud_cumulative_compaction_policy_test.cpp | 117 ++--
be/test/cloud/cloud_tablet_test.cpp | 4 +-
be/test/exec/operator/olap_scan_operator_test.cpp | 22 +-
.../pipeline/query_cache_fragment_context_test.cpp | 9 +-
be/test/exec/pipeline/query_cache_test.cpp | 6 +-
be/test/exec/scan/vgeneric_iterators_test.cpp | 14 +-
.../load/delta_writer/delta_writer_cancel_test.cpp | 16 +-
.../load/memtable/memtable_flush_executor_test.cpp | 122 ++--
.../cloud_group_rowset_builder_writer_test.cpp | 120 ++--
be/test/olap/rowset/group_rowset_builder_test.cpp | 116 ++--
be/test/olap/rowset/group_rowset_writer_test.cpp | 213 ++++---
..._binlog_ttl_test.cpp => changelog_ttl_test.cpp} | 152 ++---
...st.cpp => changelog_vmerge_compaction_test.cpp} | 100 +++-
.../storage/compaction/compaction_task_test.cpp | 6 +-
...umulative_compaction_changelog_policy_test.cpp} | 193 +++---
.../cumulative_compaction_policy_test.cpp | 25 +
.../compaction/cumulative_compaction_test.cpp | 60 +-
.../compaction/ordered_data_compaction_test.cpp | 2 +-
.../storage/compaction/segcompaction_mow_test.cpp | 2 +-
be/test/storage/compaction/segcompaction_test.cpp | 8 +-
.../compaction/vertical_compaction_test.cpp | 18 +-
.../block_reader_change_next_block_test.cpp | 350 +++++------
...block_reader_changelog_vcollect_merge_test.cpp} | 107 ++--
.../iterator/block_reader_min_delta_test.cpp | 95 +--
.../storage/iterator/olap_data_convertor_test.cpp | 18 +-
be/test/storage/mow/key_probe_test.cpp | 12 +-
be/test/storage/mow/mow_transform_test_base.h | 59 +-
be/test/storage/read_schema_test.cpp | 62 +-
be/test/storage/rowid_conversion_test.cpp | 6 +-
.../storage/rowset/rowset_meta_manager_test.cpp | 24 +-
be/test/storage/rowset/rowset_meta_test.cpp | 20 +-
.../storage/rowset/segment_flusher_format_test.cpp | 633 ++++++++++----------
.../engine_storage_migration_task_test.cpp | 156 ++---
.../segment/historical_row_retriever_test.cpp | 38 +-
.../segment/segment_iterator_expr_zonemap_test.cpp | 50 +-
be/test/storage/tablet/tablet_meta_test.cpp | 10 +-
be/test/storage/tablet/tablet_schema_test.cpp | 24 +-
be/test/storage/tablet/tablet_test.cpp | 18 +-
.../segment_0.dat | Bin
.../segment_1.dat | Bin
.../segment_0.dat | Bin
.../segment_1.dat | Bin
.../segment_0.dat | Bin
.../segment_1.dat | Bin
.../segment_0.dat | Bin
.../segment_1.dat | Bin
.../segment_0.dat | Bin
.../segment_1.dat | Bin
.../segment_0.dat | Bin
.../segment_1.dat | Bin
...g_derive_test.cpp => changelog_derive_test.cpp} | 650 +++++++++++----------
be/test/storage/transform/validate_stage_test.cpp | 46 +-
be/test/storage/txn/txn_manager_test.cpp | 88 +--
.../variant/variant_column_writer_reader_test.cpp | 2 +-
.../variant/variant_doc_mode_compaction_test.cpp | 4 +-
be/test/testutil/creators.h | 62 +-
be/test/testutil/index_storage_test_util.cpp | 2 +-
cloud/src/meta-service/meta_service_job.cpp | 12 +-
cloud/src/recycler/recycler.cpp | 6 +-
cloud/src/recycler/recycler.h | 2 +-
cloud/test/meta_service_job_test.cpp | 18 +-
cloud/test/meta_service_test.cpp | 6 +-
cloud/test/recycler_test.cpp | 29 +-
.../main/java/org/apache/doris/catalog/Column.java | 28 +-
.../main/java/org/apache/doris/common/Config.java | 15 +-
.../main/java/org/apache/doris/alter/Alter.java | 11 +-
.../org/apache/doris/alter/AlterOperations.java | 20 +-
.../doris/alter/MaterializedViewHandler.java | 2 +-
.../java/org/apache/doris/alter/RollupJobV2.java | 6 +-
.../apache/doris/alter/SchemaChangeHandler.java | 248 ++++----
.../org/apache/doris/alter/SchemaChangeJobV2.java | 8 +-
.../org/apache/doris/backup/BackupHandler.java | 7 +-
.../java/org/apache/doris/backup/RestoreJob.java | 24 +-
.../java/org/apache/doris/binlog/BinlogUtils.java | 31 -
.../org/apache/doris/catalog/BinlogConfig.java | 199 +------
.../doris/catalog/BuiltinTableValuedFunctions.java | 5 +-
.../apache/doris/catalog/CatalogRecycleBin.java | 12 +-
.../org/apache/doris/catalog/ChangelogConfig.java | 208 +++++++
.../apache/doris/catalog/CloudTabletStatMgr.java | 20 +-
.../java/org/apache/doris/catalog/Database.java | 131 +++--
.../org/apache/doris/catalog/DatabaseProperty.java | 5 +
.../main/java/org/apache/doris/catalog/Env.java | 43 +-
.../org/apache/doris/catalog/FunctionGenTable.java | 2 +-
.../org/apache/doris/catalog/LocalReplica.java | 32 +-
.../main/java/org/apache/doris/catalog/MTMV.java | 6 +-
.../apache/doris/catalog/MaterializedIndex.java | 19 +-
.../doris/catalog/MaterializedIndexMeta.java | 19 +-
...Wrapper.java => OlapChangelogTableWrapper.java} | 42 +-
.../java/org/apache/doris/catalog/OlapTable.java | 216 ++++---
.../org/apache/doris/catalog/OlapTableWrapper.java | 4 +-
.../java/org/apache/doris/catalog/Partition.java | 14 +-
.../java/org/apache/doris/catalog/Replica.java | 12 +-
.../org/apache/doris/catalog/TableProperty.java | 84 +--
.../main/java/org/apache/doris/catalog/Tablet.java | 36 +-
.../apache/doris/catalog/TabletBinlogConfig.java | 121 ++++
.../apache/doris/catalog/TabletInvertedIndex.java | 2 +-
.../java/org/apache/doris/catalog/TabletMeta.java | 10 +-
.../org/apache/doris/catalog/TabletStatMgr.java | 24 +-
.../doris/catalog/stream/BaseTableStream.java | 12 +-
.../doris/catalog/stream/OlapTableStream.java | 2 +-
.../ChangelogRepairReason.java} | 4 +-
.../ChangelogTabletLocality.java} | 201 +++----
.../ChangelogTtlDiscovery.java} | 29 +-
.../org/apache/doris/changelog/ChangelogUtils.java | 54 ++
.../clone/ColocateTableCheckerAndBalancer.java | 17 +-
.../java/org/apache/doris/clone/Rebalancer.java | 5 +-
.../java/org/apache/doris/clone/TabletChecker.java | 13 +-
.../org/apache/doris/clone/TabletSchedCtx.java | 79 +--
.../org/apache/doris/clone/TabletScheduler.java | 268 ++++-----
.../apache/doris/clone/TabletSchedulerStat.java | 8 +-
.../cloud/alter/CloudSchemaChangeHandler.java | 10 +-
.../apache/doris/cloud/backup/CloudRestoreJob.java | 10 +-
.../cloud/datasource/CloudInternalCatalog.java | 48 +-
.../transaction/CloudGlobalTransactionMgr.java | 4 +-
.../common/proc/IncompleteTabletsProcNode.java | 6 +-
.../apache/doris/common/proc/IndexInfoProcDir.java | 4 +-
.../doris/common/proc/PartitionsProcDir.java | 14 +-
.../doris/common/proc/TabletHealthProcDir.java | 58 +-
.../common/proc/TabletSchedulerDetailProcDir.java | 2 +-
.../apache/doris/common/proc/TabletsProcDir.java | 6 +-
.../apache/doris/common/util/BufferSizeUtil.java | 6 +-
.../apache/doris/common/util/PropertyAnalyzer.java | 124 ++--
.../apache/doris/datasource/InternalCatalog.java | 176 +++---
.../apache/doris/job/extensions/mtmv/MTMVTask.java | 2 +-
.../java/org/apache/doris/load/DeleteHandler.java | 4 +-
.../org/apache/doris/master/ReportHandler.java | 30 +-
.../org/apache/doris/mtmv/MTMVPartitionState.java | 2 +-
.../org/apache/doris/mtmv/MTMVPartitionUtil.java | 2 +-
.../java/org/apache/doris/mtmv/MTMVPlanUtil.java | 8 +-
.../org/apache/doris/mtmv/MTMVRelationManager.java | 2 +-
.../main/java/org/apache/doris/mtmv/ivm/AGENTS.md | 20 +-
.../doris/mtmv/ivm/IvmDeltaRewriteHelper.java | 2 +-
.../apache/doris/mtmv/ivm/IvmDupDeltaRewriter.java | 2 +-
.../apache/doris/mtmv/ivm/IvmFailureReason.java | 8 +-
.../doris/mtmv/ivm/IvmIncrRefreshManager.java | 2 +-
.../doris/mtmv/ivm/IvmNormalizeCommonChecker.java | 2 +-
.../doris/mtmv/ivm/IvmNormalizeDupKeysMTMV.java | 2 +-
.../doris/mtmv/ivm/IvmNormalizeUniqKeysMTMV.java | 20 +-
.../doris/mtmv/ivm/IvmSequenceCalculator.java | 30 +-
.../java/org/apache/doris/mtmv/ivm/IvmUtil.java | 18 +-
.../org/apache/doris/nereids/NereidsPlanner.java | 4 +-
.../org/apache/doris/nereids/StatementContext.java | 32 +-
.../glue/translator/PhysicalPlanTranslator.java | 44 +-
.../doris/nereids/rules/analysis/BindRelation.java | 84 +--
.../doris/nereids/rules/analysis/BindSink.java | 2 +-
.../nereids/rules/analysis/CollectRelation.java | 18 +-
.../nereids/rules/analysis/IvmNormalizeMTMV.java | 2 +-
.../rules/implementation/AggregateStrategies.java | 6 +-
.../rewrite/AggScalarSubQueryToWindowFunction.java | 6 +-
.../rewrite/NormalizeOlapTableStreamScan.java | 88 +--
.../table/{Binlog.java => Changelog.java} | 17 +-
.../visitor/TableValuedFunctionVisitor.java | 6 +-
.../commands/AlterDatabasePropertiesCommand.java | 5 +-
.../plans/commands/CreateDatabaseCommand.java | 17 +-
.../trees/plans/commands/CreateMTMVCommand.java | 2 +-
.../plans/commands/CreateTableLikeCommand.java | 3 +
.../trees/plans/commands/DeleteFromCommand.java | 4 +-
.../trees/plans/commands/RefreshMTMVCommand.java | 4 +-
.../trees/plans/commands/ShowDataCommand.java | 88 +--
.../trees/plans/commands/ShowTabletIdCommand.java | 8 +-
.../trees/plans/commands/info/AddColumnOp.java | 2 +-
.../trees/plans/commands/info/AddColumnsOp.java | 4 +-
.../plans/commands/info/AddPartitionLikeOp.java | 4 +-
.../trees/plans/commands/info/AddPartitionOp.java | 4 +-
.../trees/plans/commands/info/AddRollupOp.java | 4 +-
.../nereids/trees/plans/commands/info/AlterOp.java | 6 +-
.../trees/plans/commands/info/BaseViewInfo.java | 4 +-
.../plans/commands/info/ColumnDefinition.java | 2 +-
.../trees/plans/commands/info/CreateIndexOp.java | 4 +-
.../trees/plans/commands/info/CreateMTMVInfo.java | 4 +-
.../trees/plans/commands/info/CreateTableInfo.java | 28 +-
.../trees/plans/commands/info/DropColumnOp.java | 4 +-
.../trees/plans/commands/info/DropIndexOp.java | 4 +-
.../trees/plans/commands/info/DropPartitionOp.java | 4 +-
.../trees/plans/commands/info/DropRollupOp.java | 4 +-
.../plans/commands/info/ModifyColumnCommentOp.java | 4 +-
.../plans/commands/info/ModifyDistributionOp.java | 4 +-
.../plans/commands/info/ModifyPartitionOp.java | 4 +-
.../plans/commands/info/ModifyTableCommentOp.java | 4 +-
.../commands/info/ModifyTablePropertiesOp.java | 13 +-
.../plans/commands/info/RenamePartitionOp.java | 4 +-
.../trees/plans/commands/info/RenameRollupOp.java | 4 +-
.../trees/plans/commands/info/RenameTableOp.java | 4 +-
.../plans/commands/info/ReplacePartitionOp.java | 4 +-
.../trees/plans/commands/info/ReplaceTableOp.java | 2 +-
.../commands/insert/InsertIntoTableCommand.java | 4 +-
.../trees/plans/logical/LogicalOlapScan.java | 8 +-
.../plans/logical/LogicalOlapTableStreamScan.java | 16 +-
.../trees/plans/physical/PhysicalOlapScan.java | 2 +-
.../org/apache/doris/planner/OlapScanNode.java | 36 +-
.../org/apache/doris/planner/OlapTableSink.java | 61 +-
.../planner/normalize/QueryCacheNormalizer.java | 6 +-
.../apache/doris/service/FrontendServiceImpl.java | 32 +-
.../doris/statistics/util/StatisticsUtil.java | 5 +-
.../java/org/apache/doris/system/HeartbeatMgr.java | 29 +-
.../doris/tablefunction/MetadataGenerator.java | 9 +-
...ogFunction.java => TableChangelogFunction.java} | 40 +-
.../doris/tablefunction/TableValuedFunctionIf.java | 4 +-
.../org/apache/doris/task/CreateReplicaTask.java | 10 +-
.../doris/task/UpdateTabletMetaInfoTask.java | 8 +-
.../doris/transaction/DatabaseTransactionMgr.java | 12 +-
.../apache/doris/transaction/TransactionState.java | 2 +-
.../apache/doris/transaction/TransactionUtil.java | 4 +-
.../main/java/org/apache/doris/tso/TSOService.java | 8 +-
.../java/org/apache/doris/tso/TSOTimestamp.java | 2 +-
...st.java => ChangelogConfigPublicationTest.java} | 85 +--
.../doris/alter/SchemaChangeHandlerTest.java | 160 ++---
.../org/apache/doris/backup/CatalogMocker.java | 10 +-
.../org/apache/doris/binlog/BinlogGcerTest.java | 18 +-
.../org/apache/doris/binlog/BinlogManagerTest.java | 35 +-
.../org/apache/doris/binlog/BinlogTestUtils.java | 9 +-
.../doris/catalog/AlterTableStreamCommentTest.java | 6 +-
.../catalog/BinlogFormatCaseInsensitiveTest.java | 147 -----
.../doris/catalog/CatalogRecycleBinTest.java | 54 +-
.../org/apache/doris/catalog/CatalogTestUtil.java | 4 +-
.../doris/catalog/ChangelogConfigSplitTest.java | 443 ++++++++++++++
.../doris/catalog/ChangelogFeatureGateTest.java | 137 +++++
.../ChangelogLegacyBinlogPropertiesTest.java | 233 ++++++++
.../doris/catalog/ChangelogTtlConfigTest.java | 142 +++++
.../java/org/apache/doris/catalog/ColumnTest.java | 25 +
.../doris/catalog/CreateTableStreamTest.java | 20 +-
.../org/apache/doris/catalog/CreateTableTest.java | 54 +-
.../org/apache/doris/catalog/CreateViewTest.java | 15 +-
.../org/apache/doris/catalog/DatabaseTest.java | 144 ++++-
.../doris/catalog/DropMaterializedViewTest.java | 14 +-
.../apache/doris/catalog/DropTableStreamTest.java | 14 +-
.../doris/catalog/MaterializedIndexTest.java | 44 +-
...Test.java => OlapTableChangelogSchemaTest.java} | 88 +--
.../apache/doris/catalog/OlapTableWrapperTest.java | 54 +-
.../doris/catalog/RowBinlogFeatureGateTest.java | 119 ----
.../doris/catalog/RowBinlogTtlConfigTest.java | 99 ----
.../apache/doris/catalog/ShowCreateMTMVTest.java | 5 +-
.../doris/catalog/TabletBinlogConfigTest.java | 126 ++++
.../java/org/apache/doris/catalog/TabletTest.java | 70 +--
.../stream/CloudTableStreamConsumptionTest.java | 10 +-
.../stream/OlapTableStreamSnapshotReadTest.java | 2 +-
.../stream/TableStreamManagerCleanupTest.java | 7 +-
.../ChangelogTtlDiscoveryTest.java} | 66 +--
.../ChangelogUtilsTest.java} | 22 +-
...ancerTest.java => ChangelogRebalancerTest.java} | 20 +-
...yTest.java => ChangelogTabletLocalityTest.java} | 192 +++---
...Test.java => ChangelogTabletSchedulerTest.java} | 216 +++----
.../doris/clone/ClusterLoadStatisticsTest.java | 6 +-
.../clone/ColocateTableCheckerAndBalancerTest.java | 40 +-
.../org/apache/doris/clone/RebalancerTestUtil.java | 2 +-
.../cloud/alter/CloudSchemaChangeHandlerTest.java | 50 +-
.../transaction/CloudGlobalTransactionMgrTest.java | 41 +-
.../doris/cluster/DecommissionBackendTest.java | 2 +-
.../apache/doris/common/PropertyAnalyzerTest.java | 80 ++-
.../common/proc/CloudProcVersionDisplayTest.java | 2 +-
.../apache/doris/common/proc/ProcServiceTest.java | 2 +-
.../doris/common/proc/TabletHealthProcDirTest.java | 97 ++-
.../org/apache/doris/common/util/UnitTestUtil.java | 2 +-
.../doris/datasource/InternalCatalogTest.java | 13 +-
.../datasource/RoundRobinCreateTabletTest.java | 2 +-
.../org/apache/doris/http/DorisHttpTestCase.java | 2 +-
.../java/org/apache/doris/load/DeleteJobTest.java | 2 +-
...erTest.java => ChangelogReportHandlerTest.java} | 18 +-
.../java/org/apache/doris/mtmv/AlterMTMVTest.java | 84 ++-
.../org/apache/doris/mtmv/MTMVPlanUtilTest.java | 61 +-
.../java/org/apache/doris/mtmv/MTMVTaskTest.java | 14 +-
.../doris/mtmv/ivm/IvmAggDeltaHandlerTest.java | 2 +-
.../doris/mtmv/ivm/IvmBaselineRebuildTest.java | 40 +-
.../doris/mtmv/ivm/IvmDeltaRewriteHelperTest.java | 6 +-
.../doris/mtmv/ivm/IvmDeltaRewriteStateTest.java | 2 +-
.../apache/doris/mtmv/ivm/IvmDeltaTestBase.java | 19 +-
.../doris/mtmv/ivm/IvmFailureReasonTest.java | 2 +-
.../doris/mtmv/ivm/IvmIncrRefreshManagerTest.java | 18 +-
.../mtmv/ivm/IvmNormalizeCommonCheckerTest.java | 2 +-
.../mtmv/ivm/IvmNormalizeUniqKeysMTMVJoinTest.java | 4 +-
.../mtmv/ivm/IvmNormalizeUniqKeysMTMVTest.java | 71 ++-
.../ivm/IvmNormalizeUniqKeysMTMVUnionTest.java | 4 +-
.../mtmv/ivm/IvmPlanSignatureGeneratorTest.java | 2 +-
.../doris/mtmv/ivm/IvmSequenceCalculatorTest.java | 18 +-
.../org/apache/doris/mtmv/ivm/IvmUtilTest.java | 14 +
.../apache/doris/nereids/StatementContextTest.java | 14 +-
.../translator/PhysicalPlanTranslatorTest.java | 50 +-
.../nereids/rules/analysis/BindRelationTest.java | 6 +-
.../nereids/rules/analysis/CheckRowPolicyTest.java | 10 +-
...t.java => CollectRelationChangelogTtlTest.java} | 70 +--
.../rules/analysis/IvmFullRefreshMTMVTest.java | 5 +-
.../apache/doris/nereids/sqltest/SqlTestBase.java | 5 +-
.../nereids/trees/plans/CreateMTMVCommandTest.java | 171 +++---
.../trees/plans/ExplainTableStreamPlanTest.java | 70 +--
.../trees/plans/commands/ShowDataCommandTest.java | 14 +-
.../plans/commands/ShowTabletIdCommandTest.java | 2 +-
.../plans/commands/TruncateTableCommandTest.java | 8 +-
.../commands/UpdateMvByPartitionCommandTest.java | 10 +-
.../plans/commands/info/ModifyPartitionOpTest.java | 32 +-
.../InsertIntoTableCommandTableStreamTest.java | 14 +-
.../doris/service/FrontendServiceImplTest.java | 21 +-
...st.java => TableChangelogFunctionAuthTest.java} | 12 +-
.../TsoStatusMetadataGeneratorTest.java | 12 +-
.../java/org/apache/doris/task/AgentTaskTest.java | 20 +-
.../transaction/DatabaseTransactionMgrTest.java | 59 +-
.../TableStreamOffsetTransactionTest.java | 6 +-
.../java/org/apache/doris/tso/TSOServiceTest.java | 90 +--
.../apache/doris/utframe/TestWithFeService.java | 10 +-
gensrc/proto/cloud.proto | 3 +-
gensrc/proto/descriptors.proto | 6 +-
gensrc/proto/internal_service.proto | 6 +-
gensrc/proto/olap_file.proto | 30 +-
gensrc/thrift/AgentService.thrift | 8 +-
gensrc/thrift/Descriptors.thrift | 8 +-
gensrc/thrift/HeartbeatService.thrift | 3 +-
gensrc/thrift/PlanNodes.thrift | 11 +-
.../test_changelog_alter_compaction_policy.out} | 0
.../test_changelog_auto_partition.out} | 2 +-
.../test_changelog_basic.out} | 16 +-
.../test_changelog_before_column_name.out | 129 ++++
.../test_changelog_changes_syntax.out} | 3 +-
.../changelog_p0/test_changelog_cluster_key.out | 8 +
.../test_changelog_compaction.out} | 8 +-
.../test_changelog_flexible_partial_update.out} | 7 +-
...ngelog_flexible_partial_update_no_sequence.out} | 3 +-
.../test_changelog_mow_light_delete.out} | 5 +-
.../test_changelog_multi_segment.out} | 8 +-
.../changelog_p0/test_changelog_property_keys.out | 46 ++
.../test_changelog_publish_conflict.out} | 8 +-
.../test_changelog_rowset_tso_pruning.out} | 0
.../test_changelog_schema_change.out} | 6 +-
.../test_changelog_ttl.out} | 0
.../test_changelog_ttl_compaction_isolation.out} | 0
.../test_changelog_ttl_mow.out} | 0
.../test_changelog_ttl_publication.out} | 0
.../test_changelog_ttl_query_forms.out} | 0
.../test_cloud_changelog_compaction.out} | 0
.../test_cloud_changelog_publish_conflict.out} | 0
.../test_group_commit_interval_ms_property.out | 12 -
...ut => test_ivm_changelog_schema_validation.out} | 4 +-
...est_ivm_dup_excluded_no_changelog_fallback.out} | 0
..._row_binlog.out => test_ivm_mtmv_changelog.out} | 4 +-
.../data/query_p0/system/test_table_properties.out | 20 +-
.../test_binlog_format_case_insensitive.out | 21 -
.../row_binlog_p0/test_row_binlog_cluster_key.out | 8 -
.../test_show_create_table_and_views_nereids.out | 23 -
.../test_table_stream_multi_segment_mow.out | 2 +-
.../pipeline/cloud_p0/conf/be_custom.conf | 1 +
.../pipeline/cloud_p0/conf/fe_custom.conf | 1 +
regression-test/pipeline/external/conf/be.conf | 1 +
regression-test/pipeline/external/conf/fe.conf | 1 +
.../pipeline/nonConcurrent/conf/be.conf | 1 +
.../pipeline/nonConcurrent/conf/fe.conf | 1 +
regression-test/pipeline/p0/conf/be.conf | 1 +
regression-test/pipeline/p0/conf/fe.conf | 1 +
regression-test/pipeline/p1/conf/be.conf | 1 +
regression-test/pipeline/p1/conf/fe.conf | 1 +
..._auth.groovy => test_changelog_tvf_auth.groovy} | 31 +-
.../test_changelog_alter_compaction_policy.groovy} | 60 +-
.../test_changelog_append_only_projection.groovy} | 15 +-
.../test_changelog_auto_partition.groovy} | 19 +-
.../test_changelog_basic.groovy} | 211 ++++---
.../test_changelog_before_column_name.groovy | 340 +++++++++++
.../test_changelog_changes_syntax.groovy} | 185 +++---
.../test_changelog_cluster_key.groovy} | 38 +-
.../test_changelog_compaction.groovy} | 150 +++--
.../test_changelog_flexible_partial_update.groovy} | 51 +-
...log_flexible_partial_update_no_sequence.groovy} | 45 +-
.../test_changelog_mow_light_delete.groovy | 101 ++++
.../test_changelog_multi_segment.groovy} | 71 ++-
...test_changelog_partial_update_only_keys.groovy} | 43 +-
.../test_changelog_property_keys.groovy | 498 ++++++++++++++++
.../test_changelog_publish_conflict.groovy} | 73 ++-
.../test_changelog_rowset_tso_pruning.groovy} | 53 +-
.../test_changelog_schema_change.groovy | 104 ++++
.../suites/changelog_p0/test_changelog_ttl.groovy | 214 +++++++
...test_changelog_ttl_compaction_isolation.groovy} | 40 +-
.../changelog_p0/test_changelog_ttl_mow.groovy | 122 ++++
.../test_changelog_ttl_publication.groovy} | 50 +-
.../test_changelog_ttl_query_forms.groovy} | 39 +-
.../test_cloud_changelog_compaction.groovy} | 148 +++--
.../test_cloud_changelog_publish_conflict.groovy} | 63 +-
.../test_incr_without_changelog.groovy} | 12 +-
.../check_before_quit/check_before_quit.groovy | 8 +-
...est_cloud_table_stream_check_meta_orphan.groovy | 4 +-
...st_cloud_table_stream_continuous_consume.groovy | 3 +-
.../test_cloud_table_stream_ddl_metadata.groovy | 3 +-
.../test_cloud_table_stream_e2e.groovy | 3 +-
.../test_cloud_table_stream_initial_rows.groovy | 12 +-
.../test_cloud_table_stream_multi_reference.groovy | 12 +-
...test_cloud_table_stream_object_lifecycle.groovy | 15 +-
...est_cloud_table_stream_partition_consume.groovy | 3 +-
...t_cloud_table_stream_partition_lifecycle.groovy | 9 +-
.../test_cloud_table_stream_schema_change.groovy | 5 +-
.../suites/mtmv_ivm_restart_p0/load.groovy | 5 +-
.../suites/mtmv_p0/ivm/test_ivm_agg_1.groovy | 30 +-
.../suites/mtmv_p0/ivm/test_ivm_agg_2.groovy | 26 +-
.../suites/mtmv_p0/ivm/test_ivm_agg_3.groovy | 17 +-
.../suites/mtmv_p0/ivm/test_ivm_agg_4.groovy | 16 +-
.../suites/mtmv_p0/ivm/test_ivm_agg_5.groovy | 8 +-
.../suites/mtmv_p0/ivm/test_ivm_agg_6.groovy | 8 +-
.../suites/mtmv_p0/ivm/test_ivm_agg_array_1.groovy | 6 +-
.../ivm/test_ivm_agg_expr_over_agg_1.groovy | 3 +-
.../ivm/test_ivm_agg_expr_over_agg_2.groovy | 18 +-
.../ivm/test_ivm_agg_expr_over_agg_3.groovy | 3 +-
.../suites/mtmv_p0/ivm/test_ivm_agg_join_1.groovy | 30 +-
.../suites/mtmv_p0/ivm/test_ivm_agg_join_2.groovy | 39 +-
.../suites/mtmv_p0/ivm/test_ivm_agg_join_3.groovy | 6 +-
.../test_ivm_agg_join_same_name_group_key.groovy | 18 +-
.../mtmv_p0/ivm/test_ivm_agg_outer_join_1.groovy | 9 +-
...test_ivm_agg_previous_commit_not_visible.groovy | 7 +-
.../mtmv_p0/ivm/test_ivm_agg_shared_columns.groovy | 12 +-
.../ivm/test_ivm_append_only_dup_mv_1.groovy | 9 +-
.../ivm/test_ivm_append_only_dup_mv_2.groovy | 21 +-
.../ivm/test_ivm_append_only_dup_mv_3.groovy | 35 +-
.../ivm/test_ivm_base_sequence_isolation.groovy | 25 +-
.../ivm/test_ivm_baseline_marker_scope.groovy | 16 +-
.../suites/mtmv_p0/ivm/test_ivm_basic_mtmv.groovy | 43 +-
.../mtmv_p0/ivm/test_ivm_bitmap_agg_1.groovy | 3 +-
.../mtmv_p0/ivm/test_ivm_bitmap_agg_2.groovy | 6 +-
.../ivm/test_ivm_bitmap_runtime_fallback.groovy | 5 +-
.../mtmv_p0/ivm/test_ivm_chained_mtmv_1.groovy | 14 +-
.../mtmv_p0/ivm/test_ivm_chained_mtmv_2.groovy | 12 +-
.../ivm/test_ivm_chained_stream_scope.groovy | 12 +-
..._1.groovy => test_ivm_changelog_order_1.groovy} | 12 +-
...=> test_ivm_changelog_schema_validation.groovy} | 36 +-
.../ivm/test_ivm_complete_refresh_rowid.groovy | 5 +-
.../ivm/test_ivm_create_if_not_exists.groovy | 12 +-
.../mtmv_p0/ivm/test_ivm_cross_db_same_name.groovy | 10 +-
.../ivm/test_ivm_cross_db_single_base.groovy | 5 +-
.../suites/mtmv_p0/ivm/test_ivm_cube_repeat.groovy | 5 +-
.../suites/mtmv_p0/ivm/test_ivm_decimal_avg.groovy | 5 +-
.../test_ivm_drop_column_fallback_reason.groovy | 5 +-
..._drop_referenced_column_baseline_rebuild.groovy | 7 +-
..._ivm_dup_excluded_no_changelog_fallback.groovy} | 11 +-
.../mtmv_p0/ivm/test_ivm_dup_keys_lsn_rowid.groovy | 29 +-
.../mtmv_p0/ivm/test_ivm_dup_keys_mtmv.groovy | 7 +-
.../ivm/test_ivm_excluded_trigger_table.groovy | 5 +-
.../test_ivm_execution_failure_no_fallback.groovy | 5 +-
.../mtmv_p0/ivm/test_ivm_explain_refresh.groovy | 6 +-
.../test_ivm_fallback_keeps_the_whole_scope.groovy | 10 +-
.../test_ivm_fallback_stream_multi_batch.groovy | 10 +-
...test_ivm_fallback_stream_multi_batch_dup.groovy | 6 +-
..._ivm_full_keys_same_name_unprojected_key.groovy | 6 +-
...test_ivm_full_outer_join_residual_update.groovy | 10 +-
.../mtmv_p0/ivm/test_ivm_hidden_columns.groovy | 20 +-
.../mtmv_p0/ivm/test_ivm_inner_join_1.groovy | 33 +-
.../mtmv_p0/ivm/test_ivm_inner_join_2.groovy | 32 +-
.../ivm/test_ivm_inner_join_update_delete.groovy | 10 +-
.../ivm/test_ivm_internal_stream_guard.groovy | 10 +-
.../mtmv_p0/ivm/test_ivm_minmax_count.groovy | 5 +-
.../mtmv_p0/ivm/test_ivm_minmax_count_join.groovy | 10 +-
.../ivm/test_ivm_minmax_runtime_fallback.groovy | 5 +-
...inlog.groovy => test_ivm_mtmv_changelog.groovy} | 48 +-
.../ivm/test_ivm_nested_outer_join_create.groovy | 15 +-
.../mtmv_p0/ivm/test_ivm_nullable_row_id.groovy | 6 +-
.../mtmv_p0/ivm/test_ivm_one_row_relation_1.groovy | 20 +-
.../test_ivm_one_row_relation_partitioned.groovy | 10 +-
.../mtmv_p0/ivm/test_ivm_outer_join_1.groovy | 18 +-
.../mtmv_p0/ivm/test_ivm_outer_join_2.groovy | 24 +-
.../mtmv_p0/ivm/test_ivm_outer_join_3.groovy | 12 +-
.../mtmv_p0/ivm/test_ivm_outer_join_4.groovy | 6 +-
.../mtmv_p0/ivm/test_ivm_outer_join_5.groovy | 10 +-
.../mtmv_p0/ivm/test_ivm_outer_join_6.groovy | 10 +-
.../ivm/test_ivm_outer_join_union_null_side.groovy | 9 +-
...ivm_overwrite_failure_between_the_halves.groovy | 5 +-
...test_ivm_partial_rebuild_is_not_recorded.groovy | 12 +-
.../ivm/test_ivm_partition_baseline_rebuild.groovy | 5 +-
..._ivm_partition_baseline_rebuild_dup_keys.groovy | 5 +-
.../ivm/test_ivm_partition_drop_live_delta.groovy | 9 +-
.../ivm/test_ivm_partition_epoch_rebuild.groovy | 9 +-
.../ivm/test_ivm_partition_sync_limit.groovy | 12 +-
...est_ivm_partition_sync_limit_with_window.groovy | 12 +-
.../ivm/test_ivm_partition_sync_retry.groovy | 9 +-
.../ivm/test_ivm_partition_unique_key.groovy | 5 +-
.../ivm/test_ivm_partition_window_limit.groovy | 5 +-
.../ivm/test_ivm_partition_window_remove.groovy | 5 +-
...est_ivm_partitions_after_failed_complete.groovy | 5 +-
..._ivm_partitions_fallback_stream_unusable.groovy | 12 +-
.../ivm/test_ivm_plan_signature_fallback.groovy | 3 +-
.../mtmv_p0/ivm/test_ivm_refresh_dry_run.groovy | 5 +-
.../suites/mtmv_p0/ivm/test_ivm_repeat_1.groovy | 3 +-
.../suites/mtmv_p0/ivm/test_ivm_repeat_2.groovy | 5 +-
.../mtmv_p0/ivm/test_ivm_repeat_join_1.groovy | 6 +-
.../ivm/test_ivm_replace_stream_cleanup.groovy | 6 +-
.../ivm/test_ivm_replace_table_reconcile.groovy | 10 +-
.../mtmv_p0/ivm/test_ivm_rewrite_projection.groovy | 3 +-
.../ivm/test_ivm_sequence_delta_overwrite.groovy | 10 +-
.../suites/mtmv_p0/ivm/test_ivm_snapshot.groovy | 8 +-
...remental_rebuilds_invalidated_partitions.groovy | 7 +-
.../mtmv_p0/ivm/test_ivm_subquery_alias_1.groovy | 3 +-
.../mtmv_p0/ivm/test_ivm_subquery_alias_2.groovy | 3 +-
.../mtmv_p0/ivm/test_ivm_subquery_alias_3.groovy | 6 +-
.../mtmv_p0/ivm/test_ivm_task_query_id.groovy | 5 +-
..._ivm_unconsumed_dimension_is_not_current.groovy | 12 +-
.../suites/mtmv_p0/ivm/test_ivm_union_1.groovy | 30 +-
.../suites/mtmv_p0/ivm/test_ivm_union_2.groovy | 41 +-
.../suites/mtmv_p0/ivm/test_ivm_union_3.groovy | 22 +-
.../ivm/test_ivm_union_delete_same_base.groovy | 5 +-
.../mtmv_p0/ivm/test_ivm_use_full_keys_1.groovy | 12 +-
.../mtmv_p0/ivm/test_ivm_use_full_keys_2.groovy | 9 +-
.../mtmv_p0/ivm/test_ivm_use_full_keys_3.groovy | 3 +-
.../mtmv_p0/ivm/test_ivm_use_full_keys_4.groovy | 3 +-
.../mtmv_p0/ivm/test_ivm_use_full_keys_5.groovy | 6 +-
.../mtmv_p0/ivm/test_ivm_use_full_keys_6.groovy | 16 +-
.../mtmv_p0/ivm/test_ivm_varchar_key_rowid.groovy | 6 +-
.../test_stream_consumption_schema.groovy | 9 +-
.../schema_table/test_stream_info_schema.groovy | 8 +-
.../test_binlog_format_case_insensitive.groovy | 180 ------
.../test_row_binlog_mow_light_delete.groovy | 104 ----
.../test_row_binlog_schema_change.groovy | 105 ----
.../row_binlog_p0/test_row_binlog_ttl.groovy | 220 -------
.../row_binlog_p0/test_row_binlog_ttl_mow.groovy | 123 ----
.../test_binlog_property_alter_exception.groovy | 96 ---
.../test_changelog_property_alter_exception.groovy | 142 +++++
...end_only_stream_without_historical_value.groovy | 9 +-
.../test_min_delta_op_filter_correctness.groovy | 5 +-
.../table_stream_p0/test_min_delta_stream.groovy | 121 ++--
.../test_mow_min_delta_delete_before.groovy | 10 +-
...st_olap_table_stream_history_consumption.groovy | 9 +-
.../test_olap_table_stream_history_query.groovy | 8 +-
...p_table_stream_mixed_history_incremental.groovy | 3 +-
.../test_olap_table_stream_reset.groovy | 19 +-
.../test_olap_table_stream_schema_sync.groovy | 5 +-
.../test_olap_table_stream_snapshot.groovy | 22 +-
.../test_olap_table_stream_snapshot_count.groovy | 3 +-
.../test_table_stream_alter_comment.groovy | 3 +-
.../test_table_stream_input_validation.groovy | 5 +-
.../test_table_stream_multi_segment_mow.groovy | 87 ++-
.../test_table_stream_query_comprehensive.groovy | 36 +-
.../test_constant_hidden_column_statistics.groovy | 5 +-
.../test_mow_time_travel_row_policy.groovy | 9 +-
.../time_travel_p0/test_time_travel_dup.groovy | 13 +-
.../time_travel_p0/test_time_travel_mow.groovy | 8 +-
.../test_time_travel_mow_complex.groovy | 7 +-
.../suites/tso_p0/test_committed_tso.groovy | 4 +-
.../tso_p0/test_tso_rowset_commit_tso.groovy | 3 +-
...st_constant_column_partial_update_hidden.groovy | 5 +-
659 files changed, 11907 insertions(+), 9939 deletions(-)
diff --git a/be/src/agent/heartbeat_server.cpp b/be/src/agent/heartbeat_server.cpp
index 3629e462012..a3ee995020e 100644
--- a/be/src/agent/heartbeat_server.cpp
+++ b/be/src/agent/heartbeat_server.cpp
@@ -56,7 +56,8 @@ namespace doris {
namespace {
// Store the sampled value independently: bvar's sampler also runs before cluster
// initialization and after cluster teardown.
-bvar::Status<int64_t> row_binlog_ttl_reference("row_binlog_ttl_reference_tso", 0);
+// ATTN(changelog-rename): formerly "row_binlog_ttl_reference_tso"
+bvar::Status<int64_t> changelog_ttl_reference("changelog_ttl_reference_tso", 0);
} // namespace
HeartbeatServer::HeartbeatServer(ClusterInfo* cluster_info)
@@ -339,10 +340,9 @@ Status HeartbeatServer::_heartbeat(const TMasterInfo& master_info) {
}
// A delayed epoch from the same master address must not authorize cleanup.
- if (master_info.epoch == _fe_epoch && master_info.__isset.row_binlog_ttl_reference_tso) {
- _cluster_info->advance_row_binlog_ttl_reference_tso(
- master_info.row_binlog_ttl_reference_tso);
- row_binlog_ttl_reference.set_value(_cluster_info->row_binlog_ttl_reference_tso());
+ if (master_info.epoch == _fe_epoch && master_info.__isset.changelog_ttl_reference_tso) {
+ _cluster_info->advance_changelog_ttl_reference_tso(master_info.changelog_ttl_reference_tso);
+ changelog_ttl_reference.set_value(_cluster_info->changelog_ttl_reference_tso());
}
if (need_report) {
diff --git a/be/src/agent/task_worker_pool.cpp b/be/src/agent/task_worker_pool.cpp
index 2e584918976..1755dd170ce 100644
--- a/be/src/agent/task_worker_pool.cpp
+++ b/be/src/agent/task_worker_pool.cpp
@@ -909,10 +909,10 @@ void update_tablet_meta_callback(StorageEngine& engine, const TAgentTaskRequest&
need_to_save = true;
}
if (tablet_meta_info.__isset.compaction_policy) {
- // The binlog policy is set when a row binlog tablet is created and never changes.
- if (tablet->is_row_binlog_tablet()) {
+ // The changelog policy is set when a changelog tablet is created and never changes.
+ if (tablet->is_changelog_tablet()) {
status = Status::InvalidArgument(
- "can not change compaction policy of row binlog tablet {}",
+ "can not change compaction policy of changelog tablet {}",
tablet_meta_info.tablet_id);
continue;
}
diff --git a/be/src/cloud/cloud_compaction_action.cpp b/be/src/cloud/cloud_compaction_action.cpp
index 6cc4651e57d..e0400b25272 100644
--- a/be/src/cloud/cloud_compaction_action.cpp
+++ b/be/src/cloud/cloud_compaction_action.cpp
@@ -145,12 +145,12 @@ Status CloudCompactionAction::_handle_run_compaction(HttpRequest* req, std::stri
// check compaction_type equals 'base' or 'cumulative'
auto& compaction_type = req->param(PARAM_COMPACTION_TYPE);
- if (compaction_type == PARAM_COMPACTION_ROW_BINLOG_TTL) {
+ if (compaction_type == PARAM_COMPACTION_CHANGELOG_TTL) {
if (tablet_id == 0 || table_id != 0) {
- return Status::InvalidArgument("row_binlog_ttl requires a tablet_id");
+ return Status::InvalidArgument("changelog_ttl requires a tablet_id");
}
- RETURN_IF_ERROR(_engine.submit_row_binlog_ttl(tablet_id, true));
- *json_result = R"({"status":"Success","msg":"ROW binlog TTL task queued"})";
+ RETURN_IF_ERROR(_engine.submit_changelog_ttl(tablet_id, true));
+ *json_result = R"({"status":"Success","msg":"changelog TTL task queued"})";
return Status::OK();
}
if (compaction_type != PARAM_COMPACTION_BASE &&
diff --git a/be/src/cloud/cloud_cumulative_compaction.cpp b/be/src/cloud/cloud_cumulative_compaction.cpp
index e55e80003ff..e0159e8d1c7 100644
--- a/be/src/cloud/cloud_cumulative_compaction.cpp
+++ b/be/src/cloud/cloud_cumulative_compaction.cpp
@@ -166,7 +166,7 @@ Status CloudCumulativeCompaction::prepare_compact() {
st = cloud_tablet()->sync_rowsets();
RETURN_IF_ERROR(st);
}
- snapshot_row_binlog_ttl();
+ snapshot_changelog_ttl();
// pick rowsets to compact
st = pick_rowsets_to_compact();
@@ -368,7 +368,7 @@ Status CloudCumulativeCompaction::modify_rowsets() {
} else if (!_enable_parallel_cumu_compaction ||
_output_rowset->start_version() == input_cumulative_point) {
new_cumulative_point =
- _engine.cumu_compaction_policy(compaction_policy)
+ _engine.cumu_compaction_policy(compaction_policy, cloud_tablet()->tablet_id())
->new_cumulative_point(cloud_tablet(), _output_rowset,
_last_delete_version, input_cumulative_point);
}
@@ -494,8 +494,8 @@ Status CloudCumulativeCompaction::modify_rowsets() {
"CumulativeCompaction.modify_rowsets.trigger_abort_job_failed for tablet_id {}",
cloud_tablet()->tablet_id());
});
- if (_row_binlog_ttl_filtered_rows > 0) {
- compaction_job->set_row_binlog_ttl_seconds(_row_binlog_ttl_seconds);
+ if (_changelog_ttl_filtered_rows > 0) {
+ compaction_job->set_changelog_ttl_seconds(_changelog_ttl_seconds);
}
cloud::FinishTabletJobResponse resp;
auto st = _engine.meta_mgr().commit_tablet_job(job, &resp);
@@ -655,8 +655,8 @@ Status CloudCumulativeCompaction::advance_cumulative_point_before_pick(
return Status::OK();
}
- auto compaction_policy =
- _engine.cumu_compaction_policy(cloud_tablet()->tablet_meta()->compaction_policy());
+ auto compaction_policy = _engine.cumu_compaction_policy(
+ cloud_tablet()->tablet_meta()->compaction_policy(), cloud_tablet()->tablet_id());
int64_t input_cumulative_point;
int64_t output_cumulative_point;
std::vector<RowsetSharedPtr> candidates;
@@ -684,8 +684,8 @@ Status CloudCumulativeCompaction::advance_cumulative_point_before_pick(
}
auto rowset_meta = rowset->rowset_meta();
- if (_row_binlog_ttl_cutoff_tso >= 0 &&
- row_binlog_rowset_expired(*rowset_meta, _row_binlog_ttl_cutoff_tso)) {
+ if (_changelog_ttl_cutoff_tso >= 0 &&
+ changelog_rowset_expired(*rowset_meta, _changelog_ttl_cutoff_tso)) {
break;
}
if (rowset_meta->has_delete_predicate()) {
@@ -720,7 +720,7 @@ Status CloudCumulativeCompaction::pick_rowsets_to_compact() {
int64_t min_conflict_version = _min_conflict_version;
int64_t max_conflict_version = _max_conflict_version;
- if (_row_binlog_ttl_cutoff_tso >= 0) {
+ if (_changelog_ttl_cutoff_tso >= 0) {
std::vector<RowsetSharedPtr> ttl_candidates;
{
std::shared_lock rlock(_tablet->get_header_lock());
@@ -737,7 +737,7 @@ Status CloudCumulativeCompaction::pick_rowsets_to_compact() {
});
}
std::ranges::sort(ttl_candidates, Rowset::comparator);
- if (pick_expired_row_binlog_rowset(ttl_candidates)) {
+ if (pick_expired_changelog_rowset(ttl_candidates)) {
return Status::OK();
}
}
@@ -769,12 +769,12 @@ Status CloudCumulativeCompaction::pick_rowsets_to_compact() {
DCHECK(false) << st;
return st;
}
- if (pick_expired_row_binlog_rowset(candidates)) {
+ if (pick_expired_changelog_rowset(candidates)) {
return Status::OK();
}
size_t compaction_score = 0;
- _engine.cumu_compaction_policy(compaction_policy)
+ _engine.cumu_compaction_policy(compaction_policy, cloud_tablet()->tablet_id())
->pick_input_rowsets(cloud_tablet(), candidates, max_score,
config::cumulative_compaction_min_deltas, &_input_rowsets,
&_last_delete_version, &compaction_score);
@@ -857,7 +857,7 @@ Status CloudCumulativeCompaction::pick_rowsets_to_compact() {
}
Status CloudCumulativeCompaction::prepare_merge_input_rowsets(MergeInputRowsetsResult* result) {
- if (_row_binlog_ttl_filtered_rows > 0 ||
+ if (_changelog_ttl_filtered_rows > 0 ||
!_single_rowset_compaction_segment_group_size.has_value()) {
return Status::OK();
}
diff --git a/be/src/cloud/cloud_cumulative_compaction_binlog_policy.cpp b/be/src/cloud/cloud_cumulative_compaction_changelog_policy.cpp
similarity index 76%
rename from be/src/cloud/cloud_cumulative_compaction_binlog_policy.cpp
rename to be/src/cloud/cloud_cumulative_compaction_changelog_policy.cpp
index 6af62cab469..6206103da9b 100644
--- a/be/src/cloud/cloud_cumulative_compaction_binlog_policy.cpp
+++ b/be/src/cloud/cloud_cumulative_compaction_changelog_policy.cpp
@@ -15,31 +15,31 @@
// specific language governing permissions and limitations
// under the License.
-#include "cloud/cloud_cumulative_compaction_binlog_policy.h"
+#include "cloud/cloud_cumulative_compaction_changelog_policy.h"
#include "cloud/cloud_tablet.h"
#include "common/config.h"
-#include "storage/compaction/cumulative_compaction_binlog_policy.h"
+#include "storage/compaction/cumulative_compaction_changelog_policy.h"
#include "storage/tablet/tablet.h"
#include "util/time.h"
namespace doris {
-bool CloudBinlogCumulativeCompactionPolicy::is_compaction_enough(
+bool CloudChangelogCumulativeCompactionPolicy::is_compaction_enough(
const RowsetMetaSharedPtr& rowset_meta) const {
if (rowset_meta->compaction_level() !=
- BinlogCumulativeCompactionPolicy::kBinlogCompactionMaxLevel - 1) {
+ ChangelogCumulativeCompactionPolicy::kChangelogCompactionMaxLevel - 1) {
return false;
}
if (rowset_meta->start_version() == 0) {
return true;
}
return rowset_meta->data_disk_size() >=
- config::binlog_compaction_goal_size_mbytes * 1024 * 1024 ||
- rowset_meta->get_compaction_score() >= config::binlog_compaction_file_count_threshold;
+ config::changelog_compaction_goal_size_mbytes * 1024 * 1024 ||
+ rowset_meta->get_compaction_score() >= config::changelog_compaction_file_count_threshold;
}
-int64_t CloudBinlogCumulativeCompactionPolicy::new_cumulative_point(
+int64_t CloudChangelogCumulativeCompactionPolicy::new_cumulative_point(
CloudTablet* tablet, const RowsetSharedPtr& output_rowset, Version& last_delete_version,
int64_t last_cumulative_point) {
if (output_rowset->num_segments() == 0 || !is_compaction_enough(output_rowset->rowset_meta())) {
@@ -48,7 +48,7 @@ int64_t CloudBinlogCumulativeCompactionPolicy::new_cumulative_point(
return output_rowset->end_version() + 1;
}
-int64_t CloudBinlogCumulativeCompactionPolicy::get_compaction_level(
+int64_t CloudChangelogCumulativeCompactionPolicy::get_compaction_level(
CloudTablet* tablet, const std::vector<RowsetSharedPtr>& input_rowsets,
RowsetSharedPtr output_rowset) {
DCHECK(!input_rowsets.empty()) << "tablet=" << tablet->tablet_id();
@@ -65,13 +65,13 @@ int64_t CloudBinlogCumulativeCompactionPolicy::get_compaction_level(
<< "tablet=" << tablet->tablet_id();
}
- if (first_level == BinlogCumulativeCompactionPolicy::kBinlogCompactionMaxLevel - 1) {
+ if (first_level == ChangelogCumulativeCompactionPolicy::kChangelogCompactionMaxLevel - 1) {
return first_level;
}
return first_level + 1;
}
-uint32_t CloudBinlogCumulativeCompactionPolicy::calc_binlog_compaction_level_score(
+uint32_t CloudChangelogCumulativeCompactionPolicy::calc_changelog_compaction_level_score(
CloudTablet* tablet, const std::vector<RowsetSharedPtr>& candidate_rowsets,
int8_t level) const {
uint32_t score = 0;
@@ -81,7 +81,7 @@ uint32_t CloudBinlogCumulativeCompactionPolicy::calc_binlog_compaction_level_sco
if (rs_meta->compaction_level() != level) {
continue;
}
- if (level == BinlogCumulativeCompactionPolicy::kBinlogCompactionMaxLevel - 1 &&
+ if (level == ChangelogCumulativeCompactionPolicy::kChangelogCompactionMaxLevel - 1 &&
point != Tablet::K_INVALID_CUMULATIVE_POINT && rs_meta->end_version() < point) {
score += 1;
continue;
@@ -91,14 +91,14 @@ uint32_t CloudBinlogCumulativeCompactionPolicy::calc_binlog_compaction_level_sco
return score;
}
-uint32_t CloudBinlogCumulativeCompactionPolicy::calc_binlog_compaction_score(
+uint32_t CloudChangelogCumulativeCompactionPolicy::calc_changelog_compaction_score(
CloudTablet* tablet, const std::vector<RowsetSharedPtr>& candidate_rowsets,
int8_t* compaction_level) const {
uint32_t max_score = 0;
int8_t max_level = -1;
- for (int8_t level = 0; level < BinlogCumulativeCompactionPolicy::kBinlogCompactionMaxLevel;
- ++level) {
- uint32_t score = calc_binlog_compaction_level_score(tablet, candidate_rowsets, level);
+ for (int8_t level = 0;
+ level < ChangelogCumulativeCompactionPolicy::kChangelogCompactionMaxLevel; ++level) {
+ uint32_t score = calc_changelog_compaction_level_score(tablet, candidate_rowsets, level);
if (score > max_score) {
max_score = score;
max_level = level;
@@ -110,18 +110,18 @@ uint32_t CloudBinlogCumulativeCompactionPolicy::calc_binlog_compaction_score(
return max_score;
}
-void CloudBinlogCumulativeCompactionPolicy::filter_new_visible_rowsets(
+void CloudChangelogCumulativeCompactionPolicy::filter_new_visible_rowsets(
const std::vector<RowsetSharedPtr>& candidate_rowsets,
std::vector<RowsetSharedPtr>* output_rowsets) const {
output_rowsets->clear();
int64_t now = UnixSeconds();
int64_t max_processable_old_version = 0;
bool filter_new_rowset =
- candidate_rowsets.size() <= config::binlog_compaction_file_count_threshold;
+ candidate_rowsets.size() <= config::changelog_compaction_file_count_threshold;
for (const auto& rs : candidate_rowsets) {
if (filter_new_rowset && rs->rowset_meta()->is_singleton_delta() &&
rs->rowset_meta()->newest_write_timestamp() +
- config::binlog_compaction_wait_timesec_after_visible >
+ config::changelog_compaction_wait_timesec_after_visible >
now) {
continue;
}
@@ -136,7 +136,7 @@ void CloudBinlogCumulativeCompactionPolicy::filter_new_visible_rowsets(
}
}
-int64_t CloudBinlogCumulativeCompactionPolicy::pick_input_rowsets(
+int64_t CloudChangelogCumulativeCompactionPolicy::pick_input_rowsets(
CloudTablet* tablet, const std::vector<RowsetSharedPtr>& candidate_rowsets,
const int64_t max_compaction_score, const int64_t min_compaction_score,
std::vector<RowsetSharedPtr>* input_rowsets, Version* last_delete_version,
@@ -145,7 +145,7 @@ int64_t CloudBinlogCumulativeCompactionPolicy::pick_input_rowsets(
filter_new_visible_rowsets(candidate_rowsets, &filtered_rowsets);
int8_t compaction_level = -1;
- calc_binlog_compaction_score(tablet, filtered_rowsets, &compaction_level);
+ calc_changelog_compaction_score(tablet, filtered_rowsets, &compaction_level);
if (compaction_level < 0) {
*compaction_score = 0;
return 0;
@@ -160,7 +160,7 @@ int64_t CloudBinlogCumulativeCompactionPolicy::pick_input_rowsets(
if (!level_rowsets.empty() &&
rs->start_version() != level_rowsets.back()->end_version() + 1) {
LOG(WARNING) << "rowset is non-continuous in the same compaction_level "
- "of binlog compaction. tablet="
+ "of changelog compaction. tablet="
<< tablet->tablet_id()
<< ", compaction_level=" << static_cast<int>(compaction_level)
<< ", prev_version=" << level_rowsets.back()->version()
@@ -175,7 +175,8 @@ int64_t CloudBinlogCumulativeCompactionPolicy::pick_input_rowsets(
remaining_rowsets.reserve(level_rowsets.size());
const int64_t point = tablet->cumulative_layer_point();
for (const auto& rs : level_rowsets) {
- if (compaction_level == BinlogCumulativeCompactionPolicy::kBinlogCompactionMaxLevel - 1 &&
+ if (compaction_level ==
+ ChangelogCumulativeCompactionPolicy::kChangelogCompactionMaxLevel - 1 &&
point != Tablet::K_INVALID_CUMULATIVE_POINT && rs->end_version() < point) {
continue;
}
@@ -197,15 +198,15 @@ int64_t CloudBinlogCumulativeCompactionPolicy::pick_input_rowsets(
picked_score += rs->rowset_meta()->get_compaction_score();
}
- bool can_do_binlog_compaction =
- total_size >= config::binlog_compaction_goal_size_mbytes * 1024 * 1024 ||
- picked_score >= config::binlog_compaction_file_count_threshold ||
+ bool can_do_changelog_compaction =
+ total_size >= config::changelog_compaction_goal_size_mbytes * 1024 * 1024 ||
+ picked_score >= config::changelog_compaction_file_count_threshold ||
(UnixMillis() - tablet->last_cumu_compaction_success_time()) / 1000 >=
- config::binlog_compaction_time_threshold_seconds;
+ config::changelog_compaction_time_threshold_seconds;
if (transient_size < 2) {
- can_do_binlog_compaction = false;
+ can_do_changelog_compaction = false;
}
- if (can_do_binlog_compaction) {
+ if (can_do_changelog_compaction) {
input_rowsets->swap(picked_rowsets);
*compaction_score = picked_score;
return transient_size;
diff --git a/be/src/cloud/cloud_cumulative_compaction_binlog_policy.h b/be/src/cloud/cloud_cumulative_compaction_changelog_policy.h
similarity index 79%
rename from be/src/cloud/cloud_cumulative_compaction_binlog_policy.h
rename to be/src/cloud/cloud_cumulative_compaction_changelog_policy.h
index 9d170c3d2e4..0b4246c08af 100644
--- a/be/src/cloud/cloud_cumulative_compaction_binlog_policy.h
+++ b/be/src/cloud/cloud_cumulative_compaction_changelog_policy.h
@@ -18,14 +18,14 @@
#pragma once
#include "cloud/cloud_cumulative_compaction_policy.h"
-#include "storage/compaction/cumulative_compaction_binlog_policy.h"
+#include "storage/compaction/cumulative_compaction_changelog_policy.h"
namespace doris {
-class CloudBinlogCumulativeCompactionPolicy final : public CloudCumulativeCompactionPolicy {
+class CloudChangelogCumulativeCompactionPolicy final : public CloudCumulativeCompactionPolicy {
public:
- CloudBinlogCumulativeCompactionPolicy() = default;
- ~CloudBinlogCumulativeCompactionPolicy() override = default;
+ CloudChangelogCumulativeCompactionPolicy() = default;
+ ~CloudChangelogCumulativeCompactionPolicy() override = default;
int64_t new_cumulative_point(CloudTablet* tablet, const RowsetSharedPtr& output_rowset,
Version& last_delete_version,
@@ -43,18 +43,18 @@ public:
Version* last_delete_version, size_t* compaction_score,
bool allow_delete = false) override;
- std::string name() override { return std::string(CUMULATIVE_BINLOG_POLICY); }
+ std::string name() override { return std::string(CUMULATIVE_CHANGELOG_POLICY); }
private:
bool is_compaction_enough(const RowsetMetaSharedPtr& rowset_meta) const;
void filter_new_visible_rowsets(const std::vector<RowsetSharedPtr>& candidate_rowsets,
std::vector<RowsetSharedPtr>* output_rowsets) const;
- uint32_t calc_binlog_compaction_level_score(
+ uint32_t calc_changelog_compaction_level_score(
CloudTablet* tablet, const std::vector<RowsetSharedPtr>& candidate_rowsets,
int8_t level) const;
- uint32_t calc_binlog_compaction_score(CloudTablet* tablet,
- const std::vector<RowsetSharedPtr>& candidate_rowsets,
- int8_t* compaction_level) const;
+ uint32_t calc_changelog_compaction_score(CloudTablet* tablet,
+ const std::vector<RowsetSharedPtr>& candidate_rowsets,
+ int8_t* compaction_level) const;
};
} // namespace doris
diff --git a/be/src/cloud/cloud_delta_writer.cpp b/be/src/cloud/cloud_delta_writer.cpp
index 83553e176cd..ad8907e8bd6 100644
--- a/be/src/cloud/cloud_delta_writer.cpp
+++ b/be/src/cloud/cloud_delta_writer.cpp
@@ -42,14 +42,14 @@ CloudDeltaWriter::CloudDeltaWriter(CloudStorageEngine& engine, const WriteReques
CloudDeltaWriter::CloudDeltaWriter(CloudStorageEngine& engine, const WriteRequest& group_build_req,
const WriteRequest& sub_data_req,
- const WriteRequest& sub_row_binlog_req, RuntimeProfile* profile,
+ const WriteRequest& sub_changelog_req, RuntimeProfile* profile,
const UniqueId& load_id)
: BaseDeltaWriter(group_build_req, profile, load_id) {
DCHECK(group_build_req.write_req_type == WriteRequestType::GROUP &&
sub_data_req.write_req_type == WriteRequestType::DATA &&
- sub_row_binlog_req.write_req_type == WriteRequestType::ROW_BINLOG);
+ sub_changelog_req.write_req_type == WriteRequestType::CHANGELOG);
_rowset_builder = std::make_unique<CloudGroupRowsetBuilder>(
- engine, group_build_req, sub_data_req, sub_row_binlog_req, profile);
+ engine, group_build_req, sub_data_req, sub_changelog_req, profile);
_resource_ctx = thread_context()->resource_ctx();
}
diff --git a/be/src/cloud/cloud_delta_writer.h b/be/src/cloud/cloud_delta_writer.h
index cb2ec081f59..8d7b762ddb8 100644
--- a/be/src/cloud/cloud_delta_writer.h
+++ b/be/src/cloud/cloud_delta_writer.h
@@ -32,7 +32,7 @@ public:
CloudDeltaWriter(CloudStorageEngine& engine, const WriteRequest& req, RuntimeProfile* profile,
const UniqueId& load_id);
CloudDeltaWriter(CloudStorageEngine& engine, const WriteRequest& group_build_req,
- const WriteRequest& sub_data_req, const WriteRequest& sub_row_binlog_req,
+ const WriteRequest& sub_data_req, const WriteRequest& sub_changelog_req,
RuntimeProfile* profile, const UniqueId& load_id);
~CloudDeltaWriter() override;
diff --git a/be/src/cloud/cloud_engine_calc_delete_bitmap_task.cpp b/be/src/cloud/cloud_engine_calc_delete_bitmap_task.cpp
index ad49fd2dc7d..cd22c86800f 100644
--- a/be/src/cloud/cloud_engine_calc_delete_bitmap_task.cpp
+++ b/be/src/cloud/cloud_engine_calc_delete_bitmap_task.cpp
@@ -158,8 +158,8 @@ Status CloudTabletCalcDeleteBitmapTask::handle(int64_t queue_time_us) const {
return Status::Error<ErrorCode::PUSH_TABLE_NOT_EXIST>(
"can't get tablet when calculate delete bitmap. tablet_id={}", _tablet_id);
}
- if (tablet->is_row_binlog_tablet()) {
- VLOG_DEBUG << "skip calculating delete bitmap for row binlog tablet, tablet_id="
+ if (tablet->is_changelog_tablet()) {
+ VLOG_DEBUG << "skip calculating delete bitmap for changelog tablet, tablet_id="
<< _tablet_id << ", txn_id=" << _transaction_id
<< ", it will be handled with its base tablet";
return Status::OK();
@@ -341,10 +341,10 @@ Status CloudTabletCalcDeleteBitmapTask::_handle_rowset(
std::shared_ptr<PublishStatus> publish_status;
int64_t txn_expiration;
TxnPublishInfo previous_publish_info;
- RowBinlogTxnInfo attach_row_binlog;
+ ChangelogTxnInfo attach_changelog;
Status status = _engine.txn_delete_bitmap_cache().get_tablet_txn_info(
transaction_id, _tablet_id, &rowset, &delete_bitmap, &rowset_ids, &txn_expiration,
- &partial_update_info, &publish_status, &previous_publish_info, &attach_row_binlog);
+ &partial_update_info, &publish_status, &previous_publish_info, &attach_changelog);
if (status != Status::OK()) {
LOG(WARNING) << "failed to get tablet txn info. tablet_id=" << _tablet_id << ", " << txn_str
<< ", status=" << status;
@@ -363,7 +363,7 @@ Status CloudTabletCalcDeleteBitmapTask::_handle_rowset(
TabletTxnInfo txn_info;
txn_info.rowset = rowset;
txn_info.delete_bitmap = delete_bitmap;
- txn_info.attach_row_binlog = attach_row_binlog;
+ txn_info.attach_changelog = attach_changelog;
txn_info.rowset_ids = rowset_ids;
txn_info.partial_update_info = partial_update_info;
txn_info.publish_status = publish_status;
diff --git a/be/src/cloud/cloud_internal_service.cpp b/be/src/cloud/cloud_internal_service.cpp
index 24274a49451..f20a3d33821 100644
--- a/be/src/cloud/cloud_internal_service.cpp
+++ b/be/src/cloud/cloud_internal_service.cpp
@@ -125,12 +125,12 @@ void submit_sync_tablet_meta(CloudStorageEngine& engine, FifoThreadPool& work_po
int64_t failed = 0;
g_cloud_sync_tablet_meta_requests_total << 1;
for (const auto tablet_id : request->tablet_ids()) {
- if (request->discover_row_binlog_ttl()) {
- if (config::disable_auto_compaction || !config::enable_feature_binlog) {
+ if (request->discover_changelog_ttl()) {
+ if (config::disable_auto_compaction || !config::enable_feature_changelog) {
++skipped;
continue;
}
- auto st = engine->submit_row_binlog_ttl(tablet_id, true);
+ auto st = engine->submit_changelog_ttl(tablet_id, true);
if (st.ok()) {
++synced;
} else {
diff --git a/be/src/cloud/cloud_meta_mgr.cpp b/be/src/cloud/cloud_meta_mgr.cpp
index 295c3163b4f..41f856342a8 100644
--- a/be/src/cloud/cloud_meta_mgr.cpp
+++ b/be/src/cloud/cloud_meta_mgr.cpp
@@ -893,7 +893,7 @@ Status CloudMetaMgr::sync_tablet_rowsets_unlocked(CloudTablet* tablet,
sync_stats->get_remote_rowsets_num += resp.rowset_meta().size();
}
- // MOW and row-binlog tablets need delete bitmap from meta-service.
+ // MOW and changelog tablets need delete bitmap from meta-service.
if (options.sync_delete_bitmap && tablet->need_read_delete_bitmap() &&
tablet->tablet_state() == TABLET_RUNNING) {
DBUG_EXECUTE_IF("CloudMetaMgr::sync_tablet_rowsets.sync_tablet_delete_bitmap.block",
@@ -1628,18 +1628,18 @@ Status CloudMetaMgr::do_commit_rowset(RowsetMeta& rs_meta, const std::string& jo
Status CloudMetaMgr::commit_rowset(RowsetMeta& rs_meta, const std::string& job_id, int64_t table_id,
RowsetMetaSharedPtr* existed_rs_meta,
- RowsetMeta* attach_row_binlog,
- RowsetMetaSharedPtr* existed_attach_row_binlog) {
- if (attach_row_binlog == nullptr) {
+ RowsetMeta* attach_changelog,
+ RowsetMetaSharedPtr* existed_attach_changelog) {
+ if (attach_changelog == nullptr) {
return do_commit_rowset(rs_meta, job_id, table_id, existed_rs_meta);
}
- VLOG_DEBUG << "commit rowset with row binlog, tablet_id: " << rs_meta.tablet_id()
+ VLOG_DEBUG << "commit rowset with changelog, tablet_id: " << rs_meta.tablet_id()
<< ", rowset_id: " << rs_meta.rowset_id()
- << ", attach_row_binlog_tablet_id: " << attach_row_binlog->tablet_id()
- << ", attach_row_binlog_rowset_id: " << attach_row_binlog->rowset_id()
+ << ", attach_changelog_tablet_id: " << attach_changelog->tablet_id()
+ << ", attach_changelog_rowset_id: " << attach_changelog->rowset_id()
<< " txn_id: " << rs_meta.txn_id();
- Status st = do_commit_rowset(*attach_row_binlog, job_id, table_id, existed_attach_row_binlog);
+ Status st = do_commit_rowset(*attach_changelog, job_id, table_id, existed_attach_changelog);
if (!st.ok() && !st.is<ALREADY_EXIST>()) {
return st;
}
@@ -1685,18 +1685,18 @@ Status CloudMetaMgr::do_update_tmp_rowset(const RowsetMeta& rs_meta, int64_t tab
}
Status CloudMetaMgr::update_tmp_rowset(const RowsetMeta& rs_meta, int64_t table_id,
- const RowsetMeta* attach_row_binlog) {
- if (attach_row_binlog == nullptr) {
+ const RowsetMeta* attach_changelog) {
+ if (attach_changelog == nullptr) {
return do_update_tmp_rowset(rs_meta, table_id);
}
- VLOG_DEBUG << "update committed rowset with row binlog, tablet_id: " << rs_meta.tablet_id()
+ VLOG_DEBUG << "update committed rowset with changelog, tablet_id: " << rs_meta.tablet_id()
<< ", rowset_id: " << rs_meta.rowset_id()
- << ", attach_row_binlog_tablet_id: " << attach_row_binlog->tablet_id()
- << ", attach_row_binlog_rowset_id: " << attach_row_binlog->rowset_id();
+ << ", attach_changelog_tablet_id: " << attach_changelog->tablet_id()
+ << ", attach_changelog_rowset_id: " << attach_changelog->rowset_id();
DCHECK_EQ(rs_meta.tablet_schema()->num_variant_columns(),
- attach_row_binlog->tablet_schema()->num_variant_columns());
- RETURN_IF_ERROR(do_update_tmp_rowset(*attach_row_binlog, table_id));
+ attach_changelog->tablet_schema()->num_variant_columns());
+ RETURN_IF_ERROR(do_update_tmp_rowset(*attach_changelog, table_id));
return do_update_tmp_rowset(rs_meta, table_id);
}
diff --git a/be/src/cloud/cloud_meta_mgr.h b/be/src/cloud/cloud_meta_mgr.h
index d53f271aff5..c695646f473 100644
--- a/be/src/cloud/cloud_meta_mgr.h
+++ b/be/src/cloud/cloud_meta_mgr.h
@@ -90,12 +90,12 @@ public:
Status commit_rowset(RowsetMeta& rs_meta, const std::string& job_id, int64_t table_id,
std::shared_ptr<RowsetMeta>* existed_rs_meta = nullptr,
- RowsetMeta* attach_row_binlog = nullptr,
- std::shared_ptr<RowsetMeta>* existed_attach_row_binlog = nullptr);
+ RowsetMeta* attach_changelog = nullptr,
+ std::shared_ptr<RowsetMeta>* existed_attach_changelog = nullptr);
void cache_committed_rowset(RowsetMetaSharedPtr rs_meta, int64_t expiration_time);
Status update_tmp_rowset(const RowsetMeta& rs_meta, int64_t table_id,
- const RowsetMeta* attach_row_binlog = nullptr);
+ const RowsetMeta* attach_changelog = nullptr);
Status update_packed_file_info(const std::string& packed_file_path,
const cloud::PackedFileInfoPB& packed_file_info,
diff --git a/be/src/cloud/cloud_rowset_builder.cpp b/be/src/cloud/cloud_rowset_builder.cpp
index 5660fb6bbb3..d8b5971252e 100644
--- a/be/src/cloud/cloud_rowset_builder.cpp
+++ b/be/src/cloud/cloud_rowset_builder.cpp
@@ -41,14 +41,14 @@ CloudRowsetBuilder::CloudRowsetBuilder(CloudStorageEngine& engine, const WriteRe
CloudGroupRowsetBuilder::CloudGroupRowsetBuilder(CloudStorageEngine& engine,
const WriteRequest& group_build_req,
const WriteRequest& sub_data_req,
- const WriteRequest& sub_row_binlog_req,
+ const WriteRequest& sub_changelog_req,
RuntimeProfile* profile)
: CloudRowsetBuilder(engine, group_build_req, profile) {
DCHECK(group_build_req.write_req_type == WriteRequestType::GROUP &&
sub_data_req.write_req_type == WriteRequestType::DATA &&
- sub_row_binlog_req.write_req_type == WriteRequestType::ROW_BINLOG);
+ sub_changelog_req.write_req_type == WriteRequestType::CHANGELOG);
_data_builder = std::make_shared<CloudRowsetBuilder>(engine, sub_data_req, profile);
- _row_binlog_builder = std::make_shared<CloudRowsetBuilder>(engine, sub_row_binlog_req, profile);
+ _changelog_builder = std::make_shared<CloudRowsetBuilder>(engine, sub_changelog_req, profile);
}
CloudRowsetBuilder::~CloudRowsetBuilder() {
@@ -73,9 +73,9 @@ Status CloudRowsetBuilder::init() {
}
}
RETURN_IF_ERROR(init_mow_context(mow_context));
- } else if (_req.write_req_type == WriteRequestType::ROW_BINLOG) {
- // Row binlog tablets use txn_delete_bitmap_cache for local make-visible.
- // The real binlog delete bitmap is derived when the base tablet calculates delete bitmap.
+ } else if (_req.write_req_type == WriteRequestType::CHANGELOG) {
+ // Changelog tablets use txn_delete_bitmap_cache for local make-visible. The real changelog
+ // delete bitmap is derived when the base tablet calculates delete bitmap.
_delete_bitmap = std::make_shared<DeleteBitmap>(_req.tablet_id);
}
RETURN_IF_ERROR(check_tablet_version_count());
@@ -103,14 +103,14 @@ Status CloudRowsetBuilder::init() {
context.tablet_schema_hash = _req.schema_hash;
context.index_id = _req.index_id;
context.tablet = _tablet;
- if (_req.write_req_type == WriteRequestType::ROW_BINLOG || !_attach_rowset_ids.empty()) {
+ if (_req.write_req_type == WriteRequestType::CHANGELOG || !_attach_rowset_ids.empty()) {
context.enable_segcompaction = false;
}
context.write_type = DataWriteType::TYPE_DIRECT;
context.mow_context = mow_context;
context.write_file_cache = _req.write_file_cache;
context.partial_update_info = _partial_update_info;
- context.write_binlog_opt().enable = _req.write_req_type == WriteRequestType::ROW_BINLOG;
+ context.write_changelog_opt().enable = _req.write_req_type == WriteRequestType::CHANGELOG;
context.file_cache_expiration_time = _tablet->file_cache_ttl_expiration_time();
context.storage_resource = _engine.get_storage_resource(_req.storage_vault_id);
if (!context.storage_resource) {
@@ -134,23 +134,22 @@ Status CloudRowsetBuilder::init() {
}
Status CloudGroupRowsetBuilder::init() {
- RETURN_IF_ERROR(_row_binlog_builder->init());
- RETURN_IF_ERROR(
- _data_builder->attach_pending_rs_guard_to_txn(_row_binlog_builder->rowset_id()));
+ RETURN_IF_ERROR(_changelog_builder->init());
+ RETURN_IF_ERROR(_data_builder->attach_pending_rs_guard_to_txn(_changelog_builder->rowset_id()));
RETURN_IF_ERROR(_data_builder->init());
_tablet = _data_builder->tablet_sptr();
std::unique_ptr<GroupRowsetWriter> group_writer;
RETURN_IF_ERROR(RowsetFactory::create_empty_group_rowset_writer(&group_writer));
group_writer->set_data_writer(_data_builder->rowset_writer());
- group_writer->set_row_binlog_writer(_row_binlog_builder->rowset_writer());
+ group_writer->set_changelog_writer(_changelog_builder->rowset_writer());
RETURN_IF_ERROR(group_writer->init(_data_builder->rowset_writer()->context()));
{
const auto& data_ctx = _data_builder->rowset_writer()->context();
- auto& binlog_ctx =
- const_cast<RowsetWriterContext&>(_row_binlog_builder->rowset_writer()->context());
- auto& cfg = binlog_ctx.write_binlog_opt().write_binlog_config();
+ auto& changelog_ctx =
+ const_cast<RowsetWriterContext&>(_changelog_builder->rowset_writer()->context());
+ auto& cfg = changelog_ctx.write_changelog_opt().write_changelog_config();
cfg.source.tablet_schema = data_ctx.tablet_schema;
cfg.source.partial_update_info = data_ctx.partial_update_info;
cfg.source.mow_context = data_ctx.mow_context;
@@ -165,7 +164,7 @@ Status CloudGroupRowsetBuilder::init() {
}
Status CloudGroupRowsetBuilder::build_rowset() {
- RETURN_IF_ERROR(_row_binlog_builder->build_rowset());
+ RETURN_IF_ERROR(_changelog_builder->build_rowset());
return _data_builder->build_rowset();
}
@@ -179,36 +178,36 @@ Status CloudGroupRowsetBuilder::wait_calc_delete_bitmap() {
Status CloudGroupRowsetBuilder::cancel(const Status& st) {
RETURN_IF_ERROR(_data_builder->cancel(st));
- RETURN_IF_ERROR(_row_binlog_builder->cancel(st));
+ RETURN_IF_ERROR(_changelog_builder->cancel(st));
return BaseRowsetBuilder::cancel(st);
}
void CloudGroupRowsetBuilder::update_tablet_stats() {
_data_builder->update_tablet_stats();
- _row_binlog_builder->update_tablet_stats();
+ _changelog_builder->update_tablet_stats();
}
Status CloudGroupRowsetBuilder::commit_rowset(const std::string& job_id, int64_t table_id) {
return _engine.meta_mgr().commit_rowset(*_data_builder->rowset_meta(), job_id, table_id,
- nullptr, _row_binlog_builder->rowset_meta().get());
+ nullptr, _changelog_builder->rowset_meta().get());
}
Status CloudGroupRowsetBuilder::set_txn_related_info() {
- RowBinlogTxnInfo attach_row_binlog;
- attach_row_binlog.rowset = _row_binlog_builder->rowset();
- attach_row_binlog.tablet = _row_binlog_builder->tablet_sptr();
+ ChangelogTxnInfo attach_changelog;
+ attach_changelog.rowset = _changelog_builder->rowset();
+ attach_changelog.tablet = _changelog_builder->tablet_sptr();
if (_data_builder->tablet()->enable_unique_key_merge_on_write()) {
- attach_row_binlog.delete_bitmap =
- std::make_shared<DeleteBitmap>(_row_binlog_builder->tablet()->tablet_id());
+ attach_changelog.delete_bitmap =
+ std::make_shared<DeleteBitmap>(_changelog_builder->tablet()->tablet_id());
}
- RETURN_IF_ERROR(_data_builder->attach_row_binlog_to_txn(attach_row_binlog));
+ RETURN_IF_ERROR(_data_builder->attach_changelog_to_txn(attach_changelog));
RETURN_IF_ERROR(_data_builder->set_txn_related_info());
- return _row_binlog_builder->set_txn_related_info();
+ return _changelog_builder->set_txn_related_info();
}
void CloudGroupRowsetBuilder::set_skip_writing_rowset_metadata(bool skip) {
_data_builder->set_skip_writing_rowset_metadata(skip);
- _row_binlog_builder->set_skip_writing_rowset_metadata(skip);
+ _changelog_builder->set_skip_writing_rowset_metadata(skip);
}
Status CloudRowsetBuilder::check_tablet_version_count() {
@@ -261,7 +260,7 @@ Status CloudRowsetBuilder::commit_rowset(const std::string& job_id, int64_t tabl
}
Status CloudRowsetBuilder::set_txn_related_info() {
- if (_tablet->enable_unique_key_merge_on_write() || _tablet->is_row_binlog_tablet()) {
+ if (_tablet->enable_unique_key_merge_on_write() || _tablet->is_changelog_tablet()) {
// For empty rowsets when skip_writing_empty_rowset_metadata=true,
// store only a lightweight marker instead of full rowset info.
// This allows CalcDeleteBitmapTask to detect and skip gracefully,
@@ -286,7 +285,7 @@ Status CloudRowsetBuilder::set_txn_related_info() {
}
_engine.txn_delete_bitmap_cache().set_tablet_txn_info(
_req.txn_id, _tablet->tablet_id(), _delete_bitmap, *_rowset_ids, _rowset,
- _req.txn_expiration, _partial_update_info, _attach_row_binlog);
+ _req.txn_expiration, _partial_update_info, _attach_changelog);
} else {
// TSO-enabled rowsets must become visible from MS rowset meta.
if (config::enable_cloud_make_rs_visible_on_be && !_tablet_schema->is_tso_enabled()) {
diff --git a/be/src/cloud/cloud_rowset_builder.h b/be/src/cloud/cloud_rowset_builder.h
index 8d70a36fd64..4b2b89d83c8 100644
--- a/be/src/cloud/cloud_rowset_builder.h
+++ b/be/src/cloud/cloud_rowset_builder.h
@@ -63,8 +63,8 @@ protected:
class CloudGroupRowsetBuilder final : public CloudRowsetBuilder {
public:
CloudGroupRowsetBuilder(CloudStorageEngine& engine, const WriteRequest& group_build_req,
- const WriteRequest& sub_data_req,
- const WriteRequest& sub_row_binlog_req, RuntimeProfile* profile);
+ const WriteRequest& sub_data_req, const WriteRequest& sub_changelog_req,
+ RuntimeProfile* profile);
Status init() override;
@@ -100,11 +100,11 @@ public:
CloudRowsetBuilder* data_builder() { return _data_builder.get(); }
- CloudRowsetBuilder* row_binlog_builder() { return _row_binlog_builder.get(); }
+ CloudRowsetBuilder* changelog_builder() { return _changelog_builder.get(); }
private:
std::shared_ptr<CloudRowsetBuilder> _data_builder;
- std::shared_ptr<CloudRowsetBuilder> _row_binlog_builder;
+ std::shared_ptr<CloudRowsetBuilder> _changelog_builder;
};
} // namespace doris
diff --git a/be/src/cloud/cloud_rowset_writer.cpp b/be/src/cloud/cloud_rowset_writer.cpp
index 03fc3ca6682..af447147b36 100644
--- a/be/src/cloud/cloud_rowset_writer.cpp
+++ b/be/src/cloud/cloud_rowset_writer.cpp
@@ -40,9 +40,9 @@ CloudRowsetWriter::~CloudRowsetWriter() {
Status CloudRowsetWriter::init(const RowsetWriterContext& rowset_writer_context) {
_context = rowset_writer_context;
- // Row-binlog writer or a schema carrying ROW_LSN_COL needs allocated LSN.
+ // Changelog writer or a schema carrying ROW_LSN_COL needs allocated LSN.
_context._need_allocate_lsn =
- _context.write_binlog_opt().enable ||
+ _context.write_changelog_opt().enable ||
(_context.tablet_schema != nullptr && _context.tablet_schema->row_lsn_col_idx() >= 0);
_rowset_meta = std::make_shared<RowsetMeta>();
@@ -82,8 +82,8 @@ Status CloudRowsetWriter::init(const RowsetWriterContext& rowset_writer_context)
_rowset_meta->set_inverted_index_storage_format(*_context.inverted_index_storage_format);
}
_rowset_meta->set_job_id(_context.job_id);
- if (_context.write_binlog_opt().enable) {
- _rowset_meta->mark_row_binlog();
+ if (_context.write_changelog_opt().enable) {
+ _rowset_meta->mark_changelog();
}
_context.segment_collector = std::make_shared<SegmentCollectorT<BaseBetaRowsetWriter>>(this);
_context.file_writer_creator = std::make_shared<FileWriterCreatorT<BaseBetaRowsetWriter>>(this);
diff --git a/be/src/cloud/cloud_schema_change_job.cpp b/be/src/cloud/cloud_schema_change_job.cpp
index a253cf5fdfa..a926d60b40c 100644
--- a/be/src/cloud/cloud_schema_change_job.cpp
+++ b/be/src/cloud/cloud_schema_change_job.cpp
@@ -246,7 +246,7 @@ Status CloudSchemaChangeJob::process_alter_tablet(const TAlterTabletReqV2& reque
ReadSchemaSPtr read_schema = std::make_shared<ReadSchema>(_base_tablet_schema->columns());
RETURN_IF_ERROR(read_schema->init_from_tablet_schema(*_base_tablet_schema,
/*merge_by_sequence_mapping=*/false,
- /*map_row_binlog_columns=*/false));
+ /*map_changelog_columns=*/false));
// delete handlers to filter out deleted rows
DeleteHandler delete_handler;
diff --git a/be/src/cloud/cloud_storage_engine.cpp b/be/src/cloud/cloud_storage_engine.cpp
index 8275b6bc3ff..95893da18be 100644
--- a/be/src/cloud/cloud_storage_engine.cpp
+++ b/be/src/cloud/cloud_storage_engine.cpp
@@ -33,7 +33,7 @@
#include "cloud/cloud_base_compaction.h"
#include "cloud/cloud_compaction_stop_token.h"
#include "cloud/cloud_cumulative_compaction.h"
-#include "cloud/cloud_cumulative_compaction_binlog_policy.h"
+#include "cloud/cloud_cumulative_compaction_changelog_policy.h"
#include "cloud/cloud_cumulative_compaction_policy.h"
#include "cloud/cloud_full_compaction.h"
#include "cloud/cloud_index_change_compaction.h"
@@ -61,7 +61,7 @@
#include "runtime/exec_env.h"
#include "runtime/memory/cache_manager.h"
#include "service/backend_options.h"
-#include "storage/compaction/cumulative_compaction_binlog_policy.h"
+#include "storage/compaction/cumulative_compaction_changelog_policy.h"
#include "storage/compaction/cumulative_compaction_policy.h"
#include "storage/compaction/cumulative_compaction_time_series_policy.h"
#include "storage/compaction_task_tracker.h"
@@ -96,13 +96,14 @@ int get_base_thread_num() {
return std::min(std::max(int(num_cores * config::base_compaction_thread_num_factor), 1), 10);
}
-int get_binlog_thread_num() {
- if (config::max_binlog_compaction_threads > 0) {
- return config::max_binlog_compaction_threads;
+int get_changelog_thread_num() {
+ if (config::max_changelog_compaction_threads > 0) {
+ return config::max_changelog_compaction_threads;
}
int num_cores = doris::CpuInfo::num_cores();
- return std::min(std::max(int(num_cores * config::binlog_compaction_thread_num_factor), 1), 10);
+ return std::min(std::max(int(num_cores * config::changelog_compaction_thread_num_factor), 1),
+ 10);
}
CloudStorageEngine::CloudStorageEngine(const EngineOptions& options)
@@ -114,8 +115,8 @@ CloudStorageEngine::CloudStorageEngine(const EngineOptions& options)
std::make_shared<CloudSizeBasedCumulativeCompactionPolicy>();
_cumulative_compaction_policies[CUMULATIVE_TIME_SERIES_POLICY] =
std::make_shared<CloudTimeSeriesCumulativeCompactionPolicy>();
- _cumulative_compaction_policies[CUMULATIVE_BINLOG_POLICY] =
- std::make_shared<CloudBinlogCumulativeCompactionPolicy>();
+ _cumulative_compaction_policies[CUMULATIVE_CHANGELOG_POLICY] =
+ std::make_shared<CloudChangelogCumulativeCompactionPolicy>();
_startup_timepoint = std::chrono::system_clock::now();
}
@@ -298,7 +299,7 @@ void CloudStorageEngine::stop() {
_stopped = true;
_stop_background_threads_latch.count_down();
- _stop_row_binlog_ttl_scanner();
+ _stop_changelog_ttl_scanner();
for (auto&& t : _bg_threads) {
if (t) {
@@ -312,8 +313,8 @@ void CloudStorageEngine::stop() {
if (_cumu_compaction_thread_pool) {
_cumu_compaction_thread_pool->shutdown();
}
- if (_binlog_compaction_thread_pool) {
- _binlog_compaction_thread_pool->shutdown();
+ if (_changelog_compaction_thread_pool) {
+ _changelog_compaction_thread_pool->shutdown();
}
_adaptive_thread_controller.stop();
LOG(INFO) << "Cloud storage engine is stopped.";
@@ -407,7 +408,7 @@ Status CloudStorageEngine::start_bg_threads(std::shared_ptr<WorkloadGroup> wg_sp
// compaction tasks producer thread
int base_thread_num = get_base_thread_num();
int cumu_thread_num = get_cumu_thread_num();
- int binlog_thread_num = get_binlog_thread_num();
+ int changelog_thread_num = get_changelog_thread_num();
RETURN_IF_ERROR(ThreadPoolBuilder("BaseCompactionTaskThreadPool")
.set_min_threads(base_thread_num)
@@ -417,23 +418,25 @@ Status CloudStorageEngine::start_bg_threads(std::shared_ptr<WorkloadGroup> wg_sp
.set_min_threads(cumu_thread_num)
.set_max_threads(cumu_thread_num)
.build(&_cumu_compaction_thread_pool));
- RETURN_IF_ERROR(ThreadPoolBuilder("BinlogCompactionTaskThreadPool")
- .set_min_threads(binlog_thread_num)
- .set_max_threads(binlog_thread_num)
- .set_max_queue_size(binlog_thread_num)
- .build(&_binlog_compaction_thread_pool));
+ // ATTN(changelog-rename): formerly "BinlogCompactionTaskThreadPool"
+ RETURN_IF_ERROR(ThreadPoolBuilder("ChangelogCompactionTaskThreadPool")
+ .set_min_threads(changelog_thread_num)
+ .set_max_threads(changelog_thread_num)
+ .set_max_queue_size(changelog_thread_num)
+ .build(&_changelog_compaction_thread_pool));
RETURN_IF_ERROR(Thread::create(
"StorageEngine", "compaction_tasks_producer_thread",
[this]() { this->_compaction_tasks_producer_callback(); },
&_bg_threads.emplace_back()));
+ // ATTN(changelog-rename): formerly "binlog_compaction_tasks_producer_thread"
RETURN_IF_ERROR(Thread::create(
- "StorageEngine", "binlog_compaction_tasks_producer_thread",
- [this]() { this->_binlog_compaction_tasks_producer_callback(); },
+ "StorageEngine", "changelog_compaction_tasks_producer_thread",
+ [this]() { this->_changelog_compaction_tasks_producer_callback(); },
&_bg_threads.emplace_back()));
- RETURN_IF_ERROR(_start_row_binlog_ttl_scanner());
+ RETURN_IF_ERROR(_start_changelog_ttl_scanner());
LOG(INFO) << "compaction tasks producer thread started,"
<< " base thread num " << base_thread_num << " cumu thread num " << cumu_thread_num
- << " binlog thread num " << binlog_thread_num;
+ << " changelog thread num " << changelog_thread_num;
RETURN_IF_ERROR(Thread::create(
"StorageEngine", "lease_compaction_thread",
@@ -533,7 +536,7 @@ Status CloudStorageEngine::_adjust_compaction_thread_num() {
int base_thread_num = get_base_thread_num();
if (!_base_compaction_thread_pool || !_cumu_compaction_thread_pool ||
- !_binlog_compaction_thread_pool) {
+ !_changelog_compaction_thread_pool) {
LOG(WARNING) << "compaction thread pool is not created";
return Status::Error<ErrorCode::INTERNAL_ERROR, false>("");
}
@@ -573,21 +576,21 @@ Status CloudStorageEngine::_adjust_compaction_thread_num() {
}
}
- int binlog_thread_num = get_binlog_thread_num();
- if (_binlog_compaction_thread_pool->max_threads() != binlog_thread_num) {
- int old_max_threads = _binlog_compaction_thread_pool->max_threads();
- Status status = _binlog_compaction_thread_pool->set_max_threads(binlog_thread_num);
+ int changelog_thread_num = get_changelog_thread_num();
+ if (_changelog_compaction_thread_pool->max_threads() != changelog_thread_num) {
+ int old_max_threads = _changelog_compaction_thread_pool->max_threads();
+ Status status = _changelog_compaction_thread_pool->set_max_threads(changelog_thread_num);
if (status.ok()) {
- VLOG_NOTICE << "update binlog compaction thread pool max_threads from "
- << old_max_threads << " to " << binlog_thread_num;
+ VLOG_NOTICE << "update changelog compaction thread pool max_threads from "
+ << old_max_threads << " to " << changelog_thread_num;
}
}
- if (_binlog_compaction_thread_pool->min_threads() != binlog_thread_num) {
- int old_min_threads = _binlog_compaction_thread_pool->min_threads();
- Status status = _binlog_compaction_thread_pool->set_min_threads(binlog_thread_num);
+ if (_changelog_compaction_thread_pool->min_threads() != changelog_thread_num) {
+ int old_min_threads = _changelog_compaction_thread_pool->min_threads();
+ Status status = _changelog_compaction_thread_pool->set_min_threads(changelog_thread_num);
if (status.ok()) {
- VLOG_NOTICE << "update binlog compaction thread pool min_threads from "
- << old_min_threads << " to " << binlog_thread_num;
+ VLOG_NOTICE << "update changelog compaction thread pool min_threads from "
+ << old_min_threads << " to " << changelog_thread_num;
}
}
return Status::OK();
@@ -675,10 +678,10 @@ void CloudStorageEngine::_compaction_tasks_producer_callback() {
} while (!_stop_background_threads_latch.wait_for(std::chrono::milliseconds(interval)));
}
-void CloudStorageEngine::_binlog_compaction_tasks_producer_callback() {
- LOG(INFO) << "try to start binlog compaction producer process!";
+void CloudStorageEngine::_changelog_compaction_tasks_producer_callback() {
+ LOG(INFO) << "try to start changelog compaction producer process!";
- int64_t last_binlog_score_update_time = 0;
+ int64_t last_changelog_score_update_time = 0;
static const int64_t check_score_interval_ms = 5000;
int64_t interval = config::generate_compaction_tasks_interval_ms;
@@ -691,21 +694,21 @@ void CloudStorageEngine::_binlog_compaction_tasks_producer_callback() {
}
bool check_score = false;
- if (cur_time - last_binlog_score_update_time >= check_score_interval_ms) {
+ if (cur_time - last_changelog_score_update_time >= check_score_interval_ms) {
check_score = true;
- last_binlog_score_update_time = cur_time;
+ last_changelog_score_update_time = cur_time;
}
std::vector<CloudTabletSPtr> tablets_compaction = _generate_cloud_compaction_tasks(
- CompactionType::CUMU_BINLOG_COMPACTION, check_score);
+ CompactionType::CHANGELOG_COMPACTION, check_score);
for (const auto& tablet : tablets_compaction) {
Status status =
- submit_compaction_task(tablet, CompactionType::CUMU_BINLOG_COMPACTION);
+ submit_compaction_task(tablet, CompactionType::CHANGELOG_COMPACTION);
if (status.ok()) continue;
if ((!status.is<ErrorCode::BE_NO_SUITABLE_VERSION>() &&
!status.is<ErrorCode::CUMULATIVE_NO_SUITABLE_VERSION>()) ||
VLOG_DEBUG_IS_ON) {
- LOG(WARNING) << "failed to submit binlog compaction task for tablet: "
+ LOG(WARNING) << "failed to submit changelog compaction task for tablet: "
<< tablet->tablet_id() << ", err: " << status;
}
}
@@ -765,7 +768,7 @@ std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_task
CompactionType compaction_type, bool check_score) {
DCHECK(compaction_type == CompactionType::BASE_COMPACTION ||
compaction_type == CompactionType::CUMULATIVE_COMPACTION ||
- compaction_type == CompactionType::CUMU_BINLOG_COMPACTION);
+ compaction_type == CompactionType::CHANGELOG_COMPACTION);
std::vector<std::shared_ptr<CloudTablet>> tablets_compaction;
CompactionScoreStats score_stats;
@@ -773,7 +776,7 @@ std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_task
std::unordered_set<int64_t> tablet_preparing_cumu_compaction;
std::unordered_map<int64_t, std::vector<std::shared_ptr<CloudCumulativeCompaction>>>
submitted_cumu_compactions;
- int submitted_cumu_binlog_compaction_count = 0;
+ int submitted_cumu_changelog_compaction_count = 0;
std::unordered_map<int64_t, std::shared_ptr<CloudBaseCompaction>> submitted_base_compactions;
std::unordered_map<int64_t, std::shared_ptr<CloudFullCompaction>> submitted_full_compactions;
std::unordered_map<int64_t, std::shared_ptr<CloudIndexChangeCompaction>>
@@ -784,7 +787,7 @@ std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_task
std::lock_guard lock(_compaction_mtx);
tablet_preparing_cumu_compaction = _tablet_preparing_cumu_compaction;
submitted_cumu_compactions = _submitted_cumu_compactions;
- submitted_cumu_binlog_compaction_count = _submitted_cumu_binlog_compaction_count;
+ submitted_cumu_changelog_compaction_count = _submitted_cumu_changelog_compaction_count;
submitted_base_compactions = _submitted_base_compactions;
submitted_full_compactions = _submitted_full_compactions;
submitted_index_change_cumu_compactions = _submitted_index_change_cumu_compaction;
@@ -792,18 +795,18 @@ std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_task
}
bool need_pick_tablet = true;
- int thread_per_disk = compaction_type == CompactionType::CUMU_BINLOG_COMPACTION
- ? config::binlog_compaction_task_num_per_disk
+ int thread_per_disk = compaction_type == CompactionType::CHANGELOG_COMPACTION
+ ? config::changelog_compaction_task_num_per_disk
: config::compaction_task_num_per_fast_disk;
int num_cumu =
std::accumulate(submitted_cumu_compactions.begin(), submitted_cumu_compactions.end(), 0,
[](int a, auto& b) { return a + b.second.size(); });
- int num_cumu_binlog = submitted_cumu_binlog_compaction_count;
- int num_cumu_data = num_cumu - num_cumu_binlog;
+ int num_cumu_changelog = submitted_cumu_changelog_compaction_count;
+ int num_cumu_data = num_cumu - num_cumu_changelog;
int num_base =
cast_set<int>(submitted_base_compactions.size() + submitted_full_compactions.size());
- int n = compaction_type == CompactionType::CUMU_BINLOG_COMPACTION
- ? thread_per_disk - num_cumu_binlog
+ int n = compaction_type == CompactionType::CHANGELOG_COMPACTION
+ ? thread_per_disk - num_cumu_changelog
: thread_per_disk - num_cumu_data - num_base;
if (compaction_type == CompactionType::BASE_COMPACTION) {
// We need to reserve at least one thread for cumulative compaction,
@@ -824,16 +827,16 @@ std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_task
if (compaction_type == CompactionType::BASE_COMPACTION) {
filter_out = [&submitted_base_compactions, &submitted_full_compactions,
&submitted_index_change_base_compactions](CloudTablet* t) {
- return t->is_row_binlog_tablet() ||
+ return t->is_changelog_tablet() ||
submitted_base_compactions.contains(t->tablet_id()) ||
submitted_full_compactions.contains(t->tablet_id()) ||
submitted_index_change_base_compactions.contains(t->tablet_id()) ||
t->tablet_state() != TABLET_RUNNING;
};
- } else if (compaction_type == CompactionType::CUMU_BINLOG_COMPACTION) {
+ } else if (compaction_type == CompactionType::CHANGELOG_COMPACTION) {
filter_out = [&tablet_preparing_cumu_compaction, &submitted_cumu_compactions,
&submitted_index_change_cumu_compactions](CloudTablet* t) {
- return !t->is_row_binlog_tablet() ||
+ return !t->is_changelog_tablet() ||
tablet_preparing_cumu_compaction.contains(t->tablet_id()) ||
submitted_index_change_cumu_compactions.contains(t->tablet_id()) ||
submitted_cumu_compactions.contains(t->tablet_id()) ||
@@ -843,7 +846,7 @@ std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_task
} else if (config::enable_parallel_cumu_compaction) {
filter_out = [&tablet_preparing_cumu_compaction,
&submitted_index_change_cumu_compactions](CloudTablet* t) {
- return t->is_row_binlog_tablet() ||
+ return t->is_changelog_tablet() ||
tablet_preparing_cumu_compaction.contains(t->tablet_id()) ||
submitted_index_change_cumu_compactions.contains(t->tablet_id()) ||
(t->tablet_state() != TABLET_RUNNING &&
@@ -852,7 +855,7 @@ std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_task
} else {
filter_out = [&tablet_preparing_cumu_compaction, &submitted_cumu_compactions,
&submitted_index_change_cumu_compactions](CloudTablet* t) {
- return t->is_row_binlog_tablet() ||
+ return t->is_changelog_tablet() ||
tablet_preparing_cumu_compaction.contains(t->tablet_id()) ||
submitted_index_change_cumu_compactions.contains(t->tablet_id()) ||
submitted_cumu_compactions.contains(t->tablet_id()) ||
@@ -893,9 +896,9 @@ std::vector<CloudTabletSPtr> CloudStorageEngine::_generate_cloud_compaction_task
DorisMetrics::instance()->tablet_time_series_max_compaction_score->set_value(
score_stats.time_series_max_score);
}
- } else if (compaction_type == CompactionType::CUMU_BINLOG_COMPACTION &&
+ } else if (compaction_type == CompactionType::CHANGELOG_COMPACTION &&
score_stats.max_score > 0) {
- DorisMetrics::instance()->tablet_binlog_max_compaction_score->set_value(
+ DorisMetrics::instance()->tablet_changelog_max_compaction_score->set_value(
score_stats.max_score);
}
}
@@ -1063,7 +1066,7 @@ Status CloudStorageEngine::_submit_cumulative_compaction_task(const CloudTabletS
int trigger_method,
CompactionType compaction_type) {
DCHECK(compaction_type == CompactionType::CUMULATIVE_COMPACTION ||
- compaction_type == CompactionType::CUMU_BINLOG_COMPACTION);
+ compaction_type == CompactionType::CHANGELOG_COMPACTION);
using namespace std::chrono;
{
std::lock_guard lock(_compaction_mtx);
@@ -1128,8 +1131,8 @@ Status CloudStorageEngine::_submit_cumulative_compaction_task(const CloudTabletS
std::lock_guard lock(_compaction_mtx);
_tablet_preparing_cumu_compaction.erase(tablet->tablet_id());
_submitted_cumu_compactions[tablet->tablet_id()].push_back(compaction);
- if (compaction_type == CompactionType::CUMU_BINLOG_COMPACTION) {
- ++_submitted_cumu_binlog_compaction_count;
+ if (compaction_type == CompactionType::CHANGELOG_COMPACTION) {
+ ++_submitted_cumu_changelog_compaction_count;
}
}
auto erase_submitted_cumu_compaction = [=, this]() {
@@ -1140,9 +1143,9 @@ Status CloudStorageEngine::_submit_cumulative_compaction_task(const CloudTabletS
auto it1 = std::find(compactions.begin(), compactions.end(), compaction);
DCHECK(it1 != compactions.end());
compactions.erase(it1);
- if (compaction_type == CompactionType::CUMU_BINLOG_COMPACTION) {
- DCHECK_GT(_submitted_cumu_binlog_compaction_count, 0);
- --_submitted_cumu_binlog_compaction_count;
+ if (compaction_type == CompactionType::CHANGELOG_COMPACTION) {
+ DCHECK_GT(_submitted_cumu_changelog_compaction_count, 0);
+ --_submitted_cumu_changelog_compaction_count;
}
if (compactions.empty()) { // No compactions on this tablet, erase key
_submitted_cumu_compactions.erase(it);
@@ -1168,14 +1171,14 @@ Status CloudStorageEngine::_submit_cumulative_compaction_task(const CloudTabletS
tablet->last_cumu_no_suitable_version_ms = 0;
}
};
- auto& submit_thread_pool = compaction_type == CompactionType::CUMU_BINLOG_COMPACTION
- ? _binlog_compaction_thread_pool
+ auto& submit_thread_pool = compaction_type == CompactionType::CHANGELOG_COMPACTION
+ ? _changelog_compaction_thread_pool
: _cumu_compaction_thread_pool;
st = submit_thread_pool->submit_func([=, this, compaction = std::move(compaction)]() {
- if (compaction_type == CompactionType::CUMU_BINLOG_COMPACTION) {
- DorisMetrics::instance()->binlog_compaction_task_running_total->increment(1);
- DorisMetrics::instance()->binlog_compaction_task_pending_total->set_value(
- _binlog_compaction_thread_pool->get_queue_size());
+ if (compaction_type == CompactionType::CHANGELOG_COMPACTION) {
+ DorisMetrics::instance()->changelog_compaction_task_running_total->increment(1);
+ DorisMetrics::instance()->changelog_compaction_task_pending_total->set_value(
+ _changelog_compaction_thread_pool->get_queue_size());
} else {
DorisMetrics::instance()->cumulative_compaction_task_running_total->increment(1);
DorisMetrics::instance()->cumulative_compaction_task_pending_total->set_value(
@@ -1201,10 +1204,10 @@ Status CloudStorageEngine::_submit_cumulative_compaction_task(const CloudTabletS
}
g_cumu_compaction_running_task_count << -1;
erase_submitted_cumu_compaction();
- if (compaction_type == CompactionType::CUMU_BINLOG_COMPACTION) {
- DorisMetrics::instance()->binlog_compaction_task_running_total->increment(-1);
- DorisMetrics::instance()->binlog_compaction_task_pending_total->set_value(
- _binlog_compaction_thread_pool->get_queue_size());
+ if (compaction_type == CompactionType::CHANGELOG_COMPACTION) {
+ DorisMetrics::instance()->changelog_compaction_task_running_total->increment(-1);
+ DorisMetrics::instance()->changelog_compaction_task_pending_total->set_value(
+ _changelog_compaction_thread_pool->get_queue_size());
} else {
DorisMetrics::instance()->cumulative_compaction_task_running_total->increment(-1);
DorisMetrics::instance()->cumulative_compaction_task_pending_total->set_value(
@@ -1273,9 +1276,9 @@ Status CloudStorageEngine::_submit_cumulative_compaction_task(const CloudTabletS
}
erase_executing_cumu_compaction();
});
- if (compaction_type == CompactionType::CUMU_BINLOG_COMPACTION) {
- DorisMetrics::instance()->binlog_compaction_task_pending_total->set_value(
- _binlog_compaction_thread_pool->get_queue_size());
+ if (compaction_type == CompactionType::CHANGELOG_COMPACTION) {
+ DorisMetrics::instance()->changelog_compaction_task_pending_total->set_value(
+ _changelog_compaction_thread_pool->get_queue_size());
} else {
DorisMetrics::instance()->cumulative_compaction_task_pending_total->set_value(
_cumu_compaction_thread_pool->get_queue_size());
@@ -1289,10 +1292,10 @@ Status CloudStorageEngine::_submit_cumulative_compaction_task(const CloudTabletS
return st;
}
-Status CloudStorageEngine::_submit_binlog_compaction_task(const CloudTabletSPtr& tablet,
- int trigger_method) {
+Status CloudStorageEngine::_submit_changelog_compaction_task(const CloudTabletSPtr& tablet,
+ int trigger_method) {
return _submit_cumulative_compaction_task(tablet, trigger_method,
- CompactionType::CUMU_BINLOG_COMPACTION);
+ CompactionType::CHANGELOG_COMPACTION);
}
Status CloudStorageEngine::_submit_full_compaction_task(const CloudTabletSPtr& tablet,
@@ -1392,7 +1395,7 @@ Status CloudStorageEngine::submit_compaction_task(const CloudTabletSPtr& tablet,
int trigger_method) {
DCHECK(compaction_type == CompactionType::CUMULATIVE_COMPACTION ||
compaction_type == CompactionType::BASE_COMPACTION ||
- compaction_type == CompactionType::CUMU_BINLOG_COMPACTION ||
+ compaction_type == CompactionType::CHANGELOG_COMPACTION ||
compaction_type == CompactionType::FULL_COMPACTION);
switch (compaction_type) {
case CompactionType::BASE_COMPACTION:
@@ -1401,8 +1404,8 @@ Status CloudStorageEngine::submit_compaction_task(const CloudTabletSPtr& tablet,
case CompactionType::CUMULATIVE_COMPACTION:
RETURN_IF_ERROR(_submit_cumulative_compaction_task(tablet, trigger_method));
return Status::OK();
- case CompactionType::CUMU_BINLOG_COMPACTION:
- RETURN_IF_ERROR(_submit_binlog_compaction_task(tablet, trigger_method));
+ case CompactionType::CHANGELOG_COMPACTION:
+ RETURN_IF_ERROR(_submit_changelog_compaction_task(tablet, trigger_method));
return Status::OK();
case CompactionType::FULL_COMPACTION:
RETURN_IF_ERROR(_submit_full_compaction_task(tablet, trigger_method));
@@ -1525,8 +1528,11 @@ Status CloudStorageEngine::get_compaction_status_json(std::string* result) {
}
std::shared_ptr<CloudCumulativeCompactionPolicy> CloudStorageEngine::cumu_compaction_policy(
- std::string_view compaction_policy) {
+ std::string_view compaction_policy, int64_t tablet_id) {
if (!_cumulative_compaction_policies.contains(compaction_policy)) {
+ LOG_EVERY_N(WARNING, 100) << "unknown compaction policy " << compaction_policy
+ << " of tablet " << tablet_id << ", fall back to "
+ << CUMULATIVE_SIZE_BASED_POLICY;
return _cumulative_compaction_policies.at(CUMULATIVE_SIZE_BASED_POLICY);
}
return _cumulative_compaction_policies.at(compaction_policy);
diff --git a/be/src/cloud/cloud_storage_engine.h b/be/src/cloud/cloud_storage_engine.h
index 330850b5ed8..1fcb63477c8 100644
--- a/be/src/cloud/cloud_storage_engine.h
+++ b/be/src/cloud/cloud_storage_engine.h
@@ -157,7 +157,7 @@ public:
}
std::shared_ptr<CloudCumulativeCompactionPolicy> cumu_compaction_policy(
- std::string_view compaction_policy);
+ std::string_view compaction_policy, int64_t tablet_id);
void sync_storage_vault();
@@ -211,7 +211,7 @@ private:
void _vacuum_stale_rowsets_thread_callback();
void _sync_tablets_thread_callback();
void _compaction_tasks_producer_callback();
- void _binlog_compaction_tasks_producer_callback();
+ void _changelog_compaction_tasks_producer_callback();
std::vector<CloudTabletSPtr> _generate_cloud_compaction_tasks(CompactionType compaction_type,
bool check_score);
Status _adjust_compaction_thread_num();
@@ -219,7 +219,7 @@ private:
Status _submit_cumulative_compaction_task(
const CloudTabletSPtr& tablet, int trigger_method = 0,
CompactionType compaction_type = CompactionType::CUMULATIVE_COMPACTION);
- Status _submit_binlog_compaction_task(const CloudTabletSPtr& tablet, int trigger_method = 0);
+ Status _submit_changelog_compaction_task(const CloudTabletSPtr& tablet, int trigger_method = 0);
Status _submit_full_compaction_task(const CloudTabletSPtr& tablet, int trigger_method = 0);
Status _request_tablet_global_compaction_lock(ReaderType compaction_type,
const CloudTabletSPtr& tablet,
@@ -264,8 +264,8 @@ private:
// tablet_id -> submitted cumu compactions, guarded by `_compaction_mtx`
std::unordered_map<int64_t, std::vector<std::shared_ptr<CloudCumulativeCompaction>>>
_submitted_cumu_compactions;
- // submitted binlog cumu compaction task count, guarded by `_compaction_mtx`
- int _submitted_cumu_binlog_compaction_count = 0;
+ // submitted changelog cumu compaction task count, guarded by `_compaction_mtx`
+ int _submitted_cumu_changelog_compaction_count = 0;
// tablet_id -> active compaction stop tokens
std::unordered_map<int64_t, std::shared_ptr<CloudCompactionStopToken>>
_active_compaction_stop_tokens;
diff --git a/be/src/cloud/cloud_tablet.cpp b/be/src/cloud/cloud_tablet.cpp
index 8cda6e44d5d..29c360226a3 100644
--- a/be/src/cloud/cloud_tablet.cpp
+++ b/be/src/cloud/cloud_tablet.cpp
@@ -845,8 +845,8 @@ Result<std::unique_ptr<RowsetWriter>> CloudTablet::create_rowset_writer(
context.partition_id = partition_id();
context.enable_unique_key_merge_on_write = enable_unique_key_merge_on_write();
context.encrypt_algorithm = tablet_meta()->encryption_algorithm();
- if (context.write_binlog_opt().enable) {
- context.write_binlog_opt().set_need_before(binlog_config().need_historical_value());
+ if (context.write_changelog_opt().enable) {
+ context.write_changelog_opt().set_need_before(binlog_config().need_historical_value());
}
context.inverted_index_storage_format = tablet_meta()->inverted_index_storage_format();
context.persist_inverted_index_storage_format =
@@ -888,9 +888,9 @@ Result<std::unique_ptr<RowsetWriter>> CloudTablet::create_transient_rowset_write
context.write_type = DataWriteType::TYPE_DIRECT;
context.partial_update_info = std::move(partial_update_info);
context.is_transient_rowset_writer = true;
- if (rowset.rowset_meta() != nullptr && rowset.rowset_meta()->is_row_binlog()) {
- context.write_binlog_opt().enable = true;
- context.write_binlog_opt().set_need_before(binlog_config().need_historical_value());
+ if (rowset.rowset_meta() != nullptr && rowset.rowset_meta()->is_changelog()) {
+ context.write_changelog_opt().enable = true;
+ context.write_changelog_opt().set_need_before(binlog_config().need_historical_value());
}
context.rowset_id = rowset.rowset_id();
context.tablet_id = tablet_id();
@@ -1224,16 +1224,16 @@ Status CloudTablet::save_delete_bitmap(const TabletTxnInfo* txn_info, int64_t tx
int64_t next_visible_version) {
RowsetSharedPtr rowset = txn_info->rowset;
int64_t cur_version = rowset->start_version();
- const bool build_row_binlog = txn_info->attach_row_binlog.rowset != nullptr;
+ const bool build_changelog = txn_info->attach_changelog.rowset != nullptr;
const RowsetId cur_build_rid = rowset->rowset_id();
- const RowsetId binlog_rid =
- build_row_binlog ? txn_info->attach_row_binlog.rowset->rowset_id() : cur_build_rid;
- auto* row_binlog_delete_bitmap = txn_info->attach_row_binlog.delete_bitmap.get();
- if (build_row_binlog) {
- DCHECK(txn_info->attach_row_binlog.rowset->rowset_meta() != nullptr);
- DCHECK(txn_info->attach_row_binlog.rowset->rowset_meta()->is_row_binlog());
- DCHECK(txn_info->attach_row_binlog.tablet != nullptr);
- DCHECK(row_binlog_delete_bitmap != nullptr);
+ const RowsetId changelog_rid =
+ build_changelog ? txn_info->attach_changelog.rowset->rowset_id() : cur_build_rid;
+ auto* changelog_delete_bitmap = txn_info->attach_changelog.delete_bitmap.get();
+ if (build_changelog) {
+ DCHECK(txn_info->attach_changelog.rowset->rowset_meta() != nullptr);
+ DCHECK(txn_info->attach_changelog.rowset->rowset_meta()->is_changelog());
+ DCHECK(txn_info->attach_changelog.tablet != nullptr);
+ DCHECK(changelog_delete_bitmap != nullptr);
}
// update delete bitmap info, in order to avoid recalculation when trying again
@@ -1246,29 +1246,29 @@ Status CloudTablet::save_delete_bitmap(const TabletTxnInfo* txn_info, int64_t tx
return Status::InternalError<false>("injected update_tmp_rowset error.");
});
const auto& rowset_meta = rowset->rowset_meta();
- if (build_row_binlog) {
- const auto& binlog_rowset_meta = txn_info->attach_row_binlog.rowset->rowset_meta();
+ if (build_changelog) {
+ const auto& changelog_rowset_meta = txn_info->attach_changelog.rowset->rowset_meta();
RETURN_IF_ERROR(_engine.meta_mgr().update_tmp_rowset(*rowset_meta, table_id(),
- binlog_rowset_meta.get()));
+ changelog_rowset_meta.get()));
} else {
RETURN_IF_ERROR(_engine.meta_mgr().update_tmp_rowset(*rowset_meta, table_id()));
}
}
- if (build_row_binlog) {
+ if (build_changelog) {
for (const auto& [key, bitmap] : delete_bitmap->delete_bitmap) {
if (std::get<1>(key) != DeleteBitmap::INVALID_SEGMENT_ID &&
std::get<0>(key) == cur_build_rid) {
- row_binlog_delete_bitmap->merge(
- {binlog_rid, std::get<1>(key), cast_set<uint64_t>(cur_version)}, bitmap);
+ changelog_delete_bitmap->merge(
+ {changelog_rid, std::get<1>(key), cast_set<uint64_t>(cur_version)}, bitmap);
}
}
- auto* binlog_tablet = static_cast<CloudTablet*>(txn_info->attach_row_binlog.tablet.get());
- RETURN_IF_ERROR(binlog_tablet->save_delete_bitmap_to_ms(
- cur_version, txn_id, txn_info->attach_row_binlog.delete_bitmap, lock_id,
- next_visible_version, txn_info->attach_row_binlog.rowset));
+ auto* changelog_tablet = static_cast<CloudTablet*>(txn_info->attach_changelog.tablet.get());
+ RETURN_IF_ERROR(changelog_tablet->save_delete_bitmap_to_ms(
+ cur_version, txn_id, txn_info->attach_changelog.delete_bitmap, lock_id,
+ next_visible_version, txn_info->attach_changelog.rowset));
RETURN_IF_ERROR(_engine.txn_delete_bitmap_cache().update_tablet_txn_info(
- txn_id, binlog_tablet->tablet_id(), txn_info->attach_row_binlog.delete_bitmap,
+ txn_id, changelog_tablet->tablet_id(), txn_info->attach_changelog.delete_bitmap,
cur_rowset_ids, PublishStatus::SUCCEED, txn_info->publish_info));
}
diff --git a/be/src/cloud/cloud_tablet_mgr.cpp b/be/src/cloud/cloud_tablet_mgr.cpp
index c75f9e60c22..0b0f0da6732 100644
--- a/be/src/cloud/cloud_tablet_mgr.cpp
+++ b/be/src/cloud/cloud_tablet_mgr.cpp
@@ -263,7 +263,7 @@ Result<std::shared_ptr<CloudTablet>> CloudTabletMgr::get_tablet(int64_t tablet_i
set_tablet_access_time_ms(tablet_ptr);
_cache->release(insert_handle);
});
- _engine.register_row_binlog_tablet(tablet);
+ _engine.register_changelog_tablet(tablet);
_tablet_map->put(std::move(tablet));
return ret;
};
@@ -486,21 +486,21 @@ Status CloudTabletMgr::get_topn_tablets_to_compact(
std::vector<std::shared_ptr<CloudTablet>>* tablets, CompactionScoreStats* score_stats) {
DCHECK(compaction_type == CompactionType::BASE_COMPACTION ||
compaction_type == CompactionType::CUMULATIVE_COMPACTION ||
- compaction_type == CompactionType::CUMU_BINLOG_COMPACTION);
+ compaction_type == CompactionType::CHANGELOG_COMPACTION);
*score_stats = {};
score_stats->scanned = true;
int64_t max_score_tablet_id = 0;
// clang-format off
auto score = [compaction_type](CloudTablet* t) {
- if (compaction_type == CompactionType::CUMU_BINLOG_COMPACTION && !t->is_row_binlog_tablet()) {
+ if (compaction_type == CompactionType::CHANGELOG_COMPACTION && !t->is_changelog_tablet()) {
return int64_t {0};
}
- if (compaction_type != CompactionType::CUMU_BINLOG_COMPACTION && t->is_row_binlog_tablet()) {
+ if (compaction_type != CompactionType::CHANGELOG_COMPACTION && t->is_changelog_tablet()) {
return int64_t {0};
}
return compaction_type == CompactionType::BASE_COMPACTION ? t->get_cloud_base_compaction_score()
: (compaction_type == CompactionType::CUMULATIVE_COMPACTION ||
- compaction_type == CompactionType::CUMU_BINLOG_COMPACTION) ? t->get_cloud_cumu_compaction_score()
+ compaction_type == CompactionType::CHANGELOG_COMPACTION) ? t->get_cloud_cumu_compaction_score()
: 0;
};
diff --git a/be/src/cloud/cloud_tablets_channel.cpp b/be/src/cloud/cloud_tablets_channel.cpp
index 4d9f76a145d..3b03c30bd68 100644
--- a/be/src/cloud/cloud_tablets_channel.cpp
+++ b/be/src/cloud/cloud_tablets_channel.cpp
@@ -42,22 +42,22 @@ std::unique_ptr<BaseDeltaWriter> CloudTabletsChannel::create_delta_writer(
DCHECK(request.write_req_type == WriteRequestType::DATA);
DCHECK(request.table_schema_param != nullptr);
- if (request.binlog_tablet_id <= 0) {
+ if (request.changelog_tablet_id <= 0) {
return std::make_unique<CloudDeltaWriter>(_engine, request, _profile, _load_id);
}
- int64_t row_binlog_index_id = 0;
+ int64_t changelog_index_id = 0;
for (const auto* index_schema : request.table_schema_param->indexes()) {
if (index_schema->index_id == request.index_id) {
- row_binlog_index_id = index_schema->row_binlog_id;
+ changelog_index_id = index_schema->changelog_index_id;
break;
}
}
- DCHECK(row_binlog_index_id > 0);
+ DCHECK(changelog_index_id > 0);
- const auto* row_binlog_index_schema =
- request.table_schema_param->row_binlog_index_schema(row_binlog_index_id);
- DCHECK(row_binlog_index_schema != nullptr);
+ const auto* changelog_index_schema =
+ request.table_schema_param->changelog_index_schema(changelog_index_id);
+ DCHECK(changelog_index_schema != nullptr);
WriteRequest group_build_req = request;
group_build_req.write_req_type = WriteRequestType::GROUP;
@@ -65,14 +65,14 @@ std::unique_ptr<BaseDeltaWriter> CloudTabletsChannel::create_delta_writer(
WriteRequest sub_data_req = request;
sub_data_req.write_req_type = WriteRequestType::DATA;
- WriteRequest sub_row_binlog_req = request;
- sub_row_binlog_req.write_req_type = WriteRequestType::ROW_BINLOG;
- sub_row_binlog_req.tablet_id = request.binlog_tablet_id;
- sub_row_binlog_req.index_id = row_binlog_index_schema->index_id;
- sub_row_binlog_req.schema_hash = row_binlog_index_schema->schema_hash;
+ WriteRequest sub_changelog_req = request;
+ sub_changelog_req.write_req_type = WriteRequestType::CHANGELOG;
+ sub_changelog_req.tablet_id = request.changelog_tablet_id;
+ sub_changelog_req.index_id = changelog_index_schema->index_id;
+ sub_changelog_req.schema_hash = changelog_index_schema->schema_hash;
return std::make_unique<CloudDeltaWriter>(_engine, group_build_req, sub_data_req,
- sub_row_binlog_req, _profile, _load_id);
+ sub_changelog_req, _profile, _load_id);
}
Status CloudTabletsChannel::add_batch(const PTabletWriterAddBlockRequest& request,
@@ -332,12 +332,12 @@ Status CloudTabletsChannel::_close_writers(PTabletWriterAddBlockResult* res) {
tablet_info->set_schema_hash(0);
tablet_info->set_received_rows(writer->total_received_rows());
tablet_info->set_num_rows_filtered(writer->num_rows_filtered());
- if (writer->binlog_tablet_id() > 0) {
- PTabletInfo* binlog_tablet_info = tablet_vec->Add();
- binlog_tablet_info->set_tablet_id(writer->binlog_tablet_id());
- binlog_tablet_info->set_schema_hash(0);
- binlog_tablet_info->set_received_rows(writer->total_received_rows());
- binlog_tablet_info->set_num_rows_filtered(writer->num_rows_filtered());
+ if (writer->changelog_tablet_id() > 0) {
+ PTabletInfo* changelog_tablet_info = tablet_vec->Add();
+ changelog_tablet_info->set_tablet_id(writer->changelog_tablet_id());
+ changelog_tablet_info->set_schema_hash(0);
+ changelog_tablet_info->set_received_rows(writer->total_received_rows());
+ changelog_tablet_info->set_num_rows_filtered(writer->num_rows_filtered());
}
// These stats may be larger than the actual value if the txn is aborted
writer->update_tablet_stats();
diff --git a/be/src/cloud/cloud_txn_delete_bitmap_cache.cpp b/be/src/cloud/cloud_txn_delete_bitmap_cache.cpp
index 83a7528a6c6..fa37b33662f 100644
--- a/be/src/cloud/cloud_txn_delete_bitmap_cache.cpp
+++ b/be/src/cloud/cloud_txn_delete_bitmap_cache.cpp
@@ -60,7 +60,7 @@ Status CloudTxnDeleteBitmapCache::get_tablet_txn_info(
DeleteBitmapPtr* delete_bitmap, RowsetIdUnorderedSet* rowset_ids, int64_t* txn_expiration,
std::shared_ptr<PartialUpdateInfo>* partial_update_info,
std::shared_ptr<PublishStatus>* publish_status, TxnPublishInfo* previous_publish_info,
- RowBinlogTxnInfo* attach_row_binlog) {
+ ChangelogTxnInfo* attach_changelog) {
{
std::shared_lock<std::shared_mutex> rlock(_rwlock);
TxnKey key(transaction_id, tablet_id);
@@ -80,8 +80,8 @@ Status CloudTxnDeleteBitmapCache::get_tablet_txn_info(
*partial_update_info = iter->second.partial_update_info;
*publish_status = iter->second.publish_status;
*previous_publish_info = iter->second.publish_info;
- if (attach_row_binlog != nullptr) {
- *attach_row_binlog = iter->second.attach_row_binlog;
+ if (attach_changelog != nullptr) {
+ *attach_changelog = iter->second.attach_changelog;
}
}
@@ -191,7 +191,7 @@ void CloudTxnDeleteBitmapCache::set_tablet_txn_info(
TTransactionId transaction_id, int64_t tablet_id, DeleteBitmapPtr delete_bitmap,
const RowsetIdUnorderedSet& rowset_ids, RowsetSharedPtr rowset, int64_t txn_expiration,
std::shared_ptr<PartialUpdateInfo> partial_update_info,
- const RowBinlogTxnInfo& attach_row_binlog) {
+ const ChangelogTxnInfo& attach_changelog) {
int64_t txn_expiration_min =
duration_cast<std::chrono::seconds>(std::chrono::system_clock::now().time_since_epoch())
.count() +
@@ -203,7 +203,7 @@ void CloudTxnDeleteBitmapCache::set_tablet_txn_info(
std::shared_ptr<PublishStatus> publish_status =
std::make_shared<PublishStatus>(PublishStatus::INIT);
_txn_map[txn_key] = TxnVal(rowset, txn_expiration, std::move(partial_update_info),
- std::move(publish_status), attach_row_binlog);
+ std::move(publish_status), attach_changelog);
_expiration_txn.emplace(txn_expiration, txn_key);
}
std::string key_str = fmt::format("{}/{}", transaction_id, tablet_id);
diff --git a/be/src/cloud/cloud_txn_delete_bitmap_cache.h b/be/src/cloud/cloud_txn_delete_bitmap_cache.h
index 6be96990ed7..3d6f77d32e7 100644
--- a/be/src/cloud/cloud_txn_delete_bitmap_cache.h
+++ b/be/src/cloud/cloud_txn_delete_bitmap_cache.h
@@ -44,13 +44,13 @@ public:
std::shared_ptr<PartialUpdateInfo>* partial_update_info,
std::shared_ptr<PublishStatus>* publish_status,
TxnPublishInfo* previous_publish_info,
- RowBinlogTxnInfo* attach_row_binlog = nullptr);
+ ChangelogTxnInfo* attach_changelog = nullptr);
void set_tablet_txn_info(TTransactionId transaction_id, int64_t tablet_id,
DeleteBitmapPtr delete_bitmap, const RowsetIdUnorderedSet& rowset_ids,
RowsetSharedPtr rowset, int64_t txn_expirationm,
std::shared_ptr<PartialUpdateInfo> partial_update_info,
- const RowBinlogTxnInfo& attach_row_binlog = {});
+ const ChangelogTxnInfo& attach_changelog = {});
Status update_tablet_txn_info(TTransactionId transaction_id, int64_t tablet_id,
DeleteBitmapPtr delete_bitmap,
@@ -111,17 +111,17 @@ private:
std::shared_ptr<PublishStatus> publish_status = nullptr;
// used to determine if the retry needs to re-calculate the delete bitmap
TxnPublishInfo publish_info;
- RowBinlogTxnInfo attach_row_binlog;
+ ChangelogTxnInfo attach_changelog;
TxnVal() : txn_expiration(0) {};
TxnVal(RowsetSharedPtr rowset_, int64_t txn_expiration_,
std::shared_ptr<PartialUpdateInfo> partial_update_info_,
std::shared_ptr<PublishStatus> publish_status_,
- const RowBinlogTxnInfo& attach_row_binlog_)
+ const ChangelogTxnInfo& attach_changelog_)
: rowset(std::move(rowset_)),
txn_expiration(txn_expiration_),
partial_update_info(std::move(partial_update_info_)),
publish_status(std::move(publish_status_)),
- attach_row_binlog(attach_row_binlog_) {}
+ attach_changelog(attach_changelog_) {}
};
std::map<TxnKey, TxnVal> _txn_map;
diff --git a/be/src/cloud/config.cpp b/be/src/cloud/config.cpp
index 614baa31e9b..715419f6fbe 100644
--- a/be/src/cloud/config.cpp
+++ b/be/src/cloud/config.cpp
@@ -49,7 +49,8 @@ DEFINE_mInt32(lease_compaction_interval_seconds, "20");
DEFINE_mBool(enable_parallel_cumu_compaction, "false");
DEFINE_mDouble(base_compaction_thread_num_factor, "0.25");
DEFINE_mDouble(cumu_compaction_thread_num_factor, "0.5");
-DEFINE_mDouble(binlog_compaction_thread_num_factor, "0.25");
+// ATTN(changelog-rename): formerly "binlog_compaction_thread_num_factor"
+DEFINE_mDouble(changelog_compaction_thread_num_factor, "0.25");
DEFINE_mInt32(check_auto_compaction_interval_seconds, "5");
DEFINE_mInt32(max_base_compaction_task_num_per_disk, "2");
DEFINE_mBool(prioritize_query_perf_in_compaction, "false");
diff --git a/be/src/cloud/config.h b/be/src/cloud/config.h
index 9bc1b11bdc6..c7d1d11b3fe 100644
--- a/be/src/cloud/config.h
+++ b/be/src/cloud/config.h
@@ -90,7 +90,7 @@ DECLARE_mInt32(lease_compaction_interval_seconds);
DECLARE_mBool(enable_parallel_cumu_compaction);
DECLARE_mDouble(base_compaction_thread_num_factor);
DECLARE_mDouble(cumu_compaction_thread_num_factor);
-DECLARE_mDouble(binlog_compaction_thread_num_factor);
+DECLARE_mDouble(changelog_compaction_thread_num_factor);
DECLARE_mInt32(check_auto_compaction_interval_seconds);
DECLARE_mInt32(max_base_compaction_task_num_per_disk);
DECLARE_mBool(prioritize_query_perf_in_compaction);
diff --git a/be/src/cloud/pb_convert.cpp b/be/src/cloud/pb_convert.cpp
index 7bb5fede003..5945307f649 100644
--- a/be/src/cloud/pb_convert.cpp
+++ b/be/src/cloud/pb_convert.cpp
@@ -125,8 +125,8 @@ void doris_rowset_meta_to_cloud(RowsetMetaCloudPB* out, const RowsetMetaPB& in)
if (in.has_commit_tso()) {
out->mutable_commit_tso()->CopyFrom(in.commit_tso());
}
- if (in.has_is_row_binlog()) {
- out->set_is_row_binlog(in.is_row_binlog());
+ if (in.has_is_changelog()) {
+ out->set_is_changelog(in.is_changelog());
}
if (in.has_db_id()) {
out->set_db_id(in.db_id());
@@ -226,8 +226,8 @@ void doris_rowset_meta_to_cloud(RowsetMetaCloudPB* out, RowsetMetaPB&& in) {
if (in.has_commit_tso()) {
out->mutable_commit_tso()->CopyFrom(in.commit_tso());
}
- if (in.has_is_row_binlog()) {
- out->set_is_row_binlog(in.is_row_binlog());
+ if (in.has_is_changelog()) {
+ out->set_is_changelog(in.is_changelog());
}
if (in.has_db_id()) {
out->set_db_id(in.db_id());
@@ -337,8 +337,8 @@ void cloud_rowset_meta_to_doris(RowsetMetaPB* out, const RowsetMetaCloudPB& in)
if (in.has_commit_tso()) {
out->mutable_commit_tso()->CopyFrom(in.commit_tso());
}
- if (in.has_is_row_binlog()) {
- out->set_is_row_binlog(in.is_row_binlog());
+ if (in.has_is_changelog()) {
+ out->set_is_changelog(in.is_changelog());
}
if (in.has_db_id()) {
out->set_db_id(in.db_id());
@@ -437,8 +437,8 @@ void cloud_rowset_meta_to_doris(RowsetMetaPB* out, RowsetMetaCloudPB&& in) {
if (in.has_commit_tso()) {
out->mutable_commit_tso()->CopyFrom(in.commit_tso());
}
- if (in.has_is_row_binlog()) {
- out->set_is_row_binlog(in.is_row_binlog());
+ if (in.has_is_changelog()) {
+ out->set_is_changelog(in.is_changelog());
}
if (in.has_db_id()) {
out->set_db_id(in.db_id());
@@ -761,8 +761,8 @@ void doris_tablet_meta_to_cloud(TabletMetaCloudPB* out, const TabletMetaPB& in)
if (in.has_tablet_role()) {
out->set_tablet_role(in.tablet_role());
}
- if (in.has_binlog_tablet_id()) {
- out->set_binlog_tablet_id(in.binlog_tablet_id());
+ if (in.has_changelog_tablet_id()) {
+ out->set_changelog_tablet_id(in.changelog_tablet_id());
}
out->set_compaction_policy(in.compaction_policy());
out->set_time_series_compaction_goal_size_mbytes(in.time_series_compaction_goal_size_mbytes());
@@ -849,8 +849,8 @@ void doris_tablet_meta_to_cloud(TabletMetaCloudPB* out, TabletMetaPB&& in) {
if (in.has_tablet_role()) {
out->set_tablet_role(in.tablet_role());
}
- if (in.has_binlog_tablet_id()) {
- out->set_binlog_tablet_id(in.binlog_tablet_id());
+ if (in.has_changelog_tablet_id()) {
+ out->set_changelog_tablet_id(in.changelog_tablet_id());
}
out->set_compaction_policy(in.compaction_policy());
out->set_time_series_compaction_goal_size_mbytes(in.time_series_compaction_goal_size_mbytes());
@@ -944,8 +944,8 @@ void cloud_tablet_meta_to_doris(TabletMetaPB* out, const TabletMetaCloudPB& in)
if (in.has_tablet_role()) {
out->set_tablet_role(in.tablet_role());
}
- if (in.has_binlog_tablet_id()) {
- out->set_binlog_tablet_id(in.binlog_tablet_id());
+ if (in.has_changelog_tablet_id()) {
+ out->set_changelog_tablet_id(in.changelog_tablet_id());
}
out->set_compaction_policy(in.compaction_policy());
out->set_time_series_compaction_goal_size_mbytes(in.time_series_compaction_goal_size_mbytes());
@@ -1032,8 +1032,8 @@ void cloud_tablet_meta_to_doris(TabletMetaPB* out, TabletMetaCloudPB&& in) {
if (in.has_tablet_role()) {
out->set_tablet_role(in.tablet_role());
}
- if (in.has_binlog_tablet_id()) {
- out->set_binlog_tablet_id(in.binlog_tablet_id());
+ if (in.has_changelog_tablet_id()) {
+ out->set_changelog_tablet_id(in.changelog_tablet_id());
}
out->set_compaction_policy(in.compaction_policy());
out->set_time_series_compaction_goal_size_mbytes(in.time_series_compaction_goal_size_mbytes());
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index cffac653cf1..585cd07613f 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -494,14 +494,21 @@ DEFINE_mInt32(ordered_data_compaction_min_segment_size, "10485760");
DEFINE_mInt32(max_base_compaction_threads, "4");
DEFINE_mInt32(max_cumu_compaction_threads, "-1");
-// Binlog Compaction
-DEFINE_mInt64(binlog_compaction_wait_timesec_after_visible, "600");
-DEFINE_mInt64(binlog_compaction_goal_size_mbytes, "128");
-DEFINE_mInt32(binlog_compaction_task_num_per_disk, "4");
-DEFINE_mInt32(binlog_compaction_file_count_threshold, "100");
-DEFINE_mInt32(binlog_level_compaction_max_deltas, "2000");
-DEFINE_mInt64(binlog_compaction_time_threshold_seconds, "3600");
-DEFINE_mInt32(max_binlog_compaction_threads, "-1");
+// Changelog compaction
+// ATTN(changelog-rename): formerly "binlog_compaction_wait_timesec_after_visible"
+DEFINE_mInt64(changelog_compaction_wait_timesec_after_visible, "600");
+// ATTN(changelog-rename): formerly "binlog_compaction_goal_size_mbytes"
+DEFINE_mInt64(changelog_compaction_goal_size_mbytes, "128");
+// ATTN(changelog-rename): formerly "binlog_compaction_task_num_per_disk"
+DEFINE_mInt32(changelog_compaction_task_num_per_disk, "4");
+// ATTN(changelog-rename): formerly "binlog_compaction_file_count_threshold"
+DEFINE_mInt32(changelog_compaction_file_count_threshold, "100");
+// ATTN(changelog-rename): formerly "binlog_level_compaction_max_deltas"
+DEFINE_mInt32(changelog_level_compaction_max_deltas, "2000");
+// ATTN(changelog-rename): formerly "binlog_compaction_time_threshold_seconds"
+DEFINE_mInt64(changelog_compaction_time_threshold_seconds, "3600");
+// ATTN(changelog-rename): formerly "max_binlog_compaction_threads"
+DEFINE_mInt32(max_changelog_compaction_threads, "-1");
DEFINE_Bool(enable_base_compaction_idle_sched, "true");
DEFINE_mInt64(base_compaction_min_rowset_num, "5");
@@ -1533,6 +1540,9 @@ DEFINE_Bool(enable_segment_cache_prune, "false");
// enable feature binlog, default false
DEFINE_Bool(enable_feature_binlog, "false");
+// enable feature changelog, default false
+// ATTN(changelog-rename): formerly "enable_feature_binlog" (non-CCR uses)
+DEFINE_Bool(enable_feature_changelog, "false");
// enable set in BitmapValue
DEFINE_Bool(enable_set_in_bitmap_value, "true");
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 5abf4867249..4783d284495 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -583,14 +583,14 @@ DECLARE_mInt32(ordered_data_compaction_min_segment_size);
DECLARE_mInt32(max_base_compaction_threads);
DECLARE_mInt32(max_cumu_compaction_threads);
-// Binlog Compaction
-DECLARE_mInt64(binlog_compaction_wait_timesec_after_visible);
-DECLARE_mInt64(binlog_compaction_goal_size_mbytes);
-DECLARE_mInt32(binlog_compaction_task_num_per_disk);
-DECLARE_mInt32(binlog_compaction_file_count_threshold);
-DECLARE_mInt32(binlog_level_compaction_max_deltas);
-DECLARE_mInt64(binlog_compaction_time_threshold_seconds);
-DECLARE_mInt32(max_binlog_compaction_threads);
+// Changelog compaction
+DECLARE_mInt64(changelog_compaction_wait_timesec_after_visible);
+DECLARE_mInt64(changelog_compaction_goal_size_mbytes);
+DECLARE_mInt32(changelog_compaction_task_num_per_disk);
+DECLARE_mInt32(changelog_compaction_file_count_threshold);
+DECLARE_mInt32(changelog_level_compaction_max_deltas);
+DECLARE_mInt64(changelog_compaction_time_threshold_seconds);
+DECLARE_mInt32(max_changelog_compaction_threads);
DECLARE_Bool(enable_base_compaction_idle_sched);
DECLARE_mInt64(base_compaction_min_rowset_num);
@@ -1570,6 +1570,8 @@ DECLARE_mInt32(estimated_mem_per_column_reader);
// enable binlog
DECLARE_Bool(enable_feature_binlog);
+// enable changelog
+DECLARE_Bool(enable_feature_changelog);
// enable set in BitmapValue
DECLARE_Bool(enable_set_in_bitmap_value);
diff --git a/be/src/common/metrics/doris_metrics.cpp b/be/src/common/metrics/doris_metrics.cpp
index 66f34fa573a..1e6446eb5a6 100644
--- a/be/src/common/metrics/doris_metrics.cpp
+++ b/be/src/common/metrics/doris_metrics.cpp
@@ -112,10 +112,14 @@ DEFINE_COUNTER_METRIC_PROTOTYPE_5ARG(base_compaction_task_running_total, MetricU
compaction_task_state_total, Labels({{"type", "base"}}));
DEFINE_COUNTER_METRIC_PROTOTYPE_5ARG(base_compaction_task_pending_total, MetricUnit::ROWSETS, "",
compaction_task_state_total, Labels({{"type", "base"}}));
-DEFINE_COUNTER_METRIC_PROTOTYPE_5ARG(binlog_compaction_task_running_total, MetricUnit::ROWSETS, "",
- compaction_task_state_total, Labels({{"type", "binlog"}}));
-DEFINE_COUNTER_METRIC_PROTOTYPE_5ARG(binlog_compaction_task_pending_total, MetricUnit::ROWSETS, "",
- compaction_task_state_total, Labels({{"type", "binlog"}}));
+// ATTN(changelog-rename): formerly "type"="binlog"
+DEFINE_COUNTER_METRIC_PROTOTYPE_5ARG(changelog_compaction_task_running_total, MetricUnit::ROWSETS,
+ "", compaction_task_state_total,
+ Labels({{"type", "changelog"}}));
+// ATTN(changelog-rename): formerly "type"="binlog"
+DEFINE_COUNTER_METRIC_PROTOTYPE_5ARG(changelog_compaction_task_pending_total, MetricUnit::ROWSETS,
+ "", compaction_task_state_total,
+ Labels({{"type", "changelog"}}));
DEFINE_COUNTER_METRIC_PROTOTYPE_5ARG(cumulative_compaction_task_running_total, MetricUnit::ROWSETS,
"", compaction_task_state_total,
Labels({{"type", "cumulative"}}));
@@ -198,7 +202,8 @@ DEFINE_GAUGE_METRIC_PROTOTYPE_2ARG(tablet_cumulative_max_compaction_score, Metri
DEFINE_GAUGE_METRIC_PROTOTYPE_2ARG(tablet_size_based_max_compaction_score, MetricUnit::NOUNIT);
DEFINE_GAUGE_METRIC_PROTOTYPE_2ARG(tablet_time_series_max_compaction_score, MetricUnit::NOUNIT);
DEFINE_GAUGE_METRIC_PROTOTYPE_2ARG(tablet_base_max_compaction_score, MetricUnit::NOUNIT);
-DEFINE_GAUGE_METRIC_PROTOTYPE_2ARG(tablet_binlog_max_compaction_score, MetricUnit::NOUNIT);
+// ATTN(changelog-rename): formerly "tablet_binlog_max_compaction_score"
+DEFINE_GAUGE_METRIC_PROTOTYPE_2ARG(tablet_changelog_max_compaction_score, MetricUnit::NOUNIT);
DEFINE_GAUGE_METRIC_PROTOTYPE_2ARG(all_rowsets_num, MetricUnit::NOUNIT);
DEFINE_GAUGE_METRIC_PROTOTYPE_2ARG(all_segments_num, MetricUnit::NOUNIT);
@@ -358,8 +363,8 @@ DorisMetrics::DorisMetrics() : _metric_registry(_s_registry_name) {
INT_COUNTER_METRIC_REGISTER(_server_metric_entity, base_compaction_task_running_total);
INT_COUNTER_METRIC_REGISTER(_server_metric_entity, base_compaction_task_pending_total);
- INT_COUNTER_METRIC_REGISTER(_server_metric_entity, binlog_compaction_task_running_total);
- INT_COUNTER_METRIC_REGISTER(_server_metric_entity, binlog_compaction_task_pending_total);
+ INT_COUNTER_METRIC_REGISTER(_server_metric_entity, changelog_compaction_task_running_total);
+ INT_COUNTER_METRIC_REGISTER(_server_metric_entity, changelog_compaction_task_pending_total);
INT_COUNTER_METRIC_REGISTER(_server_metric_entity, cumulative_compaction_task_running_total);
INT_COUNTER_METRIC_REGISTER(_server_metric_entity, cumulative_compaction_task_pending_total);
@@ -397,7 +402,7 @@ DorisMetrics::DorisMetrics() : _metric_registry(_s_registry_name) {
INT_GAUGE_METRIC_REGISTER(_server_metric_entity, tablet_size_based_max_compaction_score);
INT_GAUGE_METRIC_REGISTER(_server_metric_entity, tablet_time_series_max_compaction_score);
INT_GAUGE_METRIC_REGISTER(_server_metric_entity, tablet_base_max_compaction_score);
- INT_GAUGE_METRIC_REGISTER(_server_metric_entity, tablet_binlog_max_compaction_score);
+ INT_GAUGE_METRIC_REGISTER(_server_metric_entity, tablet_changelog_max_compaction_score);
INT_GAUGE_METRIC_REGISTER(_server_metric_entity, all_rowsets_num);
INT_GAUGE_METRIC_REGISTER(_server_metric_entity, all_segments_num);
diff --git a/be/src/common/metrics/doris_metrics.h b/be/src/common/metrics/doris_metrics.h
index 971a55051b4..adb3853d8d1 100644
--- a/be/src/common/metrics/doris_metrics.h
+++ b/be/src/common/metrics/doris_metrics.h
@@ -119,8 +119,8 @@ public:
IntCounter* base_compaction_task_running_total = nullptr;
IntCounter* base_compaction_task_pending_total = nullptr;
- IntCounter* binlog_compaction_task_running_total = nullptr;
- IntCounter* binlog_compaction_task_pending_total = nullptr;
+ IntCounter* changelog_compaction_task_running_total = nullptr;
+ IntCounter* changelog_compaction_task_pending_total = nullptr;
IntCounter* cumulative_compaction_task_running_total = nullptr;
IntCounter* cumulative_compaction_task_pending_total = nullptr;
@@ -172,7 +172,7 @@ public:
IntGauge* tablet_size_based_max_compaction_score = nullptr;
IntGauge* tablet_time_series_max_compaction_score = nullptr;
IntGauge* tablet_base_max_compaction_score = nullptr;
- IntGauge* tablet_binlog_max_compaction_score = nullptr;
+ IntGauge* tablet_changelog_max_compaction_score = nullptr;
IntGauge* all_rowsets_num = nullptr;
IntGauge* all_segments_num = nullptr;
diff --git a/be/src/exec/operator/olap_scan_operator.cpp b/be/src/exec/operator/olap_scan_operator.cpp
index d0b4f4150cf..a287d1d479b 100644
--- a/be/src/exec/operator/olap_scan_operator.cpp
+++ b/be/src/exec/operator/olap_scan_operator.cpp
@@ -544,9 +544,9 @@ bool OlapScanLocalState::_is_key_column(const std::string& key_name) {
bool OlapScanLocalState::can_push_down_column_predicate(const SlotDescriptor* slot) {
// The Operator-level method handles static column capabilities. The LocalState-level
- // condition additionally handles the current scan range's binlog merge mode.
+ // condition additionally handles the current scan range's changelog merge mode.
return Base::can_push_down_column_predicate(slot) &&
- (!_is_binlog_merge_scan() || _is_key_column(slot->col_name()));
+ (!_is_changelog_merge_scan() || _is_key_column(slot->col_name()));
}
Status OlapScanLocalState::_should_push_down_function_filter(VectorizedFnCall* fn_call,
@@ -626,12 +626,13 @@ bool OlapScanLocalState::_check_expr_storage_filter(const VExprSPtr& expr,
bool OlapScanLocalState::_storage_no_merge() {
auto& p = _parent->cast<OlapScanOperatorX>();
- // MIN_DELTA / DETAIL binlog scans read through BlockReader, which aggregates and splits one
- // stored UPDATE into BEFORE/AFTER rows, synthesizing __DORIS_BINLOG_OP__ and the before-image
- // values above the storage layer. That is a merge, so predicates must not be pushed to the
- // SegmentIterator (they would see meaningless pre-merge values); they are kept as residual
- // conjuncts evaluated above the reader. APPEND_ONLY / NONE do not split rows and keep pushdown.
- if (_is_binlog_merge_scan()) {
+ // MIN_DELTA / DETAIL changelog scans read through BlockReader, which aggregates and splits one
+ // stored UPDATE into BEFORE/AFTER rows, synthesizing __DORIS_CHANGELOG_OP__ and the
+ // before-image values above the storage layer. That is a merge, so predicates must not be
+ // pushed to the SegmentIterator (they would see meaningless pre-merge values); they are kept
+ // as residual conjuncts evaluated above the reader. APPEND_ONLY / NONE do not split rows and
+ // keep pushdown.
+ if (_is_changelog_merge_scan()) {
return false;
}
return (p._olap_scan_node.keyType == TKeysType::DUP_KEYS ||
@@ -646,14 +647,14 @@ bool OlapScanLocalState::_read_mor_as_dup() {
return p._olap_scan_node.__isset.read_mor_as_dup && p._olap_scan_node.read_mor_as_dup;
}
-bool OlapScanLocalState::_is_binlog_merge_scan() const {
- // All ranges of one scan node share the same binlog scan type. Only MIN_DELTA / DETAIL go
+bool OlapScanLocalState::_is_changelog_merge_scan() const {
+ // All ranges of one scan node share the same changelog scan type. Only MIN_DELTA / DETAIL go
// through BlockReader's merge (op synthesis + BEFORE/AFTER split).
- if (_scan_ranges.empty() || !_scan_ranges[0]->__isset.binlog_scan_type) {
+ if (_scan_ranges.empty() || !_scan_ranges[0]->__isset.changelog_scan_type) {
return false;
}
- auto scan_type = _scan_ranges[0]->binlog_scan_type;
- return scan_type == TBinlogScanType::MIN_DELTA || scan_type == TBinlogScanType::DETAIL;
+ auto scan_type = _scan_ranges[0]->changelog_scan_type;
+ return scan_type == TChangelogScanType::MIN_DELTA || scan_type == TChangelogScanType::DETAIL;
}
void OlapScanLocalState::_prune_rowsets_by_tso(const TPaloScanRange& scan_range,
@@ -785,14 +786,14 @@ Status OlapScanLocalState::_init_scanners(std::list<ScannerSPtr>* scanners) {
}
bool enable_parallel_scan = state()->enable_parallel_scan();
- auto resolve_binlog_scan_type = [](const TPaloScanRange& scan_range) {
- if (scan_range.__isset.binlog_scan_type) {
- return scan_range.binlog_scan_type;
+ auto resolve_changelog_scan_type = [](const TPaloScanRange& scan_range) {
+ if (scan_range.__isset.changelog_scan_type) {
+ return scan_range.changelog_scan_type;
}
- return TBinlogScanType::NONE;
+ return TChangelogScanType::NONE;
};
- bool read_row_binlog =
- p._olap_scan_node.__isset.read_row_binlog && p._olap_scan_node.read_row_binlog;
+ bool read_changelog =
+ p._olap_scan_node.__isset.read_changelog && p._olap_scan_node.read_changelog;
bool has_tso_predicate = _scan_ranges[0]->__isset.start_tso || _scan_ranges[0]->__isset.end_tso;
// The flag of preagg's meaning is whether return pre agg data(or partial agg data)
@@ -813,8 +814,8 @@ Status OlapScanLocalState::_init_scanners(std::list<ScannerSPtr>* scanners) {
if (enable_parallel_scan && !p._should_run_serial &&
p._push_down_agg_type == TPushAggOp::NONE &&
(_storage_no_merge() || p._olap_scan_node.is_preaggregation)
- // binlog<row> need to be read in order
- && !read_row_binlog && !has_tso_predicate) {
+ // changelog need to be read in order
+ && !read_changelog && !has_tso_predicate) {
ParallelScannerBuilder scanner_builder(this, _tablets, _read_sources, _scan_ranges,
_scanner_profile, tablet_key_ranges, state(),
p._limit, true, p._olap_scan_node.is_preaggregation);
@@ -866,7 +867,7 @@ Status OlapScanLocalState::_init_scanners(std::list<ScannerSPtr>* scanners) {
continue;
}
const auto& palo_scan_range = *_scan_ranges[scan_range_idx];
- if (read_row_binlog &&
+ if (read_changelog &&
(palo_scan_range.__isset.start_tso || palo_scan_range.__isset.end_tso)) {
auto& read_source = _read_sources[scan_range_idx];
// The version-consistent read source and delete predicates have already been
@@ -923,8 +924,8 @@ Status OlapScanLocalState::_init_scanners(std::list<ScannerSPtr>* scanners) {
{},
p._limit,
p._olap_scan_node.is_preaggregation,
- read_row_binlog,
- resolve_binlog_scan_type(palo_scan_range),
+ read_changelog,
+ resolve_changelog_scan_type(palo_scan_range),
palo_scan_range.bucket_seq,
palo_scan_range.bucket_num,
palo_scan_range.__isset.start_tso
diff --git a/be/src/exec/operator/olap_scan_operator.h b/be/src/exec/operator/olap_scan_operator.h
index 8b317a7ca39..0c593e66f7b 100644
--- a/be/src/exec/operator/olap_scan_operator.h
+++ b/be/src/exec/operator/olap_scan_operator.h
@@ -117,10 +117,10 @@ private:
bool _storage_no_merge() override;
bool _read_mor_as_dup();
- // True for MIN_DELTA / DETAIL binlog scans, which read through BlockReader's merge (op
+ // True for MIN_DELTA / DETAIL changelog scans, which read through BlockReader's merge (op
// synthesis + BEFORE/AFTER split) and thus must keep predicates above the reader. Returns bool
- // to avoid leaking the thrift binlog-scan-type enum into this header.
- bool _is_binlog_merge_scan() const;
+ // to avoid leaking the thrift changelog-scan-type enum into this header.
+ bool _is_changelog_merge_scan() const;
bool _push_down_topn(const RuntimePredicate& predicate) override {
if (!predicate.target_is_slot(_parent->node_id())) {
return false;
diff --git a/be/src/exec/pipeline/pipeline_fragment_context.cpp b/be/src/exec/pipeline/pipeline_fragment_context.cpp
index 86c02030d40..6cab8af1b92 100644
--- a/be/src/exec/pipeline/pipeline_fragment_context.cpp
+++ b/be/src/exec/pipeline/pipeline_fragment_context.cpp
@@ -1311,7 +1311,7 @@ Status PipelineFragmentContext::_create_data_sink(ObjectPool* pool, const TDataS
int child_node_id = pipeline->operators().back()->node_id();
if (state->query_options().enable_memtable_on_sink_node &&
!_has_inverted_index_v1_or_partial_update(thrift_sink.olap_table_sink) &&
- !_has_row_binlog(thrift_sink.olap_table_sink) && !config::is_cloud_mode()) {
+ !_has_changelog(thrift_sink.olap_table_sink) && !config::is_cloud_mode()) {
_sink = std::make_shared<OlapTableSinkV2OperatorX>(
pool, next_sink_operator_id(), child_node_id + 1, row_desc, output_exprs);
} else {
@@ -1542,11 +1542,11 @@ Status PipelineFragmentContext::_create_operator(ObjectPool* pool, const TPlanNo
"cache node_id={}",
tnode.node_id, _params.fragment.query_cache_param.node_id);
}
- if (tnode.olap_scan_node.__isset.read_row_binlog &&
- tnode.olap_scan_node.read_row_binlog) {
- // Row-binlog scans read a different data stream: they must
+ if (tnode.olap_scan_node.__isset.read_changelog &&
+ tnode.olap_scan_node.read_changelog) {
+ // Changelog scans read a different data stream: they must
// neither serve nor fill the query cache.
- _query_cache_runtime->disable_for_binlog_scan();
+ _query_cache_runtime->disable_for_changelog_scan();
}
}
op = std::make_shared<OlapScanOperatorX>(
diff --git a/be/src/exec/scan/olap_scanner.cpp b/be/src/exec/scan/olap_scanner.cpp
index 58e16a02f1f..d99a641c717 100644
--- a/be/src/exec/scan/olap_scanner.cpp
+++ b/be/src/exec/scan/olap_scanner.cpp
@@ -53,7 +53,7 @@
#include "runtime/runtime_profile.h"
#include "runtime/runtime_state.h"
#include "service/backend_options.h"
-#include "storage/binlog.h"
+#include "storage/changelog.h"
#include "storage/id_manager.h"
#include "storage/index/inverted/common/inverted_index_profile.h"
#include "storage/index/inverted/reader/collection_statistics.h"
@@ -81,7 +81,7 @@ OlapScanner::OlapScanner(ScanLocalStateBase* parent, OlapScanner::Params&& param
_tablet_reader_params({.tablet = std::move(params.tablet),
.tablet_schema {},
.reader_type = ReaderType::READER_QUERY,
- .read_row_binlog = params.read_row_binlog,
+ .read_changelog = params.read_changelog,
.aggregation = params.aggregation,
.version = {0, params.version},
.start_key {},
@@ -103,7 +103,7 @@ OlapScanner::OlapScanner(ScanLocalStateBase* parent, OlapScanner::Params&& param
.collection_statistics {},
.ann_topn_runtime {},
.condition_cache_digest = parent->get_condition_cache_digest(),
- .binlog_scan_type = params.binlog_scan_type}),
+ .changelog_scan_type = params.changelog_scan_type}),
_start_tso(params.start_tso),
_end_tso(params.end_tso),
_bucket_seq(params.bucket_seq),
@@ -322,10 +322,10 @@ Status OlapScanner::_init_tso_predicates() {
}
const auto& read_schema = _tablet_reader_params.read_schema;
- int32_t tso_ordinal = _tablet_reader_params.read_row_binlog ? read_schema->tso_ordinal()
- : read_schema->commit_tso_ordinal();
+ int32_t tso_ordinal = _tablet_reader_params.read_changelog ? read_schema->tso_ordinal()
+ : read_schema->commit_tso_ordinal();
const std::string& column_name =
- _tablet_reader_params.read_row_binlog ? BINLOG_TSO_COL : COMMIT_TSO_COL;
+ _tablet_reader_params.read_changelog ? CHANGELOG_TSO_COL : COMMIT_TSO_COL;
if (tso_ordinal < 0) {
return Status::InvalidArgument(
"Column {} must be present in the FE scan schema for incremental read",
@@ -351,7 +351,7 @@ Status OlapScanner::_init_tso_predicates() {
// segment row counts without applying any column predicate. The commit-tso
// predicate injected above is row-level, so the fast path would both miscount
// (ignoring commit_tso <= snapshot_tso). Disable it here, matching the
- // binlog DETAIL/MIN_DELTA handling.
+ // changelog DETAIL/MIN_DELTA handling.
_tablet_reader_params.push_down_agg_type_opt = TPushAggOp::NONE;
// The scan-node digest does not contain the per-range TSO bounds.
_tablet_reader_params.condition_cache_digest = 0;
@@ -384,16 +384,16 @@ Status OlapScanner::_init_tablet_reader_params(
_tablet_reader_params.push_down_agg_type_opt = _local_state->get_push_down_agg_type();
- // Binlog DETAIL/MIN_DELTA scan schemas contain key/tso/op/before columns
+ // Changelog DETAIL/MIN_DELTA scan schemas contain key/tso/op/before columns
// needed by the row-level merge in BlockReader. The storage-layer
// statistics fast path (VStatisticsIterator, picked when push_down_agg_type
// is COUNT/MINMAX) bypasses SegmentIterator entirely, returning raw segment
- // row counts without binlog op filtering and with a schema that does not
+ // row counts without changelog op filtering and with a schema that does not
// match the widened read schema. The result is both wrong (raw segment
- // count != binlog row count) and unsafe (column-count DCHECK fires inside
+ // count != changelog row count) and unsafe (column-count DCHECK fires inside
// VStatisticsIterator::next_batch). Disable the fast path for these scans.
- if (_tablet_reader_params.binlog_scan_type == TBinlogScanType::DETAIL ||
- _tablet_reader_params.binlog_scan_type == TBinlogScanType::MIN_DELTA) {
+ if (_tablet_reader_params.changelog_scan_type == TChangelogScanType::DETAIL ||
+ _tablet_reader_params.changelog_scan_type == TChangelogScanType::MIN_DELTA) {
_tablet_reader_params.push_down_agg_type_opt = TPushAggOp::NONE;
}
@@ -428,13 +428,13 @@ Status OlapScanner::_init_tablet_reader_params(
_tablet_reader_params.profile = _local_state->custom_profile();
_tablet_reader_params.runtime_state = _state;
- const bool is_binlog_merge_scan =
- _tablet_reader_params.binlog_scan_type == TBinlogScanType::MIN_DELTA ||
- _tablet_reader_params.binlog_scan_type == TBinlogScanType::DETAIL;
+ const bool is_changelog_merge_scan =
+ _tablet_reader_params.changelog_scan_type == TChangelogScanType::MIN_DELTA ||
+ _tablet_reader_params.changelog_scan_type == TChangelogScanType::DETAIL;
RETURN_IF_ERROR(_init_tso_predicates());
- // Row-binlog scans must not be re-ordered or truncated by ORDER BY / TopN pushdowns,
- // so reset every reorder-related param for all binlog scan types.
+ // Changelog scans must not be re-ordered or truncated by ORDER BY / TopN pushdowns,
+ // so reset every reorder-related param for all changelog scan types.
//
// Only MIN_DELTA / DETAIL additionally force the storage layer to deliver rows strictly
// in primary-key order, so the BlockReader can group consecutive same-key changes
@@ -445,9 +445,9 @@ Status OlapScanner::_init_tablet_reader_params(
// some or even all key columns. Forcing a key-ordered merge would make the fallback
// comparator read key positions that do not exist in the projected blocks and crash
// the BE (issue #66390), so it reads unordered like a plain scan.
- if (_tablet_reader_params.binlog_scan_type != TBinlogScanType::NONE) {
- _tablet_reader_params.read_orderby_key = is_binlog_merge_scan;
- _tablet_reader_params.force_key_ordered_read = is_binlog_merge_scan;
+ if (_tablet_reader_params.changelog_scan_type != TChangelogScanType::NONE) {
+ _tablet_reader_params.read_orderby_key = is_changelog_merge_scan;
+ _tablet_reader_params.force_key_ordered_read = is_changelog_merge_scan;
_tablet_reader_params.read_orderby_key_reverse = false;
_tablet_reader_params.read_orderby_key_num_prefix_columns = 0;
_tablet_reader_params.read_orderby_key_limit = 0;
@@ -645,7 +645,7 @@ Status OlapScanner::_init_read_schema() {
}
// The FE physical scan tuple is the read-path schema. It already includes
- // every storage key, sequence, TSO and binlog column required below.
+ // every storage key, sequence, TSO and changelog column required below.
_tablet_reader_params.read_schema =
std::make_shared<ReadSchema>(std::move(read_columns), std::move(expected_types));
diff --git a/be/src/exec/scan/olap_scanner.h b/be/src/exec/scan/olap_scanner.h
index 9eb5f867be7..0d46ce31996 100644
--- a/be/src/exec/scan/olap_scanner.h
+++ b/be/src/exec/scan/olap_scanner.h
@@ -78,8 +78,8 @@ public:
io::FileCacheStatistics initial_file_cache_stats;
int64_t limit;
bool aggregation;
- bool read_row_binlog = false;
- TBinlogScanType::type binlog_scan_type = TBinlogScanType::NONE;
+ bool read_changelog = false;
+ TChangelogScanType::type changelog_scan_type = TChangelogScanType::NONE;
int32_t bucket_seq = 0;
int32_t bucket_num = 0;
std::optional<int64_t> start_tso;
diff --git a/be/src/exec/scan/parallel_scanner_builder.cpp b/be/src/exec/scan/parallel_scanner_builder.cpp
index e1650f09323..119caad6c5c 100644
--- a/be/src/exec/scan/parallel_scanner_builder.cpp
+++ b/be/src/exec/scan/parallel_scanner_builder.cpp
@@ -321,8 +321,8 @@ std::shared_ptr<OlapScanner> ParallelScannerBuilder::_build_scanner(
.initial_file_cache_stats = std::move(initial_file_cache_stats),
.limit = _limit,
.aggregation = _is_preaggregation,
- .read_row_binlog = false,
- .binlog_scan_type = TBinlogScanType::NONE,
+ .read_changelog = false,
+ .changelog_scan_type = TChangelogScanType::NONE,
.bucket_seq = scan_range.bucket_seq,
.bucket_num = scan_range.bucket_num,
.start_tso = std::nullopt,
diff --git a/be/src/exec/sink/writer/vtablet_writer.cpp b/be/src/exec/sink/writer/vtablet_writer.cpp
index 0774f441a78..fa2bff788c1 100644
--- a/be/src/exec/sink/writer/vtablet_writer.cpp
+++ b/be/src/exec/sink/writer/vtablet_writer.cpp
@@ -75,7 +75,7 @@
#include "runtime/thread_context.h"
#include "runtime/workload_group/workload_group.h"
#include "service/backend_options.h"
-#include "storage/binlog.h"
+#include "storage/changelog.h"
#include "storage/tablet_info.h"
#include "util/brpc_closure.h"
#include "util/debug_points.h"
@@ -766,11 +766,11 @@ void VNodeChannel::_open_internal(bool is_incremental) {
auto* ptablet = request->add_tablets();
ptablet->set_partition_id(tablet.partition_id);
ptablet->set_tablet_id(tablet.tablet_id);
- // only write binlog on backends that also own the binlog tablet.
- int64_t binlog_tablet_id =
- _parent->_location->get_binlog_tablet_id(tablet.tablet_id, _node_id);
- if (binlog_tablet_id > 0) {
- ptablet->set_binlog_tablet_id(binlog_tablet_id);
+ // only write changelog on backends that also own the changelog tablet.
+ int64_t changelog_tablet_id =
+ _parent->_location->get_changelog_tablet_id(tablet.tablet_id, _node_id);
+ if (changelog_tablet_id > 0) {
+ ptablet->set_changelog_tablet_id(changelog_tablet_id);
}
deduper.insert(tablet.tablet_id);
_all_tablets.push_back(std::move(tablet));
@@ -1641,11 +1641,12 @@ Status VTabletWriter::_init(RuntimeState* state, RuntimeProfile* profile) {
_write_file_cache = table_sink.write_file_cache;
_schema.reset(new OlapTableSchemaParam());
RETURN_IF_ERROR(_schema->init(table_sink.schema));
- bool has_row_binlog = std::any_of(_schema->indexes().begin(), _schema->indexes().end(),
- [](const auto* index) { return index->row_binlog_id > 0; });
- if (has_row_binlog) {
+ bool has_changelog =
+ std::any_of(_schema->indexes().begin(), _schema->indexes().end(),
+ [](const auto* index) { return index->changelog_index_id > 0; });
+ if (has_changelog) {
_allocated_lsn_buffer = GlobalAutoIncBuffers::GetInstance()->get_auto_inc_buffer(
- _schema->db_id(), _schema->table_id(), kBinlogLsnAutoIncId);
+ _schema->db_id(), _schema->table_id(), kChangelogLsnAutoIncId);
}
_schema->set_timestamp_ms(state->timestamp_ms());
_schema->set_nano_seconds(state->nano_seconds());
@@ -2159,9 +2160,9 @@ Status VTabletWriter::_generate_one_index_channel_payload(
auto& tablet_ids = row_part_tablet_id.tablet_ids;
size_t row_cnt = row_ids.size();
- bool has_row_binlog = _schema->indexes()[index_idx]->row_binlog_id > 0;
+ bool has_changelog = _schema->indexes()[index_idx]->changelog_index_id > 0;
std::vector<int64_t> allocated_lsns;
- if (has_row_binlog && row_cnt > 0) {
+ if (has_changelog && row_cnt > 0) {
DCHECK(_allocated_lsn_buffer != nullptr);
RETURN_IF_ERROR(allocate_lsn(_allocated_lsn_buffer, row_cnt, allocated_lsns));
}
@@ -2184,13 +2185,13 @@ Status VTabletWriter::_generate_one_index_channel_payload(
payload_it = tmp_it;
payload_it->second.row_ids->reserve(row_cnt);
payload_it->second.route_idxs.reserve(row_cnt);
- if (has_row_binlog) {
+ if (has_changelog) {
payload_it->second.allocated_lsns.reserve(row_cnt);
}
}
payload_it->second.row_ids->push_back(row_ids[i]);
payload_it->second.route_idxs.push_back(cast_set<uint32_t>(i));
- if (has_row_binlog) {
+ if (has_changelog) {
payload_it->second.allocated_lsns.push_back(allocated_lsns[i]);
}
continue;
@@ -2215,13 +2216,13 @@ Status VTabletWriter::_generate_one_index_channel_payload(
payload_it = tmp_it;
payload_it->second.row_ids->reserve(row_cnt);
payload_it->second.route_idxs.reserve(row_cnt);
- if (has_row_binlog) {
+ if (has_changelog) {
payload_it->second.allocated_lsns.reserve(row_cnt);
}
}
payload_it->second.row_ids->push_back(row_ids[i]);
payload_it->second.route_idxs.push_back(cast_set<uint32_t>(i));
- if (has_row_binlog) {
+ if (has_changelog) {
payload_it->second.allocated_lsns.push_back(allocated_lsns[i]);
}
}
diff --git a/be/src/load/channel/tablets_channel.cpp b/be/src/load/channel/tablets_channel.cpp
index 01e19042245..b3c7f27e3eb 100644
--- a/be/src/load/channel/tablets_channel.cpp
+++ b/be/src/load/channel/tablets_channel.cpp
@@ -263,8 +263,8 @@ Status BaseTabletsChannel::incremental_open(const PTabletWriterOpenRequest& para
wrequest.write_file_cache = params.write_file_cache();
wrequest.storage_vault_id = params.storage_vault_id();
wrequest.enable_table_memtable_backpressure = params.is_adaptive_random_bucket();
- if (tablet.has_binlog_tablet_id()) {
- wrequest.binlog_tablet_id = tablet.binlog_tablet_id();
+ if (tablet.has_changelog_tablet_id()) {
+ wrequest.changelog_tablet_id = tablet.changelog_tablet_id();
}
auto delta_writer = create_delta_writer(wrequest);
@@ -320,24 +320,24 @@ std::unique_ptr<BaseDeltaWriter> TabletsChannel::create_delta_writer(const Write
DCHECK(request.write_req_type == WriteRequestType::DATA);
DCHECK(request.table_schema_param != nullptr);
- // whether to write binlog is decided by binlog_tablet_id: it is set only when this backend
- // also owns the binlog tablet that is paired with the base tablet.
- if (request.binlog_tablet_id <= 0) {
+ // whether to write changelog is decided by changelog_tablet_id: it is set only when this
+ // backend also owns the changelog tablet that is paired with the base tablet.
+ if (request.changelog_tablet_id <= 0) {
return std::make_unique<DeltaWriter>(_engine, request, _profile, _load_id);
}
- int64_t row_binlog_index_id = 0;
+ int64_t changelog_index_id = 0;
for (const auto* index_schema : request.table_schema_param->indexes()) {
if (index_schema->index_id == request.index_id) {
- row_binlog_index_id = index_schema->row_binlog_id;
+ changelog_index_id = index_schema->changelog_index_id;
break;
}
}
- DCHECK(row_binlog_index_id > 0);
+ DCHECK(changelog_index_id > 0);
- const auto* row_binlog_index_schema =
- request.table_schema_param->row_binlog_index_schema(row_binlog_index_id);
- DCHECK(row_binlog_index_schema != nullptr);
+ const auto* changelog_index_schema =
+ request.table_schema_param->changelog_index_schema(changelog_index_id);
+ DCHECK(changelog_index_schema != nullptr);
// group_build_req is only for the group wrapper itself. It provides the group semantics and
// metadata used by BaseDeltaWriter/GroupRowsetBuilder to expose tablet_id, txn_id,
@@ -349,13 +349,13 @@ std::unique_ptr<BaseDeltaWriter> TabletsChannel::create_delta_writer(const Write
WriteRequest sub_data_req = request;
sub_data_req.write_req_type = WriteRequestType::DATA;
- WriteRequest sub_row_binlog_req = request;
- sub_row_binlog_req.write_req_type = WriteRequestType::ROW_BINLOG;
- sub_row_binlog_req.tablet_id = request.binlog_tablet_id;
- sub_row_binlog_req.index_id = row_binlog_index_schema->index_id;
- sub_row_binlog_req.schema_hash = row_binlog_index_schema->schema_hash;
+ WriteRequest sub_changelog_req = request;
+ sub_changelog_req.write_req_type = WriteRequestType::CHANGELOG;
+ sub_changelog_req.tablet_id = request.changelog_tablet_id;
+ sub_changelog_req.index_id = changelog_index_schema->index_id;
+ sub_changelog_req.schema_hash = changelog_index_schema->schema_hash;
- return std::make_unique<DeltaWriter>(_engine, group_build_req, sub_data_req, sub_row_binlog_req,
+ return std::make_unique<DeltaWriter>(_engine, group_build_req, sub_data_req, sub_changelog_req,
_profile, _load_id);
}
@@ -531,13 +531,13 @@ void TabletsChannel::_commit_txn(DeltaWriter* writer, PTabletWriterAddBlockResul
tablet_info->set_schema_hash(0);
tablet_info->set_received_rows(writer->total_received_rows());
tablet_info->set_num_rows_filtered(writer->num_rows_filtered());
- // report the row binlog tablet as a normal tablet so FE advances its version.
- if (writer->binlog_tablet_id() > 0) {
- PTabletInfo* binlog_tablet_info = tablet_vec->Add();
- binlog_tablet_info->set_tablet_id(writer->binlog_tablet_id());
- binlog_tablet_info->set_schema_hash(0);
- binlog_tablet_info->set_received_rows(writer->total_received_rows());
- binlog_tablet_info->set_num_rows_filtered(writer->num_rows_filtered());
+ // report the changelog tablet as a normal tablet so FE advances its version.
+ if (writer->changelog_tablet_id() > 0) {
+ PTabletInfo* changelog_tablet_info = tablet_vec->Add();
+ changelog_tablet_info->set_tablet_id(writer->changelog_tablet_id());
+ changelog_tablet_info->set_schema_hash(0);
+ changelog_tablet_info->set_received_rows(writer->total_received_rows());
+ changelog_tablet_info->set_num_rows_filtered(writer->num_rows_filtered());
}
_total_received_rows += writer->total_received_rows();
_num_rows_filtered += writer->num_rows_filtered();
@@ -638,8 +638,8 @@ Status BaseTabletsChannel::_open_all_writers(const PTabletWriterOpenRequest& req
.enable_table_memtable_backpressure = request.is_adaptive_random_bucket(),
.load_cancel_status = _load_cancel_status,
};
- if (tablet.has_binlog_tablet_id()) {
- wrequest.binlog_tablet_id = tablet.binlog_tablet_id();
+ if (tablet.has_changelog_tablet_id()) {
+ wrequest.changelog_tablet_id = tablet.changelog_tablet_id();
}
auto delta_writer = create_delta_writer(wrequest);
diff --git a/be/src/load/delta_writer/delta_writer.cpp b/be/src/load/delta_writer/delta_writer.cpp
index 84504805e1e..e352778a1e5 100644
--- a/be/src/load/delta_writer/delta_writer.cpp
+++ b/be/src/load/delta_writer/delta_writer.cpp
@@ -70,14 +70,14 @@ DeltaWriter::DeltaWriter(StorageEngine& engine, const WriteRequest& req, Runtime
}
DeltaWriter::DeltaWriter(StorageEngine& engine, const WriteRequest& group_build_req,
- const WriteRequest& sub_data_req, const WriteRequest& sub_row_binlog_req,
+ const WriteRequest& sub_data_req, const WriteRequest& sub_changelog_req,
RuntimeProfile* profile, const UniqueId& load_id)
: BaseDeltaWriter(group_build_req, profile, load_id), _engine(engine) {
DCHECK(group_build_req.write_req_type == WriteRequestType::GROUP &&
sub_data_req.write_req_type == WriteRequestType::DATA &&
- sub_row_binlog_req.write_req_type == WriteRequestType::ROW_BINLOG);
+ sub_changelog_req.write_req_type == WriteRequestType::CHANGELOG);
_rowset_builder = std::make_unique<GroupRowsetBuilder>(_engine, group_build_req, sub_data_req,
- sub_row_binlog_req, profile);
+ sub_changelog_req, profile);
}
void BaseDeltaWriter::_init_profile(RuntimeProfile* profile) {
diff --git a/be/src/load/delta_writer/delta_writer.h b/be/src/load/delta_writer/delta_writer.h
index a405349161a..c76450cb7f5 100644
--- a/be/src/load/delta_writer/delta_writer.h
+++ b/be/src/load/delta_writer/delta_writer.h
@@ -89,7 +89,7 @@ public:
int64_t tablet_id() const { return _req.tablet_id; }
- int64_t binlog_tablet_id() const { return _req.binlog_tablet_id; }
+ int64_t changelog_tablet_id() const { return _req.changelog_tablet_id; }
int64_t txn_id() const { return _req.txn_id; }
@@ -133,7 +133,7 @@ public:
DeltaWriter(StorageEngine& engine, const WriteRequest& req, RuntimeProfile* profile,
const UniqueId& load_id);
DeltaWriter(StorageEngine& engine, const WriteRequest& group_build_req,
- const WriteRequest& sub_data_req, const WriteRequest& sub_row_binlog_req,
+ const WriteRequest& sub_data_req, const WriteRequest& sub_changelog_req,
RuntimeProfile* profile, const UniqueId& load_id);
~DeltaWriter() override;
diff --git a/be/src/load/delta_writer/delta_writer_context.h b/be/src/load/delta_writer/delta_writer_context.h
index 88de1a171bf..e977b743999 100644
--- a/be/src/load/delta_writer/delta_writer_context.h
+++ b/be/src/load/delta_writer/delta_writer_context.h
@@ -33,9 +33,9 @@ class OlapTableSchemaParam;
class AtomicStatus;
enum class WriteRequestType {
- DATA = 0, // data write
- ROW_BINLOG = 1, // row binlog write
- GROUP = 2, // group write for data and binlog<row>
+ DATA = 0, // data write
+ CHANGELOG = 1, // changelog write
+ GROUP = 2, // group write for data and changelog
};
struct WriteRequest {
@@ -45,8 +45,8 @@ struct WriteRequest {
int64_t txn_expiration = 0; // For cloud mode
int64_t index_id = 0;
int64_t partition_id = 0;
- // the row binlog tablet id written together with this base tablet, 0 means no binlog.
- int64_t binlog_tablet_id = 0;
+ // the changelog tablet id written together with this base tablet, 0 means no changelog.
+ int64_t changelog_tablet_id = 0;
PUniqueId load_id;
TupleDescriptor* tuple_desc = nullptr;
// slots are in order of tablet's schema
diff --git a/be/src/load/memtable/memtable_flush_executor.cpp b/be/src/load/memtable/memtable_flush_executor.cpp
index 6345e5121e1..5317cfeeaf6 100644
--- a/be/src/load/memtable/memtable_flush_executor.cpp
+++ b/be/src/load/memtable/memtable_flush_executor.cpp
@@ -33,7 +33,7 @@
#include "runtime/thread_context.h"
#include "runtime/workload_group/workload_group.h"
#include "runtime/workload_management/resource_context.h"
-#include "storage/binlog.h"
+#include "storage/changelog.h"
#include "storage/rowset/group_rowset_writer.h"
#include "storage/rowset/rowset_writer.h"
#include "storage/storage_engine.h"
@@ -137,7 +137,7 @@ Status FlushToken::_submit_sub_tasks(ThreadPool* pool,
std::shared_lock rdlk(_flush_status_lock);
DBUG_EXECUTE_IF("FlushToken.submit_sub_task_error", {
if (i != 0) {
- // only affect flush binlog task
+ // only affect flush changelog task
_flush_status = Status::IOError<false>("dbug_be_memtable_submit_flush_error");
}
});
@@ -186,23 +186,23 @@ Status FlushToken::submit(std::shared_ptr<MemTable> mem_table) {
std::vector<std::shared_ptr<Runnable>> tasks;
if (group_rowset_writer != nullptr) {
auto data_writer = group_rowset_writer->data_writer();
- auto binlog_writer = group_rowset_writer->row_binlog_writer();
+ auto changelog_writer = group_rowset_writer->changelog_writer();
DCHECK(data_writer != nullptr);
- DCHECK(binlog_writer != nullptr);
+ DCHECK(changelog_writer != nullptr);
shared_memtable = std::make_shared<SharedMemtable>();
shared_memtable->memtable = mem_table;
shared_memtable->allocated_lsn_map = group_rowset_writer->context().allocated_lsn_map;
- // Keep data/binlog segment_id allocators in sync.
+ // Keep data/changelog segment_id allocators in sync.
auto segment_id = DORIS_TRY(data_writer->allocate_segment_id());
- auto binlog_segment_id = DORIS_TRY(binlog_writer->allocate_segment_id());
- DCHECK_EQ(segment_id, binlog_segment_id);
+ auto changelog_segment_id = DORIS_TRY(changelog_writer->allocate_segment_id());
+ DCHECK_EQ(segment_id, changelog_segment_id);
shared_memtable->segment_id = segment_id;
tasks.emplace_back(PartOfGroupMemtableFlushTask::create_shared(
shared_from_this(), shared_memtable, WriteRequestType::DATA, submit_task_time));
tasks.emplace_back(PartOfGroupMemtableFlushTask::create_shared(
- shared_from_this(), shared_memtable, WriteRequestType::ROW_BINLOG,
+ shared_from_this(), shared_memtable, WriteRequestType::CHANGELOG,
submit_task_time));
} else {
auto segment_id = DORIS_TRY(_rowset_writer->allocate_segment_id());
@@ -227,13 +227,13 @@ void FlushToken::_flush_group_memtable(std::shared_ptr<SharedMemtable> shared_me
DCHECK(shared_memtable != nullptr);
DCHECK(shared_memtable->memtable != nullptr);
DCHECK(write_req_type == WriteRequestType::DATA ||
- write_req_type == WriteRequestType::ROW_BINLOG);
+ write_req_type == WriteRequestType::CHANGELOG);
auto* group_rowset_writer = typeid_cast<GroupRowsetWriter*>(_rowset_writer.get());
DCHECK(group_rowset_writer != nullptr);
auto flush_writer = write_req_type == WriteRequestType::DATA
? group_rowset_writer->data_writer()
- : group_rowset_writer->row_binlog_writer();
+ : group_rowset_writer->changelog_writer();
DCHECK(flush_writer != nullptr);
_flush_memtable_impl(flush_writer.get(), shared_memtable->memtable.get(),
shared_memtable->segment_id, submit_task_time, shared_memtable.get());
diff --git a/be/src/load/memtable/memtable_flush_executor.h b/be/src/load/memtable/memtable_flush_executor.h
index 3c707439b00..12aebfa1fc9 100644
--- a/be/src/load/memtable/memtable_flush_executor.h
+++ b/be/src/load/memtable/memtable_flush_executor.h
@@ -28,7 +28,7 @@
#include "common/status.h"
#include "load/delta_writer/delta_writer_context.h"
#include "load/memtable/memtable.h"
-#include "storage/binlog.h"
+#include "storage/changelog.h"
#include "util/threadpool.h"
namespace doris {
@@ -74,7 +74,7 @@ struct SharedMemtable {
bool has_allocated_lsns = false;
std::atomic<int> finished_sub_task_count {0};
- // data + binlog
+ // data + changelog
std::atomic<int> total_sub_task_count {2};
int add_finished_sub_task() { return finished_sub_task_count.fetch_add(1); }
diff --git a/be/src/load/memtable/memtable_writer.cpp b/be/src/load/memtable/memtable_writer.cpp
index f05e5a5cde4..c4e7be7bafa 100644
--- a/be/src/load/memtable/memtable_writer.cpp
+++ b/be/src/load/memtable/memtable_writer.cpp
@@ -83,12 +83,12 @@ Status MemTableWriter::init(std::shared_ptr<RowsetWriter> rowset_writer,
_unique_key_mow = unique_key_mow;
_partial_update_info = partial_update_info;
_resource_ctx = thread_context()->resource_ctx();
- // Only the row-binlog base index needs LSN, aligned with the sink (row_binlog_id > 0).
+ // Only the changelog base index needs LSN, aligned with the sink (changelog_index_id > 0).
_need_lsn = false;
if (_req.table_schema_param != nullptr) {
for (const auto* index_schema : _req.table_schema_param->indexes()) {
if (index_schema->index_id == _req.index_id) {
- _need_lsn = index_schema->row_binlog_id > 0;
+ _need_lsn = index_schema->changelog_index_id > 0;
break;
}
}
@@ -97,7 +97,7 @@ Status MemTableWriter::init(std::shared_ptr<RowsetWriter> rowset_writer,
const auto keys_type = _tablet_schema->keys_type();
if (keys_type == KeysType::AGG_KEYS ||
(keys_type == KeysType::UNIQUE_KEYS && !_unique_key_mow)) {
- // Row-binlog LSN sidecar does not support MemTable aggregation now. For AGG tables
+ // Changelog LSN sidecar does not support MemTable aggregation now. For AGG tables
// and unique key merge-on-read tables, multiple input rows can be merged into one
// output row in MemTable, so their output LSN semantics should be implemented
// explicitly before enabling allocated LSNs on these table types.
diff --git a/be/src/runtime/cluster_info.h b/be/src/runtime/cluster_info.h
index c1907e13687..e808ee94696 100644
--- a/be/src/runtime/cluster_info.h
+++ b/be/src/runtime/cluster_info.h
@@ -29,15 +29,15 @@ namespace doris {
// These info are usually in heartbeat from Master FE.
class ClusterInfo {
public:
- int64_t row_binlog_ttl_reference_tso() const {
- return _row_binlog_ttl_reference_tso.load(std::memory_order_acquire);
+ int64_t changelog_ttl_reference_tso() const {
+ return _changelog_ttl_reference_tso.load(std::memory_order_acquire);
}
// Heartbeat identity validation must precede this call. Never persist this value:
// after a restart cleanup waits for a fresh, authenticated master heartbeat.
- void advance_row_binlog_ttl_reference_tso(int64_t tso) {
- auto previous = _row_binlog_ttl_reference_tso.load(std::memory_order_acquire);
- while (tso > previous && !_row_binlog_ttl_reference_tso.compare_exchange_weak(
+ void advance_changelog_ttl_reference_tso(int64_t tso) {
+ auto previous = _changelog_ttl_reference_tso.load(std::memory_order_acquire);
+ while (tso > previous && !_changelog_ttl_reference_tso.compare_exchange_weak(
previous, tso, std::memory_order_acq_rel)) {
}
}
@@ -59,7 +59,7 @@ public:
std::string last_auth_token = "";
private:
- std::atomic<int64_t> _row_binlog_ttl_reference_tso {0};
+ std::atomic<int64_t> _changelog_ttl_reference_tso {0};
};
} // namespace doris
diff --git a/be/src/runtime/query_cache/query_cache.cpp b/be/src/runtime/query_cache/query_cache.cpp
index d7e042e0301..14879dff950 100644
--- a/be/src/runtime/query_cache/query_cache.cpp
+++ b/be/src/runtime/query_cache/query_cache.cpp
@@ -190,8 +190,8 @@ std::shared_ptr<QueryCacheInstanceDecision> QueryCacheRuntime::get_or_make_decis
void QueryCacheRuntime::_make_decision(const std::vector<TScanRangeParams>& scan_ranges,
QueryCacheInstanceDecision* decision) {
decision->mode = QueryCacheInstanceDecision::Mode::MISS;
- if (_binlog_scan) {
- // A row-binlog scan reads a different data stream under the same plan
+ if (_changelog_scan) {
+ // A changelog scan reads a different data stream under the same plan
// digest: it must neither serve cached blocks nor write its output
// back (that would poison the cache for normal queries).
decision->key_valid = false;
diff --git a/be/src/runtime/query_cache/query_cache.h b/be/src/runtime/query_cache/query_cache.h
index ed353c08ad7..d28d03f25a9 100644
--- a/be/src/runtime/query_cache/query_cache.h
+++ b/be/src/runtime/query_cache/query_cache.h
@@ -344,10 +344,10 @@ public:
QueryCache* cache() const { return _cache; }
- // Row-binlog scans read a different data stream and must not serve or fill
+ // Changelog scans read a different data stream and must not serve or fill
// the query cache. Called while building the operator tree (single
// threaded, before any local state init), so no locking is needed.
- void disable_for_binlog_scan() { _binlog_scan = true; }
+ void disable_for_changelog_scan() { _changelog_scan = true; }
// Idempotent: the first call for a given instance (identified by the cache
// key derived from its scan ranges) makes the decision, later calls return
@@ -393,7 +393,7 @@ private:
TQueryCacheParam _param;
QueryCache* _cache = nullptr;
- bool _binlog_scan = false;
+ bool _changelog_scan = false;
std::mutex _lock;
std::map<std::string, std::shared_ptr<QueryCacheInstanceDecision>> _decisions;
diff --git a/be/src/service/http/action/compaction_action.cpp b/be/src/service/http/action/compaction_action.cpp
index c86dd30d5dc..8ceb7c0130f 100644
--- a/be/src/service/http/action/compaction_action.cpp
+++ b/be/src/service/http/action/compaction_action.cpp
@@ -129,12 +129,12 @@ Status CompactionAction::_handle_run_compaction(HttpRequest* req, std::string* j
// check compaction_type
std::string compaction_type = req->param(PARAM_COMPACTION_TYPE);
- if (compaction_type == PARAM_COMPACTION_ROW_BINLOG_TTL) {
+ if (compaction_type == PARAM_COMPACTION_CHANGELOG_TTL) {
if (tablet_id == 0 || table_id != 0) {
- return Status::InvalidArgument("row_binlog_ttl requires a tablet_id");
+ return Status::InvalidArgument("changelog_ttl requires a tablet_id");
}
- RETURN_IF_ERROR(_engine.submit_row_binlog_ttl(tablet_id));
- *json_result = R"({"status":"Success","msg":"ROW binlog TTL task queued"})";
+ RETURN_IF_ERROR(_engine.submit_changelog_ttl(tablet_id));
+ *json_result = R"({"status":"Success","msg":"changelog TTL task queued"})";
return Status::OK();
}
if (compaction_type != PARAM_COMPACTION_BASE &&
diff --git a/be/src/service/http/action/compaction_action.h b/be/src/service/http/action/compaction_action.h
index 34463380740..02643d465a2 100644
--- a/be/src/service/http/action/compaction_action.h
+++ b/be/src/service/http/action/compaction_action.h
@@ -40,7 +40,8 @@ const std::string PARAM_COMPACTION_TYPE = "compact_type";
const std::string PARAM_COMPACTION_BASE = "base";
const std::string PARAM_COMPACTION_CUMULATIVE = "cumulative";
const std::string PARAM_COMPACTION_FULL = "full";
-const std::string PARAM_COMPACTION_ROW_BINLOG_TTL = "row_binlog_ttl";
+// ATTN(changelog-rename): formerly "row_binlog_ttl"
+const std::string PARAM_COMPACTION_CHANGELOG_TTL = "changelog_ttl";
const std::string PARAM_COMPACTION_FORCE = "force";
/// This action is used for viewing the compaction status.
@@ -61,7 +62,7 @@ private:
/// param compact_type in req to distinguish the task type, base/cumulative/full
Status _handle_run_compaction(HttpRequest* req, std::string* json_result);
- /// thread callback function for the tablet to do base/cumulative/binlog compaction
+ /// thread callback function for the tablet to do base/cumulative/changelog compaction
Status _execute_compaction_callback(TabletSharedPtr tablet, const std::string& compaction_type);
/// fetch compaction running status
diff --git a/be/src/storage/binlog.h b/be/src/storage/binlog.h
index 7c37c4457be..d5507220ba2 100644
--- a/be/src/storage/binlog.h
+++ b/be/src/storage/binlog.h
@@ -19,54 +19,19 @@
#include <fmt/format.h>
-#include <limits>
-#include <map>
-#include <memory>
-#include <mutex>
+#include <cstdint>
+#include <cstdlib>
#include <string>
#include <string_view>
-#include <utility>
-#include <vector>
-
-#include "common/logging.h" // DCHECK
-#include "common/status.h"
-#include "exec/sink/autoinc_buffer.h"
-#include "storage/olap_common.h"
-#include "storage/olap_define.h" // DataWriteType
-#include "storage/rowset/rowset_fwd.h"
-#include "storage/tablet/tablet_fwd.h" // BaseTabletSPtr
-#include "storage/tablet/tablet_schema.h" // TabletSchemaSPtr
-#include "storage/utils.h" // BINLOG_*_COL
-namespace doris {
-
-class AutoIncIDBuffer;
-struct PartialUpdateInfo;
-struct MowContext;
+#include "storage/olap_common.h" // TabletUid, RowsetId
-// Row binlog op type.
-// NOTE: The value is persisted into row binlog data, so keep it stable.
-static constexpr int64_t ROW_BINLOG_APPEND = 0;
-static constexpr int64_t ROW_BINLOG_UPDATE = 1;
-static constexpr int64_t ROW_BINLOG_DELETE = 2;
+namespace doris {
constexpr std::string_view kBinlogPrefix = "binlog_";
constexpr std::string_view kBinlogMetaPrefix = "binlog_meta_";
constexpr std::string_view kBinlogDataPrefix = "binlog_data_";
-namespace binlog {
-
-inline std::string build_before_column_name(std::string_view name) {
- std::string before_name = "__BEFORE__";
- before_name.append(name.data(), name.size());
- before_name.append("__");
- return before_name;
-}
-
-} // namespace binlog
-
-constexpr int64_t kBinlogLsnAutoIncId = -1;
-
inline auto make_binlog_meta_key(const std::string_view tablet, int64_t version,
const std::string_view rowset) {
return fmt::format("{}meta_{}_{:020d}_{}", kBinlogPrefix, tablet, version, rowset);
@@ -134,106 +99,4 @@ inline std::string get_binlog_data_key_from_meta_key(const std::string_view meta
return fmt::format("{}data_{}", kBinlogPrefix, meta_key.substr(kBinlogMetaPrefix.length()));
}
-// Allocate per-row LSNs.
-// The caller must provide a valid auto-inc buffer (typically from GlobalAutoIncBuffers).
-inline Status allocate_lsn(const std::shared_ptr<AutoIncIDBuffer>& lsn_buffer, size_t num_rows,
- std::vector<int64_t>& lsn_ids) {
- if (lsn_buffer == nullptr) {
- return Status::InternalError("try to get lsn buffer but null");
- }
- DCHECK(num_rows > 0);
-
- std::vector<std::pair<int64_t, size_t>> ranges;
- RETURN_IF_ERROR(lsn_buffer->sync_request_ids(num_rows, &ranges));
-
- lsn_ids.clear();
- lsn_ids.reserve(num_rows);
- for (const auto& [start, length] : ranges) {
- for (size_t i = 0; i < length; ++i) {
- DCHECK_LE(start, std::numeric_limits<int64_t>::max() - static_cast<int64_t>(i));
- lsn_ids.push_back(start + static_cast<int64_t>(i));
- }
- }
- DCHECK_EQ(lsn_ids.size(), num_rows);
- return Status::OK();
-}
-
-constexpr int64_t kTsoLogicalBits = 18;
-
-inline int64_t extract_tso_physical_time(int64_t tso) {
- return tso <= 0 ? 0 : tso >> kTsoLogicalBits;
-}
-
-inline int64_t row_binlog_ttl_cutoff_tso(int64_t reference_tso, int64_t ttl_seconds) {
- DCHECK_GT(reference_tso, 0);
- DCHECK_GE(ttl_seconds, 0);
- const int64_t reference_ms = extract_tso_physical_time(reference_tso);
- if (ttl_seconds > reference_ms / 1000) {
- return 0;
- }
- constexpr int64_t kMaxLogical = (1L << kTsoLogicalBits) - 1;
- return ((reference_ms - ttl_seconds * 1000) << kTsoLogicalBits) | kMaxLogical;
-}
-
-namespace segment_v2 {
-
-class SegmentAllocatedLsnMap {
-public:
- void insert_segment_allocated_lsns(int64_t seg_id, ConstAllocatedLsnVectorSharedPtr lsn_ids) {
- std::lock_guard<std::mutex> l(_mutex);
- _seg_id_to_lsn_ids.emplace(seg_id, std::move(lsn_ids));
- }
-
- void remove_segment(int64_t seg_id) {
- std::lock_guard<std::mutex> l(_mutex);
- _seg_id_to_lsn_ids.erase(seg_id);
- }
-
- bool contains_segment(int64_t seg_id) const {
- std::lock_guard<std::mutex> l(_mutex);
- return _seg_id_to_lsn_ids.count(seg_id) > 0;
- }
-
- ConstAllocatedLsnVectorSharedPtr get_segment_allocated_lsns(int64_t seg_id) const {
- std::lock_guard<std::mutex> l(_mutex);
- auto it = _seg_id_to_lsn_ids.find(seg_id);
- DCHECK(it != _seg_id_to_lsn_ids.end())
- << "SegmentAllocatedLsnMap::get_segment_allocated_lsns missing seg_id=" << seg_id
- << ", existing_seg_ids=[" << ([&] {
- std::string s;
- for (const auto& [id, _] : _seg_id_to_lsn_ids) {
- if (!s.empty()) {
- s.push_back(',');
- }
- s.append(std::to_string(id));
- }
- return s;
- }())
- << "]";
- return it->second;
- }
-
-private:
- mutable std::mutex _mutex;
- std::map<int64_t, ConstAllocatedLsnVectorSharedPtr> _seg_id_to_lsn_ids;
-};
-
-struct SegmentWriteBinlogOptions {
-public:
- bool write_before = false;
-
- // source context, used for retrieving historical row and building binlog<row> block
- struct SourceWriteDataOptions {
- BaseTabletSPtr base_tablet = nullptr;
- TabletSchemaSPtr tablet_schema = nullptr;
- std::shared_ptr<PartialUpdateInfo> partial_update_info;
- std::shared_ptr<MowContext> mow_context;
- bool is_transient_rowset_writer = false;
- DataWriteType source_write_type = DataWriteType::TYPE_DEFAULT;
- // input rowset's row-binlog, read LSN from it at publish time
- RowsetSharedPtr row_binlog_rowset = nullptr;
- } source;
-};
-} // namespace segment_v2
-
} // namespace doris
diff --git a/be/src/storage/binlog_config.cpp b/be/src/storage/binlog_config.cpp
index 33b397be9ca..5f91c176c99 100644
--- a/be/src/storage/binlog_config.cpp
+++ b/be/src/storage/binlog_config.cpp
@@ -22,23 +22,23 @@
#include <gen_cpp/olap_file.pb.h>
#include "common/logging.h"
-#include "storage/binlog.h"
+#include "storage/changelog.h"
#include "storage/rowset/rowset_meta.h"
namespace doris {
-bool row_binlog_rowset_expired(const RowsetMeta& meta, int64_t cutoff) {
+bool changelog_rowset_expired(const RowsetMeta& meta, int64_t cutoff) {
return cutoff >= 0 && meta.num_rows() > 0 && meta.has_commit_tso() &&
meta.commit_tso().start_tso() > 0 &&
meta.commit_tso().end_tso() >= meta.commit_tso().start_tso() &&
meta.commit_tso().end_tso() <= cutoff;
}
-int64_t BinlogConfig::row_ttl_cutoff_tso(int64_t reference_tso) const {
- if (!has_row_ttl() || reference_tso <= 0) {
+int64_t BinlogConfig::changelog_ttl_cutoff_tso(int64_t reference_tso) const {
+ if (!has_changelog_ttl() || reference_tso <= 0) {
return -1;
}
- return doris::row_binlog_ttl_cutoff_tso(reference_tso, _ttl_seconds);
+ return doris::changelog_ttl_cutoff_tso(reference_tso, _ttl_seconds);
}
BinlogConfig& BinlogConfig::operator=(const TBinlogConfig& config) {
@@ -58,8 +58,8 @@ BinlogConfig& BinlogConfig::operator=(const TBinlogConfig& config) {
_max_history_nums = config.max_history_nums;
}
if (config.__isset.binlog_format) {
- if (config.binlog_format == TBinlogFormat::ROW) {
- _binlog_format = BinlogFormatPB::ROW;
+ if (config.binlog_format == TBinlogFormat::CHANGELOG) {
+ _binlog_format = BinlogFormatPB::CHANGELOG;
} else if (config.binlog_format == TBinlogFormat::STATEMENT_AND_SNAPSHOT) {
_binlog_format = BinlogFormatPB::STATEMENT_AND_SNAPSHOT;
} else {
diff --git a/be/src/storage/binlog_config.h b/be/src/storage/binlog_config.h
index 04bc5b1b9a3..f4899976815 100644
--- a/be/src/storage/binlog_config.h
+++ b/be/src/storage/binlog_config.h
@@ -30,7 +30,7 @@ class TBinlogConfig;
class BinlogConfigPB;
class RowsetMeta;
-bool row_binlog_rowset_expired(const RowsetMeta& meta, int64_t cutoff);
+bool changelog_rowset_expired(const RowsetMeta& meta, int64_t cutoff);
class BinlogConfig {
public:
@@ -72,13 +72,13 @@ public:
void set_need_historical_value(bool need_historical_value) {
_need_historical_value = need_historical_value;
}
- bool has_row_ttl() const { return _enable && is_row_binlog_format() && _ttl_seconds > 0; }
- int64_t row_ttl_cutoff_tso(int64_t reference_tso) const;
+ bool has_changelog_ttl() const { return _enable && is_changelog() && _ttl_seconds > 0; }
+ int64_t changelog_ttl_cutoff_tso(int64_t reference_tso) const;
bool is_ccr_binlog_format() const {
return _binlog_format == BinlogFormatPB::STATEMENT_AND_SNAPSHOT;
}
- bool is_row_binlog_format() const { return _binlog_format == BinlogFormatPB::ROW; }
+ bool is_changelog() const { return _binlog_format == BinlogFormatPB::CHANGELOG; }
BinlogConfig& operator=(const TBinlogConfig& config);
BinlogConfig& operator=(const BinlogConfigPB& config);
diff --git a/be/src/storage/binlog.h b/be/src/storage/changelog.h
similarity index 56%
copy from be/src/storage/binlog.h
copy to be/src/storage/changelog.h
index 7c37c4457be..587e75f8b7f 100644
--- a/be/src/storage/binlog.h
+++ b/be/src/storage/changelog.h
@@ -17,8 +17,6 @@
#pragma once
-#include <fmt/format.h>
-
#include <limits>
#include <map>
#include <memory>
@@ -36,7 +34,7 @@
#include "storage/rowset/rowset_fwd.h"
#include "storage/tablet/tablet_fwd.h" // BaseTabletSPtr
#include "storage/tablet/tablet_schema.h" // TabletSchemaSPtr
-#include "storage/utils.h" // BINLOG_*_COL
+#include "storage/utils.h" // CHANGELOG_*_COL
namespace doris {
@@ -44,95 +42,25 @@ class AutoIncIDBuffer;
struct PartialUpdateInfo;
struct MowContext;
-// Row binlog op type.
-// NOTE: The value is persisted into row binlog data, so keep it stable.
-static constexpr int64_t ROW_BINLOG_APPEND = 0;
-static constexpr int64_t ROW_BINLOG_UPDATE = 1;
-static constexpr int64_t ROW_BINLOG_DELETE = 2;
-
-constexpr std::string_view kBinlogPrefix = "binlog_";
-constexpr std::string_view kBinlogMetaPrefix = "binlog_meta_";
-constexpr std::string_view kBinlogDataPrefix = "binlog_data_";
+// Changelog op type.
+// NOTE: The value is persisted into changelog data, so keep it stable.
+static constexpr int64_t CHANGELOG_OP_APPEND = 0;
+static constexpr int64_t CHANGELOG_OP_UPDATE = 1;
+static constexpr int64_t CHANGELOG_OP_DELETE = 2;
-namespace binlog {
+namespace changelog {
+// ATTN(changelog-rename): formerly "__BEFORE__<col>__"
inline std::string build_before_column_name(std::string_view name) {
- std::string before_name = "__BEFORE__";
+ std::string before_name = "__DORIS_CHANGELOG_BEFORE_";
before_name.append(name.data(), name.size());
before_name.append("__");
return before_name;
}
-} // namespace binlog
-
-constexpr int64_t kBinlogLsnAutoIncId = -1;
-
-inline auto make_binlog_meta_key(const std::string_view tablet, int64_t version,
- const std::string_view rowset) {
- return fmt::format("{}meta_{}_{:020d}_{}", kBinlogPrefix, tablet, version, rowset);
-}
-
-inline auto make_binlog_meta_key(const std::string_view tablet, const std::string_view version_str,
- const std::string_view rowset) {
- // TODO(Drogon): use fmt::format not convert to version_num, only string with length prefix '0'
- int64_t version = std::atoll(version_str.data());
- return make_binlog_meta_key(tablet, version, rowset);
-}
-
-inline auto make_binlog_meta_key(const TabletUid& tablet_uid, int64_t version,
- const RowsetId& rowset_id) {
- return make_binlog_meta_key(tablet_uid.to_string(), version, rowset_id.to_string());
-}
-
-inline auto make_binlog_meta_key_prefix(const TabletUid& tablet_uid) {
- return fmt::format("{}meta_{}_", kBinlogPrefix, tablet_uid.to_string());
-}
-
-inline auto make_binlog_meta_key_prefix(const TabletUid& tablet_uid, int64_t version) {
- return fmt::format("{}meta_{}_{:020d}_", kBinlogPrefix, tablet_uid.to_string(), version);
-}
-
-inline auto make_binlog_data_key(const std::string_view tablet, int64_t version,
- const std::string_view rowset) {
- return fmt::format("{}data_{}_{:020d}_{}", kBinlogPrefix, tablet, version, rowset);
-}
-
-inline auto make_binlog_data_key(const std::string_view tablet, const std::string_view version,
- const std::string_view rowset) {
- return fmt::format("{}data_{}_{:0>20}_{}", kBinlogPrefix, tablet, version, rowset);
-}
-
-inline auto make_binlog_data_key(const TabletUid& tablet_uid, int64_t version,
- const RowsetId& rowset_id) {
- return make_binlog_data_key(tablet_uid.to_string(), version, rowset_id.to_string());
-}
-
-inline auto make_binlog_data_key(const TabletUid& tablet_uid, int64_t version,
- const std::string_view rowset_id) {
- return make_binlog_data_key(tablet_uid.to_string(), version, rowset_id);
-}
-
-inline auto make_binlog_data_key_prefix(const TabletUid& tablet_uid, int64_t version) {
- return fmt::format("{}data_{}_{:020d}_", kBinlogPrefix, tablet_uid.to_string(), version);
-}
+} // namespace changelog
-inline auto make_binlog_filename_key(const TabletUid& tablet_uid, const std::string_view version) {
- return fmt::format("{}meta_{}_{:0>20}_", kBinlogPrefix, tablet_uid.to_string(), version);
-}
-
-inline bool starts_with_binlog_meta(const std::string_view str) {
- auto prefix = kBinlogMetaPrefix;
- if (prefix.length() > str.length()) {
- return false;
- }
-
- return str.compare(0, prefix.length(), prefix) == 0;
-}
-
-inline std::string get_binlog_data_key_from_meta_key(const std::string_view meta_key) {
- // like "binlog_meta_6943f1585fe834b5-e542c2b83a21d0b7" => "binlog_data-6943f1585fe834b5-e542c2b83a21d0b7"
- return fmt::format("{}data_{}", kBinlogPrefix, meta_key.substr(kBinlogMetaPrefix.length()));
-}
+constexpr int64_t kChangelogLsnAutoIncId = -1;
// Allocate per-row LSNs.
// The caller must provide a valid auto-inc buffer (typically from GlobalAutoIncBuffers).
@@ -164,7 +92,7 @@ inline int64_t extract_tso_physical_time(int64_t tso) {
return tso <= 0 ? 0 : tso >> kTsoLogicalBits;
}
-inline int64_t row_binlog_ttl_cutoff_tso(int64_t reference_tso, int64_t ttl_seconds) {
+inline int64_t changelog_ttl_cutoff_tso(int64_t reference_tso, int64_t ttl_seconds) {
DCHECK_GT(reference_tso, 0);
DCHECK_GE(ttl_seconds, 0);
const int64_t reference_ms = extract_tso_physical_time(reference_tso);
@@ -218,11 +146,11 @@ private:
std::map<int64_t, ConstAllocatedLsnVectorSharedPtr> _seg_id_to_lsn_ids;
};
-struct SegmentWriteBinlogOptions {
+struct SegmentWriteChangelogOptions {
public:
bool write_before = false;
- // source context, used for retrieving historical row and building binlog<row> block
+ // source context, used for retrieving historical row and building changelog block
struct SourceWriteDataOptions {
BaseTabletSPtr base_tablet = nullptr;
TabletSchemaSPtr tablet_schema = nullptr;
@@ -230,8 +158,8 @@ public:
std::shared_ptr<MowContext> mow_context;
bool is_transient_rowset_writer = false;
DataWriteType source_write_type = DataWriteType::TYPE_DEFAULT;
- // input rowset's row-binlog, read LSN from it at publish time
- RowsetSharedPtr row_binlog_rowset = nullptr;
+ // input rowset's changelog rowset, read LSN from it at publish time
+ RowsetSharedPtr changelog_rowset = nullptr;
} source;
};
} // namespace segment_v2
diff --git a/be/src/storage/row_binlog_ttl_scanner.cpp b/be/src/storage/changelog_ttl_scanner.cpp
similarity index 61%
rename from be/src/storage/row_binlog_ttl_scanner.cpp
rename to be/src/storage/changelog_ttl_scanner.cpp
index 42bac49cf54..17ab7f1c0b0 100644
--- a/be/src/storage/row_binlog_ttl_scanner.cpp
+++ b/be/src/storage/changelog_ttl_scanner.cpp
@@ -34,15 +34,18 @@
namespace doris {
namespace {
-bvar::Adder<int64_t> ttl_discovered("row_binlog_ttl_discovered");
-bvar::Adder<int64_t> ttl_submitted("row_binlog_ttl_submitted");
-bvar::Adder<int64_t> ttl_failed("row_binlog_ttl_failed");
+// ATTN(changelog-rename): formerly "row_binlog_ttl_discovered"
+bvar::Adder<int64_t> ttl_discovered("changelog_ttl_discovered");
+// ATTN(changelog-rename): formerly "row_binlog_ttl_submitted"
+bvar::Adder<int64_t> ttl_submitted("changelog_ttl_submitted");
+// ATTN(changelog-rename): formerly "row_binlog_ttl_failed"
+bvar::Adder<int64_t> ttl_failed("changelog_ttl_failed");
constexpr int kScanBatchSize = 64;
-Status prepare_row_binlog_ttl(BaseStorageEngine& engine, int64_t tablet_id, bool refresh_cloud_meta,
- bool is_cloud) {
+Status prepare_changelog_ttl(BaseStorageEngine& engine, int64_t tablet_id, bool refresh_cloud_meta,
+ bool is_cloud) {
// Cloud catalog discovery and already queued work must also honor runtime switches.
- if (!config::enable_feature_binlog || config::disable_auto_compaction) {
+ if (!config::enable_feature_changelog || config::disable_auto_compaction) {
return Status::OK();
}
auto tablet = DORIS_TRY(engine.get_tablet(tablet_id));
@@ -51,7 +54,7 @@ Status prepare_row_binlog_ttl(BaseStorageEngine& engine, int64_t tablet_id, bool
RETURN_IF_ERROR(cloud_tablet->sync_meta());
RETURN_IF_ERROR(cloud_tablet->sync_rowsets());
}
- if (tablet->tablet_state() != TABLET_RUNNING || !tablet->scan_expired_row_binlog_rowsets()) {
+ if (tablet->tablet_state() != TABLET_RUNNING || !tablet->scan_expired_changelog_rowsets()) {
return Status::OK();
}
ttl_discovered << 1;
@@ -70,10 +73,10 @@ Status prepare_row_binlog_ttl(BaseStorageEngine& engine, int64_t tablet_id, bool
return Status::OK();
}
RETURN_IF_ERROR(engine.to_cloud().submit_compaction_task(
- cloud_tablet, CompactionType::CUMU_BINLOG_COMPACTION));
+ cloud_tablet, CompactionType::CHANGELOG_COMPACTION));
} else {
RETURN_IF_ERROR(engine.to_local().submit_compaction_task(
- std::static_pointer_cast<Tablet>(tablet), CompactionType::CUMU_BINLOG_COMPACTION,
+ std::static_pointer_cast<Tablet>(tablet), CompactionType::CHANGELOG_COMPACTION,
false));
}
ttl_submitted << 1;
@@ -81,76 +84,78 @@ Status prepare_row_binlog_ttl(BaseStorageEngine& engine, int64_t tablet_id, bool
}
} // namespace
-void BaseStorageEngine::register_row_binlog_tablet(const BaseTabletSPtr& tablet) {
- if (tablet->is_row_binlog_tablet()) {
- std::lock_guard lock(_row_binlog_ttl_mutex);
- _row_binlog_ttl_tablets[tablet->tablet_id()] = tablet;
+void BaseStorageEngine::register_changelog_tablet(const BaseTabletSPtr& tablet) {
+ if (tablet->is_changelog_tablet()) {
+ std::lock_guard lock(_changelog_ttl_mutex);
+ _changelog_ttl_tablets[tablet->tablet_id()] = tablet;
}
}
-Status BaseStorageEngine::submit_row_binlog_ttl(int64_t tablet_id, bool refresh_cloud_meta) {
- if (ExecEnv::GetInstance()->cluster_info()->row_binlog_ttl_reference_tso() <= 0) {
- return Status::Error<ErrorCode::SERVICE_UNAVAILABLE>("No valid ROW binlog TTL reference");
+Status BaseStorageEngine::submit_changelog_ttl(int64_t tablet_id, bool refresh_cloud_meta) {
+ if (ExecEnv::GetInstance()->cluster_info()->changelog_ttl_reference_tso() <= 0) {
+ return Status::Error<ErrorCode::SERVICE_UNAVAILABLE>("No valid changelog TTL reference");
}
- std::lock_guard lock(_row_binlog_ttl_mutex);
- if (!_row_binlog_ttl_prepare_pool || _stop_background_threads_latch.count() == 0) {
+ std::lock_guard lock(_changelog_ttl_mutex);
+ if (!_changelog_ttl_prepare_pool || _stop_background_threads_latch.count() == 0) {
return Status::Error<ErrorCode::SERVICE_UNAVAILABLE>(
- "ROW binlog TTL scanner is not running");
+ "changelog TTL scanner is not running");
}
- if (!_row_binlog_ttl_pending.insert(tablet_id).second) {
- return Status::AlreadyExist("ROW binlog TTL task already queued for {}", tablet_id);
+ if (!_changelog_ttl_pending.insert(tablet_id).second) {
+ return Status::AlreadyExist("changelog TTL task already queued for {}", tablet_id);
}
- auto st = _row_binlog_ttl_prepare_pool->submit_func([this, tablet_id, refresh_cloud_meta]() {
+ auto st = _changelog_ttl_prepare_pool->submit_func([this, tablet_id, refresh_cloud_meta]() {
SCOPED_INIT_THREAD_CONTEXT();
Defer clear_pending([this, tablet_id] {
- std::lock_guard lock(_row_binlog_ttl_mutex);
- _row_binlog_ttl_pending.erase(tablet_id);
+ std::lock_guard lock(_changelog_ttl_mutex);
+ _changelog_ttl_pending.erase(tablet_id);
});
auto result =
- prepare_row_binlog_ttl(*this, tablet_id, refresh_cloud_meta, _type == Type::CLOUD);
+ prepare_changelog_ttl(*this, tablet_id, refresh_cloud_meta, _type == Type::CLOUD);
if (!result.ok()) {
ttl_failed << 1;
LOG_EVERY_N(WARNING, 100)
- << "ROW binlog TTL task failed, tablet=" << tablet_id << ", status=" << result;
+ << "changelog TTL task failed, tablet=" << tablet_id << ", status=" << result;
}
});
if (!st.ok()) {
- _row_binlog_ttl_pending.erase(tablet_id);
+ _changelog_ttl_pending.erase(tablet_id);
}
return st;
}
-Status BaseStorageEngine::_start_row_binlog_ttl_scanner() {
- RETURN_IF_ERROR(ThreadPoolBuilder("RowBinlogTtlPrepare")
+Status BaseStorageEngine::_start_changelog_ttl_scanner() {
+ // ATTN(changelog-rename): formerly "RowBinlogTtlPrepare"
+ RETURN_IF_ERROR(ThreadPoolBuilder("ChangelogTtlPrepare")
.set_min_threads(1)
.set_max_threads(1)
.set_max_queue_size(kScanBatchSize)
- .build(&_row_binlog_ttl_prepare_pool));
+ .build(&_changelog_ttl_prepare_pool));
+ // ATTN(changelog-rename): formerly "row_binlog_ttl_scanner"
return Thread::create(
- "StorageEngine", "row_binlog_ttl_scanner",
+ "StorageEngine", "changelog_ttl_scanner",
[this] {
SCOPED_INIT_THREAD_CONTEXT();
int64_t cursor = 0;
do {
std::vector<int64_t> batch;
{
- std::lock_guard lock(_row_binlog_ttl_mutex);
- auto it = _row_binlog_ttl_tablets.upper_bound(cursor);
- for (int n = 0; n < kScanBatchSize && it != _row_binlog_ttl_tablets.end();
+ std::lock_guard lock(_changelog_ttl_mutex);
+ auto it = _changelog_ttl_tablets.upper_bound(cursor);
+ for (int n = 0; n < kScanBatchSize && it != _changelog_ttl_tablets.end();
++n) {
if (it->second.expired()) {
- it = _row_binlog_ttl_tablets.erase(it);
+ it = _changelog_ttl_tablets.erase(it);
} else {
batch.push_back(it->first);
++it;
}
}
- if (batch.empty() && it == _row_binlog_ttl_tablets.end()) {
+ if (batch.empty() && it == _changelog_ttl_tablets.end()) {
cursor = 0;
}
}
- if (!config::enable_feature_binlog || config::disable_auto_compaction ||
- ExecEnv::GetInstance()->cluster_info()->row_binlog_ttl_reference_tso() <=
+ if (!config::enable_feature_changelog || config::disable_auto_compaction ||
+ ExecEnv::GetInstance()->cluster_info()->changelog_ttl_reference_tso() <=
0) {
// Keep sweeping weak registrations even while cleanup is paused.
if (!batch.empty()) {
@@ -159,7 +164,7 @@ Status BaseStorageEngine::_start_row_binlog_ttl_scanner() {
continue;
}
for (auto tablet_id : batch) {
- auto st = submit_row_binlog_ttl(tablet_id);
+ auto st = submit_changelog_ttl(tablet_id);
if (!st.ok() && !st.is<ErrorCode::ALREADY_EXIST>()) {
ttl_failed << 1;
break; // Resume here next round; a full queue must not starve the tail.
@@ -168,15 +173,15 @@ Status BaseStorageEngine::_start_row_binlog_ttl_scanner() {
}
} while (!_stop_background_threads_latch.wait_for(std::chrono::milliseconds(1000)));
},
- &_row_binlog_ttl_scan_thread);
+ &_changelog_ttl_scan_thread);
}
-void BaseStorageEngine::_stop_row_binlog_ttl_scanner() {
- if (_row_binlog_ttl_scan_thread) {
- _row_binlog_ttl_scan_thread->join();
+void BaseStorageEngine::_stop_changelog_ttl_scanner() {
+ if (_changelog_ttl_scan_thread) {
+ _changelog_ttl_scan_thread->join();
}
- if (_row_binlog_ttl_prepare_pool) {
- _row_binlog_ttl_prepare_pool->shutdown();
+ if (_changelog_ttl_prepare_pool) {
+ _changelog_ttl_prepare_pool->shutdown();
}
}
} // namespace doris
diff --git a/be/src/storage/compaction/compaction.cpp b/be/src/storage/compaction/compaction.cpp
index ecd2abf9154..519eba74807 100644
--- a/be/src/storage/compaction/compaction.cpp
+++ b/be/src/storage/compaction/compaction.cpp
@@ -62,7 +62,7 @@
#include "runtime/memory/mem_tracker_limiter.h"
#include "runtime/thread_context.h"
#include "storage/compaction/cumulative_compaction.h"
-#include "storage/compaction/cumulative_compaction_binlog_policy.h"
+#include "storage/compaction/cumulative_compaction_changelog_policy.h"
#include "storage/compaction/cumulative_compaction_policy.h"
#include "storage/compaction/cumulative_compaction_time_series_policy.h"
#include "storage/compaction_task_tracker.h"
@@ -292,28 +292,28 @@ int64_t Compaction::merge_way_num() {
return way_num;
}
-void Compaction::snapshot_row_binlog_ttl() {
- if (!_tablet->is_row_binlog_tablet()) {
+void Compaction::snapshot_changelog_ttl() {
+ if (!_tablet->is_changelog_tablet()) {
return;
}
if (!config::is_cloud_mode()) {
- _row_binlog_ttl_visible_version = std::get<0>(
+ _changelog_ttl_visible_version = std::get<0>(
static_cast<Tablet*>(_tablet.get())->get_visible_version_and_time(false));
}
std::shared_lock lock(_tablet->get_header_lock());
const auto& config = _tablet->tablet_meta()->binlog_config();
- _row_binlog_ttl_cutoff_tso = config.row_ttl_cutoff_tso(
- ExecEnv::GetInstance()->cluster_info()->row_binlog_ttl_reference_tso());
- _row_binlog_ttl_seconds = config.ttl_seconds();
+ _changelog_ttl_cutoff_tso = config.changelog_ttl_cutoff_tso(
+ ExecEnv::GetInstance()->cluster_info()->changelog_ttl_reference_tso());
+ _changelog_ttl_seconds = config.ttl_seconds();
}
-bool Compaction::pick_expired_row_binlog_rowset(const std::vector<RowsetSharedPtr>& candidates) {
- if (_row_binlog_ttl_cutoff_tso < 0) {
+bool Compaction::pick_expired_changelog_rowset(const std::vector<RowsetSharedPtr>& candidates) {
+ if (_changelog_ttl_cutoff_tso < 0) {
return false;
}
auto expired = std::ranges::find_if(candidates, [this](const RowsetSharedPtr& rowset) {
- return rowset->end_version() <= _row_binlog_ttl_visible_version &&
- row_binlog_rowset_expired(*rowset->rowset_meta(), _row_binlog_ttl_cutoff_tso);
+ return rowset->end_version() <= _changelog_ttl_visible_version &&
+ changelog_rowset_expired(*rowset->rowset_meta(), _changelog_ttl_cutoff_tso);
});
if (expired == candidates.end()) {
return false;
@@ -333,9 +333,9 @@ bool Compaction::pick_expired_row_binlog_rowset(const std::vector<RowsetSharedPt
auto end = std::next(expired);
while (end != candidates.end() && end - first < max_inputs &&
(*end)->start_version() == (*std::prev(end))->end_version() + 1 &&
- (*end)->end_version() <= _row_binlog_ttl_visible_version &&
+ (*end)->end_version() <= _changelog_ttl_visible_version &&
((*end)->num_rows() == 0 ||
- row_binlog_rowset_expired(*(*end)->rowset_meta(), _row_binlog_ttl_cutoff_tso))) {
+ changelog_rowset_expired(*(*end)->rowset_meta(), _changelog_ttl_cutoff_tso))) {
++end;
}
{
@@ -350,10 +350,10 @@ bool Compaction::pick_expired_row_binlog_rowset(const std::vector<RowsetSharedPt
return true;
}
-void Compaction::filter_row_binlog_ttl_rowsets() {
+void Compaction::filter_changelog_ttl_rowsets() {
_data_input_rowsets.clear();
- _row_binlog_ttl_filtered_rows = 0;
- if (_row_binlog_ttl_cutoff_tso < 0) {
+ _changelog_ttl_filtered_rows = 0;
+ if (_changelog_ttl_cutoff_tso < 0) {
_data_input_rowsets = _input_rowsets;
return;
}
@@ -361,12 +361,12 @@ void Compaction::filter_row_binlog_ttl_rowsets() {
if (rowset->num_rows() == 0) {
continue;
}
- if (rowset->end_version() > _row_binlog_ttl_visible_version ||
- !row_binlog_rowset_expired(*rowset->rowset_meta(), _row_binlog_ttl_cutoff_tso)) {
+ if (rowset->end_version() > _changelog_ttl_visible_version ||
+ !changelog_rowset_expired(*rowset->rowset_meta(), _changelog_ttl_cutoff_tso)) {
_data_input_rowsets.push_back(rowset);
continue;
}
- _row_binlog_ttl_filtered_rows += rowset->num_rows();
+ _changelog_ttl_filtered_rows += rowset->num_rows();
}
}
@@ -378,7 +378,7 @@ Status Compaction::merge_input_rowsets() {
}
Status Compaction::prepare_merge_input_rowsets_execution(MergeInputRowsetsContext* context) {
- filter_row_binlog_ttl_rowsets();
+ filter_changelog_ttl_rowsets();
RETURN_IF_ERROR(prepare_merge_input_rowsets(&context->result));
context->input_rs_readers.reserve(_data_input_rowsets.size());
@@ -422,7 +422,7 @@ Status Compaction::execute_merge_input_rowsets(MergeInputRowsetsContext* context
Status Compaction::finish_merge_input_rowsets_execution(MergeInputRowsetsContext* context) {
auto& result = context->result;
- _stats.filtered_rows += _row_binlog_ttl_filtered_rows;
+ _stats.filtered_rows += _changelog_ttl_filtered_rows;
COUNTER_UPDATE(_merged_rows_counter, _stats.merged_rows);
COUNTER_UPDATE(_filtered_rows_counter, _stats.filtered_rows);
@@ -601,8 +601,8 @@ Status CompactionMixin::do_compact_ordered_rowsets() {
rowset_meta->set_index_disk_size(_input_rowsets_index_size);
rowset_meta->set_empty(_input_row_num == 0);
rowset_meta->set_num_segments(_input_num_segments);
- rowset_meta->set_segments_overlap(_trigger_quick_merge_by_binlog ? OVERLAPPING
- : NONOVERLAPPING);
+ rowset_meta->set_segments_overlap(_trigger_quick_merge_by_changelog ? OVERLAPPING
+ : NONOVERLAPPING);
rowset_meta->set_rowset_state(VISIBLE);
rowset_meta->set_segments_key_bounds_truncated(segments_key_bounds_truncated);
// If any input was already aggregated we have no way to recover per-segment
@@ -611,7 +611,7 @@ Status CompactionMixin::do_compact_ordered_rowsets() {
bool aggregate_key_bounds =
any_input_aggregated ||
(config::enable_aggregate_non_mow_key_bounds &&
- !_tablet->enable_unique_key_merge_on_write() && !tablet()->is_row_binlog_tablet());
+ !_tablet->enable_unique_key_merge_on_write() && !tablet()->is_changelog_tablet());
rowset_meta->set_segments_key_bounds(segment_key_bounds, aggregate_key_bounds);
rowset_meta->set_num_segment_rows(num_segment_rows);
if (auto commit_tso = commit_tso_range(_input_rowsets); commit_tso.has_value()) {
@@ -686,9 +686,9 @@ bool CompactionMixin::handle_ordered_data_compaction() {
if (!config::enable_ordered_data_compaction) {
return false;
}
- if (_row_binlog_ttl_cutoff_tso >= 0 &&
+ if (_changelog_ttl_cutoff_tso >= 0 &&
std::ranges::any_of(_input_rowsets, [this](const RowsetSharedPtr& rowset) {
- return row_binlog_rowset_expired(*rowset->rowset_meta(), _row_binlog_ttl_cutoff_tso);
+ return changelog_rowset_expired(*rowset->rowset_meta(), _changelog_ttl_cutoff_tso);
})) {
return false;
}
@@ -719,8 +719,8 @@ bool CompactionMixin::handle_ordered_data_compaction() {
// The remote file system and full compaction does not support to link files.
return false;
}
- bool is_binlog_compaction = _tablet->is_row_binlog_tablet();
- if (is_binlog_compaction && !_input_rowsets.empty()) {
+ bool is_changelog_compaction = _tablet->is_changelog_tablet();
+ if (is_changelog_compaction && !_input_rowsets.empty()) {
RowsetIdUnorderedSet input_rowset_ids;
input_rowset_ids.reserve(_input_rowsets.size());
for (const auto& rs : _input_rowsets) {
@@ -732,7 +732,7 @@ bool CompactionMixin::handle_ordered_data_compaction() {
}
if (_tablet->keys_type() == KeysType::UNIQUE_KEYS &&
- _tablet->enable_unique_key_merge_on_write() && !is_binlog_compaction) {
+ _tablet->enable_unique_key_merge_on_write() && !is_changelog_compaction) {
return false;
}
@@ -753,22 +753,22 @@ bool CompactionMixin::handle_ordered_data_compaction() {
}
}
- if (is_binlog_compaction) {
+ if (is_changelog_compaction) {
DCHECK(!_input_rowsets.empty()) << "tablet=" << _tablet->tablet_id();
auto compaction_level = _input_rowsets.front()->rowset_meta()->compaction_level();
- bool can_quick_merge_binlog =
+ bool can_quick_merge_changelog =
compaction_level ==
- BinlogCumulativeCompactionPolicy::kBinlogCompactionMaxLevel - 1 &&
+ ChangelogCumulativeCompactionPolicy::kChangelogCompactionMaxLevel - 1 &&
_input_rowsets.size() >= 2 && _input_rowsets[0]->start_version() == 0;
- if (!can_quick_merge_binlog) {
+ if (!can_quick_merge_changelog) {
return false;
}
- _trigger_quick_merge_by_binlog = true;
+ _trigger_quick_merge_by_changelog = true;
- // Binlog quick merge at LMax is a special meta/link compaction path selected by
- // BinlogCumulativeCompactionPolicy. It does not require the whole input to satisfy the normal
- // ordered-data tidy check, but the output rowset built by do_compact_ordered_rowsets()
- // is still NONOVERLAPPING (excluding row-binlog).
+ // Changelog quick merge at LMax is a special meta/link compaction path selected by
+ // ChangelogCumulativeCompactionPolicy. It does not require the whole input to satisfy the
+ // normal ordered-data tidy check, but the output rowset built by
+ // do_compact_ordered_rowsets() is still NONOVERLAPPING (excluding changelog).
auto st = do_compact_ordered_rowsets();
if (!st.ok()) {
LOG(WARNING) << "failed to compact ordered rowsets: " << st;
@@ -809,7 +809,7 @@ Status CompactionMixin::execute_compact() {
uint32_t checksum_before;
uint32_t checksum_after;
bool enable_compaction_checksum =
- config::enable_compaction_checksum && _row_binlog_ttl_cutoff_tso < 0;
+ config::enable_compaction_checksum && _changelog_ttl_cutoff_tso < 0;
if (enable_compaction_checksum) {
EngineChecksumTask checksum_task(_engine, _tablet->tablet_id(), _tablet->schema_hash(),
_input_rowsets.back()->end_version(), &checksum_before);
@@ -910,7 +910,7 @@ Status CompactionMixin::execute_compact_impl(int64_t permits) {
RETURN_IF_ERROR(merge_input_rowsets());
- // Currently, updates are only made in the time_series and binlog policies.
+ // Currently, updates are only made in the time_series and changelog policies.
update_compaction_level();
RETURN_IF_ERROR(modify_rowsets());
@@ -1887,18 +1887,18 @@ void CompactionMixin::find_longest_consecutive_version(std::vector<RowsetSharedP
Status CompactionMixin::construct_output_rowset_writer(RowsetWriterContext& ctx) {
// only do index compaction for dup_keys and unique_keys with mow enabled
- if (_row_binlog_ttl_filtered_rows == 0 && _enable_inverted_index_compaction &&
+ if (_changelog_ttl_filtered_rows == 0 && _enable_inverted_index_compaction &&
(((_tablet->keys_type() == KeysType::UNIQUE_KEYS &&
_tablet->enable_unique_key_merge_on_write()) ||
_tablet->keys_type() == KeysType::DUP_KEYS))) {
construct_index_compaction_columns(ctx);
}
- if (_tablet->is_row_binlog_tablet()) {
- ctx.write_binlog_opt().enable = true;
+ if (_tablet->is_changelog_tablet()) {
+ ctx.write_changelog_opt().enable = true;
}
ctx.version = _output_version;
ctx.rowset_state = VISIBLE;
- ctx.segments_overlap = _trigger_quick_merge_by_binlog ? OVERLAPPING : NONOVERLAPPING;
+ ctx.segments_overlap = _trigger_quick_merge_by_changelog ? OVERLAPPING : NONOVERLAPPING;
ctx.tablet_schema = _cur_tablet_schema;
ctx.variant_compaction_paths = _cur_variant_compaction_paths;
ctx.newest_write_timestamp = _newest_write_timestamp;
@@ -2075,12 +2075,11 @@ Status CompactionMixin::modify_rowsets() {
} else {
std::lock_guard wrlock(_tablet->get_header_lock());
SCOPED_SIMPLE_TRACE_IF_TIMEOUT(TRACE_TABLET_LOCK_THRESHOLD);
- if (_row_binlog_ttl_filtered_rows > 0) {
+ if (_changelog_ttl_filtered_rows > 0) {
const auto& current = _tablet->tablet_meta()->binlog_config();
- if (!current.has_row_ttl() || current.ttl_seconds() != _row_binlog_ttl_seconds) {
+ if (!current.has_changelog_ttl() || current.ttl_seconds() != _changelog_ttl_seconds) {
return Status::Error<CUMULATIVE_NO_SUITABLE_VERSION>(
- "ROW binlog TTL changed during compaction, tablet={}",
- _tablet->tablet_id());
+ "changelog TTL changed during compaction, tablet={}", _tablet->tablet_id());
}
}
RETURN_IF_ERROR(tablet()->modify_rowsets(output_rowsets, _input_rowsets, true));
@@ -2130,7 +2129,7 @@ bool CompactionMixin::_check_if_includes_input_rowsets(
void CompactionMixin::update_compaction_level() {
auto* cumu_policy = tablet()->cumulative_compaction_policy();
if (cumu_policy && (cumu_policy->name() == CUMULATIVE_TIME_SERIES_POLICY ||
- cumu_policy->name() == CUMULATIVE_BINLOG_POLICY)) {
+ cumu_policy->name() == CUMULATIVE_CHANGELOG_POLICY)) {
int64_t compaction_level =
cumu_policy->get_compaction_level(tablet(), _input_rowsets, _output_rowset);
_output_rowset->rowset_meta()->set_compaction_level(compaction_level);
@@ -2156,7 +2155,7 @@ int64_t CompactionMixin::get_compaction_permits() {
int64_t permits = 0;
const int64_t point = tablet()->cumulative_layer_point();
for (auto&& rowset : _input_rowsets) {
- if (tablet()->is_row_binlog_tablet() && point != Tablet::K_INVALID_CUMULATIVE_POINT &&
+ if (tablet()->is_changelog_tablet() && point != Tablet::K_INVALID_CUMULATIVE_POINT &&
rowset->end_version() < point) {
++permits;
continue;
@@ -2280,7 +2279,7 @@ Status CloudCompactionMixin::finish_execute_compact(int64_t execution_start_time
<< _output_rowset->rowset_meta()->rowset_id().to_string();
})
- // Currently, updates are only made in the time_series and binlog policies.
+ // Currently, updates are only made in the time_series and changelog policies.
update_compaction_level();
RETURN_IF_ERROR(_engine.meta_mgr().commit_rowset(*_output_rowset->rowset_meta().get(), _uuid,
@@ -2477,7 +2476,7 @@ Status CloudCompactionMixin::set_storage_resource_from_input_rowsets(RowsetWrite
Status CloudCompactionMixin::construct_output_rowset_writer(RowsetWriterContext& ctx) {
// only do index compaction for dup_keys and unique_keys with mow enabled
- if (_row_binlog_ttl_filtered_rows == 0 && _enable_inverted_index_compaction &&
+ if (_changelog_ttl_filtered_rows == 0 && _enable_inverted_index_compaction &&
(((_tablet->keys_type() == KeysType::UNIQUE_KEYS &&
_tablet->enable_unique_key_merge_on_write()) ||
_tablet->keys_type() == KeysType::DUP_KEYS))) {
@@ -2500,8 +2499,8 @@ Status CloudCompactionMixin::construct_output_rowset_writer(RowsetWriterContext&
ctx.write_type = DataWriteType::TYPE_COMPACTION;
ctx.compaction_type = compaction_type();
ctx.allow_packed_file = false;
- if (_tablet->is_row_binlog_tablet()) {
- ctx.write_binlog_opt().enable = true;
+ if (_tablet->is_changelog_tablet()) {
+ ctx.write_changelog_opt().enable = true;
}
// We presume that the data involved in cumulative compaction is sufficiently 'hot'
@@ -2580,9 +2579,9 @@ void CloudCompactionMixin::update_compaction_level() {
_input_rowsets.back()->rowset_meta()->compaction_level());
} else {
auto compaction_policy = _tablet->tablet_meta()->compaction_policy();
- auto cumu_policy = _engine.cumu_compaction_policy(compaction_policy);
+ auto cumu_policy = _engine.cumu_compaction_policy(compaction_policy, _tablet->tablet_id());
if (cumu_policy && (cumu_policy->name() == CUMULATIVE_TIME_SERIES_POLICY ||
- cumu_policy->name() == CUMULATIVE_BINLOG_POLICY)) {
+ cumu_policy->name() == CUMULATIVE_CHANGELOG_POLICY)) {
int64_t compaction_level = cumu_policy->get_compaction_level(
cloud_tablet(), _input_rowsets, _output_rowset);
_output_rowset->rowset_meta()->set_compaction_level(compaction_level);
diff --git a/be/src/storage/compaction/compaction.h b/be/src/storage/compaction/compaction.h
index 2acf0ee3212..6084f9b039c 100644
--- a/be/src/storage/compaction/compaction.h
+++ b/be/src/storage/compaction/compaction.h
@@ -122,9 +122,9 @@ protected:
Status execute_merge_input_rowsets(MergeInputRowsetsContext* context);
Status finish_merge_input_rowsets_execution(MergeInputRowsetsContext* context);
- void snapshot_row_binlog_ttl();
- bool pick_expired_row_binlog_rowset(const std::vector<RowsetSharedPtr>& candidates);
- void filter_row_binlog_ttl_rowsets();
+ void snapshot_changelog_ttl();
+ bool pick_expired_changelog_rowset(const std::vector<RowsetSharedPtr>& candidates);
+ void filter_changelog_ttl_rowsets();
virtual Status prepare_merge_input_rowsets(MergeInputRowsetsResult* /*result*/) {
return Status::OK();
@@ -182,10 +182,10 @@ protected:
std::vector<RowsetSharedPtr> _input_rowsets;
std::vector<RowsetSharedPtr> _data_input_rowsets;
- int64_t _row_binlog_ttl_cutoff_tso {-1};
- int64_t _row_binlog_ttl_seconds {0};
- int64_t _row_binlog_ttl_visible_version {std::numeric_limits<int64_t>::max()};
- int64_t _row_binlog_ttl_filtered_rows {0};
+ int64_t _changelog_ttl_cutoff_tso {-1};
+ int64_t _changelog_ttl_seconds {0};
+ int64_t _changelog_ttl_visible_version {std::numeric_limits<int64_t>::max()};
+ int64_t _changelog_ttl_filtered_rows {0};
int64_t _input_rowsets_data_size {0};
int64_t _input_rowsets_index_size {0};
int64_t _input_rowsets_total_size {0};
@@ -208,7 +208,7 @@ protected:
bool _is_vertical;
bool _is_ordered_data_compaction {false};
- bool _trigger_quick_merge_by_binlog {false};
+ bool _trigger_quick_merge_by_changelog {false};
bool _allow_delete_in_cumu_compaction;
bool _enable_vertical_compact_variant_subcolumns;
diff --git a/be/src/storage/compaction/cumulative_compaction.cpp b/be/src/storage/compaction/cumulative_compaction.cpp
index 250c8710640..5d1e6625755 100644
--- a/be/src/storage/compaction/cumulative_compaction.cpp
+++ b/be/src/storage/compaction/cumulative_compaction.cpp
@@ -69,7 +69,7 @@ Status CumulativeCompaction::prepare_compact() {
return st;
}
- snapshot_row_binlog_ttl();
+ snapshot_changelog_ttl();
tablet()->calculate_cumulative_point();
VLOG_CRITICAL << "after calculate, current cumulative point is "
<< tablet()->cumulative_layer_point() << ", tablet=" << _tablet->tablet_id();
@@ -142,7 +142,7 @@ Status CumulativeCompaction::execute_compact() {
Status CumulativeCompaction::pick_rowsets_to_compact() {
auto candidate_rowsets =
- tablet()->pick_candidate_rowsets_to_cumulative_compaction(_row_binlog_ttl_cutoff_tso);
+ tablet()->pick_candidate_rowsets_to_cumulative_compaction(_changelog_ttl_cutoff_tso);
if (candidate_rowsets.empty()) {
return Status::Error<CUMULATIVE_NO_SUITABLE_VERSION>("candidate_rowsets is empty");
}
@@ -178,18 +178,18 @@ Status CumulativeCompaction::pick_rowsets_to_compact() {
}
}
}
- if (pick_expired_row_binlog_rowset(candidate_rowsets)) {
+ if (pick_expired_changelog_rowset(candidate_rowsets)) {
return Status::OK();
}
- int64_t max_score = tablet()->is_row_binlog_tablet()
- ? config::binlog_level_compaction_max_deltas
+ int64_t max_score = tablet()->is_changelog_tablet()
+ ? config::changelog_level_compaction_max_deltas
: config::cumulative_compaction_max_deltas;
int64_t process_memory_usage = doris::GlobalMemoryArbitrator::process_memory_usage();
bool memory_usage_high = process_memory_usage > MemInfo::soft_mem_limit() * 8 / 10;
if (tablet()->last_compaction_status.is<ErrorCode::MEM_LIMIT_EXCEEDED>() || memory_usage_high) {
- if (tablet()->is_row_binlog_tablet()) {
- max_score = std::max<int64_t>(config::binlog_level_compaction_max_deltas /
+ if (tablet()->is_changelog_tablet()) {
+ max_score = std::max<int64_t>(config::changelog_level_compaction_max_deltas /
config::cumulative_compaction_max_deltas_factor,
config::cumulative_compaction_min_deltas + 1);
} else {
diff --git a/be/src/storage/compaction/cumulative_compaction_binlog_policy.cpp b/be/src/storage/compaction/cumulative_compaction_changelog_policy.cpp
similarity index 77%
rename from be/src/storage/compaction/cumulative_compaction_binlog_policy.cpp
rename to be/src/storage/compaction/cumulative_compaction_changelog_policy.cpp
index 6aa71ff4b14..f6c44a566de 100644
--- a/be/src/storage/compaction/cumulative_compaction_binlog_policy.cpp
+++ b/be/src/storage/compaction/cumulative_compaction_changelog_policy.cpp
@@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.
-#include "storage/compaction/cumulative_compaction_binlog_policy.h"
+#include "storage/compaction/cumulative_compaction_changelog_policy.h"
#include <algorithm>
#include <string>
@@ -30,20 +30,20 @@
namespace doris {
-bool BinlogCumulativeCompactionPolicy::is_compaction_enough(
+bool ChangelogCumulativeCompactionPolicy::is_compaction_enough(
const RowsetMetaSharedPtr& rowset_meta) const {
- if (rowset_meta->compaction_level() != kBinlogCompactionMaxLevel - 1) {
+ if (rowset_meta->compaction_level() != kChangelogCompactionMaxLevel - 1) {
return false;
}
if (rowset_meta->start_version() == 0) {
return true;
}
return rowset_meta->data_disk_size() >=
- config::binlog_compaction_goal_size_mbytes * 1024 * 1024 ||
- rowset_meta->get_compaction_score() >= config::binlog_compaction_file_count_threshold;
+ config::changelog_compaction_goal_size_mbytes * 1024 * 1024 ||
+ rowset_meta->get_compaction_score() >= config::changelog_compaction_file_count_threshold;
}
-void BinlogCumulativeCompactionPolicy::calculate_cumulative_point(
+void ChangelogCumulativeCompactionPolicy::calculate_cumulative_point(
Tablet* tablet, const RowsetMetaMapContainer& all_rowsets, int64_t current_cumulative_point,
int64_t* cumulative_point) {
*cumulative_point = Tablet::K_INVALID_CUMULATIVE_POINT;
@@ -72,16 +72,16 @@ void BinlogCumulativeCompactionPolicy::calculate_cumulative_point(
*cumulative_point = prev_version + 1;
}
- VLOG_NOTICE << "binlog compaction policy, calculate cumulative point value="
+ VLOG_NOTICE << "changelog compaction policy, calculate cumulative point value="
<< *cumulative_point << ", tablet=" << tablet->tablet_id();
}
-void BinlogCumulativeCompactionPolicy::update_cumulative_point(
+void ChangelogCumulativeCompactionPolicy::update_cumulative_point(
Tablet* tablet, const std::vector<RowsetSharedPtr>& input_rowsets,
RowsetSharedPtr output_rowset, Version& last_delete_version) {
if (tablet->tablet_state() != TABLET_RUNNING || input_rowsets.empty() ||
output_rowset->num_segments() == 0 ||
- output_rowset->rowset_meta()->compaction_level() != kBinlogCompactionMaxLevel - 1 ||
+ output_rowset->rowset_meta()->compaction_level() != kChangelogCompactionMaxLevel - 1 ||
!is_compaction_enough(output_rowset->rowset_meta())) {
return;
}
@@ -93,7 +93,7 @@ void BinlogCumulativeCompactionPolicy::update_cumulative_point(
tablet->set_cumulative_layer_point(output_rowset->end_version() + 1);
}
-int BinlogCumulativeCompactionPolicy::pick_input_rowsets(
+int ChangelogCumulativeCompactionPolicy::pick_input_rowsets(
Tablet* tablet, const std::vector<RowsetSharedPtr>& candidate_rowsets,
int8_t compaction_level, int64_t max_compaction_score,
std::vector<RowsetSharedPtr>* input_rowsets) const {
@@ -107,7 +107,7 @@ int BinlogCumulativeCompactionPolicy::pick_input_rowsets(
if (!level_rowsets.empty() &&
rs->start_version() != level_rowsets.back()->end_version() + 1) {
LOG(WARNING) << "rowset is non-continuous in the same compaction_level "
- "of binlog compaction. tablet="
+ "of changelog compaction. tablet="
<< tablet->tablet_id()
<< ", compaction_level=" << std::to_string(compaction_level)
<< ", prev_version=" << level_rowsets.back()->version()
@@ -127,7 +127,7 @@ int BinlogCumulativeCompactionPolicy::pick_input_rowsets(
int compact_enough_size = 0;
const int64_t point = tablet->cumulative_layer_point();
- if (compaction_level == kBinlogCompactionMaxLevel - 1 &&
+ if (compaction_level == kChangelogCompactionMaxLevel - 1 &&
point != Tablet::K_INVALID_CUMULATIVE_POINT) {
for (const auto& rs : level_rowsets) {
if (rs->end_version() < point) {
@@ -161,30 +161,30 @@ int BinlogCumulativeCompactionPolicy::pick_input_rowsets(
// 4) Trigger check
// - L0/L1: only physical rewrite if trigger is met.
// - LMax: if physical rewrite trigger is NOT met, try quick compact; if both met, compare score.
- bool can_do_binlog_compaction = false;
- if (total_size >= config::binlog_compaction_goal_size_mbytes * 1024 * 1024 ||
- compaction_score >= config::binlog_compaction_file_count_threshold) {
- can_do_binlog_compaction = true;
+ bool can_do_changelog_compaction = false;
+ if (total_size >= config::changelog_compaction_goal_size_mbytes * 1024 * 1024 ||
+ compaction_score >= config::changelog_compaction_file_count_threshold) {
+ can_do_changelog_compaction = true;
} else if ((UnixMillis() - tablet->last_cumu_compaction_success_time()) / 1000 >=
- config::binlog_compaction_time_threshold_seconds) {
- can_do_binlog_compaction = true;
+ config::changelog_compaction_time_threshold_seconds) {
+ can_do_changelog_compaction = true;
}
// 5) LMax quick compact vs physical rewrite.
- if (compaction_level == kBinlogCompactionMaxLevel - 1 && compact_enough_size > 1) {
+ if (compaction_level == kChangelogCompactionMaxLevel - 1 && compact_enough_size > 1) {
int64_t quick_merge_score = compact_enough_size;
- if (!can_do_binlog_compaction || quick_merge_score > compaction_score) {
+ if (!can_do_changelog_compaction || quick_merge_score > compaction_score) {
input_rowsets->swap(compact_enough_rowsets);
return compact_enough_size;
}
}
if (transient_size < 2) {
- can_do_binlog_compaction = false;
+ can_do_changelog_compaction = false;
}
- if (can_do_binlog_compaction) {
+ if (can_do_changelog_compaction) {
input_rowsets->swap(picked_rowsets);
return transient_size;
}
@@ -193,11 +193,11 @@ int BinlogCumulativeCompactionPolicy::pick_input_rowsets(
return 0;
}
-uint32_t BinlogCumulativeCompactionPolicy::calc_binlog_compaction_level_score(Tablet* tablet,
- int8_t level) const {
+uint32_t ChangelogCumulativeCompactionPolicy::calc_changelog_compaction_level_score(
+ Tablet* tablet, int8_t level) const {
uint32_t score = 0;
const int64_t point = tablet->cumulative_layer_point();
- // Binlog tiered compaction score (L0..LMax)
+ // Changelog tiered compaction score (L0..LMax)
//
// L0/L1/... : | rowsets ... |
//
@@ -212,8 +212,8 @@ uint32_t BinlogCumulativeCompactionPolicy::calc_binlog_compaction_level_score(Ta
if (!rs_meta->is_local() || rs_meta->compaction_level() != level) {
continue;
}
- if (level == kBinlogCompactionMaxLevel - 1 && point != Tablet::K_INVALID_CUMULATIVE_POINT &&
- rs_meta->end_version() < point) {
+ if (level == kChangelogCompactionMaxLevel - 1 &&
+ point != Tablet::K_INVALID_CUMULATIVE_POINT && rs_meta->end_version() < point) {
score += 1;
continue;
}
@@ -222,7 +222,7 @@ uint32_t BinlogCumulativeCompactionPolicy::calc_binlog_compaction_level_score(Ta
return score;
}
-uint32_t BinlogCumulativeCompactionPolicy::calc_binlog_compaction_level_score(
+uint32_t ChangelogCumulativeCompactionPolicy::calc_changelog_compaction_level_score(
Tablet* tablet, const std::vector<RowsetSharedPtr>& candidate_rowsets, int8_t level) const {
uint32_t score = 0;
const int64_t point = tablet->cumulative_layer_point();
@@ -231,8 +231,8 @@ uint32_t BinlogCumulativeCompactionPolicy::calc_binlog_compaction_level_score(
if (!rs->is_local() || rs_meta->compaction_level() != level) {
continue;
}
- if (level == kBinlogCompactionMaxLevel - 1 && point != Tablet::K_INVALID_CUMULATIVE_POINT &&
- rs_meta->end_version() < point) {
+ if (level == kChangelogCompactionMaxLevel - 1 &&
+ point != Tablet::K_INVALID_CUMULATIVE_POINT && rs_meta->end_version() < point) {
score += 1;
continue;
}
@@ -241,12 +241,12 @@ uint32_t BinlogCumulativeCompactionPolicy::calc_binlog_compaction_level_score(
return score;
}
-uint32_t BinlogCumulativeCompactionPolicy::calc_binlog_compaction_score(
+uint32_t ChangelogCumulativeCompactionPolicy::calc_changelog_compaction_score(
Tablet* tablet, int8_t* prefer_compaction_level) const {
uint32_t max_score = 0;
int8_t max_level = -1;
- for (int8_t level = 0; level < kBinlogCompactionMaxLevel; ++level) {
- uint32_t score = calc_binlog_compaction_level_score(tablet, level);
+ for (int8_t level = 0; level < kChangelogCompactionMaxLevel; ++level) {
+ uint32_t score = calc_changelog_compaction_level_score(tablet, level);
if (score > max_score) {
max_score = score;
max_level = level;
@@ -258,13 +258,13 @@ uint32_t BinlogCumulativeCompactionPolicy::calc_binlog_compaction_score(
return max_score;
}
-uint32_t BinlogCumulativeCompactionPolicy::calc_binlog_compaction_score(
+uint32_t ChangelogCumulativeCompactionPolicy::calc_changelog_compaction_score(
Tablet* tablet, const std::vector<RowsetSharedPtr>& candidate_rowsets,
int8_t* prefer_compaction_level) const {
uint32_t max_score = 0;
int8_t max_level = -1;
- for (int8_t level = 0; level < kBinlogCompactionMaxLevel; ++level) {
- uint32_t score = calc_binlog_compaction_level_score(tablet, candidate_rowsets, level);
+ for (int8_t level = 0; level < kChangelogCompactionMaxLevel; ++level) {
+ uint32_t score = calc_changelog_compaction_level_score(tablet, candidate_rowsets, level);
if (score > max_score) {
max_score = score;
max_level = level;
@@ -276,13 +276,13 @@ uint32_t BinlogCumulativeCompactionPolicy::calc_binlog_compaction_score(
return max_score;
}
-int BinlogCumulativeCompactionPolicy::pick_input_rowsets(
+int ChangelogCumulativeCompactionPolicy::pick_input_rowsets(
Tablet* tablet, const std::vector<RowsetSharedPtr>& candidate_rowsets,
const int64_t max_compaction_score, const int64_t min_compaction_score,
std::vector<RowsetSharedPtr>* input_rowsets, Version* last_delete_version,
size_t* compaction_score, bool allow_delete) {
int8_t compaction_level = -1;
- calc_binlog_compaction_score(tablet, candidate_rowsets, &compaction_level);
+ calc_changelog_compaction_score(tablet, candidate_rowsets, &compaction_level);
if (compaction_level < 0) {
*compaction_score = 0;
return 0;
@@ -294,11 +294,11 @@ int BinlogCumulativeCompactionPolicy::pick_input_rowsets(
return picked_score;
}
-uint32_t BinlogCumulativeCompactionPolicy::calc_cumulative_compaction_score(Tablet* tablet) {
- return calc_binlog_compaction_score(tablet);
+uint32_t ChangelogCumulativeCompactionPolicy::calc_cumulative_compaction_score(Tablet* tablet) {
+ return calc_changelog_compaction_score(tablet);
}
-int64_t BinlogCumulativeCompactionPolicy::get_compaction_level(
+int64_t ChangelogCumulativeCompactionPolicy::get_compaction_level(
Tablet* tablet, const std::vector<RowsetSharedPtr>& input_rowsets,
RowsetSharedPtr output_rowset) {
DCHECK(!input_rowsets.empty()) << "tablet=" << tablet->tablet_id();
@@ -319,19 +319,19 @@ int64_t BinlogCumulativeCompactionPolicy::get_compaction_level(
}
DCHECK_EQ(input_rowsets[i]->start_version(), input_rowsets[i - 1]->end_version() + 1)
- << "Binlog compaction input rowsets are non-continuous, prev_version: "
+ << "Changelog compaction input rowsets are non-continuous, prev_version: "
<< input_rowsets[i - 1]->version()
<< ", cur_version: " << input_rowsets[i]->version()
<< ", tablet: " << tablet->tablet_id();
if (input_rowsets[i]->start_version() != input_rowsets[i - 1]->end_version() + 1) {
- LOG(WARNING) << "Failed to check binlog compaction input rowsets continuity, "
+ LOG(WARNING) << "Failed to check changelog compaction input rowsets continuity, "
<< "prev_version=" << input_rowsets[i - 1]->version()
<< ", cur_version=" << input_rowsets[i]->version()
<< ", tablet=" << tablet->tablet_id();
}
}
- if (first_level == kBinlogCompactionMaxLevel - 1) {
+ if (first_level == kChangelogCompactionMaxLevel - 1) {
return first_level;
}
return first_level + 1;
diff --git a/be/src/storage/compaction/cumulative_compaction_binlog_policy.h b/be/src/storage/compaction/cumulative_compaction_changelog_policy.h
similarity index 73%
rename from be/src/storage/compaction/cumulative_compaction_binlog_policy.h
rename to be/src/storage/compaction/cumulative_compaction_changelog_policy.h
index 0503cff7f77..7e7038cbb38 100644
--- a/be/src/storage/compaction/cumulative_compaction_binlog_policy.h
+++ b/be/src/storage/compaction/cumulative_compaction_changelog_policy.h
@@ -27,16 +27,17 @@ namespace doris {
class Tablet;
-inline constexpr std::string_view CUMULATIVE_BINLOG_POLICY = "binlog";
+// ATTN(changelog-rename): formerly "binlog"
+inline constexpr std::string_view CUMULATIVE_CHANGELOG_POLICY = "changelog";
-class BinlogCumulativeCompactionPolicy final : public CumulativeCompactionPolicy {
+class ChangelogCumulativeCompactionPolicy final : public CumulativeCompactionPolicy {
public:
- BinlogCumulativeCompactionPolicy() = default;
- ~BinlogCumulativeCompactionPolicy() override = default;
+ ChangelogCumulativeCompactionPolicy() = default;
+ ~ChangelogCumulativeCompactionPolicy() override = default;
- static constexpr int8_t kBinlogCompactionMaxLevel = 3;
+ static constexpr int8_t kChangelogCompactionMaxLevel = 3;
- // Binlog compaction selection rules (tiered, L0..LMax)
+ // Changelog compaction selection rules (tiered, L0..LMax)
//
// Score / Permits
// - L0/L1 use RowsetMeta::get_compaction_score().
@@ -44,9 +45,9 @@ public:
// others use RowsetMeta::get_compaction_score().
//
// Trigger (all levels): merge when ANY holds
- // - size >= binlog_compaction_goal_size_mbytes * 1MB
- // - score >= binlog_compaction_file_count_threshold
- // - time >= binlog_compaction_time_threshold_seconds
+ // - size >= changelog_compaction_goal_size_mbytes * 1MB
+ // - score >= changelog_compaction_file_count_threshold
+ // - time >= changelog_compaction_time_threshold_seconds
//
// LMax model (oldest -> newest):
// version: 0 point
@@ -63,13 +64,13 @@ public:
int8_t compaction_level, int64_t max_compaction_score,
std::vector<RowsetSharedPtr>* input_rowsets) const;
- uint32_t calc_binlog_compaction_score(Tablet* tablet,
- int8_t* prefer_compaction_level = nullptr) const;
- uint32_t calc_binlog_compaction_score(Tablet* tablet,
- const std::vector<RowsetSharedPtr>& candidate_rowsets,
- int8_t* prefer_compaction_level) const;
- uint32_t calc_binlog_compaction_level_score(Tablet* tablet, int8_t level) const;
- uint32_t calc_binlog_compaction_level_score(
+ uint32_t calc_changelog_compaction_score(Tablet* tablet,
+ int8_t* prefer_compaction_level = nullptr) const;
+ uint32_t calc_changelog_compaction_score(Tablet* tablet,
+ const std::vector<RowsetSharedPtr>& candidate_rowsets,
+ int8_t* prefer_compaction_level) const;
+ uint32_t calc_changelog_compaction_level_score(Tablet* tablet, int8_t level) const;
+ uint32_t calc_changelog_compaction_level_score(
Tablet* tablet, const std::vector<RowsetSharedPtr>& candidate_rowsets,
int8_t level) const;
@@ -94,7 +95,7 @@ public:
int64_t get_compaction_level(Tablet* tablet, const std::vector<RowsetSharedPtr>& input_rowsets,
RowsetSharedPtr output_rowset) override;
- std::string_view name() override { return CUMULATIVE_BINLOG_POLICY; }
+ std::string_view name() override { return CUMULATIVE_CHANGELOG_POLICY; }
};
} // namespace doris
diff --git a/be/src/storage/compaction/cumulative_compaction_policy.cpp b/be/src/storage/compaction/cumulative_compaction_policy.cpp
index 07e7488d8e5..c205df1bc8e 100644
--- a/be/src/storage/compaction/cumulative_compaction_policy.cpp
+++ b/be/src/storage/compaction/cumulative_compaction_policy.cpp
@@ -23,9 +23,10 @@
#include <ostream>
#include <string>
+#include "common/check.h"
#include "common/config.h"
#include "common/logging.h"
-#include "storage/compaction/cumulative_compaction_binlog_policy.h"
+#include "storage/compaction/cumulative_compaction_changelog_policy.h"
#include "storage/compaction/cumulative_compaction_time_series_policy.h"
#include "storage/olap_common.h"
#include "storage/rowset/rowset.h"
@@ -436,9 +437,32 @@ CumulativeCompactionPolicyFactory::create_cumulative_compaction_policy(
return std::make_shared<TimeSeriesCumulativeCompactionPolicy>();
} else if (compaction_policy == CUMULATIVE_SIZE_BASED_POLICY) {
return std::make_shared<SizeBasedCumulativeCompactionPolicy>();
- } else if (compaction_policy == CUMULATIVE_BINLOG_POLICY) {
- return std::make_shared<BinlogCumulativeCompactionPolicy>();
+ } else if (compaction_policy == CUMULATIVE_CHANGELOG_POLICY) {
+ return std::make_shared<ChangelogCumulativeCompactionPolicy>();
}
+ LOG_EVERY_N(WARNING, 100) << "unknown compaction policy " << compaction_policy
+ << ", fall back to " << CUMULATIVE_SIZE_BASED_POLICY;
return std::make_shared<SizeBasedCumulativeCompactionPolicy>();
}
+
+std::shared_ptr<CumulativeCompactionPolicy>
+CumulativeCompactionPolicyFactory::get_cumulative_compaction_policy(
+ const std::unordered_map<std::string_view, std::shared_ptr<CumulativeCompactionPolicy>>&
+ policies,
+ std::string_view compaction_policy, int64_t tablet_id) {
+ auto it = policies.find(compaction_policy);
+ if (it != policies.end()) {
+ return it->second;
+ }
+ // An unknown name must not reach unordered_map::at(): its std::out_of_range is not caught
+ // on the compaction scheduler threads, so the BE would abort, and abort again after every
+ // restart. Example: a tablet meta persisted by a build that used another policy name.
+ LOG_EVERY_N(WARNING, 100) << "unknown compaction policy " << compaction_policy << " of tablet "
+ << tablet_id << ", fall back to " << CUMULATIVE_SIZE_BASED_POLICY;
+ // Every policy table registers the size-based policy, see
+ // StorageEngine::_update_cumulative_compaction_policy().
+ auto size_based = policies.find(CUMULATIVE_SIZE_BASED_POLICY);
+ DORIS_CHECK(size_based != policies.end());
+ return size_based->second;
+}
} // namespace doris
diff --git a/be/src/storage/compaction/cumulative_compaction_policy.h b/be/src/storage/compaction/cumulative_compaction_policy.h
index 45ea6265c10..87ab3e8237d 100644
--- a/be/src/storage/compaction/cumulative_compaction_policy.h
+++ b/be/src/storage/compaction/cumulative_compaction_policy.h
@@ -22,6 +22,8 @@
#include <memory>
#include <string>
+#include <string_view>
+#include <unordered_map>
#include <vector>
#include "common/config.h"
@@ -192,6 +194,15 @@ public:
/// to construct the policy. Now it can product size based and num based policies.
static std::shared_ptr<CumulativeCompactionPolicy> create_cumulative_compaction_policy(
const std::string_view& compaction_policy);
+
+ /// Returns the policy registered under `compaction_policy` in `policies`, which must hold
+ /// the size-based policy. The name comes from the meta of tablet `tablet_id`; a name this
+ /// build does not know falls back to the size-based policy with a warning, the same as
+ /// create_cumulative_compaction_policy() does.
+ static std::shared_ptr<CumulativeCompactionPolicy> get_cumulative_compaction_policy(
+ const std::unordered_map<std::string_view, std::shared_ptr<CumulativeCompactionPolicy>>&
+ policies,
+ std::string_view compaction_policy, int64_t tablet_id);
};
} // namespace doris
diff --git a/be/src/storage/compaction_task_tracker.h b/be/src/storage/compaction_task_tracker.h
index 1b5ce7e8fed..b4856d5bd18 100644
--- a/be/src/storage/compaction_task_tracker.h
+++ b/be/src/storage/compaction_task_tracker.h
@@ -130,7 +130,7 @@ struct CompactionTaskInfo {
int64_t output_segments_num {0};
std::string output_version; // e.g. "[0-5]"
bool is_ordered_data_compaction {
- false}; // ordered/meta-link compaction path, including binlog quick path
+ false}; // ordered/meta-link compaction path, including changelog quick path
// ===== Merge performance =====
int64_t merge_latency_ms {0}; // merge rowsets latency (ms; 0 for ordered path)
diff --git a/be/src/storage/data_dir.cpp b/be/src/storage/data_dir.cpp
index 6419351700a..33f071bcc8f 100644
--- a/be/src/storage/data_dir.cpp
+++ b/be/src/storage/data_dir.cpp
@@ -519,12 +519,13 @@ Status DataDir::load() {
rowset_partition_id_eq_0_num, config::ignore_invalid_partition_id_rowset_num));
}
- // Row binlog rowset is now a normal rowset under its own binlog tablet, loaded above.
- // Index them by txn and tablet id so each base rowset can re-attach its paired binlog rowset.
- std::map<std::pair<int64_t, int64_t>, RowsetMetaSharedPtr> row_binlog_metas;
+ // Changelog rowset is now a normal rowset under its own changelog tablet, loaded above.
+ // Index them by txn and tablet id so each base rowset can re-attach its paired changelog
+ // rowset.
+ std::map<std::pair<int64_t, int64_t>, RowsetMetaSharedPtr> changelog_metas;
for (auto&& rowset_meta : dir_rowset_metas) {
- if (rowset_meta->is_row_binlog()) {
- row_binlog_metas[{rowset_meta->txn_id(), rowset_meta->tablet_id()}] = rowset_meta;
+ if (rowset_meta->is_changelog()) {
+ changelog_metas[{rowset_meta->txn_id(), rowset_meta->tablet_id()}] = rowset_meta;
}
}
@@ -551,42 +552,42 @@ Status DataDir::load() {
continue;
}
- // Committed row binlog rowset is recovered with base rowset.
- if (rowset_meta->is_row_binlog() &&
+ // Committed changelog rowset is recovered with base rowset.
+ if (rowset_meta->is_changelog() &&
rowset_meta->rowset_state() == RowsetStatePB::COMMITTED) {
continue;
}
- RowBinlogTxnInfo attach_row_binlog;
- if (auto it = row_binlog_metas.find(
- {rowset_meta->txn_id(), tablet->tablet_meta()->binlog_tablet_id()});
- it != row_binlog_metas.end()) {
- const RowsetMetaSharedPtr& attach_row_binlog_rowset_meta = it->second;
- DCHECK_EQ(attach_row_binlog_rowset_meta->rowset_state(), rowset_meta->rowset_state());
- TabletSharedPtr binlog_tablet = _engine.tablet_manager()->get_tablet(
- attach_row_binlog_rowset_meta->tablet_id());
- if (binlog_tablet == nullptr) {
- LOG(WARNING) << "could not find binlog tablet: "
- << attach_row_binlog_rowset_meta->tablet_id()
- << " for binlog<row> rowset: "
- << attach_row_binlog_rowset_meta->rowset_id();
+ ChangelogTxnInfo attach_changelog;
+ if (auto it = changelog_metas.find(
+ {rowset_meta->txn_id(), tablet->tablet_meta()->changelog_tablet_id()});
+ it != changelog_metas.end()) {
+ const RowsetMetaSharedPtr& attach_changelog_rowset_meta = it->second;
+ DCHECK_EQ(attach_changelog_rowset_meta->rowset_state(), rowset_meta->rowset_state());
+ TabletSharedPtr changelog_tablet =
+ _engine.tablet_manager()->get_tablet(attach_changelog_rowset_meta->tablet_id());
+ if (changelog_tablet == nullptr) {
+ LOG(WARNING) << "could not find changelog tablet: "
+ << attach_changelog_rowset_meta->tablet_id()
+ << " for changelog rowset: "
+ << attach_changelog_rowset_meta->rowset_id();
++invalid_rowset_counter;
continue;
}
- if (!attach_row_binlog_rowset_meta->tablet_schema()) {
- attach_row_binlog_rowset_meta->set_tablet_schema(binlog_tablet->tablet_schema());
+ if (!attach_changelog_rowset_meta->tablet_schema()) {
+ attach_changelog_rowset_meta->set_tablet_schema(changelog_tablet->tablet_schema());
}
- Status attach_create_status = binlog_tablet->create_rowset(
- attach_row_binlog_rowset_meta, &attach_row_binlog.rowset);
+ Status attach_create_status = changelog_tablet->create_rowset(
+ attach_changelog_rowset_meta, &attach_changelog.rowset);
if (!attach_create_status.ok()) {
- LOG(WARNING) << "could not create rowset from binlog<row> rowset meta: "
- << " rowset_id: " << attach_row_binlog_rowset_meta->rowset_id()
- << " rowset_type: " << attach_row_binlog_rowset_meta->rowset_type()
- << " rowset_state: " << attach_row_binlog_rowset_meta->rowset_state();
+ LOG(WARNING) << "could not create rowset from changelog rowset meta: "
+ << " rowset_id: " << attach_changelog_rowset_meta->rowset_id()
+ << " rowset_type: " << attach_changelog_rowset_meta->rowset_type()
+ << " rowset_state: " << attach_changelog_rowset_meta->rowset_state();
++invalid_rowset_counter;
continue;
}
- attach_row_binlog.tablet = std::move(binlog_tablet);
+ attach_changelog.tablet = std::move(changelog_tablet);
}
RowsetSharedPtr rowset;
@@ -600,65 +601,62 @@ Status DataDir::load() {
}
std::optional<BinlogFormatPB> binlog_format = std::nullopt;
- std::optional<RowsetMetaPB> attach_row_binlog_rowset_meta;
- if (attach_row_binlog.rowset != nullptr) {
- binlog_format = BinlogFormatPB::ROW;
- attach_row_binlog_rowset_meta =
- attach_row_binlog.rowset->rowset_meta()->get_rowset_pb();
+ std::optional<RowsetMetaPB> attach_changelog_rowset_meta;
+ if (attach_changelog.rowset != nullptr) {
+ binlog_format = BinlogFormatPB::CHANGELOG;
+ attach_changelog_rowset_meta = attach_changelog.rowset->rowset_meta()->get_rowset_pb();
}
- std::string attach_binlog_rowset_id =
- attach_row_binlog.rowset != nullptr
- ? attach_row_binlog.rowset->rowset_id().to_string()
+ std::string attach_changelog_rowset_id =
+ attach_changelog.rowset != nullptr
+ ? attach_changelog.rowset->rowset_id().to_string()
: "0";
if (rowset_meta->rowset_state() == RowsetStatePB::COMMITTED &&
rowset_meta->tablet_uid() == tablet->tablet_uid()) {
if (!rowset_meta->tablet_schema()) {
rowset_meta->set_tablet_schema(tablet->tablet_schema());
- RETURN_IF_ERROR(RowsetMetaManager::save(_meta, rowset_meta->tablet_uid(),
- rowset_meta->rowset_id(),
- rowset_meta->get_rowset_pb(), binlog_format,
- attach_row_binlog_rowset_meta));
+ RETURN_IF_ERROR(RowsetMetaManager::save(
+ _meta, rowset_meta->tablet_uid(), rowset_meta->rowset_id(),
+ rowset_meta->get_rowset_pb(), binlog_format, attach_changelog_rowset_meta));
}
std::vector<RowsetId> rowset_ids {rowset_meta->rowset_id()};
- if (attach_row_binlog.rowset != nullptr) {
- rowset_ids.emplace_back(attach_row_binlog.rowset->rowset_id());
+ if (attach_changelog.rowset != nullptr) {
+ rowset_ids.emplace_back(attach_changelog.rowset->rowset_id());
}
Status commit_txn_status = _engine.txn_manager()->commit_txn(
_meta, rowset_meta->partition_id(), rowset_meta->txn_id(),
rowset_meta->tablet_id(), rowset_meta->tablet_uid(), rowset_meta->load_id(),
rowset, _engine.pending_local_rowsets().add(rowset_ids), true, nullptr,
- attach_row_binlog);
+ attach_changelog);
if (commit_txn_status || commit_txn_status.is<PUSH_TRANSACTION_ALREADY_EXIST>()) {
LOG(INFO) << "successfully to add committed rowset: " << rowset_meta->rowset_id()
<< " to tablet: " << rowset_meta->tablet_id()
<< " schema hash: " << rowset_meta->tablet_schema_hash()
<< " for txn: " << rowset_meta->txn_id()
- << ", binlog<row> rowset: " << attach_binlog_rowset_id;
+ << ", changelog rowset: " << attach_changelog_rowset_id;
} else if (commit_txn_status.is<ErrorCode::INTERNAL_ERROR>()) {
LOG(WARNING) << "failed to add committed rowset: " << rowset_meta->rowset_id()
<< " to tablet: " << rowset_meta->tablet_id()
<< " for txn: " << rowset_meta->txn_id()
<< " error: " << commit_txn_status
- << ", binlog<row> rowset: " << attach_binlog_rowset_id;
+ << ", changelog rowset: " << attach_changelog_rowset_id;
return commit_txn_status;
} else {
LOG(WARNING) << "failed to add committed rowset: " << rowset_meta->rowset_id()
<< " to tablet: " << rowset_meta->tablet_id()
<< " for txn: " << rowset_meta->txn_id()
<< " error: " << commit_txn_status
- << ", binlog<row> rowset: " << attach_binlog_rowset_id;
+ << ", changelog rowset: " << attach_changelog_rowset_id;
}
} else if (rowset_meta->rowset_state() == RowsetStatePB::VISIBLE &&
rowset_meta->tablet_uid() == tablet->tablet_uid()) {
if (!rowset_meta->tablet_schema()) {
rowset_meta->set_tablet_schema(tablet->tablet_schema());
- RETURN_IF_ERROR(RowsetMetaManager::save(_meta, rowset_meta->tablet_uid(),
- rowset_meta->rowset_id(),
- rowset_meta->get_rowset_pb(), binlog_format,
- attach_row_binlog_rowset_meta));
+ RETURN_IF_ERROR(RowsetMetaManager::save(
+ _meta, rowset_meta->tablet_uid(), rowset_meta->rowset_id(),
+ rowset_meta->get_rowset_pb(), binlog_format, attach_changelog_rowset_meta));
}
Status publish_status = tablet->add_rowset(rowset);
if (!publish_status && !publish_status.is<PUSH_VERSION_ALREADY_EXIST>()) {
@@ -667,7 +665,7 @@ Status DataDir::load() {
<< " txn id:" << rowset_meta->txn_id()
<< " start_version: " << rowset_meta->version().first
<< " end_version: " << rowset_meta->version().second
- << ", binlog<row> rowset: " << attach_binlog_rowset_id;
+ << ", changelog rowset: " << attach_changelog_rowset_id;
}
} else {
LOG(WARNING) << "find invalid rowset: " << rowset_meta->rowset_id()
@@ -676,7 +674,7 @@ Status DataDir::load() {
<< " schema hash: " << rowset_meta->tablet_schema_hash()
<< " txn: " << rowset_meta->txn_id()
<< " current valid tablet uid: " << tablet->tablet_uid()
- << ", binlog<row> rowset: " << attach_binlog_rowset_id;
+ << ", changelog rowset: " << attach_changelog_rowset_id;
++invalid_rowset_counter;
}
}
diff --git a/be/src/storage/iterator/block_reader.cpp b/be/src/storage/iterator/block_reader.cpp
index 285cee951f2..9a3bb3c88cb 100644
--- a/be/src/storage/iterator/block_reader.cpp
+++ b/be/src/storage/iterator/block_reader.cpp
@@ -40,8 +40,8 @@
#include "core/data_type/data_type_number.h"
#include "exprs/aggregate/aggregate_function_reader.h"
#include "runtime/runtime_state.h"
-#include "storage/binlog.h"
-#include "storage/iterator/binlog_block_reader_utils.h"
+#include "storage/changelog.h"
+#include "storage/iterator/changelog_block_reader_utils.h"
#include "storage/iterator/vcollect_iterator.h"
#include "storage/olap_common.h"
#include "storage/olap_define.h"
@@ -76,11 +76,11 @@ Status BlockReader::next_block_with_aggregation(Block* block, bool* eof) {
return res;
}
-int64_t BlockReader::_read_binlog_op(const IColumn& col, size_t row) const {
+int64_t BlockReader::_read_changelog_op(const IColumn& col, size_t row) const {
const IColumn* cur = &col;
if (const auto* nullable = check_and_get_column<ColumnNullable>(*cur)) {
if (nullable->is_null_at(row)) {
- return binlog::ROW_BINLOG_UNKNOWN;
+ return changelog::CHANGELOG_OP_UNKNOWN;
}
cur = &nullable->get_nested_column();
}
@@ -89,10 +89,10 @@ int64_t BlockReader::_read_binlog_op(const IColumn& col, size_t row) const {
return int64_col->get_element(row);
}
- return binlog::ROW_BINLOG_UNKNOWN;
+ return changelog::CHANGELOG_OP_UNKNOWN;
}
-Status BlockReader::_write_binlog_op(IColumn& col, int64_t op) const {
+Status BlockReader::_write_changelog_op(IColumn& col, int64_t op) const {
IColumn* cur = &col;
ColumnNullable* nullable = nullptr;
if (auto* n = typeid_cast<ColumnNullable*>(cur)) {
@@ -112,16 +112,16 @@ Status BlockReader::_write_binlog_op(IColumn& col, int64_t op) const {
return Status::OK();
}
-// Resolves which source-block column to read from for a given binlog row position.
-// When use_before is true, return the ordinal of its __BEFORE__ mirror.
-// Binlog meta columns map to themselves.
+// Resolves which source-block column to read from for a given changelog row position.
+// When use_before is true, return the ordinal of its before-image mirror.
+// Changelog meta columns map to themselves.
uint32_t BlockReader::_resolve_source_column_ordinal(uint32_t ordinal, bool use_before) const {
return use_before ? _read_schema->before_column_ordinal(ordinal) : ordinal;
}
bool BlockReader::_min_delta_values_equal(size_t last_row) {
- const auto& value_column_pairs = _read_schema->row_binlog_value_column_pairs();
- if (!_read_schema->row_binlog_value_pairs_complete() || value_column_pairs.empty() ||
+ const auto& value_column_pairs = _read_schema->changelog_value_column_pairs();
+ if (!_read_schema->changelog_value_pairs_complete() || value_column_pairs.empty() ||
_min_delta_value_compare_unsupported) {
return false;
}
@@ -185,7 +185,7 @@ Status BlockReader::_append_change_row(MutableColumns& target_columns, const Blo
const int32_t op_ordinal = _read_schema->op_ordinal();
for (size_t ordinal = 0; ordinal < target_columns.size(); ++ordinal) {
if (static_cast<int32_t>(ordinal) == op_ordinal) {
- RETURN_IF_ERROR(_write_binlog_op(*target_columns[ordinal], output_op));
+ RETURN_IF_ERROR(_write_changelog_op(*target_columns[ordinal], output_op));
continue;
}
auto source_ordinal =
@@ -216,7 +216,7 @@ Status BlockReader::_min_delta_next_block(Block* block, bool* eof) {
const int32_t tso_ordinal = _read_schema->tso_ordinal();
const int32_t lsn_ordinal = _read_schema->lsn_ordinal();
const int32_t op_ordinal = _read_schema->op_ordinal();
- // A group is a run of consecutive rows sharing the same user key. Row-binlog reads are
+ // A group is a run of consecutive rows sharing the same user key. Changelog reads are
// globally key-ordered (ReaderParams::force_key_ordered_read), and the key columns are the
// leading num_key_columns() columns of every read block, so a group boundary is exactly
// where the user key changes. _stored_data_columns keeps the group's first row at index 0.
@@ -266,17 +266,17 @@ Status BlockReader::_min_delta_next_block(Block* block, bool* eof) {
}
}
size_t group_size = _stored_data_columns[0]->size();
- auto first_op = _read_binlog_op(*_stored_data_columns[op_ordinal], 0);
- auto last_op = _read_binlog_op(*_stored_data_columns[op_ordinal], group_size - 1);
- auto result = binlog::AggregateFunctionMinDelta::calculate_result(first_op, last_op);
+ auto first_op = _read_changelog_op(*_stored_data_columns[op_ordinal], 0);
+ auto last_op = _read_changelog_op(*_stored_data_columns[op_ordinal], group_size - 1);
+ auto result = changelog::AggregateFunctionMinDelta::calculate_result(first_op, last_op);
switch (result) {
- case binlog::AggregateFunctionMinDelta::ResultType::SKIP:
+ case changelog::AggregateFunctionMinDelta::ResultType::SKIP:
break;
- case binlog::AggregateFunctionMinDelta::ResultType::INSERT:
+ case changelog::AggregateFunctionMinDelta::ResultType::INSERT:
for (size_t ordinal = 0; ordinal < target_columns.size(); ++ordinal) {
if (static_cast<int32_t>(ordinal) == op_ordinal) {
- RETURN_IF_ERROR(_write_binlog_op(*target_columns[ordinal],
- binlog::STREAM_CHANGE_INSERT));
+ RETURN_IF_ERROR(_write_changelog_op(*target_columns[ordinal],
+ changelog::STREAM_CHANGE_INSERT));
} else {
// insert should use most recent value
target_columns[ordinal]->insert_from(*_stored_data_columns[ordinal],
@@ -285,12 +285,12 @@ Status BlockReader::_min_delta_next_block(Block* block, bool* eof) {
}
output_row_count++;
break;
- case binlog::AggregateFunctionMinDelta::ResultType::DELETE:
+ case changelog::AggregateFunctionMinDelta::ResultType::DELETE:
for (size_t ordinal = 0; ordinal < target_columns.size(); ++ordinal) {
const auto read_ordinal = static_cast<int32_t>(ordinal);
if (read_ordinal == op_ordinal) {
- RETURN_IF_ERROR(_write_binlog_op(*target_columns[ordinal],
- binlog::STREAM_CHANGE_DELETE));
+ RETURN_IF_ERROR(_write_changelog_op(*target_columns[ordinal],
+ changelog::STREAM_CHANGE_DELETE));
} else if (read_ordinal == lsn_ordinal || read_ordinal == tso_ordinal) {
target_columns[ordinal]->insert_from(*_stored_data_columns[ordinal],
group_size - 1);
@@ -303,16 +303,16 @@ Status BlockReader::_min_delta_next_block(Block* block, bool* eof) {
}
output_row_count++;
break;
- case binlog::AggregateFunctionMinDelta::ResultType::UPDATE_BEFORE_AFTER:
- if (binlog::is_valid_row_binlog_op(first_op) &&
- binlog::is_valid_row_binlog_op(last_op) &&
+ case changelog::AggregateFunctionMinDelta::ResultType::UPDATE_BEFORE_AFTER:
+ if (changelog::is_valid_changelog_op(first_op) &&
+ changelog::is_valid_changelog_op(last_op) &&
_min_delta_values_equal(group_size - 1)) {
break;
}
for (size_t ordinal = 0; ordinal < target_columns.size(); ++ordinal) {
if (static_cast<int32_t>(ordinal) == op_ordinal) {
- RETURN_IF_ERROR(_write_binlog_op(*target_columns[ordinal],
- binlog::STREAM_CHANGE_UPDATE_BEFORE));
+ RETURN_IF_ERROR(_write_changelog_op(*target_columns[ordinal],
+ changelog::STREAM_CHANGE_UPDATE_BEFORE));
} else {
auto source_ordinal =
_resolve_source_column_ordinal(static_cast<uint32_t>(ordinal), true);
@@ -323,8 +323,8 @@ Status BlockReader::_min_delta_next_block(Block* block, bool* eof) {
if (output_row_count >= batch_max_rows()) {
for (size_t ordinal = 0; ordinal < target_columns.size(); ++ordinal) {
if (static_cast<int32_t>(ordinal) == op_ordinal) {
- RETURN_IF_ERROR(_write_binlog_op(*_pending_row_columns[ordinal],
- binlog::STREAM_CHANGE_UPDATE_AFTER));
+ RETURN_IF_ERROR(_write_changelog_op(*_pending_row_columns[ordinal],
+ changelog::STREAM_CHANGE_UPDATE_AFTER));
} else {
_pending_row_columns[ordinal]->insert_from(*_stored_data_columns[ordinal],
group_size - 1);
@@ -334,8 +334,8 @@ Status BlockReader::_min_delta_next_block(Block* block, bool* eof) {
} else {
for (size_t ordinal = 0; ordinal < target_columns.size(); ++ordinal) {
if (static_cast<int32_t>(ordinal) == op_ordinal) {
- RETURN_IF_ERROR(_write_binlog_op(*target_columns[ordinal],
- binlog::STREAM_CHANGE_UPDATE_AFTER));
+ RETURN_IF_ERROR(_write_changelog_op(*target_columns[ordinal],
+ changelog::STREAM_CHANGE_UPDATE_AFTER));
} else {
target_columns[ordinal]->insert_from(*_stored_data_columns[ordinal],
group_size - 1);
@@ -354,7 +354,7 @@ Status BlockReader::_min_delta_next_block(Block* block, bool* eof) {
return Status::OK();
}
-// DETAIL reader: emits every recorded binlog change verbatim. APPEND -> single INSERT row,
+// DETAIL reader: emits every recorded changelog change verbatim. APPEND -> single INSERT row,
// DELETE -> single DELETE row, UPDATE -> a BEFORE+AFTER pair. When the AFTER row would
// overflow batch_max_rows(), it is parked in _pending_row_columns and flushed next call.
Status BlockReader::_detail_change_next_block(Block* block, bool* eof) {
@@ -379,27 +379,27 @@ Status BlockReader::_detail_change_next_block(Block* block, bool* eof) {
}
const Block& source_block = *_next_row.block;
const size_t row = _next_row.row_pos;
- int64_t op = _read_binlog_op(*source_block.get_by_position(op_ordinal).column, row);
- if (op == ROW_BINLOG_UPDATE) {
+ int64_t op = _read_changelog_op(*source_block.get_by_position(op_ordinal).column, row);
+ if (op == CHANGELOG_OP_UPDATE) {
RETURN_IF_ERROR(_append_change_row(target_columns, source_block, row,
- binlog::STREAM_CHANGE_UPDATE_BEFORE, true));
+ changelog::STREAM_CHANGE_UPDATE_BEFORE, true));
output_row_count++;
if (output_row_count >= batch_max_rows()) {
RETURN_IF_ERROR(_append_change_row(_pending_row_columns, source_block, row,
- binlog::STREAM_CHANGE_UPDATE_AFTER, false));
+ changelog::STREAM_CHANGE_UPDATE_AFTER, false));
_has_pending_row = true;
} else {
RETURN_IF_ERROR(_append_change_row(target_columns, source_block, row,
- binlog::STREAM_CHANGE_UPDATE_AFTER, false));
+ changelog::STREAM_CHANGE_UPDATE_AFTER, false));
output_row_count++;
}
- } else if (op == ROW_BINLOG_APPEND) {
+ } else if (op == CHANGELOG_OP_APPEND) {
RETURN_IF_ERROR(_append_change_row(target_columns, source_block, row,
- binlog::STREAM_CHANGE_INSERT, false));
+ changelog::STREAM_CHANGE_INSERT, false));
output_row_count++;
- } else if (op == ROW_BINLOG_DELETE) {
+ } else if (op == CHANGELOG_OP_DELETE) {
RETURN_IF_ERROR(_append_change_row(target_columns, source_block, row,
- binlog::STREAM_CHANGE_DELETE, true));
+ changelog::STREAM_CHANGE_DELETE, true));
output_row_count++;
}
@@ -458,7 +458,8 @@ Status BlockReader::_init_collect_iter(const ReaderParams& read_params) {
{
SCOPED_RAW_TIMER(&_stats.block_reader_vcollect_iter_init_timer_ns);
_is_rowsets_overlapping = _rowsets_not_mono_asc_disjoint(read_params);
- const bool is_min_delta_stream = read_params.binlog_scan_type == TBinlogScanType::MIN_DELTA;
+ const bool is_min_delta_stream =
+ read_params.changelog_scan_type == TChangelogScanType::MIN_DELTA;
const bool force_merge = read_params.read_orderby_key || is_min_delta_stream;
const bool is_reverse = !is_min_delta_stream && read_params.read_orderby_key_reverse;
_vcollect_iter.init(this, _is_rowsets_overlapping, force_merge, is_reverse);
@@ -552,19 +553,19 @@ Status BlockReader::init(const ReaderParams& read_params) {
SCOPED_RAW_TIMER(&_stats.tablet_reader_init_timer_ns);
RETURN_IF_ERROR(TabletReader::init(read_params));
- // A Row Binlog scan maps the before-image columns; every other read of this reader, which is
+ // A changelog scan maps the before-image columns; every other read of this reader, which is
// the one that merges rows across rowsets, builds the sequence mapping instead.
- const bool map_row_binlog_columns =
- read_params.binlog_scan_type == TBinlogScanType::MIN_DELTA ||
- read_params.binlog_scan_type == TBinlogScanType::DETAIL;
+ const bool map_changelog_columns =
+ read_params.changelog_scan_type == TChangelogScanType::MIN_DELTA ||
+ read_params.changelog_scan_type == TChangelogScanType::DETAIL;
const bool merge_by_sequence_mapping =
- !map_row_binlog_columns && _tablet_schema->has_seq_map() &&
+ !map_changelog_columns && _tablet_schema->has_seq_map() &&
_tablet_schema->keys_type() == UNIQUE_KEYS && !_direct_mode &&
!(read_params.reader_type == ReaderType::READER_QUERY &&
_tablet->enable_unique_key_merge_on_write());
RETURN_IF_ERROR(_read_schema->init_from_tablet_schema(
- *_tablet_schema, merge_by_sequence_mapping, map_row_binlog_columns));
- if (map_row_binlog_columns) {
+ *_tablet_schema, merge_by_sequence_mapping, map_changelog_columns));
+ if (map_changelog_columns) {
_min_delta_value_compare_unsupported = false;
}
@@ -588,13 +589,13 @@ Status BlockReader::init(const ReaderParams& read_params) {
// MIN_DELTA: collapse consecutive same-key changes to the minimum equivalent change set
// (e.g. INSERT+DELETE -> SKIP, INSERT+UPDATE -> INSERT). Reduces downstream traffic.
- if (read_params.binlog_scan_type == TBinlogScanType::MIN_DELTA) {
+ if (read_params.changelog_scan_type == TChangelogScanType::MIN_DELTA) {
_next_block_func = &BlockReader::_min_delta_next_block;
return Status::OK();
}
// DETAIL: emit every recorded change as-is, with BEFORE+AFTER rows for UPDATE.
// Used when the consumer needs full change history rather than the net delta.
- if (read_params.binlog_scan_type == TBinlogScanType::DETAIL) {
+ if (read_params.changelog_scan_type == TChangelogScanType::DETAIL) {
_next_block_func = &BlockReader::_detail_change_next_block;
return Status::OK();
}
diff --git a/be/src/storage/iterator/block_reader.h b/be/src/storage/iterator/block_reader.h
index d3615369454..fb70bbd8c9f 100644
--- a/be/src/storage/iterator/block_reader.h
+++ b/be/src/storage/iterator/block_reader.h
@@ -72,9 +72,9 @@ private:
Status _detail_change_next_block(Block* block, bool* eof);
- int64_t _read_binlog_op(const IColumn& col, size_t row) const;
+ int64_t _read_changelog_op(const IColumn& col, size_t row) const;
- Status _write_binlog_op(IColumn& col, int64_t op) const;
+ Status _write_changelog_op(IColumn& col, int64_t op) const;
uint32_t _resolve_source_column_ordinal(uint32_t ordinal, bool use_before) const;
diff --git a/be/src/storage/iterator/binlog_block_reader_utils.h b/be/src/storage/iterator/changelog_block_reader_utils.h
similarity index 79%
rename from be/src/storage/iterator/binlog_block_reader_utils.h
rename to be/src/storage/iterator/changelog_block_reader_utils.h
index 5b3bf53ce02..8a836892d5f 100644
--- a/be/src/storage/iterator/binlog_block_reader_utils.h
+++ b/be/src/storage/iterator/changelog_block_reader_utils.h
@@ -21,11 +21,11 @@
#include <cstddef>
#include <cstdint>
-#include "storage/binlog.h"
+#include "storage/changelog.h"
-namespace doris::binlog {
+namespace doris::changelog {
-constexpr int64_t ROW_BINLOG_UNKNOWN = 3;
+constexpr int64_t CHANGELOG_OP_UNKNOWN = 3;
constexpr int64_t STREAM_CHANGE_INSERT = 0;
constexpr int64_t STREAM_CHANGE_DELETE = 1;
@@ -33,13 +33,14 @@ constexpr int64_t STREAM_CHANGE_UPDATE_BEFORE = 2;
constexpr int64_t STREAM_CHANGE_UPDATE_AFTER = 3;
enum class MinDeltaResultType { SKIP, INSERT, DELETE, UPDATE_BEFORE_AFTER };
-// MIN_DELTA uses row binlog op codes as indices into a 2D lookup table, so we guard the op layout here.
-static_assert(doris::ROW_BINLOG_APPEND == 0 && doris::ROW_BINLOG_UPDATE == 1 &&
- doris::ROW_BINLOG_DELETE == 2,
- "row binlog op layout changed; update min-delta transition matrix");
+// MIN_DELTA uses changelog op codes as indices into a 2D lookup table, so we guard the op layout
+// here.
+static_assert(doris::CHANGELOG_OP_APPEND == 0 && doris::CHANGELOG_OP_UPDATE == 1 &&
+ doris::CHANGELOG_OP_DELETE == 2,
+ "changelog op layout changed; update min-delta transition matrix");
-inline bool is_valid_row_binlog_op(int64_t op) {
- return op >= doris::ROW_BINLOG_APPEND && op <= doris::ROW_BINLOG_DELETE;
+inline bool is_valid_changelog_op(int64_t op) {
+ return op >= doris::CHANGELOG_OP_APPEND && op <= doris::CHANGELOG_OP_DELETE;
}
inline MinDeltaResultType calculate_min_delta_result(int64_t first_op, int64_t last_op) {
@@ -67,7 +68,7 @@ inline MinDeltaResultType calculate_min_delta_result(int64_t first_op, int64_t l
}};
// Fallback for unknown/invalid op codes: avoid out-of-bounds and keep changes conservatively.
- if (!is_valid_row_binlog_op(first_op) || !is_valid_row_binlog_op(last_op)) {
+ if (!is_valid_changelog_op(first_op) || !is_valid_changelog_op(last_op)) {
return ResultType::UPDATE_BEFORE_AFTER;
}
@@ -76,7 +77,7 @@ inline MinDeltaResultType calculate_min_delta_result(int64_t first_op, int64_t l
/**
* MIN_DELTA result helper:
- * Given the first/last row binlog op within the same key group, returns the min-delta change type.
+ * Given the first/last changelog op within the same key group, returns the min-delta change type.
*/
class AggregateFunctionMinDelta {
public:
@@ -87,4 +88,4 @@ public:
}
};
-} // namespace doris::binlog
+} // namespace doris::changelog
diff --git a/be/src/storage/iterator/olap_data_convertor.h b/be/src/storage/iterator/olap_data_convertor.h
index 1fdcb93005a..5999a7fdfa9 100644
--- a/be/src/storage/iterator/olap_data_convertor.h
+++ b/be/src/storage/iterator/olap_data_convertor.h
@@ -320,7 +320,7 @@ private:
assert(column_data);
_values = column_data->get_data().data() + _row_pos;
- // DATA and ROW_BINLOG flush tasks may convert the same source Block concurrently,
+ // DATA and CHANGELOG flush tasks may convert the same source Block concurrently,
// so this converter must never mutate the source column or its nullable nested column.
// Most simple types can expose the requested source slice directly with zero copy.
// FLOAT/DOUBLE are special because storage requires a canonical NaN representation:
diff --git a/be/src/storage/iterator/vcollect_iterator.cpp b/be/src/storage/iterator/vcollect_iterator.cpp
index f761fe84639..50acc1c5ed3 100644
--- a/be/src/storage/iterator/vcollect_iterator.cpp
+++ b/be/src/storage/iterator/vcollect_iterator.cpp
@@ -45,7 +45,7 @@
#include "runtime/runtime_predicate.h"
#include "runtime/runtime_profile.h"
#include "runtime/runtime_state.h"
-#include "storage/binlog.h"
+#include "storage/changelog.h"
#include "storage/olap_common.h"
#include "storage/olap_define.h"
#include "storage/rowset/rowset.h"
@@ -80,7 +80,7 @@ void VCollectIterator::init(TabletReader* reader, bool ori_data_overlapping, boo
_merge = false;
}
- if (_reader->_reader_context.read_row_binlog) {
+ if (_reader->_reader_context.read_changelog) {
_merge = false;
}
@@ -236,7 +236,7 @@ bool VCollectIterator::LevelIteratorComparator::operator()(LevelIterator* lhs, L
// for AGG_KEYS if a version is deleted, the lower version no need to agg_update
// Tie-break direction depends on `_small_seq_first`:
// - false (UNIQUE_KEYS sequence column): larger value sorts first (cmp_res < 0 => lhs sorts first).
- // - true (row binlog TSO column): smaller value sorts first (cmp_res > 0 => lhs sorts first).
+ // - true (changelog TSO column): smaller value sorts first (cmp_res > 0 => lhs sorts first).
bool lower = (cmp_res != 0) ? (_small_seq_first ? (cmp_res > 0) : (cmp_res < 0))
: (_use_insert_order_when_same ? (lhs->version() > rhs->version())
: (lhs->version() < rhs->version()));
diff --git a/be/src/storage/iterator/vcollect_iterator.h b/be/src/storage/iterator/vcollect_iterator.h
index 6f0f9d1a6f4..8c3d853797a 100644
--- a/be/src/storage/iterator/vcollect_iterator.h
+++ b/be/src/storage/iterator/vcollect_iterator.h
@@ -171,7 +171,7 @@ private:
bool _use_insert_order_when_same = false;
// Ordering direction of the tie-break column when keys are equal:
// - false : larger value sorts first (UNIQUE_KEYS sequence column).
- // - true : smaller value sorts first (row binlog TSO column).
+ // - true : smaller value sorts first (changelog TSO column).
// The two cases are mutually exclusive (DCHECK at construction site).
bool _small_seq_first = false;
};
diff --git a/be/src/storage/iterator/vertical_block_reader.cpp b/be/src/storage/iterator/vertical_block_reader.cpp
index 659e9494db5..2913b144cf9 100644
--- a/be/src/storage/iterator/vertical_block_reader.cpp
+++ b/be/src/storage/iterator/vertical_block_reader.cpp
@@ -271,7 +271,7 @@ Status VerticalBlockReader::init(const ReaderParams& read_params,
RETURN_IF_ERROR(_read_schema->init_from_tablet_schema(*_tablet_schema,
/*merge_by_sequence_mapping=*/false,
- /*map_row_binlog_columns=*/false));
+ /*map_changelog_columns=*/false));
auto status = _init_collect_iter(read_params, sample_info);
if (!status.ok()) [[unlikely]] {
diff --git a/be/src/storage/iterator/vgeneric_iterators.h b/be/src/storage/iterator/vgeneric_iterators.h
index 9176f1daf9d..876bb620aa9 100644
--- a/be/src/storage/iterator/vgeneric_iterators.h
+++ b/be/src/storage/iterator/vgeneric_iterators.h
@@ -171,7 +171,7 @@ private:
bool _use_insert_order_when_same = false;
// Tie-break direction on the sequence column when keys are equal:
// false = larger value sorts first (UNIQUE_KEYS sequence column);
- // true = smaller value sorts first (row binlog TSO column).
+ // true = smaller value sorts first (changelog TSO column).
bool _small_seq_first = false;
bool _valid = false;
mutable bool _skip = false;
diff --git a/be/src/storage/iterators.h b/be/src/storage/iterators.h
index 5c369b685b2..cad005f5c3e 100644
--- a/be/src/storage/iterators.h
+++ b/be/src/storage/iterators.h
@@ -136,8 +136,8 @@ public:
// For rows with the same key, use ascending order (small-to-large) for tie-breakers.
// For example, use lower rowset version / segment id first.
bool use_insert_order_when_same = false;
- bool read_row_binlog = false;
- int binlog_tso_idx = -1;
+ bool read_changelog = false;
+ int changelog_tso_idx = -1;
// columns for orderby keys
std::vector<uint32_t>* read_orderby_key_columns = nullptr;
io::IOContext io_ctx;
diff --git a/be/src/storage/merger.cpp b/be/src/storage/merger.cpp
index b89ec47b064..440fe565c1f 100644
--- a/be/src/storage/merger.cpp
+++ b/be/src/storage/merger.cpp
@@ -139,13 +139,13 @@ Status Merger::vmerge_rowsets(BaseTabletSPtr tablet, ReaderType reader_type,
TabletReader::ReaderParams reader_params;
reader_params.tablet = tablet;
reader_params.reader_type = reader_type;
- reader_params.read_row_binlog = tablet->is_row_binlog_tablet();
+ reader_params.read_changelog = tablet->is_changelog_tablet();
reader_params.runtime_state = runtime_state;
if (key_range.has_value()) {
RETURN_IF_ERROR(set_key_range(&reader_params, cur_tablet_schema, *key_range));
}
- if (reader_params.read_row_binlog) {
- // Row-binlog horizontal (non-vertical) compaction must produce a globally
+ if (reader_params.read_changelog) {
+ // Changelog horizontal (non-vertical) compaction must produce a globally
// (key, TSO)-ordered output.
reader_params.read_orderby_key = true;
reader_params.force_key_ordered_read = true;
@@ -171,7 +171,7 @@ Status Merger::vmerge_rowsets(BaseTabletSPtr tablet, ReaderType reader_type,
if (!tablet->tablet_schema()->cluster_key_uids().empty()) {
reader_params.delete_bitmap = tablet->tablet_meta()->delete_bitmap_ptr();
}
- if (reader_params.read_row_binlog) {
+ if (reader_params.read_changelog) {
reader_params.delete_bitmap = tablet->tablet_meta()->delete_bitmap_ptr();
}
@@ -354,7 +354,7 @@ Status Merger::vertical_compact_one_group(
reader_params.key_group_cluster_key_idxes = key_group_cluster_key_idxes;
reader_params.tablet = tablet;
reader_params.reader_type = reader_type;
- reader_params.read_row_binlog = tablet->is_row_binlog_tablet();
+ reader_params.read_changelog = tablet->is_changelog_tablet();
reader_params.runtime_state = runtime_state;
reader_params.enable_sparse_optimization = enable_sparse_optimization;
if (key_range.has_value()) {
@@ -383,7 +383,7 @@ Status Merger::vertical_compact_one_group(
reader_params.delete_bitmap = tablet->tablet_meta()->delete_bitmap_ptr();
has_cluster_key = true;
}
- if (reader_params.read_row_binlog) {
+ if (reader_params.read_changelog) {
reader_params.delete_bitmap = tablet->tablet_meta()->delete_bitmap_ptr();
}
diff --git a/be/src/storage/mow/historical_row_fetcher.h b/be/src/storage/mow/historical_row_fetcher.h
index c3a7422f7a4..ae967d9693d 100644
--- a/be/src/storage/mow/historical_row_fetcher.h
+++ b/be/src/storage/mow/historical_row_fetcher.h
@@ -48,7 +48,7 @@ public:
void plan_fixed_read(const RowLocation& loc, size_t dst_pos);
void plan_flexible_read(const RowLocation& loc, size_t dst_pos, const BitmapValue& skip_bitmap);
- // Fixed partial update and the binlog AFTER image. Same behavior as
+ // Fixed partial update and the changelog AFTER image. Same behavior as
// FixedReadPlan::fill_missing_columns, including its default / null / auto-inc order and its
// handling of delete-signed old rows.
Status fill_missing_columns(const TabletSchema& tablet_schema, Block& full_block,
@@ -71,7 +71,7 @@ public:
uint32_t block_start_pos, const Block* block,
std::vector<BitmapValue>* skip_bitmaps) const;
- // A raw read on the fixed plan: no fill steps at all. Used by the binlog BEFORE image, where
+ // A raw read on the fixed plan: no fill steps at all. Used by the changelog BEFORE image, where
// rows missing from *read_index stay NULL.
Status read_columns(const TabletSchema& tablet_schema, std::vector<uint32_t> cids_to_read,
Block& dst_block, std::map<uint32_t, uint32_t>* read_index,
diff --git a/be/src/storage/mow/key_probe.h b/be/src/storage/mow/key_probe.h
index 74224ee8872..424949e0da3 100644
--- a/be/src/storage/mow/key_probe.h
+++ b/be/src/storage/mow/key_probe.h
@@ -73,7 +73,7 @@ class MowKeyProbe {
public:
// Which rows this probe marks deleted in mow_context->delete_bitmap.
enum class MarkDeleted : uint8_t {
- // Pure lookup: marking is somebody else's job (the row binlog history read).
+ // Pure lookup: marking is somebody else's job (the changelog history read).
NONE = 0,
// Mark the old row, i.e. the row already in the table that the incoming row replaces. On a
// sequence loss nothing is marked at all: the caller of this mode drops the losing row
@@ -128,11 +128,11 @@ public:
}};
}
- // The row binlog history lookup: marks nothing, and keeps the old values of a sequence loser,
- // and of a delete-signed row when a __BEFORE__* image is wanted.
- static MowKeyProbe for_row_binlog(BaseTablet* tablet, TabletSchema* lookup_schema,
- bool has_sequence_col,
- std::shared_ptr<MowContext> mow_context, bool write_before) {
+ // The changelog history lookup: marks nothing, and keeps the old values of a sequence loser,
+ // and of a delete-signed row when a before image is wanted.
+ static MowKeyProbe for_changelog(BaseTablet* tablet, TabletSchema* lookup_schema,
+ bool has_sequence_col, std::shared_ptr<MowContext> mow_context,
+ bool write_before) {
return MowKeyProbe {tablet,
lookup_schema,
has_sequence_col,
diff --git a/be/src/storage/olap_common.h b/be/src/storage/olap_common.h
index 16931f0a4e5..f001b10e094 100644
--- a/be/src/storage/olap_common.h
+++ b/be/src/storage/olap_common.h
@@ -59,8 +59,8 @@ enum CompactionType {
BASE_COMPACTION = 1,
CUMULATIVE_COMPACTION = 2,
FULL_COMPACTION = 3,
- // Only used by scheduler to route row-binlog tablets to the binlog thread pool.
- CUMU_BINLOG_COMPACTION = 4
+ // Only used by scheduler to route changelog tablets to the changelog thread pool.
+ CHANGELOG_COMPACTION = 4
};
struct CompactionScoreStats {
diff --git a/be/src/storage/olap_server.cpp b/be/src/storage/olap_server.cpp
index d1667998c72..3061c6b782c 100644
--- a/be/src/storage/olap_server.cpp
+++ b/be/src/storage/olap_server.cpp
@@ -63,7 +63,7 @@
#include "storage/compaction/cold_data_compaction.h"
#include "storage/compaction/compaction_permit_limiter.h"
#include "storage/compaction/cumulative_compaction.h"
-#include "storage/compaction/cumulative_compaction_binlog_policy.h"
+#include "storage/compaction/cumulative_compaction_changelog_policy.h"
#include "storage/compaction/cumulative_compaction_policy.h"
#include "storage/compaction/cumulative_compaction_time_series_policy.h"
#include "storage/compaction_task_tracker.h"
@@ -106,7 +106,7 @@ CompactionSubmitRegistry::CompactionSubmitRegistry(CompactionSubmitRegistry&& r)
std::swap(_tablet_submitted_cumu_compaction, r._tablet_submitted_cumu_compaction);
std::swap(_tablet_submitted_base_compaction, r._tablet_submitted_base_compaction);
std::swap(_tablet_submitted_full_compaction, r._tablet_submitted_full_compaction);
- std::swap(_tablet_submitted_binlog_compaction, r._tablet_submitted_binlog_compaction);
+ std::swap(_tablet_submitted_changelog_compaction, r._tablet_submitted_changelog_compaction);
}
CompactionSubmitRegistry CompactionSubmitRegistry::create_snapshot() {
@@ -115,7 +115,7 @@ CompactionSubmitRegistry CompactionSubmitRegistry::create_snapshot() {
CompactionSubmitRegistry registry;
registry._tablet_submitted_base_compaction = _tablet_submitted_base_compaction;
registry._tablet_submitted_cumu_compaction = _tablet_submitted_cumu_compaction;
- registry._tablet_submitted_binlog_compaction = _tablet_submitted_binlog_compaction;
+ registry._tablet_submitted_changelog_compaction = _tablet_submitted_changelog_compaction;
return registry;
}
@@ -125,7 +125,7 @@ void CompactionSubmitRegistry::reset(const std::vector<DataDir*>& stores) {
for (const auto& store : stores) {
_tablet_submitted_cumu_compaction[store] = {};
_tablet_submitted_base_compaction[store] = {};
- _tablet_submitted_binlog_compaction[store] = {};
+ _tablet_submitted_changelog_compaction[store] = {};
}
}
@@ -185,8 +185,8 @@ CompactionSubmitRegistry::TabletSet& CompactionSubmitRegistry::_get_tablet_set(
return _tablet_submitted_cumu_compaction[dir];
case CompactionType::FULL_COMPACTION:
return _tablet_submitted_full_compaction[dir];
- case CompactionType::CUMU_BINLOG_COMPACTION:
- return _tablet_submitted_binlog_compaction[dir];
+ case CompactionType::CHANGELOG_COMPACTION:
+ return _tablet_submitted_changelog_compaction[dir];
default:
CHECK(false) << "invalid compaction type";
}
@@ -211,8 +211,8 @@ static int32_t get_base_compaction_threads_num(size_t data_dirs_num) {
return threads_num;
}
-static int32_t get_binlog_compaction_threads_num(size_t data_dirs_num) {
- int32_t threads_num = config::max_binlog_compaction_threads;
+static int32_t get_changelog_compaction_threads_num(size_t data_dirs_num) {
+ int32_t threads_num = config::max_changelog_compaction_threads;
if (threads_num == -1) {
threads_num = cast_set<int32_t>(data_dirs_num);
}
@@ -250,7 +250,7 @@ Status StorageEngine::start_bg_threads(std::shared_ptr<WorkloadGroup> wg_sptr) {
auto base_compaction_threads = get_base_compaction_threads_num(data_dirs.size());
auto cumu_compaction_threads = get_cumu_compaction_threads_num(data_dirs.size());
- auto binlog_compaction_threads = get_binlog_compaction_threads_num(data_dirs.size());
+ auto changelog_compaction_threads = get_changelog_compaction_threads_num(data_dirs.size());
RETURN_IF_ERROR(ThreadPoolBuilder("BaseCompactionTaskThreadPool")
.set_min_threads(base_compaction_threads)
@@ -260,11 +260,12 @@ Status StorageEngine::start_bg_threads(std::shared_ptr<WorkloadGroup> wg_sptr) {
.set_min_threads(cumu_compaction_threads)
.set_max_threads(cumu_compaction_threads)
.build(&_cumu_compaction_thread_pool));
- RETURN_IF_ERROR(ThreadPoolBuilder("BinlogCompactionTaskThreadPool")
- .set_min_threads(binlog_compaction_threads)
- .set_max_threads(binlog_compaction_threads)
- .set_max_queue_size(binlog_compaction_threads)
- .build(&_binlog_compaction_thread_pool));
+ // ATTN(changelog-rename): formerly "BinlogCompactionTaskThreadPool"
+ RETURN_IF_ERROR(ThreadPoolBuilder("ChangelogCompactionTaskThreadPool")
+ .set_min_threads(changelog_compaction_threads)
+ .set_max_threads(changelog_compaction_threads)
+ .set_max_queue_size(changelog_compaction_threads)
+ .build(&_changelog_compaction_thread_pool));
if (config::enable_segcompaction) {
RETURN_IF_ERROR(ThreadPoolBuilder("SegCompactionTaskThreadPool")
@@ -286,12 +287,13 @@ Status StorageEngine::start_bg_threads(std::shared_ptr<WorkloadGroup> wg_sptr) {
&_compaction_tasks_producer_thread));
LOG(INFO) << "compaction tasks producer thread started";
+ // ATTN(changelog-rename): formerly "binlog_compaction_tasks_producer_thread"
RETURN_IF_ERROR(Thread::create(
- "StorageEngine", "binlog_compaction_tasks_producer_thread",
- [this]() { this->_binlog_compaction_tasks_producer_callback(); },
- &_binlog_compaction_tasks_producer_thread));
- LOG(INFO) << "binlog compaction tasks producer thread started";
- RETURN_IF_ERROR(_start_row_binlog_ttl_scanner());
+ "StorageEngine", "changelog_compaction_tasks_producer_thread",
+ [this]() { this->_changelog_compaction_tasks_producer_callback(); },
+ &_changelog_compaction_tasks_producer_thread));
+ LOG(INFO) << "changelog compaction tasks producer thread started";
+ RETURN_IF_ERROR(_start_changelog_ttl_scanner());
int32_t max_checkpoint_thread_num = config::max_meta_checkpoint_threads;
if (max_checkpoint_thread_num < 0) {
@@ -631,23 +633,23 @@ void StorageEngine::_adjust_compaction_thread_num() {
}
}
- auto binlog_compaction_threads_num = get_binlog_compaction_threads_num(_store_map.size());
- if (_binlog_compaction_thread_pool->max_threads() != binlog_compaction_threads_num) {
- int old_max_threads = _binlog_compaction_thread_pool->max_threads();
- Status status =
- _binlog_compaction_thread_pool->set_max_threads(binlog_compaction_threads_num);
+ auto changelog_compaction_threads_num = get_changelog_compaction_threads_num(_store_map.size());
+ if (_changelog_compaction_thread_pool->max_threads() != changelog_compaction_threads_num) {
+ int old_max_threads = _changelog_compaction_thread_pool->max_threads();
+ Status status = _changelog_compaction_thread_pool->set_max_threads(
+ changelog_compaction_threads_num);
if (status.ok()) {
- VLOG_NOTICE << "update binlog compaction thread pool max_threads from "
- << old_max_threads << " to " << binlog_compaction_threads_num;
+ VLOG_NOTICE << "update changelog compaction thread pool max_threads from "
+ << old_max_threads << " to " << changelog_compaction_threads_num;
}
}
- if (_binlog_compaction_thread_pool->min_threads() != binlog_compaction_threads_num) {
- int old_min_threads = _binlog_compaction_thread_pool->min_threads();
- Status status =
- _binlog_compaction_thread_pool->set_min_threads(binlog_compaction_threads_num);
+ if (_changelog_compaction_thread_pool->min_threads() != changelog_compaction_threads_num) {
+ int old_min_threads = _changelog_compaction_thread_pool->min_threads();
+ Status status = _changelog_compaction_thread_pool->set_min_threads(
+ changelog_compaction_threads_num);
if (status.ok()) {
- VLOG_NOTICE << "update binlog compaction thread pool min_threads from "
- << old_min_threads << " to " << binlog_compaction_threads_num;
+ VLOG_NOTICE << "update changelog compaction thread pool min_threads from "
+ << old_min_threads << " to " << changelog_compaction_threads_num;
}
}
}
@@ -767,12 +769,12 @@ void StorageEngine::_compaction_tasks_producer_callback() {
} while (!_stop_background_threads_latch.wait_for(std::chrono::milliseconds(interval)));
}
-void StorageEngine::_binlog_compaction_tasks_producer_callback() {
- LOG(INFO) << "try to start binlog compaction producer process!";
+void StorageEngine::_changelog_compaction_tasks_producer_callback() {
+ LOG(INFO) << "try to start changelog compaction producer process!";
std::vector<DataDir*> data_dirs = get_stores();
- int64_t last_binlog_score_update_time = 0;
+ int64_t last_changelog_score_update_time = 0;
static const int64_t check_score_interval_ms = 5000;
int64_t interval = config::generate_compaction_tasks_interval_ms;
@@ -784,20 +786,20 @@ void StorageEngine::_binlog_compaction_tasks_producer_callback() {
_adjust_compaction_thread_num();
bool check_score = false;
- if (cur_time - last_binlog_score_update_time >= check_score_interval_ms) {
+ if (cur_time - last_changelog_score_update_time >= check_score_interval_ms) {
check_score = true;
- last_binlog_score_update_time = cur_time;
+ last_changelog_score_update_time = cur_time;
}
std::vector<TabletCompactionContext> tablet_compaction_contexts =
- _generate_compaction_tasks(CompactionType::CUMU_BINLOG_COMPACTION, data_dirs,
+ _generate_compaction_tasks(CompactionType::CHANGELOG_COMPACTION, data_dirs,
check_score);
for (const auto& tablet_compaction_context : tablet_compaction_contexts) {
const auto& tablet = tablet_compaction_context.tablet;
- Status st = _submit_compaction_task(tablet, CompactionType::CUMU_BINLOG_COMPACTION,
+ Status st = _submit_compaction_task(tablet, CompactionType::CHANGELOG_COMPACTION,
false, 0);
if (!st.ok()) {
- LOG(WARNING) << "failed to submit binlog compaction task for tablet: "
+ LOG(WARNING) << "failed to submit changelog compaction task for tablet: "
<< tablet->tablet_id() << ", err: " << st;
}
}
@@ -826,7 +828,7 @@ void StorageEngine::get_tablet_rowset_versions(const PGetTabletVersionsRequest*
bool need_generate_compaction_tasks(int task_cnt_per_disk, int thread_per_disk,
CompactionType compaction_type, bool all_base) {
- if (compaction_type == CompactionType::CUMU_BINLOG_COMPACTION) {
+ if (compaction_type == CompactionType::CHANGELOG_COMPACTION) {
return task_cnt_per_disk < thread_per_disk;
}
@@ -874,8 +876,8 @@ int get_concurrent_per_disk(int64_t max_score, int thread_per_disk) {
}
int32_t disk_compaction_slot_num(const DataDir& data_dir, CompactionType compaction_type) {
- if (compaction_type == CompactionType::CUMU_BINLOG_COMPACTION) {
- return config::binlog_compaction_task_num_per_disk;
+ if (compaction_type == CompactionType::CHANGELOG_COMPACTION) {
+ return config::changelog_compaction_task_num_per_disk;
}
return data_dir.is_ssd_disk() ? config::compaction_task_num_per_fast_disk
: config::compaction_task_num_per_disk;
@@ -893,7 +895,7 @@ std::vector<TabletCompactionContext> StorageEngine::_generate_compaction_tasks(
CompactionType compaction_type, std::vector<DataDir*>& data_dirs, bool check_score) {
DCHECK(compaction_type == CompactionType::BASE_COMPACTION ||
compaction_type == CompactionType::CUMULATIVE_COMPACTION ||
- compaction_type == CompactionType::CUMU_BINLOG_COMPACTION);
+ compaction_type == CompactionType::CHANGELOG_COMPACTION);
TEST_SYNC_POINT_RETURN_WITH_VALUE("olap_server::_generate_compaction_tasks.return_empty",
std::vector<TabletCompactionContext> {});
_update_cumulative_compaction_policy();
@@ -912,9 +914,9 @@ std::vector<TabletCompactionContext> StorageEngine::_generate_compaction_tasks(
for (auto* data_dir : data_dirs) {
bool need_pick_tablet = true;
uint32_t executing_task_num =
- compaction_type == CompactionType::CUMU_BINLOG_COMPACTION
+ compaction_type == CompactionType::CHANGELOG_COMPACTION
? compaction_registry_snapshot.count_executing_compaction(
- data_dir, CompactionType::CUMU_BINLOG_COMPACTION)
+ data_dir, CompactionType::CHANGELOG_COMPACTION)
: compaction_registry_snapshot.count_executing_cumu_and_base(data_dir);
need_pick_tablet = has_free_compaction_slot(&compaction_registry_snapshot, data_dir,
compaction_type, executing_task_num);
@@ -975,9 +977,9 @@ std::vector<TabletCompactionContext> StorageEngine::_generate_compaction_tasks(
max_score_stats.size_based_max_score);
update_policy_score(DorisMetrics::instance()->tablet_time_series_max_compaction_score,
max_score_stats.time_series_max_score);
- } else if (compaction_type == CompactionType::CUMU_BINLOG_COMPACTION &&
+ } else if (compaction_type == CompactionType::CHANGELOG_COMPACTION &&
max_score_stats.max_score > 0) {
- DorisMetrics::instance()->tablet_binlog_max_compaction_score->set_value(
+ DorisMetrics::instance()->tablet_changelog_max_compaction_score->set_value(
max_score_stats.max_score);
}
}
@@ -993,9 +995,9 @@ void StorageEngine::_update_cumulative_compaction_policy() {
_cumulative_compaction_policies[CUMULATIVE_TIME_SERIES_POLICY] =
CumulativeCompactionPolicyFactory::create_cumulative_compaction_policy(
CUMULATIVE_TIME_SERIES_POLICY);
- _cumulative_compaction_policies[CUMULATIVE_BINLOG_POLICY] =
+ _cumulative_compaction_policies[CUMULATIVE_CHANGELOG_POLICY] =
CumulativeCompactionPolicyFactory::create_cumulative_compaction_policy(
- CUMULATIVE_BINLOG_POLICY);
+ CUMULATIVE_CHANGELOG_POLICY);
}
}
@@ -1005,9 +1007,10 @@ CumuCompactionPolicyTable StorageEngine::_snapshot_cumulative_compaction_policy(
}
std::shared_ptr<CumulativeCompactionPolicy> StorageEngine::_get_cumulative_compaction_policy(
- std::string_view compaction_policy) {
+ std::string_view compaction_policy, int64_t tablet_id) {
std::lock_guard<std::mutex> lock(_cumulative_compaction_policy_mtx);
- return _cumulative_compaction_policies.at(compaction_policy);
+ return CumulativeCompactionPolicyFactory::get_cumulative_compaction_policy(
+ _cumulative_compaction_policies, compaction_policy, tablet_id);
}
void StorageEngine::_pop_tablet_from_submitted_compaction(TabletSharedPtr tablet,
@@ -1035,11 +1038,11 @@ Status StorageEngine::_submit_compaction_task(TabletSharedPtr tablet,
compaction, permits);
if (st.ok() && permits > 0) {
if (!force) {
- if (compaction_type == CompactionType::CUMU_BINLOG_COMPACTION) {
+ if (compaction_type == CompactionType::CHANGELOG_COMPACTION) {
if (!_permit_limiter.try_request(permits)) {
_pop_tablet_from_submitted_compaction(tablet, compaction_type);
tablet->compaction_stage = CompactionStage::NOT_SCHEDULED;
- return Status::TooManyTasks("Insufficient binlog compaction permits");
+ return Status::TooManyTasks("Insufficient changelog compaction permits");
}
} else {
_permit_limiter.request(permits);
@@ -1064,7 +1067,7 @@ Status StorageEngine::_submit_compaction_task(TabletSharedPtr tablet,
case CompactionType::FULL_COMPACTION:
info.compaction_type = CompactionProfileType::FULL;
break;
- case CompactionType::CUMU_BINLOG_COMPACTION:
+ case CompactionType::CHANGELOG_COMPACTION:
info.compaction_type = CompactionProfileType::CUMULATIVE;
break;
default:
@@ -1095,9 +1098,9 @@ Status StorageEngine::_submit_compaction_task(TabletSharedPtr tablet,
thread_pool = &_cumu_compaction_thread_pool;
compaction_type_name = "CUMU";
break;
- case CompactionType::CUMU_BINLOG_COMPACTION:
- thread_pool = &_binlog_compaction_thread_pool;
- compaction_type_name = "BINLOG";
+ case CompactionType::CHANGELOG_COMPACTION:
+ thread_pool = &_changelog_compaction_thread_pool;
+ compaction_type_name = "CHANGELOG";
break;
case CompactionType::BASE_COMPACTION:
case CompactionType::FULL_COMPACTION:
@@ -1168,10 +1171,10 @@ void StorageEngine::_handle_compaction(TabletSharedPtr tablet,
DorisMetrics::instance()->base_compaction_task_running_total->increment(1);
DorisMetrics::instance()->base_compaction_task_pending_total->set_value(
_base_compaction_thread_pool->get_queue_size());
- } else if (compaction_type == CompactionType::CUMU_BINLOG_COMPACTION) {
- DorisMetrics::instance()->binlog_compaction_task_running_total->increment(1);
- DorisMetrics::instance()->binlog_compaction_task_pending_total->set_value(
- _binlog_compaction_thread_pool->get_queue_size());
+ } else if (compaction_type == CompactionType::CHANGELOG_COMPACTION) {
+ DorisMetrics::instance()->changelog_compaction_task_running_total->increment(1);
+ DorisMetrics::instance()->changelog_compaction_task_pending_total->set_value(
+ _changelog_compaction_thread_pool->get_queue_size());
}
bool is_large_task = true;
Defer defer {[&]() {
@@ -1196,10 +1199,10 @@ void StorageEngine::_handle_compaction(TabletSharedPtr tablet,
DorisMetrics::instance()->base_compaction_task_running_total->increment(-1);
DorisMetrics::instance()->base_compaction_task_pending_total->set_value(
_base_compaction_thread_pool->get_queue_size());
- } else if (compaction_type == CompactionType::CUMU_BINLOG_COMPACTION) {
- DorisMetrics::instance()->binlog_compaction_task_running_total->increment(-1);
- DorisMetrics::instance()->binlog_compaction_task_pending_total->set_value(
- _binlog_compaction_thread_pool->get_queue_size());
+ } else if (compaction_type == CompactionType::CHANGELOG_COMPACTION) {
+ DorisMetrics::instance()->changelog_compaction_task_running_total->increment(-1);
+ DorisMetrics::instance()->changelog_compaction_task_pending_total->set_value(
+ _changelog_compaction_thread_pool->get_queue_size());
}
}};
do {
@@ -1293,8 +1296,8 @@ Status StorageEngine::submit_compaction_task(TabletSharedPtr tablet, CompactionT
if (tablet->get_cumulative_compaction_policy() == nullptr ||
tablet->get_cumulative_compaction_policy()->name() !=
tablet->tablet_meta()->compaction_policy()) {
- tablet->set_cumulative_compaction_policy(
- _get_cumulative_compaction_policy(tablet->tablet_meta()->compaction_policy()));
+ tablet->set_cumulative_compaction_policy(_get_cumulative_compaction_policy(
+ tablet->tablet_meta()->compaction_policy(), tablet->tablet_id()));
}
tablet->set_skip_compaction(false);
return _submit_compaction_task(tablet, compaction_type, force, trigger_method);
@@ -1664,8 +1667,8 @@ void StorageEngine::_handle_cold_data_compaction(TabletSharedPtr t) {
_update_cumulative_compaction_policy();
if (t->get_cumulative_compaction_policy() == nullptr ||
t->get_cumulative_compaction_policy()->name() != t->tablet_meta()->compaction_policy()) {
- t->set_cumulative_compaction_policy(
- _get_cumulative_compaction_policy(t->tablet_meta()->compaction_policy()));
+ t->set_cumulative_compaction_policy(_get_cumulative_compaction_policy(
+ t->tablet_meta()->compaction_policy(), t->tablet_id()));
}
auto st = compaction->prepare_compact();
diff --git a/be/src/storage/partial_update_info.cpp b/be/src/storage/partial_update_info.cpp
index a44f8ded9f7..9c469928e19 100644
--- a/be/src/storage/partial_update_info.cpp
+++ b/be/src/storage/partial_update_info.cpp
@@ -1373,7 +1373,7 @@ Status BlockAggregator::aggregate_for_flexible_partial_update(
std::vector<std::unique_ptr<SegmentCacheHandle>>& segment_caches,
std::vector<int64_t>* row_lsns, std::vector<uint8_t>* insert_after_delete_flags) {
if (row_lsns != nullptr && row_lsns->size() != num_rows) {
- return Status::InvalidArgument("row binlog LSN count {} does not match row count {}",
+ return Status::InvalidArgument("changelog LSN count {} does not match row count {}",
row_lsns->size(), num_rows);
}
std::vector<IOlapColumnDataAccessor*> key_columns {};
diff --git a/be/src/storage/partial_update_info.h b/be/src/storage/partial_update_info.h
index eeb3b6b2976..aa70e6f7fb1 100644
--- a/be/src/storage/partial_update_info.h
+++ b/be/src/storage/partial_update_info.h
@@ -287,7 +287,7 @@ private:
const segment_v2::MowKeyProbe& _probe;
HistoricalRowFetcher& _fetcher;
- // Optional sidecar used by Row Binlog. Flexible aggregation can remove or merge input rows;
+ // Optional sidecar used by changelog. Flexible aggregation can remove or merge input rows;
// keep the persisted LSN aligned with the surviving row and retain the largest LSN of all
// merged changes.
const std::vector<int64_t>* _input_row_lsns = nullptr;
diff --git a/be/src/storage/predicate/null_predicate.cpp b/be/src/storage/predicate/null_predicate.cpp
index 892a1712e86..ebea4a58f93 100644
--- a/be/src/storage/predicate/null_predicate.cpp
+++ b/be/src/storage/predicate/null_predicate.cpp
@@ -27,8 +27,6 @@
#include "storage/index/inverted/common/inverted_index_cache.h"
#include "storage/index/inverted/inverted_index_reader.h"
-using namespace doris;
-
namespace doris {
NullPredicate::NullPredicate(uint32_t column_id, std::string col_name, bool is_null,
diff --git a/be/src/storage/rowset/beta_rowset_reader.cpp b/be/src/storage/rowset/beta_rowset_reader.cpp
index bed808d461c..3cd7b88cc99 100644
--- a/be/src/storage/rowset/beta_rowset_reader.cpp
+++ b/be/src/storage/rowset/beta_rowset_reader.cpp
@@ -34,7 +34,7 @@
#include "runtime/descriptors.h"
#include "runtime/query_context.h"
#include "runtime/runtime_profile.h"
-#include "storage/binlog.h"
+#include "storage/changelog.h"
#include "storage/delete/delete_handler.h"
#include "storage/iterator/vgeneric_iterators.h"
#include "storage/olap_define.h"
@@ -206,11 +206,11 @@ Status BetaRowsetReader::get_segment_iterators(RowsetReaderContext* read_context
_read_options.topn_filter_source_node_ids = _read_context->topn_filter_source_node_ids;
_read_options.read_orderby_key_reverse = _read_context->read_orderby_key_reverse;
_read_options.use_insert_order_when_same = _read_context->use_insert_order_when_same;
- _read_options.read_row_binlog = _read_context->read_row_binlog;
- // Binlog TSO drives the merge tie-break and is therefore part of the exact
+ _read_options.read_changelog = _read_context->read_changelog;
+ // Changelog TSO drives the merge tie-break and is therefore part of the exact
// read schema shared by the rowset merge.
if (int32_t tso_ordinal = read_schema->tso_ordinal(); tso_ordinal >= 0) {
- _read_options.binlog_tso_idx = tso_ordinal;
+ _read_options.changelog_tso_idx = tso_ordinal;
}
_read_options.read_orderby_key_columns = _read_context->read_orderby_key_columns;
_read_options.io_ctx.reader_type = _read_context->reader_type;
@@ -342,13 +342,13 @@ Status BetaRowsetReader::_init_iterator() {
int32_t sequence_loc = _read_context->use_sequence_column_for_merge_order
? _read_context->read_schema->sequence_ordinal()
: -1;
- if (_read_options.binlog_tso_idx != -1) {
- sequence_loc = _read_options.binlog_tso_idx;
+ if (_read_options.changelog_tso_idx != -1) {
+ sequence_loc = _read_options.changelog_tso_idx;
}
_iterator = new_merge_iterator(std::move(iterators), sequence_loc, _read_context->is_unique,
_read_context->read_orderby_key_reverse,
_read_context->merged_rows, _read_context->read_schema,
- _read_options.binlog_tso_idx != -1);
+ _read_options.changelog_tso_idx != -1);
} else {
if (_read_context->read_orderby_key_reverse) {
// reverse iterators to read backward for ORDER BY key DESC
diff --git a/be/src/storage/rowset/beta_rowset_writer.cpp b/be/src/storage/rowset/beta_rowset_writer.cpp
index 0b78127f68b..ed2800d6fdb 100644
--- a/be/src/storage/rowset/beta_rowset_writer.cpp
+++ b/be/src/storage/rowset/beta_rowset_writer.cpp
@@ -153,8 +153,8 @@ void build_rowset_meta_with_spec_field(RowsetMeta& rowset_meta,
if (spec_rowset_meta.has_commit_tso()) {
rowset_meta.set_commit_tso(spec_rowset_meta.commit_tso());
}
- if (spec_rowset_meta.is_row_binlog()) {
- rowset_meta.mark_row_binlog();
+ if (spec_rowset_meta.is_changelog()) {
+ rowset_meta.mark_changelog();
}
}
@@ -336,7 +336,7 @@ BaseBetaRowsetWriter::BaseBetaRowsetWriter()
BetaRowsetWriter::BetaRowsetWriter(StorageEngine& engine)
: _engine(engine), _segcompaction_worker(std::make_shared<SegcompactionWorker>(this)) {}
-RowBinlogRowsetWriter::RowBinlogRowsetWriter(StorageEngine& engine) : BetaRowsetWriter(engine) {}
+ChangelogRowsetWriter::ChangelogRowsetWriter(StorageEngine& engine) : BetaRowsetWriter(engine) {}
BaseBetaRowsetWriter::~BaseBetaRowsetWriter() {
// Finish callbacks before removing files they can still read on cancellation.
@@ -381,9 +381,9 @@ void BaseBetaRowsetWriter::cancel_calc_delete_bitmap(const Status& st) {
Status BaseBetaRowsetWriter::init(const RowsetWriterContext& rowset_writer_context) {
_context = rowset_writer_context;
DCHECK(_context.tablet_schema != nullptr);
- // Row-binlog writer or a schema carrying ROW_LSN_COL needs allocated LSN.
+ // Changelog writer or a schema carrying ROW_LSN_COL needs allocated LSN.
_context._need_allocate_lsn =
- _context.write_binlog_opt().enable || _context.tablet_schema->row_lsn_col_idx() >= 0;
+ _context.write_changelog_opt().enable || _context.tablet_schema->row_lsn_col_idx() >= 0;
_rowset_meta.reset(new RowsetMeta);
if (_context.storage_resource) {
_rowset_meta->set_remote_storage_resource(*_context.storage_resource);
@@ -413,8 +413,8 @@ Status BaseBetaRowsetWriter::init(const RowsetWriterContext& rowset_writer_conte
_rowset_meta->set_inverted_index_storage_format(*_context.inverted_index_storage_format);
}
_rowset_meta->set_compaction_level(_context.compaction_level);
- if (_context.write_binlog_opt().enable) {
- _rowset_meta->mark_row_binlog();
+ if (_context.write_changelog_opt().enable) {
+ _rowset_meta->mark_changelog();
}
_context.segment_collector = std::make_shared<SegmentCollectorT<BaseBetaRowsetWriter>>(this);
_context.file_writer_creator = std::make_shared<FileWriterCreatorT<BaseBetaRowsetWriter>>(this);
@@ -1123,8 +1123,8 @@ Status BaseBetaRowsetWriter::_build_rowset_meta(RowsetMeta* rowset_meta, bool ch
rowset_meta->set_segments_overlap(NONOVERLAPPING);
} else if (!segments_encoded_key_bounds.empty() &&
is_segment_overlapping(segments_encoded_key_bounds) &&
- _rowset_meta->is_row_binlog()) {
- // A row-binlog horizontal merge output can hold the same user key in more than
+ _rowset_meta->is_changelog()) {
+ // A changelog horizontal merge output can hold the same user key in more than
// one output segment (a forced segment boundary splits a key range), so the
// segments genuinely overlap. The compaction ctx default is NONOVERLAPPING for
// the non-quick-merge path, and the branch above only ever downgrades, so upgrade
@@ -1165,7 +1165,7 @@ Status BaseBetaRowsetWriter::_build_rowset_meta(RowsetMeta* rowset_meta, bool ch
rowset_meta->set_index_disk_size(total_index_size + _total_index_size);
bool aggregate_key_bounds = config::enable_aggregate_non_mow_key_bounds &&
!_context.enable_unique_key_merge_on_write &&
- !_rowset_meta->is_row_binlog();
+ !_rowset_meta->is_changelog();
rowset_meta->set_segments_key_bounds(segments_encoded_key_bounds, aggregate_key_bounds);
// TODO write zonemap to meta
rowset_meta->set_empty((num_rows_written + _num_rows_written) == 0);
diff --git a/be/src/storage/rowset/beta_rowset_writer.h b/be/src/storage/rowset/beta_rowset_writer.h
index 3887919f0da..6ff653b08db 100644
--- a/be/src/storage/rowset/beta_rowset_writer.h
+++ b/be/src/storage/rowset/beta_rowset_writer.h
@@ -340,9 +340,9 @@ private:
std::atomic<int> _segcompaction_status {ErrorCode::OK};
};
-class RowBinlogRowsetWriter : public BetaRowsetWriter {
+class ChangelogRowsetWriter : public BetaRowsetWriter {
public:
- RowBinlogRowsetWriter(StorageEngine& engine);
+ ChangelogRowsetWriter(StorageEngine& engine);
};
} // namespace doris
diff --git a/be/src/storage/rowset/beta_rowset_writer_v2.cpp b/be/src/storage/rowset/beta_rowset_writer_v2.cpp
index 6c03d66d652..1e707dfa169 100644
--- a/be/src/storage/rowset/beta_rowset_writer_v2.cpp
+++ b/be/src/storage/rowset/beta_rowset_writer_v2.cpp
@@ -61,9 +61,9 @@ BetaRowsetWriterV2::~BetaRowsetWriterV2() = default;
Status BetaRowsetWriterV2::init(const RowsetWriterContext& rowset_writer_context) {
_context = rowset_writer_context;
- // Row-binlog writer or a schema carrying ROW_LSN_COL needs allocated LSN.
+ // Changelog writer or a schema carrying ROW_LSN_COL needs allocated LSN.
_context._need_allocate_lsn =
- _context.write_binlog_opt().enable ||
+ _context.write_changelog_opt().enable ||
(_context.tablet_schema != nullptr && _context.tablet_schema->row_lsn_col_idx() >= 0);
_context.segment_collector = std::make_shared<SegmentCollectorT<BetaRowsetWriterV2>>(this);
_context.file_writer_creator = std::make_shared<FileWriterCreatorT<BetaRowsetWriterV2>>(this);
diff --git a/be/src/storage/rowset/group_rowset_writer.cpp b/be/src/storage/rowset/group_rowset_writer.cpp
index 088f4008768..a2fdf0557e3 100644
--- a/be/src/storage/rowset/group_rowset_writer.cpp
+++ b/be/src/storage/rowset/group_rowset_writer.cpp
@@ -27,32 +27,31 @@ void GroupRowsetWriter::set_data_writer(const RowsetWriterSharedPtr& txn_rowset_
_txn_rowset_writer = txn_rowset_writer;
}
-void GroupRowsetWriter::set_row_binlog_writer(
- const RowsetWriterSharedPtr& row_binlog_rowset_writer) {
- _row_binlog_rowset_writer = row_binlog_rowset_writer;
+void GroupRowsetWriter::set_changelog_writer(const RowsetWriterSharedPtr& changelog_rowset_writer) {
+ _changelog_rowset_writer = changelog_rowset_writer;
}
Status GroupRowsetWriter::init(const RowsetWriterContext& rowset_writer_context) {
DCHECK(_txn_rowset_writer != nullptr);
- DCHECK(_row_binlog_rowset_writer != nullptr);
+ DCHECK(_changelog_rowset_writer != nullptr);
_context = rowset_writer_context;
auto& data_ctx = const_cast<RowsetWriterContext&>(_txn_rowset_writer->context());
- auto& row_binlog_ctx = const_cast<RowsetWriterContext&>(_row_binlog_rowset_writer->context());
+ auto& changelog_ctx = const_cast<RowsetWriterContext&>(_changelog_rowset_writer->context());
_context._need_allocate_lsn =
- data_ctx.need_allocated_lsn() || row_binlog_ctx.need_allocated_lsn();
+ data_ctx.need_allocated_lsn() || changelog_ctx.need_allocated_lsn();
if (_context.need_allocated_lsn()) {
- // Share segment LSNs between data and row-binlog writers.
+ // Share segment LSNs between data and changelog writers.
_context.allocated_lsn_map = std::make_shared<segment_v2::SegmentAllocatedLsnMap>();
data_ctx.allocated_lsn_map = _context.allocated_lsn_map;
- row_binlog_ctx.allocated_lsn_map = _context.allocated_lsn_map;
+ changelog_ctx.allocated_lsn_map = _context.allocated_lsn_map;
}
return Status::OK();
}
Status GroupRowsetWriter::flush_rowsets() {
RETURN_IF_ERROR(_txn_rowset_writer->flush());
- RETURN_IF_ERROR(_row_binlog_rowset_writer->flush());
+ RETURN_IF_ERROR(_changelog_rowset_writer->flush());
return Status::OK();
}
@@ -61,13 +60,13 @@ Status GroupRowsetWriter::build_rowsets(std::vector<RowsetSharedPtr>& rowsets) {
rowsets.reserve(2);
RowsetSharedPtr txn_rowset;
- RowsetSharedPtr row_binlog_rowset;
+ RowsetSharedPtr changelog_rowset;
RETURN_IF_ERROR(_txn_rowset_writer->build(txn_rowset));
Status st = Status::OK();
- DBUG_EXECUTE_IF("GroupRowsetWriter::build_rowsets.row_binlog_build_failed",
- { st = Status::InternalError("debug row binlog build failed"); });
+ DBUG_EXECUTE_IF("GroupRowsetWriter::build_rowsets.changelog_build_failed",
+ { st = Status::InternalError("debug changelog build failed"); });
if (st.ok()) {
- st = _row_binlog_rowset_writer->build(row_binlog_rowset);
+ st = _changelog_rowset_writer->build(changelog_rowset);
}
if (!st.ok()) {
RETURN_IF_ERROR(_txn_rowset_writer->force_rollback());
@@ -75,13 +74,13 @@ Status GroupRowsetWriter::build_rowsets(std::vector<RowsetSharedPtr>& rowsets) {
}
rowsets.emplace_back(std::move(txn_rowset));
- rowsets.emplace_back(std::move(row_binlog_rowset));
+ rowsets.emplace_back(std::move(changelog_rowset));
return Status::OK();
}
Status GroupRowsetWriter::flush_memtable(Block* block, int32_t segment_id, int64_t* flush_size) {
RETURN_IF_ERROR(_txn_rowset_writer->flush_memtable(block, segment_id, flush_size));
- RETURN_IF_ERROR(_row_binlog_rowset_writer->flush_memtable(block, segment_id, flush_size));
+ RETURN_IF_ERROR(_changelog_rowset_writer->flush_memtable(block, segment_id, flush_size));
return Status::OK();
}
@@ -92,16 +91,16 @@ Status GroupRowsetWriter::flush_single_block(const Block* block) {
Status GroupRowsetWriter::flush_single_block(const Block* block, int32_t segment_id) {
RETURN_IF_ERROR(_txn_rowset_writer->flush_single_block(block, segment_id));
- RETURN_IF_ERROR(_row_binlog_rowset_writer->flush_single_block(block, segment_id));
+ RETURN_IF_ERROR(_changelog_rowset_writer->flush_single_block(block, segment_id));
return Status::OK();
}
Result<int32_t> GroupRowsetWriter::allocate_segment_id() {
DCHECK(_txn_rowset_writer != nullptr);
- DCHECK(_row_binlog_rowset_writer != nullptr);
+ DCHECK(_changelog_rowset_writer != nullptr);
auto segment_id = DORIS_TRY(_txn_rowset_writer->allocate_segment_id());
- auto binlog_segment_id = DORIS_TRY(_row_binlog_rowset_writer->allocate_segment_id());
- DCHECK_EQ(segment_id, binlog_segment_id);
+ auto changelog_segment_id = DORIS_TRY(_changelog_rowset_writer->allocate_segment_id());
+ DCHECK_EQ(segment_id, changelog_segment_id);
return segment_id;
}
diff --git a/be/src/storage/rowset/group_rowset_writer.h b/be/src/storage/rowset/group_rowset_writer.h
index 4c0520067ff..93b805829f9 100644
--- a/be/src/storage/rowset/group_rowset_writer.h
+++ b/be/src/storage/rowset/group_rowset_writer.h
@@ -27,7 +27,7 @@ public:
void set_data_writer(const RowsetWriterSharedPtr& txn_rowset_writer);
- void set_row_binlog_writer(const RowsetWriterSharedPtr& row_binlog_rowset_writer);
+ void set_changelog_writer(const RowsetWriterSharedPtr& changelog_rowset_writer);
~GroupRowsetWriter() = default;
@@ -35,7 +35,7 @@ public:
Status build_rowsets(std::vector<RowsetSharedPtr>& rowsets);
- RowsetWriterSharedPtr row_binlog_writer() { return _row_binlog_rowset_writer; }
+ RowsetWriterSharedPtr changelog_writer() { return _changelog_rowset_writer; }
RowsetWriterSharedPtr data_writer() { return _txn_rowset_writer; }
@@ -109,9 +109,9 @@ public:
int32_t get_allocated_segment_id() override {
DCHECK(_txn_rowset_writer != nullptr);
- DCHECK(_row_binlog_rowset_writer != nullptr);
+ DCHECK(_changelog_rowset_writer != nullptr);
auto seg_id = _txn_rowset_writer->get_allocated_segment_id();
- DCHECK_EQ(seg_id, _row_binlog_rowset_writer->get_allocated_segment_id());
+ DCHECK_EQ(seg_id, _changelog_rowset_writer->get_allocated_segment_id());
return seg_id;
}
@@ -132,7 +132,7 @@ public:
private:
RowsetWriterSharedPtr _txn_rowset_writer;
- RowsetWriterSharedPtr _row_binlog_rowset_writer;
+ RowsetWriterSharedPtr _changelog_rowset_writer;
};
} // namespace doris
diff --git a/be/src/storage/rowset/rowset_factory.cpp b/be/src/storage/rowset/rowset_factory.cpp
index e6d0c7a8e37..63b3d87f464 100644
--- a/be/src/storage/rowset/rowset_factory.cpp
+++ b/be/src/storage/rowset/rowset_factory.cpp
@@ -55,14 +55,14 @@ Result<std::unique_ptr<RowsetWriter>> RowsetFactory::create_rowset_writer(
return ResultError(Status::Error<ROWSET_INVALID>("invalid rowset_type"));
}
- if (context.write_binlog_opt().enable) {
+ if (context.write_changelog_opt().enable) {
std::unique_ptr<RowsetWriter> writer;
if (is_vertical) {
- writer = std::make_unique<VerticalBetaRowsetWriter<RowBinlogRowsetWriter>>(engine);
+ writer = std::make_unique<VerticalBetaRowsetWriter<ChangelogRowsetWriter>>(engine);
RETURN_IF_ERROR_RESULT(writer->init(context));
return writer;
} else {
- writer = std::make_unique<RowBinlogRowsetWriter>(engine);
+ writer = std::make_unique<ChangelogRowsetWriter>(engine);
RETURN_IF_ERROR_RESULT(writer->init(context));
return writer;
}
diff --git a/be/src/storage/rowset/rowset_meta.h b/be/src/storage/rowset/rowset_meta.h
index e4dc1b1e5c6..72e07a88caf 100644
--- a/be/src/storage/rowset/rowset_meta.h
+++ b/be/src/storage/rowset/rowset_meta.h
@@ -150,10 +150,10 @@ public:
_rowset_meta_pb.set_tablet_schema_hash(tablet_schema_hash);
}
- void mark_row_binlog() { _rowset_meta_pb.set_is_row_binlog(true); }
+ void mark_changelog() { _rowset_meta_pb.set_is_changelog(true); }
- bool is_row_binlog() const {
- return _rowset_meta_pb.has_is_row_binlog() && _rowset_meta_pb.is_row_binlog();
+ bool is_changelog() const {
+ return _rowset_meta_pb.has_is_changelog() && _rowset_meta_pb.is_changelog();
}
RowsetTypePB rowset_type() const { return _rowset_meta_pb.rowset_type(); }
@@ -373,11 +373,11 @@ public:
// 1. The rowset contains more than one segment.
// 2. segments_overlap() is not NONOVERLAPPING (OVERLAP_UNKNOWN, OVERLAPPING, and
// NONOVERLAPPING_WITHIN_GROUP are considered overlapping).
- // 3. The rowset has a singleton version, except row-binlog LMax quick merge rowsets that
+ // 3. The rowset has a singleton version, except changelog LMax quick merge rowsets that
// explicitly set segments_overlap() to OVERLAPPING.
bool is_segments_overlapping() const {
return num_segments() > 1 && segments_overlap() != NONOVERLAPPING &&
- (is_singleton_delta() || (is_row_binlog() && segments_overlap() == OVERLAPPING));
+ (is_singleton_delta() || (is_changelog() && segments_overlap() == OVERLAPPING));
}
bool produced_by_compaction() const {
diff --git a/be/src/storage/rowset/rowset_meta_manager.cpp b/be/src/storage/rowset/rowset_meta_manager.cpp
index d442c56328d..e3880e9470f 100644
--- a/be/src/storage/rowset/rowset_meta_manager.cpp
+++ b/be/src/storage/rowset/rowset_meta_manager.cpp
@@ -74,7 +74,7 @@ Status RowsetMetaManager::get_rowset_meta(OlapMeta* meta, TabletUid tablet_uid,
Status RowsetMetaManager::save(OlapMeta* meta, TabletUid tablet_uid, const RowsetId& rowset_id,
const RowsetMetaPB& rowset_meta_pb,
std::optional<BinlogFormatPB> binlog_format,
- const std::optional<RowsetMetaPB>& attach_row_binlog_rowset_meta) {
+ const std::optional<RowsetMetaPB>& attach_changelog_rowset_meta) {
if (rowset_meta_pb.partition_id() <= 0) {
LOG(WARNING) << "invalid partition id " << rowset_meta_pb.partition_id() << " tablet "
<< rowset_meta_pb.tablet_id();
@@ -95,10 +95,10 @@ Status RowsetMetaManager::save(OlapMeta* meta, TabletUid tablet_uid, const Rowse
if (*binlog_format == BinlogFormatPB::STATEMENT_AND_SNAPSHOT) {
return _save_with_ccr_binlog(meta, tablet_uid, rowset_id, rowset_meta_pb);
}
- DCHECK_EQ(*binlog_format, BinlogFormatPB::ROW);
- DCHECK(attach_row_binlog_rowset_meta.has_value());
- return _save_with_row_binlog(meta, tablet_uid, rowset_id, rowset_meta_pb,
- *attach_row_binlog_rowset_meta);
+ DCHECK_EQ(*binlog_format, BinlogFormatPB::CHANGELOG);
+ DCHECK(attach_changelog_rowset_meta.has_value());
+ return _save_with_changelog(meta, tablet_uid, rowset_id, rowset_meta_pb,
+ *attach_changelog_rowset_meta);
}
Status RowsetMetaManager::_save(OlapMeta* meta, TabletUid tablet_uid, const RowsetId& rowset_id,
@@ -160,10 +160,10 @@ Status RowsetMetaManager::_save_with_ccr_binlog(OlapMeta* meta, TabletUid tablet
return meta->put(META_COLUMN_FAMILY_INDEX, entries);
}
-Status RowsetMetaManager::_save_with_row_binlog(OlapMeta* meta, TabletUid tablet_uid,
- const RowsetId& rowset_id,
- const RowsetMetaPB& rowset_meta_pb,
- const RowsetMetaPB& attach_row_binlog_rowset_meta) {
+Status RowsetMetaManager::_save_with_changelog(OlapMeta* meta, TabletUid tablet_uid,
+ const RowsetId& rowset_id,
+ const RowsetMetaPB& rowset_meta_pb,
+ const RowsetMetaPB& attach_changelog_rowset_meta) {
std::string rowset_key =
fmt::format("{}{}_{}", ROWSET_PREFIX, tablet_uid.to_string(), rowset_id.to_string());
std::string rowset_value;
@@ -172,22 +172,22 @@ Status RowsetMetaManager::_save_with_row_binlog(OlapMeta* meta, TabletUid tablet
rowset_key);
}
- // save binlog rowset meta as a normal rowset under its own tablet uid in the same batch.
- TabletUid row_binlog_tablet_uid(attach_row_binlog_rowset_meta.tablet_uid());
- RowsetId row_binlog_rowset_id;
- row_binlog_rowset_id.init(attach_row_binlog_rowset_meta.rowset_id_v2());
- std::string row_binlog_rowset_key =
- fmt::format("{}{}_{}", ROWSET_PREFIX, row_binlog_tablet_uid.to_string(),
- row_binlog_rowset_id.to_string());
- std::string row_binlog_rowset_value;
- if (!attach_row_binlog_rowset_meta.SerializeToString(&row_binlog_rowset_value)) {
+ // save changelog rowset meta as a normal rowset under its own tablet uid in the same batch.
+ TabletUid changelog_tablet_uid(attach_changelog_rowset_meta.tablet_uid());
+ RowsetId changelog_rowset_id;
+ changelog_rowset_id.init(attach_changelog_rowset_meta.rowset_id_v2());
+ std::string changelog_rowset_key =
+ fmt::format("{}{}_{}", ROWSET_PREFIX, changelog_tablet_uid.to_string(),
+ changelog_rowset_id.to_string());
+ std::string changelog_rowset_value;
+ if (!attach_changelog_rowset_meta.SerializeToString(&changelog_rowset_value)) {
return Status::Error<SERIALIZE_PROTOBUF_ERROR>("serialize rowset pb failed. rowset id:{}",
- row_binlog_rowset_key);
+ changelog_rowset_key);
}
std::vector<OlapMeta::BatchEntry> entries = {
{std::cref(rowset_key), std::cref(rowset_value)},
- {std::cref(row_binlog_rowset_key), std::cref(row_binlog_rowset_value)}};
+ {std::cref(changelog_rowset_key), std::cref(changelog_rowset_value)}};
return meta->put(META_COLUMN_FAMILY_INDEX, entries);
}
diff --git a/be/src/storage/rowset/rowset_meta_manager.h b/be/src/storage/rowset/rowset_meta_manager.h
index 6afdd80aa3c..afa571f7373 100644
--- a/be/src/storage/rowset/rowset_meta_manager.h
+++ b/be/src/storage/rowset/rowset_meta_manager.h
@@ -61,7 +61,7 @@ public:
OlapMeta* meta, TabletUid tablet_uid, const RowsetId& rowset_id,
const RowsetMetaPB& rowset_meta_pb,
std::optional<BinlogFormatPB> binlog_format = std::nullopt,
- const std::optional<RowsetMetaPB>& attach_row_binlog_rowset_meta = std::nullopt);
+ const std::optional<RowsetMetaPB>& attach_changelog_rowset_meta = std::nullopt);
// STATEMENT_AND_SNAPSHOT
static Status save(OlapMeta* meta, TabletUid tablet_uid, const RowsetId& rowset_id,
@@ -120,10 +120,10 @@ private:
static Status _save_with_ccr_binlog(OlapMeta* meta, TabletUid tablet_uid,
const RowsetId& rowset_id,
const RowsetMetaPB& rowset_meta_pb);
- static Status _save_with_row_binlog(OlapMeta* meta, TabletUid tablet_uid,
- const RowsetId& rowset_id,
- const RowsetMetaPB& rowset_meta_pb,
- const RowsetMetaPB& attach_row_binlog_rowset_meta);
+ static Status _save_with_changelog(OlapMeta* meta, TabletUid tablet_uid,
+ const RowsetId& rowset_id,
+ const RowsetMetaPB& rowset_meta_pb,
+ const RowsetMetaPB& attach_changelog_rowset_meta);
static Status _get_rowset_binlog_metas(OlapMeta* meta, const TabletUid tablet_uid,
const std::vector<int64_t>& binlog_versions,
RowsetBinlogMetasPB* metas_pb);
diff --git a/be/src/storage/rowset/rowset_reader_context.h b/be/src/storage/rowset/rowset_reader_context.h
index 17b5938a680..3c24a4c1b6e 100644
--- a/be/src/storage/rowset/rowset_reader_context.h
+++ b/be/src/storage/rowset/rowset_reader_context.h
@@ -43,7 +43,7 @@ class TabletSchema;
struct RowsetReaderContext {
ReaderType reader_type = ReaderType::READER_QUERY;
- bool read_row_binlog = false;
+ bool read_changelog = false;
Version version {-1, -1};
// Set only by compaction, alongside its extended tablet_schema.
VariantCompactionPathsSPtr variant_compaction_paths;
diff --git a/be/src/storage/rowset/rowset_writer_context.h b/be/src/storage/rowset/rowset_writer_context.h
index 3a29a8a4b66..2032b6dbad7 100644
--- a/be/src/storage/rowset/rowset_writer_context.h
+++ b/be/src/storage/rowset/rowset_writer_context.h
@@ -36,7 +36,7 @@
#include "io/fs/local_file_system.h"
#include "io/fs/packed_file_system.h"
#include "runtime/exec_env.h"
-#include "storage/binlog.h"
+#include "storage/changelog.h"
#include "storage/olap_define.h"
#include "storage/partial_update_info.h"
#include "storage/segment/historical_row_retriever.h"
@@ -196,7 +196,7 @@ struct RowsetWriterContext {
std::string job_id;
// Per-segment LSNs allocated before memtable flush. The same storage feeds
- // both the base row LSN column and row-binlog LSN column.
+ // both the base row LSN column and changelog LSN column.
std::shared_ptr<segment_v2::SegmentAllocatedLsnMap> allocated_lsn_map = nullptr;
bool _need_allocate_lsn = false;
@@ -323,31 +323,31 @@ struct RowsetWriterContext {
return opts;
}
- struct BinlogOptions {
+ struct ChangelogOptions {
public:
bool enable = false;
void set_need_before(bool need_before) {
this->_need_before = need_before;
- _segment_write_binlog_opt.write_before = need_before;
+ _segment_write_changelog_opt.write_before = need_before;
}
- segment_v2::SegmentWriteBinlogOptions& write_binlog_config() {
- return _segment_write_binlog_opt;
+ segment_v2::SegmentWriteChangelogOptions& write_changelog_config() {
+ return _segment_write_changelog_opt;
}
- const segment_v2::SegmentWriteBinlogOptions& write_binlog_config() const {
- return _segment_write_binlog_opt;
+ const segment_v2::SegmentWriteChangelogOptions& write_changelog_config() const {
+ return _segment_write_changelog_opt;
}
private:
bool _need_before = false;
- segment_v2::SegmentWriteBinlogOptions _segment_write_binlog_opt;
- } _write_binlog_opt;
+ segment_v2::SegmentWriteChangelogOptions _segment_write_changelog_opt;
+ } _write_changelog_opt;
- BinlogOptions& write_binlog_opt() { return _write_binlog_opt; }
+ ChangelogOptions& write_changelog_opt() { return _write_changelog_opt; }
- const BinlogOptions& write_binlog_opt() const { return _write_binlog_opt; }
+ const ChangelogOptions& write_changelog_opt() const { return _write_changelog_opt; }
};
inline segment_v2::HistoricalRowRetrieverContext
diff --git a/be/src/storage/rowset/vertical_beta_rowset_writer.cpp b/be/src/storage/rowset/vertical_beta_rowset_writer.cpp
index 719b90b62be..8ed10ada725 100644
--- a/be/src/storage/rowset/vertical_beta_rowset_writer.cpp
+++ b/be/src/storage/rowset/vertical_beta_rowset_writer.cpp
@@ -46,7 +46,7 @@ using namespace ErrorCode;
template class VerticalBetaRowsetWriter<BetaRowsetWriter>;
template class VerticalBetaRowsetWriter<CloudRowsetWriter>;
-template class VerticalBetaRowsetWriter<RowBinlogRowsetWriter>;
+template class VerticalBetaRowsetWriter<ChangelogRowsetWriter>;
template <class T>
requires std::is_base_of_v<BaseBetaRowsetWriter, T>
diff --git a/be/src/storage/rowset_builder.cpp b/be/src/storage/rowset_builder.cpp
index c4b7530ce97..7f2a080be8e 100644
--- a/be/src/storage/rowset_builder.cpp
+++ b/be/src/storage/rowset_builder.cpp
@@ -77,7 +77,7 @@ RowsetBuilder::RowsetBuilder(StorageEngine& engine, const WriteRequest& req,
RuntimeProfile* profile)
: BaseRowsetBuilder(req, profile), _engine(engine) {}
-RowBinlogRowsetBuilder::RowBinlogRowsetBuilder(StorageEngine& engine, const WriteRequest& req,
+ChangelogRowsetBuilder::ChangelogRowsetBuilder(StorageEngine& engine, const WriteRequest& req,
RuntimeProfile* profile)
: RowsetBuilder(engine, req, profile) {}
@@ -90,9 +90,8 @@ void BaseRowsetBuilder::_init_profile(RuntimeProfile* profile) {
}
_profile = profile->create_child(
- fmt::format(
- "RowsetBuilder {} {}", _req.tablet_id,
- _req.write_req_type == WriteRequestType::ROW_BINLOG ? "binlog<row>" : "data"),
+ fmt::format("RowsetBuilder {} {}", _req.tablet_id,
+ _req.write_req_type == WriteRequestType::CHANGELOG ? "changelog" : "data"),
true, true);
_build_rowset_timer = ADD_TIMER(_profile, "BuildRowsetTime");
_submit_delete_bitmap_timer = ADD_TIMER(_profile, "DeleteBitmapSubmitTime");
@@ -139,8 +138,8 @@ void RowsetBuilder::_garbage_collection(bool cancel_txn) {
// when rollback failed should not delete rowset
if (need_clean) {
_engine.add_unused_rowset(_rowset);
- if (_attach_row_binlog.rowset != nullptr) {
- _engine.add_unused_rowset(_attach_row_binlog.rowset);
+ if (_attach_changelog.rowset != nullptr) {
+ _engine.add_unused_rowset(_attach_changelog.rowset);
}
}
}
@@ -279,7 +278,7 @@ Status BaseRowsetBuilder::_init_context_common_fields(RowsetWriterContext& conte
context.index_id = _req.index_id;
context.tablet = _tablet;
context.enable_segcompaction = true;
- if (_req.write_req_type == WriteRequestType::ROW_BINLOG || !_attach_rowset_ids.empty()) {
+ if (_req.write_req_type == WriteRequestType::CHANGELOG || !_attach_rowset_ids.empty()) {
context.enable_segcompaction = false;
}
context.write_type = DataWriteType::TYPE_DIRECT;
@@ -300,10 +299,10 @@ Status BaseRowsetBuilder::build_rowset() {
}
Status GroupRowsetBuilder::build_rowset() {
- // build binlog rowset first, then data rowset
- // If data rowset build fails after binlog has built some segments, we rely on the
+ // build changelog rowset first, then data rowset
+ // If data rowset build fails after changelog has built some segments, we rely on the
// RowsetBuilder/StorageEngine garbage-collection paths to clean up built-but-uncommitted files.
- RETURN_IF_ERROR(_row_binlog_rowset_builder->build_rowset());
+ RETURN_IF_ERROR(_changelog_rowset_builder->build_rowset());
return _txn_rs_builder->build_rowset();
}
@@ -397,7 +396,7 @@ Status RowsetBuilder::commit_txn() {
// Transfer ownership of `PendingRowsetGuard` to `TxnManager`
Status res = _engine.txn_manager()->commit_txn(
_req.partition_id, *tablet(), _req.txn_id, _req.load_id, _rowset,
- std::move(_pending_rs_guard), false, _partial_update_info, _attach_row_binlog);
+ std::move(_pending_rs_guard), false, _partial_update_info, _attach_changelog);
if (!res && !res.is<PUSH_TRANSACTION_ALREADY_EXIST>()) {
LOG(WARNING) << "Failed to commit txn: " << _req.txn_id
@@ -405,7 +404,7 @@ Status RowsetBuilder::commit_txn() {
return res;
}
if (_tablet->enable_unique_key_merge_on_write()) {
- // no need to update binlog tablet delete bitmap, it'll be updated in publish phase
+ // no need to update changelog tablet delete bitmap, it'll be updated in publish phase
_engine.txn_manager()->set_txn_related_delete_bitmap(
_req.partition_id, _req.txn_id, tablet()->tablet_id(), tablet()->tablet_uid(), true,
_delete_bitmap, *_rowset_ids, _partial_update_info);
@@ -435,10 +434,10 @@ Status BaseRowsetBuilder::_build_current_tablet_schema(
int64_t index_id, const OlapTableSchemaParam* table_schema_param,
const TabletSchema& ori_tablet_schema) {
const OlapTableIndexSchema* index_schema = nullptr;
- if (_req.write_req_type == WriteRequestType::ROW_BINLOG) {
- const auto* row_binlog_index_schema = table_schema_param->row_binlog_index_schema(index_id);
- DCHECK(row_binlog_index_schema != nullptr);
- index_schema = row_binlog_index_schema;
+ if (_req.write_req_type == WriteRequestType::CHANGELOG) {
+ const auto* changelog_index_schema = table_schema_param->changelog_index_schema(index_id);
+ DCHECK(changelog_index_schema != nullptr);
+ index_schema = changelog_index_schema;
} else {
for (const auto* schema : table_schema_param->indexes()) {
if (schema->index_id == index_id) {
@@ -510,64 +509,64 @@ Status BaseRowsetBuilder::_build_current_tablet_schema(
GroupRowsetBuilder::GroupRowsetBuilder(StorageEngine& engine, const WriteRequest& group_build_req,
const WriteRequest& sub_data_req,
- const WriteRequest& sub_row_binlog_req,
+ const WriteRequest& sub_changelog_req,
RuntimeProfile* profile)
: BaseRowsetBuilder(group_build_req, profile),
_engine(engine),
- _row_binlog_tablet_id(sub_row_binlog_req.tablet_id) {
+ _changelog_tablet_id(sub_changelog_req.tablet_id) {
DCHECK(group_build_req.write_req_type == WriteRequestType::GROUP &&
sub_data_req.write_req_type == WriteRequestType::DATA &&
- sub_row_binlog_req.write_req_type == WriteRequestType::ROW_BINLOG);
- _row_binlog_rowset_builder =
- std::make_shared<RowBinlogRowsetBuilder>(engine, sub_row_binlog_req, profile);
+ sub_changelog_req.write_req_type == WriteRequestType::CHANGELOG);
+ _changelog_rowset_builder =
+ std::make_shared<ChangelogRowsetBuilder>(engine, sub_changelog_req, profile);
_txn_rs_builder = std::make_shared<RowsetBuilder>(engine, sub_data_req, profile);
}
Status GroupRowsetBuilder::init() {
{
- TabletSharedPtr row_binlog_tablet =
- _engine.tablet_manager()->get_tablet(_row_binlog_tablet_id);
- if (row_binlog_tablet == nullptr) {
+ TabletSharedPtr changelog_tablet =
+ _engine.tablet_manager()->get_tablet(_changelog_tablet_id);
+ if (changelog_tablet == nullptr) {
return Status::Error<TABLE_NOT_FOUND>(
- "row-binlog tablet not found when initializing group rowset builder, "
+ "changelog tablet not found when initializing group rowset builder, "
"tablet_id={}",
- _row_binlog_tablet_id);
+ _changelog_tablet_id);
}
- // Hold the row-binlog migration lock until its relation is visible from the prepared base
+ // Hold the changelog migration lock until its relation is visible from the prepared base
// transaction. Migration can then drain the transaction instead of replacing this object.
- std::shared_lock<std::shared_timed_mutex> row_binlog_migration_lock(
- row_binlog_tablet->get_migration_lock(), std::defer_lock);
- if (!row_binlog_migration_lock.try_lock_for(
+ std::shared_lock<std::shared_timed_mutex> changelog_migration_lock(
+ changelog_tablet->get_migration_lock(), std::defer_lock);
+ if (!changelog_migration_lock.try_lock_for(
std::chrono::milliseconds(config::migration_lock_timeout_ms))) {
return Status::ObtainLockFailed(
- "try_lock row-binlog migration lock failed after {}ms, tablet_id={}",
- config::migration_lock_timeout_ms, _row_binlog_tablet_id);
+ "try_lock changelog migration lock failed after {}ms, tablet_id={}",
+ config::migration_lock_timeout_ms, _changelog_tablet_id);
}
- if (row_binlog_tablet->tablet_state() == TABLET_SHUTDOWN) {
+ if (changelog_tablet->tablet_state() == TABLET_SHUTDOWN) {
return Status::InternalError<false>(
- "row-binlog tablet is shutdown and may have been dropped or migrated, "
+ "changelog tablet is shutdown and may have been dropped or migrated, "
"tablet_id={}",
- _row_binlog_tablet_id);
+ _changelog_tablet_id);
}
- // Init binlog builder first so that its rowset id can be added into PendingLocalRowsets
+ // Init changelog builder first so that its rowset id can be added into PendingLocalRowsets
// before txn builder init.
- RETURN_IF_ERROR(_row_binlog_rowset_builder->init());
- if (_row_binlog_rowset_builder->tablet_sptr().get() != row_binlog_tablet.get()) {
+ RETURN_IF_ERROR(_changelog_rowset_builder->init());
+ if (_changelog_rowset_builder->tablet_sptr().get() != changelog_tablet.get()) {
return Status::InternalError(
- "row-binlog tablet changed while initializing group rowset builder, "
+ "changelog tablet changed while initializing group rowset builder, "
"tablet_id={}",
- _row_binlog_tablet_id);
+ _changelog_tablet_id);
}
// Before init txn, add all rowset ids into PendingLocalRowsets.
// See https://github.com/apache/doris/pull/25921.
RETURN_IF_ERROR(_txn_rs_builder->attach_pending_rs_guard_to_txn(
- _row_binlog_rowset_builder->rowset_id()));
+ _changelog_rowset_builder->rowset_id()));
RETURN_IF_ERROR(_txn_rs_builder->init());
- RETURN_IF_ERROR(_engine.txn_manager()->attach_row_binlog_tablet_to_txn(
+ RETURN_IF_ERROR(_engine.txn_manager()->attach_changelog_tablet_to_txn(
_req.partition_id, _req.txn_id, _txn_rs_builder->tablet_sptr()->get_tablet_info(),
- row_binlog_tablet));
+ changelog_tablet));
}
// Create a GroupRowsetWriter that forwards flush to both underlying
@@ -575,14 +574,14 @@ Status GroupRowsetBuilder::init() {
std::unique_ptr<doris::GroupRowsetWriter> group_writer;
RETURN_IF_ERROR(RowsetFactory::create_empty_group_rowset_writer(&group_writer));
group_writer->set_data_writer(_txn_rs_builder->rowset_writer());
- group_writer->set_row_binlog_writer(_row_binlog_rowset_builder->rowset_writer());
+ group_writer->set_changelog_writer(_changelog_rowset_builder->rowset_writer());
RETURN_IF_ERROR(group_writer->init(_txn_rs_builder->rowset_writer()->context()));
{
const auto& data_ctx = _txn_rs_builder->rowset_writer()->context();
- auto& binlog_ctx = const_cast<RowsetWriterContext&>(
- _row_binlog_rowset_builder->rowset_writer()->context());
- auto& cfg = binlog_ctx.write_binlog_opt().write_binlog_config();
+ auto& changelog_ctx = const_cast<RowsetWriterContext&>(
+ _changelog_rowset_builder->rowset_writer()->context());
+ auto& cfg = changelog_ctx.write_changelog_opt().write_changelog_config();
cfg.source.tablet_schema = data_ctx.tablet_schema;
cfg.source.partial_update_info = data_ctx.partial_update_info;
cfg.source.mow_context = data_ctx.mow_context;
@@ -606,27 +605,27 @@ Status GroupRowsetBuilder::wait_calc_delete_bitmap() {
Status GroupRowsetBuilder::cancel(const Status& st) {
RETURN_IF_ERROR(_txn_rs_builder->cancel(st));
- RETURN_IF_ERROR(_row_binlog_rowset_builder->cancel(st));
+ RETURN_IF_ERROR(_changelog_rowset_builder->cancel(st));
return BaseRowsetBuilder::cancel(st);
}
Status GroupRowsetBuilder::commit_txn() {
- // Attach binlog rowset (and its independent binlog tablet) to txn rowset, so that
+ // Attach changelog rowset (and its independent changelog tablet) to txn rowset, so that
// commit/rollback and clean-up are all handled by txn rowset builder.
- RowBinlogTxnInfo attach_row_binlog;
- attach_row_binlog.rowset = _row_binlog_rowset_builder->rowset();
- attach_row_binlog.tablet =
- std::static_pointer_cast<Tablet>(_row_binlog_rowset_builder->tablet_sptr());
- RETURN_IF_ERROR(_txn_rs_builder->attach_row_binlog_to_txn(attach_row_binlog));
+ ChangelogTxnInfo attach_changelog;
+ attach_changelog.rowset = _changelog_rowset_builder->rowset();
+ attach_changelog.tablet =
+ std::static_pointer_cast<Tablet>(_changelog_rowset_builder->tablet_sptr());
+ RETURN_IF_ERROR(_txn_rs_builder->attach_changelog_to_txn(attach_changelog));
auto st = _txn_rs_builder->commit_txn();
if (st.ok()) {
- // Avoid RowBinlogRowsetBuilder being cleaned in its base dtor.
- RETURN_IF_ERROR(_row_binlog_rowset_builder->commit_txn());
+ // Avoid ChangelogRowsetBuilder being cleaned in its base dtor.
+ RETURN_IF_ERROR(_changelog_rowset_builder->commit_txn());
}
return st;
}
-Status RowBinlogRowsetBuilder::init() {
+Status ChangelogRowsetBuilder::init() {
RowsetWriterContext context;
RETURN_IF_ERROR(_init_context_common_fields(context));
@@ -635,7 +634,7 @@ Status RowBinlogRowsetBuilder::init() {
RETURN_IF_ERROR(_build_current_tablet_schema(_req.index_id, _req.table_schema_param.get(),
*_tablet->tablet_schema()));
context.tablet_schema = _tablet_schema;
- context.write_binlog_opt().enable = true;
+ context.write_changelog_opt().enable = true;
_rowset_writer = DORIS_TRY(_tablet->create_rowset_writer(context, false));
// need to attach PendingRowsetGuard after txn_rs_builder init
@@ -645,11 +644,11 @@ Status RowBinlogRowsetBuilder::init() {
return Status::OK();
}
-Status BaseRowsetBuilder::attach_row_binlog_to_txn(const RowBinlogTxnInfo& attach_row_binlog) {
+Status BaseRowsetBuilder::attach_changelog_to_txn(const ChangelogTxnInfo& attach_changelog) {
if (!is_data_builder()) {
return Status::RuntimeError("the rowset isn't allowed to manage txn");
}
- _attach_row_binlog = attach_row_binlog;
+ _attach_changelog = attach_changelog;
return Status::OK();
}
diff --git a/be/src/storage/rowset_builder.h b/be/src/storage/rowset_builder.h
index a0eb9f40d54..de83888a679 100644
--- a/be/src/storage/rowset_builder.h
+++ b/be/src/storage/rowset_builder.h
@@ -88,8 +88,8 @@ public:
bool is_data_builder() const { return _req.write_req_type == WriteRequestType::DATA; }
- // Attach the binlog rowset and its independent binlog tablet to the same txn.
- Status attach_row_binlog_to_txn(const RowBinlogTxnInfo& attach_row_binlog);
+ // Attach the changelog rowset and its independent changelog tablet to the same txn.
+ Status attach_changelog_to_txn(const ChangelogTxnInfo& attach_changelog);
// Attach an extra pending rowset id so that PendingLocalRowsets can be
// cleaned up together with the primary rowset.
@@ -114,8 +114,8 @@ protected:
WriteRequest _req;
BaseTabletSPtr _tablet;
RowsetSharedPtr _rowset;
- // The binlog rowset and its independent binlog tablet attached to the same txn.
- RowBinlogTxnInfo _attach_row_binlog;
+ // The changelog rowset and its independent changelog tablet attached to the same txn.
+ ChangelogTxnInfo _attach_changelog;
std::shared_ptr<RowsetWriter> _rowset_writer;
PendingRowsetGuard _pending_rs_guard;
// Extra rowset ids that share the same PendingRowsetGuard.
@@ -167,19 +167,19 @@ private:
RuntimeProfile::Counter* _commit_txn_timer = nullptr;
};
-// Rowset builder dedicated for row_binlog rowset. It writes to the independent
-// row_binlog tablet while being attached to the base tablet txn by GroupRowsetBuilder.
-class RowBinlogRowsetBuilder : public RowsetBuilder {
+// Rowset builder dedicated for changelog rowset. It writes to the independent
+// changelog tablet while being attached to the base tablet txn by GroupRowsetBuilder.
+class ChangelogRowsetBuilder : public RowsetBuilder {
public:
- RowBinlogRowsetBuilder(StorageEngine& engine, const WriteRequest& req, RuntimeProfile* profile);
+ ChangelogRowsetBuilder(StorageEngine& engine, const WriteRequest& req, RuntimeProfile* profile);
// just attach rowset to txn_rs_builder in GroupRowsetBuilder, then rely on
// txn_rs_builder's clean logic.
- ~RowBinlogRowsetBuilder() override = default;
+ ~ChangelogRowsetBuilder() override = default;
Status init() override;
- // before commit, binlog rowset builder is responsible for cleaning rowset.
+ // before commit, changelog rowset builder is responsible for cleaning rowset.
// after commit, rowset will be attached to data(txn) rowset builder, and
// the owner of rowset will be changed, so cleaning rowset is handed to the
// data(txn) rowset builder.
@@ -191,11 +191,11 @@ public:
};
// Manage one transaction with multiple rowset_builders.
-// eg. normal data rowset + row_binlog rowset.
+// eg. normal data rowset + changelog rowset.
class GroupRowsetBuilder : public BaseRowsetBuilder {
public:
GroupRowsetBuilder(StorageEngine& engine, const WriteRequest& group_build_req,
- const WriteRequest& sub_data_req, const WriteRequest& sub_row_binlog_req,
+ const WriteRequest& sub_data_req, const WriteRequest& sub_changelog_req,
RuntimeProfile* profile);
Status init() override;
@@ -223,15 +223,15 @@ public:
}
RowsetBuilder* txn_rowset_builder() { return _txn_rs_builder.get(); }
- RowsetBuilder* row_binlog_builder() { return _row_binlog_rowset_builder.get(); }
+ RowsetBuilder* changelog_builder() { return _changelog_rowset_builder.get(); }
private:
// txn rowset builder will manage txn; other builders will add their
// rowsets into here.
StorageEngine& _engine;
- int64_t _row_binlog_tablet_id;
+ int64_t _changelog_tablet_id;
std::shared_ptr<RowsetBuilder> _txn_rs_builder;
- std::shared_ptr<RowsetBuilder> _row_binlog_rowset_builder;
+ std::shared_ptr<RowsetBuilder> _changelog_rowset_builder;
};
} // namespace doris
diff --git a/be/src/storage/schema.cpp b/be/src/storage/schema.cpp
index 642430b0de5..1d6df92e2c1 100644
--- a/be/src/storage/schema.cpp
+++ b/be/src/storage/schema.cpp
@@ -37,21 +37,20 @@
#include "core/data_type/data_type_variant_v2.h"
#include "core/data_type/primitive_type.h"
#include "runtime/descriptors.h"
-#include "storage/binlog.h"
+#include "storage/changelog.h"
namespace doris {
namespace {
-bool row_binlog_value_columns_have_same_type(const TabletColumn& lhs, const TabletColumn& rhs) {
+bool changelog_value_columns_have_same_type(const TabletColumn& lhs, const TabletColumn& rhs) {
if (lhs.type() != rhs.type() || lhs.is_nullable() != rhs.is_nullable() ||
lhs.length() != rhs.length() || lhs.precision() != rhs.precision() ||
lhs.frac() != rhs.frac() || lhs.get_subtype_count() != rhs.get_subtype_count()) {
return false;
}
for (uint32_t i = 0; i < lhs.get_subtype_count(); ++i) {
- if (!row_binlog_value_columns_have_same_type(lhs.get_sub_column(i),
- rhs.get_sub_column(i))) {
+ if (!changelog_value_columns_have_same_type(lhs.get_sub_column(i), rhs.get_sub_column(i))) {
return false;
}
}
@@ -210,26 +209,26 @@ void ReadSchema::_init_before_column_ordinals() {
_before_column_ordinals[ordinal] = ordinal;
continue;
}
- auto before_name = binlog::build_before_column_name(_read_columns[ordinal]->name());
+ auto before_name = changelog::build_before_column_name(_read_columns[ordinal]->name());
auto before = name_to_ordinal.find(before_name);
_before_column_ordinals[ordinal] =
before == name_to_ordinal.end() ? ordinal : before->second;
}
}
-void ReadSchema::_init_row_binlog_column_mappings(const TabletSchema& tablet_schema) {
+void ReadSchema::_init_changelog_column_mappings(const TabletSchema& tablet_schema) {
DORIS_CHECK_GE(_op_ordinal, 0);
DORIS_CHECK_EQ(_before_column_ordinals.size(), _num_block_columns);
- _row_binlog_value_column_pairs.clear();
- _row_binlog_value_pairs_complete = false;
+ _changelog_value_column_pairs.clear();
+ _changelog_value_pairs_complete = false;
std::vector<ColumnId> value_column_ids;
value_column_ids.reserve(tablet_schema.num_columns() - tablet_schema.num_key_columns());
for (ColumnId cid = 0; cid < tablet_schema.num_columns(); ++cid) {
- if (static_cast<int32_t>(cid) == tablet_schema.binlog_tso_col_idx() ||
- static_cast<int32_t>(cid) == tablet_schema.binlog_lsn_col_idx() ||
- static_cast<int32_t>(cid) == tablet_schema.binlog_op_col_idx() ||
+ if (static_cast<int32_t>(cid) == tablet_schema.changelog_tso_col_idx() ||
+ static_cast<int32_t>(cid) == tablet_schema.changelog_lsn_col_idx() ||
+ static_cast<int32_t>(cid) == tablet_schema.changelog_op_col_idx() ||
tablet_schema.column(cid).is_key()) {
continue;
}
@@ -244,20 +243,20 @@ void ReadSchema::_init_row_binlog_column_mappings(const TabletSchema& tablet_sch
for (size_t i = 0; i < value_column_count; ++i) {
const auto& after = tablet_schema.column(value_column_ids[i]);
const auto& before = tablet_schema.column(value_column_ids[i + value_column_count]);
- if (before.name() != binlog::build_before_column_name(after.name()) ||
- !row_binlog_value_columns_have_same_type(after, before)) {
+ if (before.name() != changelog::build_before_column_name(after.name()) ||
+ !changelog_value_columns_have_same_type(after, before)) {
return;
}
}
- // A valid physical row-binlog layout starts from an identity mapping. Only AFTER value
+ // A valid physical changelog layout starts from an identity mapping. Only AFTER value
// columns map to their BEFORE companions; keys, metadata and BEFORE columns map to themselves.
for (ColumnId ordinal = 0; ordinal < _num_block_columns; ++ordinal) {
_before_column_ordinals[ordinal] = ordinal;
}
bool complete = true;
- _row_binlog_value_column_pairs.reserve(value_column_count);
+ _changelog_value_column_pairs.reserve(value_column_count);
for (size_t i = 0; i < value_column_count; ++i) {
const auto after_cid = value_column_ids[i];
const auto before_cid = value_column_ids[i + value_column_count];
@@ -277,10 +276,10 @@ void ReadSchema::_init_row_binlog_column_mappings(const TabletSchema& tablet_sch
complete = false;
continue;
}
- _row_binlog_value_column_pairs.emplace_back(after, before);
+ _changelog_value_column_pairs.emplace_back(after, before);
}
- _row_binlog_value_pairs_complete =
- complete && _row_binlog_value_column_pairs.size() == value_column_count;
+ _changelog_value_pairs_complete =
+ complete && _changelog_value_column_pairs.size() == value_column_count;
}
Block ReadSchema::create_read_block() const {
@@ -322,8 +321,8 @@ std::string ReadSchema::read_columns_to_string() const {
Status ReadSchema::init_from_tablet_schema(const TabletSchema& tablet_schema,
bool merge_by_sequence_mapping,
- bool map_row_binlog_columns) {
- DORIS_CHECK(!(merge_by_sequence_mapping && map_row_binlog_columns));
+ bool map_changelog_columns) {
+ DORIS_CHECK(!(merge_by_sequence_mapping && map_changelog_columns));
_tablet_has_sequence_map = tablet_schema.has_seq_map();
_tablet_has_extracted_variant_columns =
std::ranges::any_of(tablet_schema.columns(),
@@ -331,8 +330,8 @@ Status ReadSchema::init_from_tablet_schema(const TabletSchema& tablet_schema,
if (merge_by_sequence_mapping) {
RETURN_IF_ERROR(_init_sequence_map(tablet_schema));
}
- if (map_row_binlog_columns) {
- _init_row_binlog_column_mappings(tablet_schema);
+ if (map_changelog_columns) {
+ _init_changelog_column_mappings(tablet_schema);
}
return Status::OK();
}
diff --git a/be/src/storage/schema.h b/be/src/storage/schema.h
index 21b6a614b2c..540bbc1aa1d 100644
--- a/be/src/storage/schema.h
+++ b/be/src/storage/schema.h
@@ -60,7 +60,7 @@ std::vector<TabletColumnPtr> project_columns_by_ordinal(
class ReadSchema {
public:
using SequenceMap = std::unordered_map<ColumnId, std::vector<ColumnId>>;
- using RowBinlogValueColumnPairs = std::vector<std::pair<ColumnId, ColumnId>>;
+ using ChangelogValueColumnPairs = std::vector<std::pair<ColumnId, ColumnId>>;
explicit ReadSchema(std::vector<TabletColumnPtr> columns);
@@ -94,10 +94,10 @@ public:
std::string read_columns_to_string() const;
- // Always sets the two facts below; the flags add the sequence mapping and the row-binlog
+ // Always sets the two facts below; the flags add the sequence mapping and the changelog
// before-image mapping.
Status init_from_tablet_schema(const TabletSchema& tablet_schema,
- bool merge_by_sequence_mapping, bool map_row_binlog_columns);
+ bool merge_by_sequence_mapping, bool map_changelog_columns);
// Whether the tablet schema defines a sequence mapping -- not whether this read merges by
// one. A MoW query does not merge, yet a value predicate still must not be pushed down.
@@ -111,19 +111,19 @@ public:
const SequenceMap& sequence_map() const { return _sequence_map; }
- // Return the matching before-image ordinal for a Row Binlog value column. For example, in
- // [v1, v2, __BEFORE__v1__, __BEFORE__v2__], 0 maps to 2 and 1 maps to 3. Columns without a
- // before image, including TSO/LSN/OP, map to themselves.
+ // Return the matching before-image ordinal for a changelog value column. For example, in
+ // [v1, v2, __DORIS_CHANGELOG_BEFORE_v1__, __DORIS_CHANGELOG_BEFORE_v2__], 0 maps to 2 and 1
+ // maps to 3. Columns without a before image, including TSO/LSN/OP, map to themselves.
ColumnId before_column_ordinal(ColumnId ordinal) const {
DCHECK_LT(ordinal, _before_column_ordinals.size());
return _before_column_ordinals[ordinal];
}
- const RowBinlogValueColumnPairs& row_binlog_value_column_pairs() const {
- return _row_binlog_value_column_pairs;
+ const ChangelogValueColumnPairs& changelog_value_column_pairs() const {
+ return _changelog_value_column_pairs;
}
- bool row_binlog_value_pairs_complete() const { return _row_binlog_value_pairs_complete; }
+ bool changelog_value_pairs_complete() const { return _changelog_value_pairs_complete; }
const TabletColumn* column(size_t ordinal) const { return _read_columns[ordinal].get(); }
@@ -141,8 +141,8 @@ public:
size_t num_key_columns() const { return _num_key_columns; }
// All special-column ordinals below address the caller-visible Block prefix and are -1 when
- // absent. A Row Binlog layout may be [k1, v1, __BEFORE__v1__, TSO, LSN, OP]; a snapshot layout
- // may instead contain COMMIT_TSO.
+ // absent. A changelog layout may be [k1, v1, __DORIS_CHANGELOG_BEFORE_v1__, TSO, LSN, OP]; a
+ // snapshot layout may instead contain COMMIT_TSO.
// Logical-delete marker used by unique-key reads.
int32_t delete_sign_ordinal() const { return _delete_sign_ordinal; }
// Sequence column used to choose the winning row during merge.
@@ -151,11 +151,11 @@ public:
int32_t rowid_ordinal() const { return _rowid_ordinal; }
// Rowset version synthesized for single-version reads.
int32_t version_ordinal() const { return _version_ordinal; }
- // Row Binlog transaction timestamp.
+ // Changelog transaction timestamp.
int32_t tso_ordinal() const { return _tso_ordinal; }
- // Row Binlog log-sequence number.
+ // Changelog log-sequence number.
int32_t lsn_ordinal() const { return _lsn_ordinal; }
- // Row Binlog operation kind, such as INSERT, UPDATE, or DELETE.
+ // Changelog operation kind, such as INSERT, UPDATE, or DELETE.
int32_t op_ordinal() const { return _op_ordinal; }
// Snapshot commit timestamp.
int32_t commit_tso_ordinal() const { return _commit_tso_ordinal; }
@@ -170,7 +170,7 @@ private:
void _init_read_types();
void _init_before_column_ordinals();
Status _init_sequence_map(const TabletSchema& tablet_schema);
- void _init_row_binlog_column_mappings(const TabletSchema& tablet_schema);
+ void _init_changelog_column_mappings(const TabletSchema& tablet_schema);
void _init_descriptors() {
DORIS_CHECK_LE(_num_block_columns, _read_columns.size());
@@ -210,13 +210,13 @@ private:
if (col.name() == VERSION_COL) {
_version_ordinal = i;
}
- if (col.name() == BINLOG_TSO_COL) {
+ if (col.name() == CHANGELOG_TSO_COL) {
_tso_ordinal = i;
}
- if (col.name() == BINLOG_LSN_COL) {
+ if (col.name() == CHANGELOG_LSN_COL) {
_lsn_ordinal = i;
}
- if (col.name() == BINLOG_OP_COL) {
+ if (col.name() == CHANGELOG_OP_COL) {
_op_ordinal = i;
}
if (col.name() == COMMIT_TSO_COL) {
@@ -257,8 +257,8 @@ private:
std::unordered_map<int32_t, int32_t> _uid_to_ordinal;
SequenceMap _sequence_map;
std::vector<ColumnId> _before_column_ordinals;
- RowBinlogValueColumnPairs _row_binlog_value_column_pairs;
- bool _row_binlog_value_pairs_complete = false;
+ ChangelogValueColumnPairs _changelog_value_column_pairs;
+ bool _changelog_value_pairs_complete = false;
bool _tablet_has_sequence_map = false;
bool _tablet_has_extracted_variant_columns = false;
};
diff --git a/be/src/storage/schema_change/schema_change.cpp b/be/src/storage/schema_change/schema_change.cpp
index b1eb451304d..05d7c446645 100644
--- a/be/src/storage/schema_change/schema_change.cpp
+++ b/be/src/storage/schema_change/schema_change.cpp
@@ -990,7 +990,7 @@ Status SchemaChangeJob::_do_process_alter_tablet(const TAlterTabletReqV2& reques
ReadSchemaSPtr read_schema = std::make_shared<ReadSchema>(std::move(read_columns));
RETURN_IF_ERROR(read_schema->init_from_tablet_schema(*_base_tablet_schema,
/*merge_by_sequence_mapping=*/false,
- /*map_row_binlog_columns=*/false));
+ /*map_changelog_columns=*/false));
std::vector<uint32_t> cluster_key_idxes;
DBUG_EXECUTE_IF("SchemaChangeJob::_do_process_alter_tablet.block", DBUG_BLOCK);
diff --git a/be/src/storage/segment/historical_row_retriever.cpp b/be/src/storage/segment/historical_row_retriever.cpp
index 56c7389f876..a3aafd88591 100644
--- a/be/src/storage/segment/historical_row_retriever.cpp
+++ b/be/src/storage/segment/historical_row_retriever.cpp
@@ -31,7 +31,7 @@
#include "core/data_type/data_type.h"
#include "core/string_ref.h"
#include "runtime/exec_env.h"
-#include "storage/binlog.h"
+#include "storage/changelog.h"
#include "storage/data_dir.h"
#include "storage/iterator/olap_data_convertor.h"
#include "storage/key/row_key_encoder.h"
@@ -91,14 +91,14 @@ Status PrimaryKeyModelRowRetriever::retrieve_historical_row(const Int8* delete_s
CHECK(_context.rowset_writer_ctx != nullptr);
bool write_before =
- _context.rowset_writer_ctx->write_binlog_opt().write_binlog_config().write_before;
+ _context.rowset_writer_ctx->write_changelog_opt().write_changelog_config().write_before;
// The lookup uses the tablet's latest schema, the delete-sign rule the source schema. Hold the
// shared pointer: tablet_schema() hands out a copy and the probe only borrows it.
TabletSchemaSPtr lookup_schema = _context.tablet->tablet_schema();
- MowKeyProbe probe = MowKeyProbe::for_row_binlog(_context.tablet.get(), lookup_schema.get(),
- tablet_schema->has_sequence_col(), _mow_context,
- write_before);
- // the binlog retriever reports no partial update counters
+ MowKeyProbe probe = MowKeyProbe::for_changelog(_context.tablet.get(), lookup_schema.get(),
+ tablet_schema->has_sequence_col(), _mow_context,
+ write_before);
+ // the changelog retriever reports no partial update counters
PartialUpdateStats discarded_stats;
for (size_t block_pos = row_pos; block_pos < row_pos + num_rows; block_pos++) {
@@ -120,13 +120,13 @@ Status PrimaryKeyModelRowRetriever::retrieve_historical_row(const Int8* delete_s
// it's an insert row
_has_default_or_nullable = true;
_use_default_or_null_flag.emplace_back(true);
- _operators.emplace_back(have_delete_sign ? ROW_BINLOG_DELETE : ROW_BINLOG_APPEND);
+ _operators.emplace_back(have_delete_sign ? CHANGELOG_OP_DELETE : CHANGELOG_OP_APPEND);
continue;
}
if (outcome.use_default_or_null) {
_has_default_or_nullable = true;
_use_default_or_null_flag.emplace_back(true);
- _operators.emplace_back(ROW_BINLOG_DELETE);
+ _operators.emplace_back(CHANGELOG_OP_DELETE);
} else {
// partial update should not contain invisible columns
_use_default_or_null_flag.emplace_back(false);
@@ -134,7 +134,7 @@ Status PrimaryKeyModelRowRetriever::retrieve_historical_row(const Int8* delete_s
// currently we think row_pos must be zero, so we won't consider row_pos > 0
DCHECK(row_pos == 0);
_row_fetcher->plan_fixed_read(outcome.loc, delta_pos);
- _operators.emplace_back(have_delete_sign ? ROW_BINLOG_DELETE : ROW_BINLOG_UPDATE);
+ _operators.emplace_back(have_delete_sign ? CHANGELOG_OP_DELETE : CHANGELOG_OP_UPDATE);
}
}
@@ -175,11 +175,11 @@ Status PrimaryKeyModelRowRetriever::materialize_flexible_partial_update(
CHECK(_context.rowset_writer_ctx != nullptr);
const bool write_before =
- _context.rowset_writer_ctx->write_binlog_opt().write_binlog_config().write_before;
+ _context.rowset_writer_ctx->write_changelog_opt().write_changelog_config().write_before;
TabletSchemaSPtr lookup_schema = _context.tablet->tablet_schema();
- MowKeyProbe probe = MowKeyProbe::for_row_binlog(_context.tablet.get(), lookup_schema.get(),
- tablet_schema->has_sequence_col(), _mow_context,
- write_before);
+ MowKeyProbe probe = MowKeyProbe::for_changelog(_context.tablet.get(), lookup_schema.get(),
+ tablet_schema->has_sequence_col(), _mow_context,
+ write_before);
BlockAggregator aggregator(*tablet_schema, _context.tablet, _mow_context,
*_context.partial_update_info, *_key_encoder, probe, *_row_fetcher);
@@ -265,8 +265,8 @@ Status PrimaryKeyModelRowRetriever::materialize_flexible_partial_update(
}
_operators.emplace_back(
outcome.result == KeyProbeResult::NOT_FOUND
- ? (have_delete_sign ? ROW_BINLOG_DELETE : ROW_BINLOG_APPEND)
- : (have_delete_sign ? ROW_BINLOG_DELETE : ROW_BINLOG_UPDATE));
+ ? (have_delete_sign ? CHANGELOG_OP_DELETE : CHANGELOG_OP_APPEND)
+ : (have_delete_sign ? CHANGELOG_OP_DELETE : CHANGELOG_OP_UPDATE));
}
RETURN_IF_ERROR(_row_fetcher->fill_non_primary_key_columns(
@@ -354,8 +354,8 @@ Status PrimaryKeyModelRowRetriever::revise_operators_by_old_delete_sign(size_t n
DCHECK_EQ(_old_delete_signs.size(), num_rows);
for (size_t idx = 0; idx < num_rows; ++idx) {
- if (_operators[idx] == ROW_BINLOG_UPDATE && _old_delete_signs[idx] != 0) {
- _operators[idx] = ROW_BINLOG_APPEND;
+ if (_operators[idx] == CHANGELOG_OP_UPDATE && _old_delete_signs[idx] != 0) {
+ _operators[idx] = CHANGELOG_OP_APPEND;
}
}
return Status::OK();
diff --git a/be/src/storage/segment/historical_row_retriever.h b/be/src/storage/segment/historical_row_retriever.h
index d4042d0e3be..7fbc00582bb 100644
--- a/be/src/storage/segment/historical_row_retriever.h
+++ b/be/src/storage/segment/historical_row_retriever.h
@@ -81,7 +81,7 @@ public:
Status retrieve_historical_row(const Int8* delete_sign_column_data, size_t row_pos,
size_t num_rows) override;
- // Row Binlog receives the original flexible-partial-update block, while the base writer fills
+ // Changelog receives the original flexible-partial-update block, while the base writer fills
// its own copy. Reuse the common flexible aggregator/read plans with a read-only MOW probe to
// build the same full AFTER rows without changing the base tablet's delete bitmap. The optional
// LSN sidecar is merged and filtered together with the rows.
@@ -115,7 +115,7 @@ private:
std::vector<bool> _use_default_or_null_flag;
bool _has_default_or_nullable = false;
- // cache operator for fill_binlog_columns
+ // cache operator for fill_changelog_columns
std::vector<int64_t> _operators;
// Aligned by row position in the current append batch.
std::vector<signed char> _old_delete_signs;
diff --git a/be/src/storage/segment/segment.cpp b/be/src/storage/segment/segment.cpp
index 0e60f76cc2a..30d03a2457c 100644
--- a/be/src/storage/segment/segment.cpp
+++ b/be/src/storage/segment/segment.cpp
@@ -1042,12 +1042,12 @@ Status Segment::_get_column_reader_for_read(const TabletColumn& col,
&read_options.io_ctx);
}
- if (col.name() == BINLOG_TSO_COL || col.name() == COMMIT_TSO_COL) {
+ if (col.name() == CHANGELOG_TSO_COL || col.name() == COMMIT_TSO_COL) {
const int64_t start_tso = read_options.commit_tso.start_tso();
const int64_t end_tso = read_options.commit_tso.end_tso();
// Version [0-0] identifies a pre-publish physical read, such as segment compaction while a
// RowsetWriter is still open. Its TSO is intentionally [-1--1], so preserve the on-disk
- // COMMIT_TSO_COL=0 or BINLOG_TSO_COL=NULL placeholder instead of synthesizing a value.
+ // COMMIT_TSO_COL=0 or CHANGELOG_TSO_COL=NULL placeholder instead of synthesizing a value.
if (read_options.version == Version(0, 0) && start_tso == -1 && end_tso == -1) {
if (!_column_meta_accessor->has_column_uid(col_uid)) {
return Status::InternalError("could not find {} column", col.name());
diff --git a/be/src/storage/segment/segment_iterator.cpp b/be/src/storage/segment/segment_iterator.cpp
index f5895aac905..3b19fb44fde 100644
--- a/be/src/storage/segment/segment_iterator.cpp
+++ b/be/src/storage/segment/segment_iterator.cpp
@@ -76,7 +76,7 @@
#include "runtime/runtime_state.h"
#include "runtime/thread_context.h"
#include "service/backend_options.h"
-#include "storage/binlog.h"
+#include "storage/changelog.h"
#include "storage/id_manager.h"
#include "storage/index/ann/ann_index.h"
#include "storage/index/ann/ann_index_iterator.h"
@@ -1781,13 +1781,13 @@ bool SegmentIterator::_need_read_data(ColumnId cid) {
// occurring, return true here that column data needs to be read
return true;
}
- // Row-binlog incremental reads force-push the TSO range predicate (see
+ // Changelog incremental reads force-push the TSO range predicate (see
// OlapScanner::_init_tso_predicate), and the merge iterator uses the TSO column as its
- // sequence sort key (BetaRowsetReader sets binlog_tso_idx). On a cross-version rowset
- // (e.g. produced by binlog LMax quick-merge) the TSO zonemap can be always-true, so the
+ // sequence sort key (BetaRowsetReader sets changelog_tso_idx). On a cross-version rowset
+ // (e.g. produced by changelog LMax quick-merge) the TSO zonemap can be always-true, so the
// pruning below would skip reading it and fill placeholder zeros, breaking the merge
// ordering. The TSO column carries real values on disk, so force it to be read.
- if (_opts.read_row_binlog && cid == _schema->tso_ordinal()) {
+ if (_opts.read_changelog && cid == _schema->tso_ordinal()) {
return true;
}
const auto& column = *_schema->column(cid);
diff --git a/be/src/storage/snapshot/snapshot_manager.cpp b/be/src/storage/snapshot/snapshot_manager.cpp
index dab2df21e16..d9800528501 100644
--- a/be/src/storage/snapshot/snapshot_manager.cpp
+++ b/be/src/storage/snapshot/snapshot_manager.cpp
@@ -355,8 +355,8 @@ Status SnapshotManager::_rename_rowset_id(const RowsetMetaPB& rs_meta_pb,
// keep segments_overlap same as origin rowset
context.segments_overlap = rowset_meta->segments_overlap();
context.compaction_level = org_rowset_meta->compaction_level();
- if (org_rowset_meta->is_row_binlog()) {
- context.write_binlog_opt().enable = true;
+ if (org_rowset_meta->is_changelog()) {
+ context.write_changelog_opt().enable = true;
}
// propagate MOW flag so that non-MOW key-bounds aggregation is not applied
// when restoring a MOW tablet's rowset
diff --git a/be/src/storage/storage_engine.cpp b/be/src/storage/storage_engine.cpp
index fbe50f1caf1..da3fe14d36b 100644
--- a/be/src/storage/storage_engine.cpp
+++ b/be/src/storage/storage_engine.cpp
@@ -740,14 +740,14 @@ void StorageEngine::stop() {
}
_stop_background_threads_latch.count_down();
- _stop_row_binlog_ttl_scanner();
+ _stop_changelog_ttl_scanner();
#define THREAD_JOIN(thread) \
if (thread) { \
thread->join(); \
}
THREAD_JOIN(_compaction_tasks_producer_thread);
- THREAD_JOIN(_binlog_compaction_tasks_producer_thread);
+ THREAD_JOIN(_changelog_compaction_tasks_producer_thread);
THREAD_JOIN(_unused_rowset_monitor_thread);
THREAD_JOIN(_garbage_sweeper_thread);
THREAD_JOIN(_disk_stat_monitor_thread);
@@ -775,8 +775,8 @@ void StorageEngine::stop() {
if (_cumu_compaction_thread_pool) {
_cumu_compaction_thread_pool->shutdown();
}
- if (_binlog_compaction_thread_pool) {
- _binlog_compaction_thread_pool->shutdown();
+ if (_changelog_compaction_thread_pool) {
+ _changelog_compaction_thread_pool->shutdown();
}
if (_seg_compaction_thread_pool) {
diff --git a/be/src/storage/storage_engine.h b/be/src/storage/storage_engine.h
index 481e4f8e4b5..be70ac819de 100644
--- a/be/src/storage/storage_engine.h
+++ b/be/src/storage/storage_engine.h
@@ -126,9 +126,9 @@ public:
virtual Status get_tablet_meta(int64_t tablet_id, TabletMetaSharedPtr* tablet_meta,
bool force_use_only_cached = false) = 0;
- void register_row_binlog_tablet(const BaseTabletSPtr& tablet);
+ void register_changelog_tablet(const BaseTabletSPtr& tablet);
// Catalog discovery refreshes cached Cloud metadata too, including tablets with no recent reads.
- Status submit_row_binlog_ttl(int64_t tablet_id, bool refresh_cloud_meta = false);
+ Status submit_changelog_ttl(int64_t tablet_id, bool refresh_cloud_meta = false);
void register_report_listener(ReportWorker* listener);
void deregister_report_listener(ReportWorker* listener);
@@ -185,13 +185,13 @@ protected:
std::unique_ptr<CalcDeleteBitmapExecutor> _calc_delete_bitmap_executor_for_load;
CountDownLatch _stop_background_threads_latch;
- Status _start_row_binlog_ttl_scanner();
- void _stop_row_binlog_ttl_scanner();
- std::mutex _row_binlog_ttl_mutex;
- std::map<int64_t, std::weak_ptr<BaseTablet>> _row_binlog_ttl_tablets;
- std::unordered_set<int64_t> _row_binlog_ttl_pending;
- std::unique_ptr<ThreadPool> _row_binlog_ttl_prepare_pool;
- std::shared_ptr<Thread> _row_binlog_ttl_scan_thread;
+ Status _start_changelog_ttl_scanner();
+ void _stop_changelog_ttl_scanner();
+ std::mutex _changelog_ttl_mutex;
+ std::map<int64_t, std::weak_ptr<BaseTablet>> _changelog_ttl_tablets;
+ std::unordered_set<int64_t> _changelog_ttl_pending;
+ std::unique_ptr<ThreadPool> _changelog_ttl_prepare_pool;
+ std::shared_ptr<Thread> _changelog_ttl_scan_thread;
std::shared_ptr<Thread> _id_file_map_gc_thread;
@@ -206,7 +206,7 @@ protected:
std::unique_ptr<ThreadPool> _base_compaction_thread_pool;
std::unique_ptr<ThreadPool> _cumu_compaction_thread_pool;
- std::unique_ptr<ThreadPool> _binlog_compaction_thread_pool;
+ std::unique_ptr<ThreadPool> _changelog_compaction_thread_pool;
int _cumu_compaction_thread_pool_used_threads {0};
int _cumu_compaction_thread_pool_small_tasks_running {0};
};
@@ -248,7 +248,7 @@ private:
Registry _tablet_submitted_cumu_compaction;
Registry _tablet_submitted_base_compaction;
Registry _tablet_submitted_full_compaction;
- Registry _tablet_submitted_binlog_compaction;
+ Registry _tablet_submitted_changelog_compaction;
};
class StorageEngine final : public BaseStorageEngine {
@@ -459,14 +459,14 @@ private:
void _start_disk_stat_monitor();
void _compaction_tasks_producer_callback();
- void _binlog_compaction_tasks_producer_callback();
+ void _changelog_compaction_tasks_producer_callback();
std::vector<TabletCompactionContext> _generate_compaction_tasks(
CompactionType compaction_type, std::vector<DataDir*>& data_dirs, bool check_score);
void _update_cumulative_compaction_policy();
CumuCompactionPolicyTable _snapshot_cumulative_compaction_policy();
std::shared_ptr<CumulativeCompactionPolicy> _get_cumulative_compaction_policy(
- std::string_view compaction_policy);
+ std::string_view compaction_policy, int64_t tablet_id);
void _pop_tablet_from_submitted_compaction(TabletSharedPtr tablet,
CompactionType compaction_type);
@@ -543,7 +543,7 @@ private:
std::shared_ptr<Thread> _disk_stat_monitor_thread;
// thread to produce both base and cumulative compaction tasks
std::shared_ptr<Thread> _compaction_tasks_producer_thread;
- std::shared_ptr<Thread> _binlog_compaction_tasks_producer_thread;
+ std::shared_ptr<Thread> _changelog_compaction_tasks_producer_thread;
std::shared_ptr<Thread> _cache_clean_thread;
// threads to clean all file descriptor not actively in use
std::vector<std::shared_ptr<Thread>> _path_gc_threads;
diff --git a/be/src/storage/tablet/base_tablet.cpp b/be/src/storage/tablet/base_tablet.cpp
index b3bbf336ab6..f831d3bd48f 100644
--- a/be/src/storage/tablet/base_tablet.cpp
+++ b/be/src/storage/tablet/base_tablet.cpp
@@ -45,7 +45,7 @@
#include "runtime/cluster_info.h"
#include "runtime/exec_env.h"
#include "service/point_query_executor.h"
-#include "storage/binlog.h"
+#include "storage/changelog.h"
#include "storage/compaction/cumulative_compaction_time_series_policy.h"
#include "storage/delete/calc_delete_bitmap_executor.h"
#include "storage/delete/delete_bitmap_calculator.h"
@@ -286,33 +286,33 @@ RowsetSharedPtr BaseTablet::get_stale_rowset_by_version(const Version& version)
return iter->second;
}
-bool BaseTablet::scan_expired_row_binlog_rowsets() {
- if (!is_row_binlog_tablet()) {
+bool BaseTablet::scan_expired_changelog_rowsets() {
+ if (!is_changelog_tablet()) {
return false;
}
// The cursor is a version key, never an iterator retained across publish or compaction.
// At most 256 metadata entries are visited while holding the header lock.
std::unique_lock lock(_meta_lock);
- const auto reference = ExecEnv::GetInstance()->cluster_info()->row_binlog_ttl_reference_tso();
- const auto cutoff = _tablet_meta->binlog_config().row_ttl_cutoff_tso(reference);
+ const auto reference = ExecEnv::GetInstance()->cluster_info()->changelog_ttl_reference_tso();
+ const auto cutoff = _tablet_meta->binlog_config().changelog_ttl_cutoff_tso(reference);
if (cutoff < 0 || _tablet_meta->disable_auto_compaction()) {
return false;
}
const auto& metas = _tablet_meta->all_rs_metas();
- auto it = metas.find(_row_binlog_ttl_scan_cursor);
+ auto it = metas.find(_changelog_ttl_scan_cursor);
if (it == metas.end()) {
it = metas.begin();
} else {
++it;
}
for (int n = 0; n < 256 && it != metas.end(); ++n, ++it) {
- _row_binlog_ttl_scan_cursor = it->first;
- if (row_binlog_rowset_expired(*it->second, cutoff)) {
+ _changelog_ttl_scan_cursor = it->first;
+ if (changelog_rowset_expired(*it->second, cutoff)) {
return true;
}
}
if (it == metas.end()) {
- _row_binlog_ttl_scan_cursor = Version(-1, -1);
+ _changelog_ttl_scan_cursor = Version(-1, -1);
}
return false;
}
@@ -688,19 +688,19 @@ Status BaseTablet::calc_segment_delete_bitmap(RowsetSharedPtr rowset,
// use for partial update
FixedReadPlan read_plan_ori;
FixedReadPlan read_plan_update;
- // used to read sink-time LSN from input row-binlog rowset
+ // used to read sink-time LSN from input changelog rowset
FixedReadPlan read_plan_lsn;
- RowsetSharedPtr row_binlog_rowset;
+ RowsetSharedPtr changelog_rowset;
if (rowset_writer != nullptr) {
if (auto* group_writer = typeid_cast<GroupRowsetWriter*>(rowset_writer);
group_writer != nullptr) {
- auto& binlog_ctx =
- const_cast<RowsetWriterContext&>(group_writer->row_binlog_writer()->context());
- if (binlog_ctx.write_binlog_opt().enable) {
- row_binlog_rowset = binlog_ctx.write_binlog_opt()
- .write_binlog_config()
- .source.row_binlog_rowset;
- DCHECK(row_binlog_rowset != nullptr);
+ auto& changelog_ctx =
+ const_cast<RowsetWriterContext&>(group_writer->changelog_writer()->context());
+ if (changelog_ctx.write_changelog_opt().enable) {
+ changelog_rowset = changelog_ctx.write_changelog_opt()
+ .write_changelog_config()
+ .source.changelog_rowset;
+ DCHECK(changelog_rowset != nullptr);
}
}
}
@@ -853,9 +853,9 @@ Status BaseTablet::calc_segment_delete_bitmap(RowsetSharedPtr rowset,
read_plan_ori.prepare_to_read(loc, pos);
}
read_plan_update.prepare_to_read(RowLocation {rowset_id, seg->id(), row_id}, pos);
- if (row_binlog_rowset != nullptr) {
+ if (changelog_rowset != nullptr) {
read_plan_lsn.prepare_to_read(
- RowLocation {row_binlog_rowset->rowset_id(), seg->id(), row_id}, pos);
+ RowLocation {changelog_rowset->rowset_id(), seg->id(), row_id}, pos);
}
// For flexible partial update, we should use skip bitmap to determine wheather
@@ -916,18 +916,18 @@ Status BaseTablet::calc_segment_delete_bitmap(RowsetSharedPtr rowset,
rsid_to_rowset, &block));
}
- // read sink-time LSN from input row-binlog rowset, aligned with `block` rows.
+ // read sink-time LSN from input changelog rowset, aligned with `block` rows.
Block lsn_block;
- if (row_binlog_rowset != nullptr) {
- auto row_binlog_schema = row_binlog_rowset->tablet_schema();
+ if (changelog_rowset != nullptr) {
+ auto changelog_schema = changelog_rowset->tablet_schema();
std::vector<uint32_t> lsn_cids = {
- static_cast<uint32_t>(row_binlog_schema->binlog_lsn_col_idx())};
- lsn_block = row_binlog_schema->create_storage_block(lsn_cids);
- std::map<RowsetId, RowsetSharedPtr> rsid_to_row_binlog {
- {row_binlog_rowset->rowset_id(), row_binlog_rowset}};
+ static_cast<uint32_t>(changelog_schema->changelog_lsn_col_idx())};
+ lsn_block = changelog_schema->create_storage_block(lsn_cids);
+ std::map<RowsetId, RowsetSharedPtr> rsid_to_changelog {
+ {changelog_rowset->rowset_id(), changelog_rowset}};
std::map<uint32_t, uint32_t> read_index;
RETURN_IF_ERROR(read_plan_lsn.read_columns_by_plan(
- *row_binlog_schema, lsn_cids, rsid_to_row_binlog, lsn_block, &read_index,
+ *changelog_schema, lsn_cids, rsid_to_changelog, lsn_block, &read_index,
FixedReadPlan::ReadStrategy::PREFER_ROW_STORE, false));
}
@@ -936,13 +936,13 @@ Status BaseTablet::calc_segment_delete_bitmap(RowsetSharedPtr rowset,
auto segment_id = DORIS_TRY(rowset_writer->allocate_segment_id());
// Publish-phase partial update may flush transient segments to a GroupRowsetWriter.
- // For row-binlog writing, the derive stage requires `seg_id -> lsn_ids` to be
+ // For changelog writing, the derive stage requires `seg_id -> lsn_ids` to be
// registered before the block is flushed.
if (auto* group_writer = typeid_cast<GroupRowsetWriter*>(rowset_writer);
group_writer != nullptr) {
- auto binlog_writer = group_writer->row_binlog_writer();
- auto& binlog_ctx = const_cast<RowsetWriterContext&>(binlog_writer->context());
- if (binlog_ctx.write_binlog_opt().enable) {
+ auto changelog_writer = group_writer->changelog_writer();
+ auto& changelog_ctx = const_cast<RowsetWriterContext&>(changelog_writer->context());
+ if (changelog_ctx.write_changelog_opt().enable) {
const auto& src =
assert_cast<const ColumnInt64&>(*lsn_block.get_by_position(0).column);
auto lsn_ids = std::make_shared<std::vector<int64_t>>();
@@ -950,7 +950,7 @@ Status BaseTablet::calc_segment_delete_bitmap(RowsetSharedPtr rowset,
for (auto p : sort_perm) {
lsn_ids->emplace_back(src.get_data()[p]);
}
- binlog_ctx.insert_segment_allocated_lsns(segment_id, std::move(lsn_ids));
+ changelog_ctx.insert_segment_allocated_lsns(segment_id, std::move(lsn_ids));
}
}
RETURN_IF_ERROR(rowset_writer->flush_single_block(&ordered_block, segment_id));
@@ -1683,25 +1683,43 @@ Status BaseTablet::update_delete_bitmap(const BaseTabletSPtr& self, TabletTxnInf
RowsetIdUnorderedSet rowset_ids_to_add;
RowsetIdUnorderedSet rowset_ids_to_del;
RowsetSharedPtr rowset = txn_info->rowset;
- RowsetSharedPtr row_binlog_rowset;
- bool build_row_binlog = false;
+ RowsetSharedPtr changelog_rowset;
+ bool build_changelog = false;
int64_t cur_version = rowset->start_version();
std::unique_ptr<RowsetWriter> transient_rs_writer;
DeleteBitmapPtr delete_bitmap = txn_info->delete_bitmap;
bool is_partial_update =
txn_info->partial_update_info && txn_info->partial_update_info->is_partial_update();
- const auto& binlog_rs = txn_info->attach_row_binlog.rowset;
- if (binlog_rs != nullptr && binlog_rs->rowset_meta() != nullptr &&
- binlog_rs->rowset_meta()->is_row_binlog()) {
- DCHECK(txn_info->attach_row_binlog.tablet != nullptr);
- row_binlog_rowset = binlog_rs;
- build_row_binlog =
+ const auto& changelog_rs = txn_info->attach_changelog.rowset;
+ if (changelog_rs != nullptr && changelog_rs->rowset_meta() != nullptr &&
+ changelog_rs->rowset_meta()->is_changelog()) {
+ DCHECK(txn_info->attach_changelog.tablet != nullptr);
+ changelog_rowset = changelog_rs;
+ build_changelog =
is_partial_update ||
- txn_info->attach_row_binlog.tablet->binlog_config().need_historical_value();
+ txn_info->attach_changelog.tablet->binlog_config().need_historical_value();
+ }
+ // The rewrite below reads the LSN of conflicting rows from the attached changelog rowset and
+ // derives TSO/LSN/OP for new changelog segments, at positions found by the hidden column
+ // names. Example: a load committed by a build that named those columns differently, then
+ // published after the upgrade (the FE resends the publish after every restart): its rowset
+ // has none of them, and the LSN read would index past the schema. Fail the publish instead.
+ if (build_changelog) {
+ const auto& changelog_schema = *changelog_rowset->tablet_schema();
+ if (changelog_schema.changelog_tso_col_idx() < 0 ||
+ changelog_schema.changelog_lsn_col_idx() < 0 ||
+ changelog_schema.changelog_op_col_idx() < 0) {
+ return Status::InternalError<false>(
+ "changelog rowset {} misses its hidden columns {}, {} or {}: changelog "
+ "tablet={}, base tablet={}, txn_id={}",
+ changelog_rowset->rowset_id().to_string(), CHANGELOG_TSO_COL, CHANGELOG_LSN_COL,
+ CHANGELOG_OP_COL, txn_info->attach_changelog.tablet->tablet_id(),
+ self->tablet_id(), txn_id);
+ }
}
// rewrite conflict only when partial update or need before
- if (is_partial_update || build_row_binlog) {
+ if (is_partial_update || build_changelog) {
if (txn_info->partial_update_info == nullptr) {
txn_info->partial_update_info = std::make_shared<PartialUpdateInfo>();
}
@@ -1818,43 +1836,45 @@ Status BaseTablet::update_delete_bitmap(const BaseTabletSPtr& self, TabletTxnInf
RETURN_IF_ERROR(token->wait());
}
- // Publish-phase partial update or upsert rewrite may generate transient segments. If row binlog is enabled for
- // this txn (row binlog rowset attached), we need to build row binlog segments together with
+ // Publish-phase partial update or upsert rewrite may generate transient segments. If changelog is enabled for
+ // this txn (changelog rowset attached), we need to build changelog segments together with
// transient data segments:
- // 1) create a transient row binlog writer that appends to the same version row binlog rowset
- // 2) pre-allocate per-row LSNs for each transient segment and register them in binlog options
+ // 1) create a transient changelog writer that appends to the same version changelog rowset
+ // 2) pre-allocate per-row LSNs for each transient segment and register them in changelog
+ // options
// 3) wrap both writers into a GroupRowsetWriter so calc_delete_bitmap writes to both.
- if (build_row_binlog) {
+ if (build_changelog) {
DCHECK(transient_rs_writer != nullptr);
- // Create transient row binlog writer for publish-phase segment appending.
- auto transient_row_binlog_writer =
- DORIS_TRY(txn_info->attach_row_binlog.tablet->create_transient_rowset_writer(
- *row_binlog_rowset, txn_info->partial_update_info, txn_expiration));
+ // Create transient changelog writer for publish-phase segment appending.
+ auto transient_changelog_writer =
+ DORIS_TRY(txn_info->attach_changelog.tablet->create_transient_rowset_writer(
+ *changelog_rowset, txn_info->partial_update_info, txn_expiration));
- // Prepare source MOW context for historical row retrieval in binlog writer.
+ // Prepare source MOW context for historical row retrieval in changelog writer.
auto& data_ctx = const_cast<RowsetWriterContext&>(transient_rs_writer->context());
data_ctx.mow_context = std::make_shared<MowContext>(
cur_version - 1, txn_id, std::make_shared<RowsetIdUnorderedSet>(),
specified_rowsets, nullptr);
- auto& binlog_ctx = const_cast<RowsetWriterContext&>(transient_row_binlog_writer->context());
- auto& cfg = binlog_ctx.write_binlog_opt().write_binlog_config();
+ auto& changelog_ctx =
+ const_cast<RowsetWriterContext&>(transient_changelog_writer->context());
+ auto& cfg = changelog_ctx.write_changelog_opt().write_changelog_config();
cfg.source.tablet_schema = data_ctx.tablet_schema;
cfg.source.partial_update_info = data_ctx.partial_update_info;
cfg.source.mow_context = data_ctx.mow_context;
cfg.source.is_transient_rowset_writer = data_ctx.is_transient_rowset_writer;
cfg.source.source_write_type = data_ctx.write_type;
- cfg.source.row_binlog_rowset = row_binlog_rowset;
+ cfg.source.changelog_rowset = changelog_rowset;
cfg.source.base_tablet = self;
// Wrap two transient writers into a group writer for dual flush/build.
RowsetWriterSharedPtr data_writer_sp(std::move(transient_rs_writer));
- RowsetWriterSharedPtr row_binlog_writer_sp(std::move(transient_row_binlog_writer));
+ RowsetWriterSharedPtr changelog_writer_sp(std::move(transient_changelog_writer));
std::unique_ptr<GroupRowsetWriter> group_writer;
RETURN_IF_ERROR(RowsetFactory::create_empty_group_rowset_writer(&group_writer));
group_writer->set_data_writer(data_writer_sp);
- group_writer->set_row_binlog_writer(row_binlog_writer_sp);
+ group_writer->set_changelog_writer(changelog_writer_sp);
RETURN_IF_ERROR(group_writer->init(group_writer->data_writer()->context()));
transient_rs_writer = std::move(group_writer);
}
@@ -1902,14 +1922,14 @@ Status BaseTablet::update_delete_bitmap(const BaseTabletSPtr& self, TabletTxnInf
// build rowset writer and merge transient rowset
RETURN_IF_ERROR(transient_rs_writer->flush());
RowsetSharedPtr transient_rowset;
- RowsetSharedPtr transient_row_binlog;
- if (build_row_binlog) {
+ RowsetSharedPtr transient_changelog;
+ if (build_changelog) {
auto* group_rowset_writer = typeid_cast<GroupRowsetWriter*>(transient_rs_writer.get());
DCHECK(group_rowset_writer != nullptr);
std::vector<RowsetSharedPtr> waited_build_rowsets;
RETURN_IF_ERROR(group_rowset_writer->build_rowsets(waited_build_rowsets));
transient_rowset = waited_build_rowsets.at(0);
- transient_row_binlog = waited_build_rowsets.at(1);
+ transient_changelog = waited_build_rowsets.at(1);
} else {
RETURN_IF_ERROR(transient_rs_writer->build(transient_rowset));
}
@@ -1921,16 +1941,16 @@ Status BaseTablet::update_delete_bitmap(const BaseTabletSPtr& self, TabletTxnInf
<< ", new segment num: " << new_segments << ")"
<< ", cost:" << watch.get_elapse_time_us() - t4 << "(us)";
- if (build_row_binlog) {
- DCHECK(row_binlog_rowset != nullptr);
- old_segments = row_binlog_rowset->num_segments();
- row_binlog_rowset->merge_rowset_meta(*transient_row_binlog->rowset_meta());
- new_segments = row_binlog_rowset->num_segments();
+ if (build_changelog) {
+ DCHECK(changelog_rowset != nullptr);
+ old_segments = changelog_rowset->num_segments();
+ changelog_rowset->merge_rowset_meta(*transient_changelog->rowset_meta());
+ new_segments = changelog_rowset->num_segments();
ss << ", " << txn_info->partial_update_info->partial_update_mode_str()
- << " flush binlog<row> (old segment num: " << old_segments
+ << " flush changelog (old segment num: " << old_segments
<< ", new segment num: " << new_segments << ")";
- SegmentLoader::instance()->erase_segments(*row_binlog_rowset->rowset_meta());
+ SegmentLoader::instance()->erase_segments(*changelog_rowset->rowset_meta());
}
// update the shared_ptr to new bitmap, which is consistent with current rowset.
diff --git a/be/src/storage/tablet/base_tablet.h b/be/src/storage/tablet/base_tablet.h
index 29474f91271..e570fc79944 100644
--- a/be/src/storage/tablet/base_tablet.h
+++ b/be/src/storage/tablet/base_tablet.h
@@ -95,10 +95,10 @@ public:
bool need_read_delete_bitmap() const {
return _tablet_meta->enable_unique_key_merge_on_write() ||
- _tablet_meta->is_row_binlog_tablet();
+ _tablet_meta->is_changelog_tablet();
}
- bool is_row_binlog_tablet() const { return _tablet_meta->is_row_binlog_tablet(); }
+ bool is_changelog_tablet() const { return _tablet_meta->is_changelog_tablet(); }
// Property encapsulated in TabletMeta
const TabletMetaSharedPtr& tablet_meta() const { return _tablet_meta; }
@@ -165,7 +165,7 @@ public:
void generate_tablet_meta_copy_unlocked(TabletMeta& new_tablet_meta,
bool cloud_get_rowset_meta) const;
- bool scan_expired_row_binlog_rowsets();
+ bool scan_expired_changelog_rowsets();
virtual int64_t max_version_unlocked() const { return _tablet_meta->max_version().second; }
@@ -372,7 +372,7 @@ public:
protected:
// Protected by the tablet header lock; discovery resumes after this version.
- Version _row_binlog_ttl_scan_cursor {-1, -1};
+ Version _changelog_ttl_scan_cursor {-1, -1};
// Find the missed versions until the spec_version.
//
diff --git a/be/src/storage/tablet/tablet.cpp b/be/src/storage/tablet/tablet.cpp
index 05b3ebf6c5e..78414b8a138 100644
--- a/be/src/storage/tablet/tablet.cpp
+++ b/be/src/storage/tablet/tablet.cpp
@@ -80,9 +80,10 @@
#include "load/memtable/memtable.h"
#include "service/point_query_executor.h"
#include "storage/binlog.h"
+#include "storage/changelog.h"
#include "storage/compaction/base_compaction.h"
#include "storage/compaction/cumulative_compaction.h"
-#include "storage/compaction/cumulative_compaction_binlog_policy.h"
+#include "storage/compaction/cumulative_compaction_changelog_policy.h"
#include "storage/compaction/cumulative_compaction_policy.h"
#include "storage/compaction/cumulative_compaction_time_series_policy.h"
#include "storage/compaction/full_compaction.h"
@@ -1009,7 +1010,7 @@ Versions Tablet::calc_missed_versions(int64_t spec_version, Versions existing_ve
}
bool Tablet::can_do_compaction(size_t path_hash, CompactionType compaction_type) {
- if (is_row_binlog_tablet() && compaction_type != CompactionType::CUMU_BINLOG_COMPACTION) {
+ if (is_changelog_tablet() && compaction_type != CompactionType::CHANGELOG_COMPACTION) {
return false;
}
@@ -1019,7 +1020,7 @@ bool Tablet::can_do_compaction(size_t path_hash, CompactionType compaction_type)
return false;
}
- if (compaction_type == CompactionType::CUMU_BINLOG_COMPACTION && !is_row_binlog_tablet()) {
+ if (compaction_type == CompactionType::CHANGELOG_COMPACTION && !is_changelog_tablet()) {
return false;
}
@@ -1033,7 +1034,7 @@ bool Tablet::can_do_compaction(size_t path_hash, CompactionType compaction_type)
}
uint32_t Tablet::calc_compaction_score(CompactionType compaction_type) {
- if (compaction_type == CompactionType::CUMU_BINLOG_COMPACTION) {
+ if (compaction_type == CompactionType::CHANGELOG_COMPACTION) {
std::shared_lock rdlock(_meta_lock);
return _calc_cumulative_compaction_score(_cumulative_compaction_policy);
}
@@ -1067,7 +1068,7 @@ bool Tablet::suitable_for_compaction(
std::shared_ptr<CumulativeCompactionPolicy> cumulative_compaction_policy) {
#ifndef BE_TEST
if ((compaction_type == CompactionType::CUMULATIVE_COMPACTION ||
- compaction_type == CompactionType::CUMU_BINLOG_COMPACTION) &&
+ compaction_type == CompactionType::CHANGELOG_COMPACTION) &&
cumulative_compaction_policy != nullptr) {
std::lock_guard wrlock(_meta_lock);
if (_cumulative_compaction_policy == nullptr ||
@@ -1081,7 +1082,7 @@ bool Tablet::suitable_for_compaction(
std::shared_lock rdlock(_meta_lock);
int32_t score = -1;
if (compaction_type == CompactionType::CUMULATIVE_COMPACTION ||
- compaction_type == CompactionType::CUMU_BINLOG_COMPACTION) {
+ compaction_type == CompactionType::CHANGELOG_COMPACTION) {
score = _calc_cumulative_compaction_score(cumulative_compaction_policy);
} else if (compaction_type == CompactionType::BASE_COMPACTION) {
DCHECK_EQ(compaction_type, CompactionType::BASE_COMPACTION);
@@ -1256,16 +1257,16 @@ Status Tablet::_contains_version(const Version& version) {
}
std::vector<RowsetSharedPtr> Tablet::pick_candidate_rowsets_to_cumulative_compaction(
- std::optional<int64_t> row_binlog_ttl_cutoff_tso) {
- if (is_row_binlog_tablet()) {
- return pick_candidate_rowsets_to_binlog_compaction(row_binlog_ttl_cutoff_tso);
+ std::optional<int64_t> changelog_ttl_cutoff_tso) {
+ if (is_changelog_tablet()) {
+ return pick_candidate_rowsets_to_changelog_compaction(changelog_ttl_cutoff_tso);
}
std::shared_lock rlock(_meta_lock);
return pick_candidate_rowsets_to_cumulative_compaction_unlocked();
}
std::vector<RowsetSharedPtr> Tablet::pick_candidate_rowsets_to_cumulative_compaction_unlocked() {
- DORIS_CHECK(!is_row_binlog_tablet());
+ DORIS_CHECK(!is_changelog_tablet());
std::vector<RowsetSharedPtr> candidate_rowsets;
if (_cumulative_point == K_INVALID_CUMULATIVE_POINT) {
@@ -1331,8 +1332,8 @@ std::vector<RowsetSharedPtr> Tablet::pick_candidate_rowsets_to_full_compaction()
return candidate_rowsets;
}
-std::vector<RowsetSharedPtr> Tablet::pick_candidate_rowsets_to_binlog_compaction(
- std::optional<int64_t> row_binlog_ttl_cutoff_tso) {
+std::vector<RowsetSharedPtr> Tablet::pick_candidate_rowsets_to_changelog_compaction(
+ std::optional<int64_t> changelog_ttl_cutoff_tso) {
auto [visible_version, _] = get_visible_version_and_time();
int64_t now = UnixSeconds();
std::vector<RowsetSharedPtr> candidate_rowsets;
@@ -1341,15 +1342,14 @@ std::vector<RowsetSharedPtr> Tablet::pick_candidate_rowsets_to_binlog_compaction
{
std::shared_lock rlock(_meta_lock);
bool filter_new_rowset =
- _rs_version_map.size() <= config::binlog_compaction_file_count_threshold;
+ _rs_version_map.size() <= config::changelog_compaction_file_count_threshold;
for (const auto& [version, rs] : _rs_version_map) {
max_version = std::max(max_version, version.second);
- bool expired =
- row_binlog_ttl_cutoff_tso.has_value() &&
- row_binlog_rowset_expired(*rs->rowset_meta(), *row_binlog_ttl_cutoff_tso);
+ bool expired = changelog_ttl_cutoff_tso.has_value() &&
+ changelog_rowset_expired(*rs->rowset_meta(), *changelog_ttl_cutoff_tso);
if (!expired && filter_new_rowset && rs->rowset_meta()->is_singleton_delta() &&
rs->rowset_meta()->newest_write_timestamp() +
- config::binlog_compaction_wait_timesec_after_visible >
+ config::changelog_compaction_wait_timesec_after_visible >
now) {
continue;
}
@@ -1885,7 +1885,7 @@ Status Tablet::prepare_compaction_and_calculate_permits(
// And because we set permits to 0, so even if we return OK here, nothing will be done.
return Status::OK();
}
- } else if (compaction_type == CompactionType::CUMU_BINLOG_COMPACTION) {
+ } else if (compaction_type == CompactionType::CHANGELOG_COMPACTION) {
compaction = std::make_shared<CumulativeCompaction>(tablet->_engine, tablet);
Status res = compaction->prepare_compact();
@@ -1893,7 +1893,7 @@ Status Tablet::prepare_compaction_and_calculate_permits(
tablet->set_last_cumu_compaction_failure_time(UnixMillis());
permits = 0;
if (!res.is<CUMULATIVE_NO_SUITABLE_VERSION>()) {
- return Status::InternalError("prepare binlog compaction with err: {}",
+ return Status::InternalError("prepare changelog compaction with err: {}",
res.to_string());
}
// return OK if OLAP_ERR_BE_NO_SUITABLE_VERSION, so that we don't need to
@@ -2034,8 +2034,8 @@ Result<std::unique_ptr<RowsetWriter>> Tablet::create_transient_rowset_writer(
context.write_type = DataWriteType::TYPE_DIRECT;
context.partial_update_info = std::move(partial_update_info);
context.is_transient_rowset_writer = true;
- if (rowset.rowset_meta() != nullptr && rowset.rowset_meta()->is_row_binlog()) {
- context.write_binlog_opt().enable = true;
+ if (rowset.rowset_meta() != nullptr && rowset.rowset_meta()->is_changelog()) {
+ context.write_changelog_opt().enable = true;
}
return create_transient_rowset_writer(context, rowset.rowset_id())
.transform([&](auto&& writer) {
@@ -2080,8 +2080,8 @@ void Tablet::_init_context_common_fields(RowsetWriterContext& context) {
context.encrypt_algorithm = tablet_meta()->encryption_algorithm();
- if (context.write_binlog_opt().enable) {
- context.write_binlog_opt().set_need_before(
+ if (context.write_changelog_opt().enable) {
+ context.write_changelog_opt().set_need_before(
tablet_meta()->binlog_config().need_historical_value());
}
}
@@ -2636,17 +2636,17 @@ Status Tablet::save_delete_bitmap(const TabletTxnInfo* txn_info, int64_t txn_id,
RowsetSharedPtr rowset = txn_info->rowset;
int64_t cur_version = rowset->start_version();
- // For binlog<Row> publish, sync current rowset delete bitmap deltas to binlog tablet.
- // so row binlog reads can skip rows deleted by MOW conflict resolution.
- const bool build_row_binlog = txn_info->attach_row_binlog.rowset != nullptr;
+ // For changelog publish, sync current rowset delete bitmap deltas to changelog tablet.
+ // so changelog reads can skip rows deleted by MOW conflict resolution.
+ const bool build_changelog = txn_info->attach_changelog.rowset != nullptr;
const RowsetId cur_build_rid = txn_info->rowset->rowset_id();
- const RowsetId binlog_rid =
- build_row_binlog ? txn_info->attach_row_binlog.rowset->rowset_id() : cur_build_rid;
- auto* row_binlog_delete_bitmap = txn_info->attach_row_binlog.delete_bitmap.get();
- if (build_row_binlog) {
- DCHECK(txn_info->attach_row_binlog.rowset->rowset_meta() != nullptr);
- DCHECK(txn_info->attach_row_binlog.rowset->rowset_meta()->is_row_binlog());
- DCHECK(row_binlog_delete_bitmap != nullptr);
+ const RowsetId changelog_rid =
+ build_changelog ? txn_info->attach_changelog.rowset->rowset_id() : cur_build_rid;
+ auto* changelog_delete_bitmap = txn_info->attach_changelog.delete_bitmap.get();
+ if (build_changelog) {
+ DCHECK(txn_info->attach_changelog.rowset->rowset_meta() != nullptr);
+ DCHECK(txn_info->attach_changelog.rowset->rowset_meta()->is_changelog());
+ DCHECK(changelog_delete_bitmap != nullptr);
}
// update version without write lock, compaction and publish_txn
@@ -2658,12 +2658,12 @@ Status Tablet::save_delete_bitmap(const TabletTxnInfo* txn_info, int64_t txn_id,
_tablet_meta->delete_bitmap().merge({std::get<0>(key), std::get<1>(key), cur_version},
bitmap);
- if (build_row_binlog && std::get<0>(key) == cur_build_rid) {
+ if (build_changelog && std::get<0>(key) == cur_build_rid) {
const DeleteBitmap::SegmentId& sid = std::get<1>(key);
- // Merge current delete bitmap deltas to binlog tablet because publish-phase
- // partial-update and seq replace will influence the binlog rowset.
- row_binlog_delete_bitmap->merge({binlog_rid, sid, cast_set<uint64_t>(cur_version)},
- bitmap);
+ // Merge current delete bitmap deltas to changelog tablet because publish-phase
+ // partial-update and seq replace will influence the changelog rowset.
+ changelog_delete_bitmap->merge(
+ {changelog_rid, sid, cast_set<uint64_t>(cur_version)}, bitmap);
}
}
}
@@ -2691,7 +2691,7 @@ void Tablet::set_skip_compaction(bool skip, CompactionType compaction_type, int6
if (!skip) {
_skip_cumu_compaction = false;
_skip_base_compaction = false;
- _skip_binlog_compaction = false;
+ _skip_changelog_compaction = false;
return;
}
if (compaction_type == CompactionType::CUMULATIVE_COMPACTION) {
@@ -2701,9 +2701,9 @@ void Tablet::set_skip_compaction(bool skip, CompactionType compaction_type, int6
_skip_base_compaction = true;
_skip_base_compaction_ts = start;
} else {
- DCHECK(compaction_type == CompactionType::CUMU_BINLOG_COMPACTION);
- _skip_binlog_compaction = true;
- _skip_binlog_compaction_ts = start;
+ DCHECK(compaction_type == CompactionType::CHANGELOG_COMPACTION);
+ _skip_changelog_compaction = true;
+ _skip_changelog_compaction_ts = start;
}
}
@@ -2714,9 +2714,9 @@ bool Tablet::should_skip_compaction(CompactionType compaction_type, int64_t now)
} else if (compaction_type == CompactionType::BASE_COMPACTION && _skip_base_compaction &&
now < _skip_base_compaction_ts + config::skip_tablet_compaction_second) {
return true;
- } else if (compaction_type == CompactionType::CUMU_BINLOG_COMPACTION &&
- _skip_binlog_compaction &&
- now < _skip_binlog_compaction_ts + config::skip_tablet_compaction_second) {
+ } else if (compaction_type == CompactionType::CHANGELOG_COMPACTION &&
+ _skip_changelog_compaction &&
+ now < _skip_changelog_compaction_ts + config::skip_tablet_compaction_second) {
return true;
}
return false;
diff --git a/be/src/storage/tablet/tablet.h b/be/src/storage/tablet/tablet.h
index 012ee6f731c..ea63a8378f5 100644
--- a/be/src/storage/tablet/tablet.h
+++ b/be/src/storage/tablet/tablet.h
@@ -39,9 +39,9 @@
#include "common/metrics/metrics.h"
#include "common/status.h"
#include "gen_cpp/AgentService_types.h"
-#include "storage/binlog.h"
#include "storage/binlog_config.h"
-#include "storage/compaction/cumulative_compaction_binlog_policy.h"
+#include "storage/changelog.h"
+#include "storage/compaction/cumulative_compaction_changelog_policy.h"
#include "storage/data_dir.h"
#include "storage/olap_common.h"
#include "storage/partial_update_info.h"
@@ -282,15 +282,15 @@ public:
void check_tablet_path_exists();
std::vector<RowsetSharedPtr> pick_candidate_rowsets_to_cumulative_compaction(
- std::optional<int64_t> row_binlog_ttl_cutoff_tso = std::nullopt);
+ std::optional<int64_t> changelog_ttl_cutoff_tso = std::nullopt);
// MUST hold shared `_meta_lock`. Use this when the caller already holds the
// header lock (e.g. the time-series cumulative score path under
// suitable_for_compaction) to avoid recursive shared acquisition.
std::vector<RowsetSharedPtr> pick_candidate_rowsets_to_cumulative_compaction_unlocked();
std::vector<RowsetSharedPtr> pick_candidate_rowsets_to_base_compaction();
std::vector<RowsetSharedPtr> pick_candidate_rowsets_to_full_compaction();
- std::vector<RowsetSharedPtr> pick_candidate_rowsets_to_binlog_compaction(
- std::optional<int64_t> row_binlog_ttl_cutoff_tso = std::nullopt);
+ std::vector<RowsetSharedPtr> pick_candidate_rowsets_to_changelog_compaction(
+ std::optional<int64_t> changelog_ttl_cutoff_tso = std::nullopt);
std::vector<RowsetSharedPtr> pick_candidate_rowsets_to_build_inverted_index(
const std::set<int64_t>& alter_index_uids, bool is_drop_op);
@@ -629,8 +629,8 @@ private:
bool _skip_base_compaction = false;
int64_t _skip_base_compaction_ts;
- bool _skip_binlog_compaction = false;
- int64_t _skip_binlog_compaction_ts;
+ bool _skip_changelog_compaction = false;
+ int64_t _skip_changelog_compaction_ts;
// cooldown related
CooldownConf _cooldown_conf;
diff --git a/be/src/storage/tablet/tablet_manager.cpp b/be/src/storage/tablet/tablet_manager.cpp
index 27cc5bcdf9d..dedd0bfe006 100644
--- a/be/src/storage/tablet/tablet_manager.cpp
+++ b/be/src/storage/tablet/tablet_manager.cpp
@@ -228,7 +228,7 @@ Status TabletManager::_add_tablet_to_map_unlocked(TTabletId tablet_id,
tablet->register_tablet_into_dir();
tablet_map_t& tablet_map = _get_tablet_map(tablet_id);
tablet_map[tablet_id] = tablet;
- _engine.register_row_binlog_tablet(tablet);
+ _engine.register_changelog_tablet(tablet);
_add_tablet_to_partition(tablet);
g_tablet_meta_schema_columns_count << tablet->tablet_meta()->tablet_columns_num();
COUNTER_UPDATE(ADD_CHILD_TIMER(profile, "RegisterTabletInfo", "AddTablet"),
@@ -266,9 +266,9 @@ Status TabletManager::create_tablet(const TCreateTabletReq& request, std::vector
std::unique_lock two_tablet_lock(_two_tablet_mtx, std::defer_lock);
bool in_restore_mode = request.__isset.in_restore_mode && request.in_restore_mode;
bool has_base_tablet = request.__isset.base_tablet_id && request.base_tablet_id > 0;
- bool is_colocated_row_binlog = has_base_tablet && request.__isset.tablet_role &&
- request.tablet_role == TTabletRole::TABLET_ROLE_ROW_BINLOG;
- bool is_schema_change_or_atomic_restore = has_base_tablet && !is_colocated_row_binlog;
+ bool is_colocated_changelog = has_base_tablet && request.__isset.tablet_role &&
+ request.tablet_role == TTabletRole::TABLET_ROLE_CHANGELOG;
+ bool is_schema_change_or_atomic_restore = has_base_tablet && !is_colocated_changelog;
bool need_two_lock = has_base_tablet && ((_tablets_shards_mask & request.base_tablet_id) !=
(_tablets_shards_mask & tablet_id));
if (need_two_lock) {
@@ -293,14 +293,14 @@ Status TabletManager::create_tablet(const TCreateTabletReq& request, std::vector
{
SCOPED_TIMER(ADD_TIMER(profile, "GetTabletUnlocked"));
tablet_exists = _get_tablet_unlocked(tablet_id) != nullptr;
- if (tablet_exists && !is_colocated_row_binlog) {
+ if (tablet_exists && !is_colocated_changelog) {
LOG(INFO) << "success to create tablet. tablet already exist. tablet_id=" << tablet_id;
return Status::OK();
}
}
TabletSharedPtr base_tablet = nullptr;
- // base_tablet_id is used by alter/restore and colocated row-binlog creation.
+ // base_tablet_id is used by alter/restore and colocated changelog creation.
if (has_base_tablet) {
// if base_tablet_id's lock diffrent with new_tablet_id, we need lock it.
if (need_two_lock) {
@@ -323,8 +323,8 @@ Status TabletManager::create_tablet(const TCreateTabletReq& request, std::vector
// entering this method
//
// ATTN: Since all restored replicas will be saved to HDD, so no storage_medium check here.
- // Row-binlog tablet must be on the same disk as its base tablet.
- if (in_restore_mode || is_colocated_row_binlog ||
+ // Changelog tablet must be on the same disk as its base tablet.
+ if (in_restore_mode || is_colocated_changelog ||
request.storage_medium == base_tablet->data_dir()->storage_medium()) {
LOG(INFO) << "create tablet use the base tablet data dir. tablet_id=" << tablet_id
<< ", base tablet_id=" << request.base_tablet_id
@@ -334,34 +334,34 @@ Status TabletManager::create_tablet(const TCreateTabletReq& request, std::vector
}
}
- auto persist_row_binlog_pair = [&]() {
- CHECK(is_colocated_row_binlog);
+ auto persist_changelog_pair = [&]() {
+ CHECK(is_colocated_changelog);
std::lock_guard base_tablet_wlock(base_tablet->get_header_lock());
- CHECK(base_tablet->tablet_meta()->binlog_tablet_id() == 0 ||
- base_tablet->tablet_meta()->binlog_tablet_id() == tablet_id)
+ CHECK(base_tablet->tablet_meta()->changelog_tablet_id() == 0 ||
+ base_tablet->tablet_meta()->changelog_tablet_id() == tablet_id)
<< "base tablet " << base_tablet->tablet_id()
- << " is already paired with row-binlog tablet "
- << base_tablet->tablet_meta()->binlog_tablet_id() << ", new row-binlog tablet "
+ << " is already paired with changelog tablet "
+ << base_tablet->tablet_meta()->changelog_tablet_id() << ", new changelog tablet "
<< tablet_id;
- base_tablet->tablet_meta()->set_binlog_tablet_id(tablet_id);
+ base_tablet->tablet_meta()->set_changelog_tablet_id(tablet_id);
base_tablet->save_meta();
};
if (tablet_exists) {
- persist_row_binlog_pair();
+ persist_changelog_pair();
LOG(INFO) << "success to create tablet. tablet already exist. tablet_id=" << tablet_id;
return Status::OK();
}
TabletSharedPtr tablet = _internal_create_tablet_unlocked(
- request, is_schema_change_or_atomic_restore, is_colocated_row_binlog, base_tablet.get(),
+ request, is_schema_change_or_atomic_restore, is_colocated_changelog, base_tablet.get(),
stores, profile);
if (tablet == nullptr) {
DorisMetrics::instance()->create_tablet_requests_failed->increment(1);
return Status::Error<CE_CMD_PARAMS_ERROR>("fail to create tablet. tablet_id={}",
request.tablet_id);
}
- if (is_colocated_row_binlog) {
- persist_row_binlog_pair();
+ if (is_colocated_changelog) {
+ persist_changelog_pair();
}
LOG(INFO) << "success to create tablet. tablet_id=" << tablet_id
@@ -371,10 +371,10 @@ Status TabletManager::create_tablet(const TCreateTabletReq& request, std::vector
TabletSharedPtr TabletManager::_internal_create_tablet_unlocked(
const TCreateTabletReq& request, const bool is_schema_change,
- const bool is_colocated_row_binlog, const Tablet* base_tablet,
+ const bool is_colocated_changelog, const Tablet* base_tablet,
const std::vector<DataDir*>& data_dirs, RuntimeProfile* profile) {
DCHECK((is_schema_change && base_tablet) ||
- (!is_schema_change && (base_tablet == nullptr || is_colocated_row_binlog)));
+ (!is_schema_change && (base_tablet == nullptr || is_colocated_changelog)));
// NOTE: The existence of tablet_id and schema_hash has already been checked,
// no need check again here.
@@ -765,8 +765,8 @@ std::vector<TabletCompactionContext> TabletManager::find_best_tablets_to_compact
score_stats->scanned = true;
int64_t now_ms = UnixMillis();
const string& compaction_type_str = compaction_type == CompactionType::BASE_COMPACTION ? "base"
- : compaction_type == CompactionType::CUMU_BINLOG_COMPACTION
- ? "binlog"
+ : compaction_type == CompactionType::CHANGELOG_COMPACTION
+ ? "changelog"
: "cumulative";
uint32_t highest_score = 0;
TabletScore best_tablet_context;
@@ -825,7 +825,9 @@ std::vector<TabletCompactionContext> TabletManager::find_best_tablets_to_compact
}
const auto& compaction_policy = tablet_ptr->tablet_meta()->compaction_policy();
auto cumulative_compaction_policy =
- all_cumulative_compaction_policies.at(compaction_policy);
+ CumulativeCompactionPolicyFactory::get_cumulative_compaction_policy(
+ all_cumulative_compaction_policies, compaction_policy,
+ tablet_ptr->tablet_id());
uint32_t current_compaction_score = tablet_ptr->calc_compaction_score(compaction_type);
if (current_compaction_score < 5) {
tablet_ptr->set_skip_compaction(true, compaction_type, UnixSeconds());
diff --git a/be/src/storage/tablet/tablet_manager.h b/be/src/storage/tablet/tablet_manager.h
index d1c6097cb42..085ef4c7366 100644
--- a/be/src/storage/tablet/tablet_manager.h
+++ b/be/src/storage/tablet/tablet_manager.h
@@ -207,7 +207,7 @@ private:
TabletSharedPtr _internal_create_tablet_unlocked(const TCreateTabletReq& request,
const bool is_schema_change,
- const bool is_colocated_row_binlog,
+ const bool is_colocated_changelog,
const Tablet* base_tablet,
const std::vector<DataDir*>& data_dirs,
RuntimeProfile* profile);
diff --git a/be/src/storage/tablet/tablet_meta.cpp b/be/src/storage/tablet/tablet_meta.cpp
index f39f7daca97..450476b4614 100644
--- a/be/src/storage/tablet/tablet_meta.cpp
+++ b/be/src/storage/tablet/tablet_meta.cpp
@@ -41,6 +41,7 @@
#include "common/config.h"
#include "io/fs/file_writer.h"
#include "io/fs/local_file_system.h"
+#include "storage/compaction/cumulative_compaction_changelog_policy.h"
#include "storage/data_dir.h"
#include "storage/file_header.h"
#include "storage/olap_common.h"
@@ -195,8 +196,8 @@ TabletMeta::TabletMeta(int64_t table_id, int64_t partition_id, int64_t tablet_id
time_series_compaction_level_threshold);
tablet_meta_pb.set_vertical_compaction_num_columns_per_group(
vertical_compaction_num_columns_per_group);
- tablet_meta_pb.set_tablet_role(tablet_role == TTabletRole::TABLET_ROLE_ROW_BINLOG
- ? TabletRolePB::TABLET_ROLE_ROW_BINLOG
+ tablet_meta_pb.set_tablet_role(tablet_role == TTabletRole::TABLET_ROLE_CHANGELOG
+ ? TabletRolePB::TABLET_ROLE_CHANGELOG
: TabletRolePB::TABLET_ROLE_DATA);
SchemaCreateOptions schema_create_options_for_data = {
.col_ordinal_to_unique_id = col_ordinal_to_unique_id,
@@ -270,7 +271,7 @@ TabletMeta::TabletMeta(const TabletMeta& b)
_delete_bitmap(b._delete_bitmap),
_binlog_config(b._binlog_config),
_tablet_role(b._tablet_role),
- _binlog_tablet_id(b._binlog_tablet_id),
+ _changelog_tablet_id(b._changelog_tablet_id),
_compaction_policy(b._compaction_policy),
_time_series_compaction_goal_size_mbytes(b._time_series_compaction_goal_size_mbytes),
_time_series_compaction_file_count_threshold(
@@ -920,8 +921,16 @@ void TabletMeta::init_from_pb(const TabletMetaPB& tablet_meta_pb) {
_binlog_config = tablet_meta_pb.binlog_config();
}
_tablet_role = tablet_meta_pb.tablet_role();
- _binlog_tablet_id = tablet_meta_pb.binlog_tablet_id();
- _compaction_policy = tablet_meta_pb.compaction_policy();
+ _changelog_tablet_id = tablet_meta_pb.changelog_tablet_id();
+ // The policy of a changelog tablet is chosen by its role, which is persisted as a number,
+ // not by the persisted policy string, which can name a policy this build does not know.
+ // Example: a changelog tablet whose meta was written with an older policy name. Looked
+ // up by that name it would get no policy in local mode (the lookup used to abort the BE)
+ // and the size-based policy in cloud mode, which breaks its level and quick-merge rules.
+ // Every way a meta is built (create, load, clone, cloud sync) comes through here, and a
+ // changelog tablet refuses later policy changes, so the field cannot drift afterwards.
+ _compaction_policy = is_changelog_tablet() ? std::string(CUMULATIVE_CHANGELOG_POLICY)
+ : tablet_meta_pb.compaction_policy();
_time_series_compaction_goal_size_mbytes =
tablet_meta_pb.time_series_compaction_goal_size_mbytes();
_time_series_compaction_file_count_threshold =
@@ -1025,8 +1034,8 @@ void TabletMeta::to_meta_pb(TabletMetaPB* tablet_meta_pb, bool cloud_get_rowset_
}
_binlog_config.to_pb(tablet_meta_pb->mutable_binlog_config());
tablet_meta_pb->set_tablet_role(_tablet_role);
- if (_binlog_tablet_id > 0) {
- tablet_meta_pb->set_binlog_tablet_id(_binlog_tablet_id);
+ if (_changelog_tablet_id > 0) {
+ tablet_meta_pb->set_changelog_tablet_id(_changelog_tablet_id);
}
tablet_meta_pb->set_compaction_policy(compaction_policy());
tablet_meta_pb->set_time_series_compaction_goal_size_mbytes(
@@ -1275,7 +1284,7 @@ bool operator==(const TabletMeta& a, const TabletMeta& b) {
if (a._in_restore_mode != b._in_restore_mode) return false;
if (a._preferred_rowset_type != b._preferred_rowset_type) return false;
if (a._storage_policy_id != b._storage_policy_id) return false;
- if (a._binlog_tablet_id != b._binlog_tablet_id) return false;
+ if (a._changelog_tablet_id != b._changelog_tablet_id) return false;
if (a._compaction_policy != b._compaction_policy) return false;
if (a._time_series_compaction_goal_size_mbytes != b._time_series_compaction_goal_size_mbytes)
return false;
diff --git a/be/src/storage/tablet/tablet_meta.h b/be/src/storage/tablet/tablet_meta.h
index 74c59980ffc..2865eda72f1 100644
--- a/be/src/storage/tablet/tablet_meta.h
+++ b/be/src/storage/tablet/tablet_meta.h
@@ -282,12 +282,12 @@ public:
_binlog_config = std::move(binlog_config);
}
TabletRolePB tablet_role() const { return _tablet_role; }
- bool is_row_binlog_tablet() const {
- return _tablet_role == TabletRolePB::TABLET_ROLE_ROW_BINLOG;
- }
+ bool is_changelog_tablet() const { return _tablet_role == TabletRolePB::TABLET_ROLE_CHANGELOG; }
void set_tablet_role(TabletRolePB tablet_role) { _tablet_role = tablet_role; }
- int64_t binlog_tablet_id() const { return _binlog_tablet_id; }
- void set_binlog_tablet_id(int64_t binlog_tablet_id) { _binlog_tablet_id = binlog_tablet_id; }
+ int64_t changelog_tablet_id() const { return _changelog_tablet_id; }
+ void set_changelog_tablet_id(int64_t changelog_tablet_id) {
+ _changelog_tablet_id = changelog_tablet_id;
+ }
void set_compaction_policy(std::string compaction_policy) {
_compaction_policy = compaction_policy;
@@ -416,7 +416,7 @@ private:
// binlog config
BinlogConfig _binlog_config {};
TabletRolePB _tablet_role = TabletRolePB::TABLET_ROLE_DATA;
- int64_t _binlog_tablet_id = 0;
+ int64_t _changelog_tablet_id = 0;
// meta for compaction
std::string _compaction_policy;
diff --git a/be/src/storage/tablet/tablet_reader.cpp b/be/src/storage/tablet/tablet_reader.cpp
index 1b78fe010e6..e26c87f2a73 100644
--- a/be/src/storage/tablet/tablet_reader.cpp
+++ b/be/src/storage/tablet/tablet_reader.cpp
@@ -48,6 +48,7 @@
#include "storage/tablet/tablet.h"
#include "storage/tablet/tablet_meta.h"
#include "storage/tablet/tablet_schema.h"
+#include "storage/utils.h"
namespace doris {
using namespace ErrorCode;
@@ -157,14 +158,14 @@ Status TabletReader::_capture_rs_readers(const ReaderParams& read_params) {
}
_reader_context.reader_type = read_params.reader_type;
- _reader_context.read_row_binlog = read_params.read_row_binlog;
+ _reader_context.read_changelog = read_params.read_changelog;
_reader_context.version = read_params.version;
_reader_context.variant_compaction_paths = read_params.variant_compaction_paths;
_reader_context.need_ordered_result = need_ordered_result || read_params.force_key_ordered_read;
_reader_context.topn_filter_source_node_ids = read_params.topn_filter_source_node_ids;
_reader_context.read_orderby_key_reverse = read_params.read_orderby_key_reverse;
_reader_context.use_insert_order_when_same =
- read_params.use_insert_order_when_same || read_params.read_row_binlog;
+ read_params.use_insert_order_when_same || read_params.read_changelog;
_reader_context.read_orderby_key_limit = read_params.read_orderby_key_limit;
_reader_context.read_schema = _read_schema;
_reader_context.read_orderby_key_columns =
@@ -224,6 +225,16 @@ Status TabletReader::_init_params(const ReaderParams& read_params) {
_tablet = read_params.tablet;
_tablet_schema = read_params.tablet_schema;
_read_schema = read_params.read_schema;
+ // A changelog read finds the TSO column by name: a singleton rowset stores a NULL placeholder
+ // there, and only a column named CHANGELOG_TSO_COL is given the rowset's commit TSO instead.
+ // Example: a changelog tablet whose hidden columns were named by another build. Its reads
+ // would return NULL TSOs and lose the TSO merge order, and a compaction would write those
+ // NULLs into its output for good, so both fail here.
+ if ((_tablet->is_changelog_tablet() || read_params.read_changelog) &&
+ _tablet_schema->changelog_tso_col_idx() < 0) {
+ return Status::InternalError<false>("changelog tablet {} has no {} column in its schema",
+ _tablet->tablet_id(), CHANGELOG_TSO_COL);
+ }
_reader_context.runtime_state = read_params.runtime_state;
_reader_context.target_cast_type_for_variants = read_params.target_cast_type_for_variants;
diff --git a/be/src/storage/tablet/tablet_reader.h b/be/src/storage/tablet/tablet_reader.h
index 90434934ea2..62159f6572d 100644
--- a/be/src/storage/tablet/tablet_reader.h
+++ b/be/src/storage/tablet/tablet_reader.h
@@ -131,7 +131,7 @@ public:
// Set only by compaction, alongside its extended tablet_schema.
VariantCompactionPathsSPtr variant_compaction_paths = nullptr;
ReaderType reader_type = ReaderType::READER_QUERY;
- bool read_row_binlog = false;
+ bool read_changelog = false;
bool direct_mode = false;
bool aggregation = false;
// for compaction, schema_change, check_sum: we don't use page cache
@@ -187,7 +187,7 @@ public:
// For rows with the same key, use ascending order (small-to-large) for tie-breakers.
// For example, use lower rowset version / segment id first.
bool use_insert_order_when_same = false;
- // Force globally key-ordered reading for row-binlog scans (e.g. so MIN_DELTA can
+ // Force globally key-ordered reading for changelog scans (e.g. so MIN_DELTA can
// group consecutive same-key changes across segments). Overlapping segments use the
// merge iterator; segments proven non-overlapping use an ordered union.
// See BetaRowsetReader::is_merge_iterator() in beta_rowset_reader.h:62.
@@ -226,7 +226,7 @@ public:
// General LIMIT budget forwarded to SegmentIterator. -1 means no limit.
int64_t general_read_limit = -1;
- TBinlogScanType::type binlog_scan_type = TBinlogScanType::NONE;
+ TChangelogScanType::type changelog_scan_type = TChangelogScanType::NONE;
};
TabletReader() = default;
diff --git a/be/src/storage/tablet/tablet_schema.cpp b/be/src/storage/tablet/tablet_schema.cpp
index 5fceaa50594..a3a522c36ef 100644
--- a/be/src/storage/tablet/tablet_schema.cpp
+++ b/be/src/storage/tablet/tablet_schema.cpp
@@ -914,12 +914,12 @@ void TabletSchema::append_column(TabletColumn column, ColumnType col_type) {
_commit_tso_col_idx = _num_columns;
} else if (UNLIKELY(column.name() == ROW_LSN_COL)) {
_row_lsn_col_idx = _num_columns;
- } else if (UNLIKELY(column.name() == BINLOG_TSO_COL)) {
- _binlog_tso_col_idx = _num_columns;
- } else if (UNLIKELY(column.name() == BINLOG_LSN_COL)) {
- _binlog_lsn_col_idx = _num_columns;
- } else if (UNLIKELY(column.name() == BINLOG_OP_COL)) {
- _binlog_op_col_idx = _num_columns;
+ } else if (UNLIKELY(column.name() == CHANGELOG_TSO_COL)) {
+ _changelog_tso_col_idx = _num_columns;
+ } else if (UNLIKELY(column.name() == CHANGELOG_LSN_COL)) {
+ _changelog_lsn_col_idx = _num_columns;
+ } else if (UNLIKELY(column.name() == CHANGELOG_OP_COL)) {
+ _changelog_op_col_idx = _num_columns;
} else if (UNLIKELY(column.name().starts_with(BeConsts::VIRTUAL_COLUMN_PREFIX))) {
_vir_col_idx_to_unique_id[_num_columns] = column.unique_id();
}
@@ -1124,10 +1124,10 @@ void TabletSchema::init_from_pb(const TabletSchemaPB& schema, bool ignore_extrac
_skip_bitmap_col_idx = schema.skip_bitmap_col_idx();
_commit_tso_col_idx = schema.commit_tso_col_idx();
_row_lsn_col_idx = schema.row_lsn_col_idx();
- // The binlog column cids are not persisted; derive them from the column names.
- _binlog_tso_col_idx = field_index(BINLOG_TSO_COL);
- _binlog_lsn_col_idx = field_index(BINLOG_LSN_COL);
- _binlog_op_col_idx = field_index(BINLOG_OP_COL);
+ // The changelog column cids are not persisted; derive them from the column names.
+ _changelog_tso_col_idx = field_index(CHANGELOG_TSO_COL);
+ _changelog_lsn_col_idx = field_index(CHANGELOG_LSN_COL);
+ _changelog_op_col_idx = field_index(CHANGELOG_OP_COL);
_sort_type = schema.sort_type();
_sort_col_num = schema.sort_col_num();
_compression_type = schema.compression_type();
@@ -1232,9 +1232,9 @@ void TabletSchema::shawdow_copy_without_columns(const TabletSchema& tablet_schem
_skip_bitmap_col_idx = -1;
_commit_tso_col_idx = -1;
_row_lsn_col_idx = -1;
- _binlog_tso_col_idx = -1;
- _binlog_lsn_col_idx = -1;
- _binlog_op_col_idx = -1;
+ _changelog_tso_col_idx = -1;
+ _changelog_lsn_col_idx = -1;
+ _changelog_op_col_idx = -1;
}
void TabletSchema::update_index_info_from(const TabletSchema& tablet_schema) {
@@ -1301,9 +1301,9 @@ void TabletSchema::build_current_tablet_schema(int64_t index_id, int32_t version
_skip_bitmap_col_idx = -1;
_commit_tso_col_idx = -1;
_row_lsn_col_idx = -1;
- _binlog_tso_col_idx = -1;
- _binlog_lsn_col_idx = -1;
- _binlog_op_col_idx = -1;
+ _changelog_tso_col_idx = -1;
+ _changelog_lsn_col_idx = -1;
+ _changelog_op_col_idx = -1;
_cluster_key_uids.clear();
for (const auto& i : ori_tablet_schema._cluster_key_uids) {
_cluster_key_uids.push_back(i);
@@ -1333,12 +1333,12 @@ void TabletSchema::build_current_tablet_schema(int64_t index_id, int32_t version
_commit_tso_col_idx = _num_columns;
} else if (UNLIKELY(column->name() == ROW_LSN_COL)) {
_row_lsn_col_idx = _num_columns;
- } else if (UNLIKELY(column->name() == BINLOG_TSO_COL)) {
- _binlog_tso_col_idx = _num_columns;
- } else if (UNLIKELY(column->name() == BINLOG_LSN_COL)) {
- _binlog_lsn_col_idx = _num_columns;
- } else if (UNLIKELY(column->name() == BINLOG_OP_COL)) {
- _binlog_op_col_idx = _num_columns;
+ } else if (UNLIKELY(column->name() == CHANGELOG_TSO_COL)) {
+ _changelog_tso_col_idx = _num_columns;
+ } else if (UNLIKELY(column->name() == CHANGELOG_LSN_COL)) {
+ _changelog_lsn_col_idx = _num_columns;
+ } else if (UNLIKELY(column->name() == CHANGELOG_OP_COL)) {
+ _changelog_op_col_idx = _num_columns;
}
// Reuse TabletColumn object from pool to reduce memory consumption
TabletColumnPtr new_column;
@@ -1815,9 +1815,9 @@ bool operator==(const TabletSchema& a, const TabletSchema& b) {
if (a._skip_bitmap_col_idx != b._skip_bitmap_col_idx) return false;
if (a._commit_tso_col_idx != b._commit_tso_col_idx) return false;
if (a._row_lsn_col_idx != b._row_lsn_col_idx) return false;
- if (a._binlog_tso_col_idx != b._binlog_tso_col_idx) return false;
- if (a._binlog_lsn_col_idx != b._binlog_lsn_col_idx) return false;
- if (a._binlog_op_col_idx != b._binlog_op_col_idx) return false;
+ if (a._changelog_tso_col_idx != b._changelog_tso_col_idx) return false;
+ if (a._changelog_lsn_col_idx != b._changelog_lsn_col_idx) return false;
+ if (a._changelog_op_col_idx != b._changelog_op_col_idx) return false;
if (a._disable_auto_compaction != b._disable_auto_compaction) return false;
if (a._store_row_column != b._store_row_column) return false;
if (a._row_store_page_size != b._row_store_page_size) return false;
diff --git a/be/src/storage/tablet/tablet_schema.h b/be/src/storage/tablet/tablet_schema.h
index b5ccfbe26f4..61f6b05755b 100644
--- a/be/src/storage/tablet/tablet_schema.h
+++ b/be/src/storage/tablet/tablet_schema.h
@@ -523,12 +523,14 @@ public:
int32_t version_col_idx() const { return _version_col_idx; }
bool has_skip_bitmap_col() const { return _skip_bitmap_col_idx != -1; }
int32_t skip_bitmap_col_idx() const { return _skip_bitmap_col_idx; }
- bool is_tso_enabled() const { return _commit_tso_col_idx != -1 || _binlog_tso_col_idx != -1; }
+ bool is_tso_enabled() const {
+ return _commit_tso_col_idx != -1 || _changelog_tso_col_idx != -1;
+ }
int32_t commit_tso_col_idx() const { return _commit_tso_col_idx; }
int32_t row_lsn_col_idx() const { return _row_lsn_col_idx; }
- int32_t binlog_tso_col_idx() const { return _binlog_tso_col_idx; }
- int32_t binlog_lsn_col_idx() const { return _binlog_lsn_col_idx; }
- int32_t binlog_op_col_idx() const { return _binlog_op_col_idx; }
+ int32_t changelog_tso_col_idx() const { return _changelog_tso_col_idx; }
+ int32_t changelog_lsn_col_idx() const { return _changelog_lsn_col_idx; }
+ int32_t changelog_op_col_idx() const { return _changelog_op_col_idx; }
segment_v2::CompressionTypePB compression_type() const { return _compression_type; }
void set_row_store_page_size(long page_size) { _row_store_page_size = page_size; }
long row_store_page_size() const { return _row_store_page_size; }
@@ -801,9 +803,9 @@ private:
int32_t _skip_bitmap_col_idx = -1;
int32_t _commit_tso_col_idx = -1;
int32_t _row_lsn_col_idx = -1;
- int32_t _binlog_tso_col_idx = -1;
- int32_t _binlog_lsn_col_idx = -1;
- int32_t _binlog_op_col_idx = -1;
+ int32_t _changelog_tso_col_idx = -1;
+ int32_t _changelog_lsn_col_idx = -1;
+ int32_t _changelog_op_col_idx = -1;
int32_t _schema_version = -1;
int64_t _table_id = -1;
int64_t _db_id = -1;
diff --git a/be/src/storage/tablet_info.cpp b/be/src/storage/tablet_info.cpp
index 3a4bcac1db7..b617e7ace6b 100644
--- a/be/src/storage/tablet_info.cpp
+++ b/be/src/storage/tablet_info.cpp
@@ -61,8 +61,8 @@
namespace doris {
-const OlapTableIndexSchema* OlapTableSchemaParam::row_binlog_index_schema(int64_t index_id) const {
- for (auto* schema : _row_binlog_index_schemas) {
+const OlapTableIndexSchema* OlapTableSchemaParam::changelog_index_schema(int64_t index_id) const {
+ for (auto* schema : _changelog_index_schemas) {
if (schema->index_id == index_id) {
return schema;
}
@@ -73,8 +73,8 @@ const OlapTableIndexSchema* OlapTableSchemaParam::row_binlog_index_schema(int64_
void OlapTableIndexSchema::to_protobuf(POlapTableIndexSchema* pindex) const {
pindex->set_id(index_id);
pindex->set_schema_hash(schema_hash);
- if (row_binlog_id > 0) {
- pindex->set_row_binlog_id(row_binlog_id);
+ if (changelog_index_id > 0) {
+ pindex->set_changelog_index_id(changelog_index_id);
}
for (auto* slot : slots) {
pindex->add_columns(slot->col_name());
@@ -195,8 +195,8 @@ Status OlapTableSchemaParam::init(const POlapTableSchemaParam& pschema) {
auto* index = _obj_pool.add(new OlapTableIndexSchema());
index->index_id = p_index.id();
index->schema_hash = p_index.schema_hash();
- if (p_index.has_row_binlog_id()) {
- index->row_binlog_id = p_index.row_binlog_id();
+ if (p_index.has_changelog_index_id()) {
+ index->changelog_index_id = p_index.changelog_index_id();
}
for (const auto& pcolumn_desc : p_index.columns_desc()) {
if (_unique_key_update_mode != UniqueKeyUpdateModePB::UPDATE_FIXED_COLUMNS ||
@@ -235,12 +235,12 @@ Status OlapTableSchemaParam::init(const POlapTableSchemaParam& pschema) {
_indexes.emplace_back(index);
}
- for (const auto& p_index : pschema.row_binlog_index_schemas()) {
+ for (const auto& p_index : pschema.changelog_index_schemas()) {
auto* index = _obj_pool.add(new OlapTableIndexSchema());
index->index_id = p_index.id();
index->schema_hash = p_index.schema_hash();
- if (p_index.has_row_binlog_id()) {
- index->row_binlog_id = p_index.row_binlog_id();
+ if (p_index.has_changelog_index_id()) {
+ index->changelog_index_id = p_index.changelog_index_id();
}
for (const auto& pcolumn_desc : p_index.columns_desc()) {
TabletColumn* tc = _obj_pool.add(new TabletColumn());
@@ -252,7 +252,7 @@ Status OlapTableSchemaParam::init(const POlapTableSchemaParam& pschema) {
ti->init_from_pb(pindex_desc);
index->indexes.emplace_back(ti);
}
- _row_binlog_index_schemas.emplace_back(index);
+ _changelog_index_schemas.emplace_back(index);
}
std::sort(_indexes.begin(), _indexes.end(),
@@ -356,8 +356,8 @@ Status OlapTableSchemaParam::init(const TOlapTableSchemaParam& tschema) {
auto* index = _obj_pool.add(new OlapTableIndexSchema());
index->index_id = t_index.id;
index->schema_hash = t_index.schema_hash;
- if (t_index.__isset.row_binlog_id) {
- index->row_binlog_id = t_index.row_binlog_id;
+ if (t_index.__isset.changelog_index_id) {
+ index->changelog_index_id = t_index.changelog_index_id;
}
for (const auto& tcolumn_desc : t_index.columns_desc) {
if (_unique_key_update_mode != UniqueKeyUpdateModePB::UPDATE_FIXED_COLUMNS ||
@@ -411,20 +411,20 @@ Status OlapTableSchemaParam::init(const TOlapTableSchemaParam& tschema) {
_indexes.emplace_back(index);
}
- if (tschema.__isset.row_binlog_index_schemas) {
- for (const auto& t_index : tschema.row_binlog_index_schemas) {
+ if (tschema.__isset.changelog_index_schemas) {
+ for (const auto& t_index : tschema.changelog_index_schemas) {
auto* index = _obj_pool.add(new OlapTableIndexSchema());
index->index_id = t_index.id;
index->schema_hash = t_index.schema_hash;
- if (t_index.__isset.row_binlog_id) {
- index->row_binlog_id = t_index.row_binlog_id;
+ if (t_index.__isset.changelog_index_id) {
+ index->changelog_index_id = t_index.changelog_index_id;
}
for (const auto& tcolumn_desc : t_index.columns_desc) {
TabletColumn* tc = _obj_pool.add(new TabletColumn());
tc->init_from_thrift(tcolumn_desc);
index->columns.emplace_back(tc);
}
- _row_binlog_index_schemas.emplace_back(index);
+ _changelog_index_schemas.emplace_back(index);
}
}
@@ -462,8 +462,8 @@ void OlapTableSchemaParam::to_protobuf(POlapTableSchemaParam* pschema) const {
for (auto* index : _indexes) {
index->to_protobuf(pschema->add_indexes());
}
- for (auto* index : _row_binlog_index_schemas) {
- index->to_protobuf(pschema->add_row_binlog_index_schemas());
+ for (auto* index : _changelog_index_schemas) {
+ index->to_protobuf(pschema->add_changelog_index_schemas());
}
}
diff --git a/be/src/storage/tablet_info.h b/be/src/storage/tablet_info.h
index 1ea346844d8..8bddc147f24 100644
--- a/be/src/storage/tablet_info.h
+++ b/be/src/storage/tablet_info.h
@@ -57,7 +57,7 @@ class TupleDescriptor;
struct OlapTableIndexSchema {
int64_t index_id;
- int64_t row_binlog_id = 0;
+ int64_t changelog_index_id = 0;
std::vector<SlotDescriptor*> slots;
int32_t schema_hash;
std::vector<TabletColumn*> columns;
@@ -81,9 +81,9 @@ public:
TupleDescriptor* tuple_desc() const { return _tuple_desc; }
const std::vector<OlapTableIndexSchema*>& indexes() const { return _indexes; }
- const OlapTableIndexSchema* row_binlog_index_schema(int64_t index_id) const;
- const std::vector<OlapTableIndexSchema*>& row_binlog_index_schemas() const {
- return _row_binlog_index_schemas;
+ const OlapTableIndexSchema* changelog_index_schema(int64_t index_id) const;
+ const std::vector<OlapTableIndexSchema*>& changelog_index_schemas() const {
+ return _changelog_index_schemas;
}
void to_protobuf(POlapTableSchemaParam* pschema) const;
@@ -137,7 +137,7 @@ private:
TupleDescriptor* _tuple_desc = nullptr;
mutable POlapTableSchemaParam* _proto_schema = nullptr;
std::vector<OlapTableIndexSchema*> _indexes;
- std::vector<OlapTableIndexSchema*> _row_binlog_index_schemas;
+ std::vector<OlapTableIndexSchema*> _changelog_index_schemas;
mutable ObjectPool _obj_pool;
UniqueKeyUpdateModePB _unique_key_update_mode {UniqueKeyUpdateModePB::UPSERT};
PartialUpdateNewRowPolicyPB _partial_update_new_row_policy {
@@ -378,7 +378,7 @@ public:
OlapTableLocationParam(const TOlapTableLocationParam& t_param) : _t_param(t_param) {
for (auto& location : _t_param.tablets) {
_tablets.emplace(location.tablet_id, &location);
- _update_base_to_binlog_tablet(location);
+ _update_base_to_changelog_tablet(location);
}
}
@@ -394,15 +394,15 @@ public:
return nullptr;
}
- // used by base tablet on node_id to mark whether to write binlog, 0 means no binlog.
- int64_t get_binlog_tablet_id(int64_t base_tablet_id, int64_t node_id) const {
- auto it = _base_to_binlog_tablet.find({base_tablet_id, node_id});
- if (it != _base_to_binlog_tablet.end()) {
+ // used by base tablet on node_id to mark whether to write changelog, 0 means no changelog.
+ int64_t get_changelog_tablet_id(int64_t base_tablet_id, int64_t node_id) const {
+ auto it = _base_to_changelog_tablet.find({base_tablet_id, node_id});
+ if (it != _base_to_changelog_tablet.end()) {
return it->second;
}
if (config::is_cloud_mode()) {
- auto cloud_it = _base_to_binlog_tablet.lower_bound({base_tablet_id, INT64_MIN});
- if (cloud_it != _base_to_binlog_tablet.end() &&
+ auto cloud_it = _base_to_changelog_tablet.lower_bound({base_tablet_id, INT64_MIN});
+ if (cloud_it != _base_to_changelog_tablet.end() &&
cloud_it->first.first == base_tablet_id) {
return cloud_it->second;
}
@@ -415,27 +415,27 @@ public:
if (_tablets.find(location.tablet_id) == _tablets.end()) {
_tablets[location.tablet_id] = &location;
}
- _update_base_to_binlog_tablet(location);
+ _update_base_to_changelog_tablet(location);
}
}
private:
- void _update_base_to_binlog_tablet(const TTabletLocation& location) {
+ void _update_base_to_changelog_tablet(const TTabletLocation& location) {
if (!location.__isset.base_tablet_id) {
return;
}
- // the binlog tablet is written by its base tablet on the nodes that own both.
+ // the changelog tablet is written by its base tablet on the nodes that own both.
for (int64_t node_id : location.node_ids) {
- _base_to_binlog_tablet.emplace(std::make_pair(location.base_tablet_id, node_id),
- location.tablet_id);
+ _base_to_changelog_tablet.emplace(std::make_pair(location.base_tablet_id, node_id),
+ location.tablet_id);
}
}
TOlapTableLocationParam _t_param;
// [tablet_id, tablet]. tablet has id, also.
std::unordered_map<int64_t, TabletLocation*> _tablets;
- // [(base_tablet_id, node_id), row_binlog_tablet_id]
- std::map<std::pair<int64_t, int64_t>, int64_t> _base_to_binlog_tablet;
+ // [(base_tablet_id, node_id), changelog_tablet_id]
+ std::map<std::pair<int64_t, int64_t>, int64_t> _base_to_changelog_tablet;
};
struct NodeInfo {
diff --git a/be/src/storage/task/engine_publish_version_task.cpp b/be/src/storage/task/engine_publish_version_task.cpp
index aacd8bdc84d..1cea9da30af 100644
--- a/be/src/storage/task/engine_publish_version_task.cpp
+++ b/be/src/storage/task/engine_publish_version_task.cpp
@@ -250,7 +250,7 @@ Status EnginePublishVersionTask::execute() {
}
auto tablet_publish_txn_ptr = std::make_shared<TabletPublishTxnTask>(
- _engine, this, tablet, rowset, txn_info_it->second->attach_row_binlog,
+ _engine, this, tablet, rowset, txn_info_it->second->attach_changelog,
partition_id, transaction_id, version, tablet_info, par_ver_info.commit_tso);
tablet_tasks.push_back(tablet_publish_txn_ptr);
auto submit_st = token->submit_func([=]() { tablet_publish_txn_ptr->handle(); });
@@ -409,13 +409,13 @@ void EnginePublishVersionTask::_calculate_tbl_num_delta_rows(
TabletPublishTxnTask::TabletPublishTxnTask(
StorageEngine& engine, EnginePublishVersionTask* engine_task, TabletSharedPtr tablet,
- RowsetSharedPtr rowset, const RowBinlogTxnInfo& attach_row_binlog, int64_t partition_id,
+ RowsetSharedPtr rowset, const ChangelogTxnInfo& attach_changelog, int64_t partition_id,
int64_t transaction_id, Version version, const TabletInfo& tablet_info, int64_t commit_tso)
: _engine(engine),
_engine_publish_version_task(engine_task),
_tablet(std::move(tablet)),
_rowset(std::move(rowset)),
- _attach_row_binlog(attach_row_binlog),
+ _attach_changelog(attach_changelog),
_partition_id(partition_id),
_transaction_id(transaction_id),
_version(version),
@@ -480,12 +480,12 @@ Status publish_version_and_add_rowset(StorageEngine& engine, int64_t partition_i
return result;
}
- // the row binlog rowset is added to its own independent binlog tablet version chain.
+ // the changelog rowset is added to its own independent changelog tablet version chain.
if (extend_tablet_txn_info_lifetime != nullptr &&
- extend_tablet_txn_info_lifetime->attach_row_binlog.rowset != nullptr) {
- const auto& attach_row_binlog = extend_tablet_txn_info_lifetime->attach_row_binlog;
- auto binlog_tablet = std::static_pointer_cast<Tablet>(attach_row_binlog.tablet);
- result = add_inc_rowset(binlog_tablet, attach_row_binlog.rowset);
+ extend_tablet_txn_info_lifetime->attach_changelog.rowset != nullptr) {
+ const auto& attach_changelog = extend_tablet_txn_info_lifetime->attach_changelog;
+ auto changelog_tablet = std::static_pointer_cast<Tablet>(attach_changelog.tablet);
+ result = add_inc_rowset(changelog_tablet, attach_changelog.rowset);
if (!result.ok() && !result.is<PUSH_VERSION_ALREADY_EXIST>()) {
return result;
}
@@ -511,13 +511,14 @@ static Status try_lock_migration(const TabletSharedPtr& tablet, int64_t transact
void TabletPublishTxnTask::handle() {
SCOPED_ATTACH_TASK(_mem_tracker);
- // the row binlog is published to its own binlog tablet together with the base tablet, acquire
- // both tablets' locks in binlog-first order.
- auto binlog_tablet = std::static_pointer_cast<Tablet>(_attach_row_binlog.tablet);
- std::shared_lock<std::shared_timed_mutex> binlog_migration_rlock;
+ // the changelog is published to its own changelog tablet together with the base tablet, acquire
+ // both tablets' locks in changelog-first order.
+ auto changelog_tablet = std::static_pointer_cast<Tablet>(_attach_changelog.tablet);
+ std::shared_lock<std::shared_timed_mutex> changelog_migration_rlock;
std::shared_lock<std::shared_timed_mutex> migration_rlock;
- if (binlog_tablet != nullptr) {
- if (_result = try_lock_migration(binlog_tablet, _transaction_id, binlog_migration_rlock);
+ if (changelog_tablet != nullptr) {
+ if (_result = try_lock_migration(changelog_tablet, _transaction_id,
+ changelog_migration_rlock);
!_result.ok()) {
return;
}
@@ -526,10 +527,10 @@ void TabletPublishTxnTask::handle() {
return;
}
- std::unique_lock<std::mutex> binlog_rowset_update_lock;
- if (binlog_tablet != nullptr) {
- binlog_rowset_update_lock =
- std::unique_lock<std::mutex>(binlog_tablet->get_rowset_update_lock());
+ std::unique_lock<std::mutex> changelog_rowset_update_lock;
+ if (changelog_tablet != nullptr) {
+ changelog_rowset_update_lock =
+ std::unique_lock<std::mutex>(changelog_tablet->get_rowset_update_lock());
}
std::unique_lock<std::mutex> rowset_update_lock(_tablet->get_rowset_update_lock(),
std::defer_lock);
@@ -574,24 +575,25 @@ void AsyncTabletPublishTask::handle() {
RowsetSharedPtr rowset = iter->second;
Version version(_version, _version);
- // the row binlog is published to its own binlog tablet together with the base tablet, acquire
- // both tablets' locks in binlog-first order.
- auto binlog_tablet =
- std::static_pointer_cast<Tablet>(txn_info_it->second->attach_row_binlog.tablet);
- std::shared_lock<std::shared_timed_mutex> binlog_migration_rlock;
+ // the changelog is published to its own changelog tablet together with the base tablet, acquire
+ // both tablets' locks in changelog-first order.
+ auto changelog_tablet =
+ std::static_pointer_cast<Tablet>(txn_info_it->second->attach_changelog.tablet);
+ std::shared_lock<std::shared_timed_mutex> changelog_migration_rlock;
std::shared_lock<std::shared_timed_mutex> migration_rlock;
- if (binlog_tablet != nullptr) {
- if (!try_lock_migration(binlog_tablet, _transaction_id, binlog_migration_rlock).ok()) {
+ if (changelog_tablet != nullptr) {
+ if (!try_lock_migration(changelog_tablet, _transaction_id, changelog_migration_rlock)
+ .ok()) {
return;
}
}
if (!try_lock_migration(_tablet, _transaction_id, migration_rlock).ok()) {
return;
}
- std::unique_lock<std::mutex> binlog_rowset_update_lock;
- if (binlog_tablet != nullptr) {
- binlog_rowset_update_lock =
- std::unique_lock<std::mutex>(binlog_tablet->get_rowset_update_lock());
+ std::unique_lock<std::mutex> changelog_rowset_update_lock;
+ if (changelog_tablet != nullptr) {
+ changelog_rowset_update_lock =
+ std::unique_lock<std::mutex>(changelog_tablet->get_rowset_update_lock());
}
std::lock_guard<std::mutex> wrlock(_tablet->get_rowset_update_lock());
_stats.schedule_time_us = MonotonicMicros() - _stats.submit_time_us;
diff --git a/be/src/storage/task/engine_publish_version_task.h b/be/src/storage/task/engine_publish_version_task.h
index a83d7196291..afc7b112428 100644
--- a/be/src/storage/task/engine_publish_version_task.h
+++ b/be/src/storage/task/engine_publish_version_task.h
@@ -66,7 +66,7 @@ class TabletPublishTxnTask {
public:
TabletPublishTxnTask(StorageEngine& engine, EnginePublishVersionTask* engine_task,
TabletSharedPtr tablet, RowsetSharedPtr rowset,
- const RowBinlogTxnInfo& attach_row_binlog, int64_t partition_id,
+ const ChangelogTxnInfo& attach_changelog, int64_t partition_id,
int64_t transaction_id, Version version, const TabletInfo& tablet_info,
int64_t commit_tso);
~TabletPublishTxnTask();
@@ -80,8 +80,8 @@ private:
TabletSharedPtr _tablet;
RowsetSharedPtr _rowset;
- // the row binlog published together with the base tablet.
- RowBinlogTxnInfo _attach_row_binlog;
+ // the changelog published together with the base tablet.
+ ChangelogTxnInfo _attach_changelog;
int64_t _partition_id;
int64_t _transaction_id;
Version _version;
diff --git a/be/src/storage/transform/block_transform.cpp b/be/src/storage/transform/block_transform.cpp
index 3520d259f10..5207cd7194e 100644
--- a/be/src/storage/transform/block_transform.cpp
+++ b/be/src/storage/transform/block_transform.cpp
@@ -27,8 +27,8 @@
#include "storage/partial_update_info.h"
#include "storage/rowset/rowset_writer_context.h"
#include "storage/tablet/tablet_schema.h"
+#include "storage/transform/changelog_derive.h"
#include "storage/transform/partial_update_fill.h"
-#include "storage/transform/row_binlog_derive.h"
#include "util/jsonb/serialize.h"
namespace doris::segment_v2 {
@@ -164,20 +164,20 @@ BlockTransformChain build_transform_chain(const RowsetWriterContext& context) {
if (context.write_type == DataWriteType::TYPE_COMPACTION) {
return BlockTransformChain {};
}
- if (context.write_binlog_opt().enable) {
+ if (context.write_changelog_opt().enable) {
if (context.write_type != DataWriteType::TYPE_DIRECT) {
return BlockTransformChain {};
}
- // binlog<row> sub-writer: only the source -> binlog-schema derivation.
+ // changelog sub-writer: only the source -> changelog-schema derivation.
// No parse or row-store stage is needed: FE rejects a VARIANT column on
- // a binlog<row> table, and the hidden row-store column is a hidden
- // non-key column, which the binlog schema drops. Plain (no probe) for
+ // a changelog table, and the hidden row-store column is a hidden
+ // non-key column, which the changelog schema drops. Plain (no probe) for
// DUP and no-BEFORE upserts; MoW (with probe) for partial update or the
// BEFORE image.
- if (binlog_needs_historical_lookup(context)) {
- return BlockTransformChain {{std::make_shared<MowRowBinlogDeriveStage>()}};
+ if (changelog_needs_historical_lookup(context)) {
+ return BlockTransformChain {{std::make_shared<MowChangelogDeriveStage>()}};
}
- return BlockTransformChain {{std::make_shared<PlainRowBinlogDeriveStage>()}};
+ return BlockTransformChain {{std::make_shared<PlainChangelogDeriveStage>()}};
}
std::vector<std::shared_ptr<const BlockTransform>> stages;
stages.push_back(std::make_shared<ValidateStage>());
diff --git a/be/src/storage/transform/block_transform.h b/be/src/storage/transform/block_transform.h
index 7fc0a4f668a..a9665c00770 100644
--- a/be/src/storage/transform/block_transform.h
+++ b/be/src/storage/transform/block_transform.h
@@ -120,8 +120,8 @@ private:
// The single place that decides which transforms a write path gets:
// - compaction: empty (rows are already final)
-// - binlog<row> sub-writer: [PlainRowBinlogDerive] or [MowRowBinlogDerive]
-// for a direct write, empty otherwise (rows are already binlog shaped)
+// - changelog sub-writer: [PlainChangelogDerive] or [MowChangelogDerive]
+// for a direct write, empty otherwise (rows are already changelog shaped)
// - fixed partial update: [Validate, FixedPartialUpdateFill, RowStoreFill]
// - flexible partial update: [Validate, FlexiblePartialUpdateFill, RowStoreFill]
// - direct / schema change / transient flush: [Validate, RowStoreFill]
diff --git a/be/src/storage/transform/row_binlog_derive.cpp b/be/src/storage/transform/changelog_derive.cpp
similarity index 72%
rename from be/src/storage/transform/row_binlog_derive.cpp
rename to be/src/storage/transform/changelog_derive.cpp
index 69dde8a293f..c613be4950b 100644
--- a/be/src/storage/transform/row_binlog_derive.cpp
+++ b/be/src/storage/transform/changelog_derive.cpp
@@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.
-#include "storage/transform/row_binlog_derive.h"
+#include "storage/transform/changelog_derive.h"
#include <algorithm>
#include <optional>
@@ -27,7 +27,7 @@
#include "core/block/column_with_type_and_name.h"
#include "core/column/column_nullable.h"
#include "core/column/column_vector.h"
-#include "storage/binlog.h"
+#include "storage/changelog.h"
#include "storage/iterator/olap_data_convertor.h"
#include "storage/key/row_key_encoder.h"
#include "storage/mow/historical_row_fetcher.h"
@@ -42,21 +42,22 @@ namespace doris::segment_v2 {
namespace {
-// Number of row-binlog system columns: TSO, LSN, op.
-constexpr uint32_t BINLOG_COLNUM = 3;
+// Number of changelog system columns: TSO, LSN, op.
+constexpr uint32_t CHANGELOG_COLNUM = 3;
// Runs the primary-key historical lookup over `block`'s source key (+seq)
// columns, leaving the planned reads and the op for each row in `retriever` for
// building AFTER/BEFORE. `seq_pos` is the seq column's position in the input
// block (-1 if absent). `convertor` must live longer than `retriever`: its
// accessors back the lookup plan.
-Status setup_retriever_and_lookup(TransformExecContext& ctx, const SegmentWriteBinlogOptions& cfg,
+Status setup_retriever_and_lookup(TransformExecContext& ctx,
+ const SegmentWriteChangelogOptions& cfg,
const TabletSchemaSPtr& source_schema, const Block* block,
int32_t seq_pos, const Int8* delete_signs, size_t num_rows,
OlapBlockDataConvertor& convertor,
std::unique_ptr<PrimaryKeyModelRowRetriever>& retriever) {
retriever = std::make_unique<PrimaryKeyModelRowRetriever>();
- // Row binlog lives in its own tablet, so ctx.tablet is that binlog tablet
+ // Changelog lives in its own tablet, so ctx.tablet is that changelog tablet
// while the history to look up sits in the source tablet. Falling back to
// ctx.tablet would silently search a tablet that holds no source rows, so
// the caller has to name the source tablet -- every site that fills
@@ -87,7 +88,7 @@ Status setup_retriever_and_lookup(TransformExecContext& ctx, const SegmentWriteB
// Fills the BEFORE value columns into `out` from the retriever's historical
// reads. No-op when the source schema has no value columns. The retriever is
// not const: reading BEFORE also loads the old delete signs it keeps.
-Status fill_before_columns(Block& out, const TabletSchema& binlog_schema,
+Status fill_before_columns(Block& out, const TabletSchema& changelog_schema,
PrimaryKeyModelRowRetriever* retriever, uint32_t before_col_start,
const std::vector<uint32_t>& value_source_cids, size_t num_rows) {
size_t value_column_num = value_source_cids.size();
@@ -99,7 +100,7 @@ Status fill_before_columns(Block& out, const TabletSchema& binlog_schema,
cid < before_col_start + cast_set<uint32_t>(value_column_num); ++cid) {
before_cids.emplace_back(cid);
}
- Block before_block = binlog_schema.create_storage_block(before_cids);
+ Block before_block = changelog_schema.create_storage_block(before_cids);
DCHECK(retriever != nullptr);
DCHECK_EQ(before_cids.size(), value_source_cids.size());
RETURN_IF_ERROR(retriever->build_before_block(&before_block, value_source_cids, 0, num_rows));
@@ -112,39 +113,39 @@ Status fill_before_columns(Block& out, const TabletSchema& binlog_schema,
// Fills the TSO / LSN / op columns into `out`. TSO is unknown until publish,
// LSN is the per-row auto-inc value, and op describes the row change.
-void fill_binlog_system_columns(Block& out, const TabletSchema& binlog_schema,
- uint32_t binlog_tso_cid, uint32_t binlog_lsn_cid,
- uint32_t binlog_op_cid, const std::vector<int64_t>& lsn_ids,
- const std::vector<int64_t>& operators, size_t num_rows) {
- std::vector<uint32_t> binlog_cids = {binlog_tso_cid, binlog_lsn_cid, binlog_op_cid};
- Block binlog_system_block = binlog_schema.create_storage_block(binlog_cids);
+void fill_changelog_system_columns(Block& out, const TabletSchema& changelog_schema,
+ uint32_t changelog_tso_cid, uint32_t changelog_lsn_cid,
+ uint32_t changelog_op_cid, const std::vector<int64_t>& lsn_ids,
+ const std::vector<int64_t>& operators, size_t num_rows) {
+ std::vector<uint32_t> changelog_cids = {changelog_tso_cid, changelog_lsn_cid, changelog_op_cid};
+ Block changelog_system_block = changelog_schema.create_storage_block(changelog_cids);
{
- auto binlog_system_columns_guard = binlog_system_block.mutate_columns_scoped();
- auto& binlog_system_columns = binlog_system_columns_guard.mutable_columns();
+ auto changelog_system_columns_guard = changelog_system_block.mutate_columns_scoped();
+ auto& changelog_system_columns = changelog_system_columns_guard.mutable_columns();
// The reader replaces this placeholder with the real commit_tso.
- IColumn* tso_col_ptr = binlog_system_columns[0].get();
+ IColumn* tso_col_ptr = changelog_system_columns[0].get();
auto* tso_nullable_column = check_and_get_column<ColumnNullable>(tso_col_ptr);
DCHECK(tso_nullable_column != nullptr);
tso_nullable_column->insert_many_defaults(num_rows);
// Auto-inc LSN keeps row order until the rowset is published.
- IColumn* lsn_col_ptr = binlog_system_columns[1].get();
+ IColumn* lsn_col_ptr = changelog_system_columns[1].get();
for (size_t i = 0; i < num_rows; i++) {
assert_cast<ColumnInt64*>(lsn_col_ptr)->insert_value(lsn_ids.at(i));
}
// wrong op only happens under partial update, it is fixed by the
// delete bitmap at publish
- const FieldType op_col_type = binlog_schema.column(binlog_op_cid).type();
- IColumn* op_col_ptr = binlog_system_columns[2].get();
+ const FieldType op_col_type = changelog_schema.column(changelog_op_cid).type();
+ IColumn* op_col_ptr = changelog_system_columns[2].get();
auto* op_nullable_column = check_and_get_column<ColumnNullable>(op_col_ptr);
IColumn* op_nested_column = op_nullable_column != nullptr
? &op_nullable_column->get_nested_column()
: op_col_ptr;
CHECK(operators.size() >= num_rows) << operators.size() << " vs " << num_rows;
CHECK(op_col_type == FieldType::OLAP_FIELD_TYPE_BIGINT)
- << "row binlog op column type must be BIGINT, actual="
+ << "changelog op column type must be BIGINT, actual="
<< static_cast<int>(op_col_type);
auto* op_int64_column = assert_cast<ColumnInt64*>(op_nested_column);
for (size_t i = 0; i < num_rows; i++) {
@@ -159,28 +160,27 @@ void fill_binlog_system_columns(Block& out, const TabletSchema& binlog_schema,
}
}
size_t col_pos_in_block = 0;
- for (auto cid : binlog_cids) {
+ for (auto cid : changelog_cids) {
out.replace_by_position(cid,
- binlog_system_block.get_by_position(col_pos_in_block++).column);
+ changelog_system_block.get_by_position(col_pos_in_block++).column);
}
}
-// Shared setup step: work out the binlog schema layout and consume this
+// Shared setup step: work out the changelog schema layout and consume this
// segment's LSN range (exactly once).
-Status resolve_binlog_context(TransformExecContext& ctx, const Block* block,
- BinlogDeriveContext* c) {
- c->binlog_schema = ctx.tablet_schema;
- auto& cfg = ctx.rowset_ctx->write_binlog_opt().write_binlog_config();
+Status resolve_changelog_context(TransformExecContext& ctx, const Block* block,
+ ChangelogDeriveContext* c) {
+ c->changelog_schema = ctx.tablet_schema;
+ auto& cfg = ctx.rowset_ctx->write_changelog_opt().write_changelog_config();
c->source_schema = cfg.source.tablet_schema;
if (UNLIKELY(c->source_schema == nullptr)) {
- return Status::InternalError(
- "binlog<row> derive missing source_tablet_schema, tablet_id={}",
- ctx.rowset_ctx->tablet_id);
+ return Status::InternalError("changelog derive missing source_tablet_schema, tablet_id={}",
+ ctx.rowset_ctx->tablet_id);
}
if (UNLIKELY(cfg.source.base_tablet == nullptr &&
- binlog_needs_historical_lookup(*ctx.rowset_ctx))) {
+ changelog_needs_historical_lookup(*ctx.rowset_ctx))) {
return Status::InternalError(
- "binlog<row> derive needs the source tablet to look history up, tablet_id={}",
+ "changelog derive needs the source tablet to look history up, tablet_id={}",
ctx.rowset_ctx->tablet_id);
}
c->num_rows = block->rows();
@@ -189,7 +189,7 @@ Status resolve_binlog_context(TransformExecContext& ctx, const Block* block,
// flush; consume it exactly once.
if (ctx.segment_id < 0) {
return Status::InternalError<false>(
- "binlog<row> blocks must be flushed through flush_single_block");
+ "changelog blocks must be flushed through flush_single_block");
}
c->lsn_ids = ctx.rowset_ctx->get_segment_allocated_lsns(ctx.segment_id);
ctx.rowset_ctx->remove_segment_allocated_lsns(ctx.segment_id);
@@ -198,16 +198,22 @@ Status resolve_binlog_context(TransformExecContext& ctx, const Block* block,
// Preserve the source writer's layout: system columns may be a prefix or
// suffix, and source hidden non-key columns are omitted while hidden keys
// remain part of the normal row image.
- int tso_col_id = c->binlog_schema->binlog_tso_col_idx();
- int lsn_col_id = c->binlog_schema->binlog_lsn_col_idx();
- int op_col_id = c->binlog_schema->binlog_op_col_idx();
- CHECK(tso_col_id >= 0) << "binlog<row> schema missing " << BINLOG_TSO_COL;
- CHECK(lsn_col_id >= 0) << "binlog<row> schema missing " << BINLOG_LSN_COL;
- CHECK(op_col_id >= 0) << "binlog<row> schema missing " << BINLOG_OP_COL;
- c->binlog_tso_cid = cast_set<uint32_t>(tso_col_id);
- c->binlog_lsn_cid = cast_set<uint32_t>(lsn_col_id);
- c->binlog_op_cid = cast_set<uint32_t>(op_col_id);
- c->normal_col_start = tso_col_id == 0 ? BINLOG_COLNUM : 0;
+ int tso_col_id = c->changelog_schema->changelog_tso_col_idx();
+ int lsn_col_id = c->changelog_schema->changelog_lsn_col_idx();
+ int op_col_id = c->changelog_schema->changelog_op_col_idx();
+ // The three positions are derived from the column names, so a schema whose hidden columns
+ // carry other names has none of them. Example: a load into a changelog tablet created by a
+ // build that named them differently. Fail that load instead of aborting the BE.
+ if (tso_col_id < 0 || lsn_col_id < 0 || op_col_id < 0) {
+ return Status::InternalError<false>(
+ "changelog schema misses its hidden columns: {}={}, {}={}, {}={}, tablet_id={}",
+ CHANGELOG_TSO_COL, tso_col_id, CHANGELOG_LSN_COL, lsn_col_id, CHANGELOG_OP_COL,
+ op_col_id, ctx.rowset_ctx->tablet_id);
+ }
+ c->changelog_tso_cid = cast_set<uint32_t>(tso_col_id);
+ c->changelog_lsn_cid = cast_set<uint32_t>(lsn_col_id);
+ c->changelog_op_cid = cast_set<uint32_t>(op_col_id);
+ c->normal_col_start = tso_col_id == 0 ? CHANGELOG_COLNUM : 0;
for (uint32_t cid = 0; cid < c->source_schema->num_columns(); ++cid) {
const auto& column = c->source_schema->column(cid);
if (column.visible() || column.is_key()) {
@@ -226,13 +232,14 @@ Status resolve_binlog_context(TransformExecContext& ctx, const Block* block,
// block whose columns hold different row counts.
const size_t expected_before_columns = c->write_before ? c->value_source_cids.size() : 0;
const size_t expected_columns =
- c->normal_source_cids.size() + expected_before_columns + BINLOG_COLNUM;
- if (c->binlog_schema->num_columns() != expected_columns) {
+ c->normal_source_cids.size() + expected_before_columns + CHANGELOG_COLNUM;
+ if (c->changelog_schema->num_columns() != expected_columns) {
return Status::InternalError<false>(
- "binlog<row> schema does not match the derived block: num_columns={}, expected={} "
+ "changelog schema does not match the derived block: num_columns={}, expected={} "
"(normal={}, before={}, system={}), write_before={}, tablet_id={}",
- c->binlog_schema->num_columns(), expected_columns, c->normal_source_cids.size(),
- expected_before_columns, BINLOG_COLNUM, c->write_before, ctx.rowset_ctx->tablet_id);
+ c->changelog_schema->num_columns(), expected_columns, c->normal_source_cids.size(),
+ expected_before_columns, CHANGELOG_COLNUM, c->write_before,
+ ctx.rowset_ctx->tablet_id);
}
return Status::OK();
}
@@ -252,33 +259,33 @@ std::optional<std::span<const Int8>> read_delete_signs(const Block* block, int32
return std::nullopt;
}
-// Shared final step: build the final binlog block -- AFTER columns (COW from
+// Shared final step: build the final changelog block -- AFTER columns (COW from
// after_src), optional BEFORE columns, the TSO/LSN/op columns -- and swap
// it into `block`.
-Status emit_binlog_block(const BinlogDeriveContext& c, const Block* after_src,
- PrimaryKeyModelRowRetriever* retriever,
- const std::vector<int64_t>* plain_operators, Block* block) {
- Block out = c.binlog_schema->create_storage_block();
+Status emit_changelog_block(const ChangelogDeriveContext& c, const Block* after_src,
+ PrimaryKeyModelRowRetriever* retriever,
+ const std::vector<int64_t>* plain_operators, Block* block) {
+ Block out = c.changelog_schema->create_storage_block();
// every index below is a cid, so the block has to carry every schema column
- if (out.columns() != c.binlog_schema->num_columns()) {
+ if (out.columns() != c.changelog_schema->num_columns()) {
return Status::InternalError<false>(
- "binlog<row> block width {} does not match its schema's {} columns", out.columns(),
- c.binlog_schema->num_columns());
+ "changelog block width {} does not match its schema's {} columns", out.columns(),
+ c.changelog_schema->num_columns());
}
// key + AFTER columns: COW pointers of the source-layout visible columns
for (size_t ordinal = 0; ordinal < c.normal_source_cids.size(); ++ordinal) {
const uint32_t source_cid = c.normal_source_cids[ordinal];
ColumnPtr after_column = after_src->get_by_position(source_cid).column;
- // The binlog schema declares every AFTER value column nullable, so a NOT
+ // The changelog schema declares every AFTER value column nullable, so a NOT
// NULL source column has to be wrapped: the segment writer converts by
- // the binlog schema. Keys keep their own nullability.
+ // the changelog schema. Keys keep their own nullability.
if (!c.source_schema->column(source_cid).is_key()) {
after_column = make_nullable(after_column);
}
out.replace_by_position(c.normal_col_start + ordinal, std::move(after_column));
}
if (c.write_before) {
- RETURN_IF_ERROR(fill_before_columns(out, *c.binlog_schema, retriever, c.before_col_start,
+ RETURN_IF_ERROR(fill_before_columns(out, *c.changelog_schema, retriever, c.before_col_start,
c.value_source_cids, c.num_rows));
}
// An insert whose old row is a tombstone is an append, not an update. Run
@@ -288,16 +295,17 @@ Status emit_binlog_block(const BinlogDeriveContext& c, const Block* after_src,
}
const std::vector<int64_t>& operators =
retriever != nullptr ? retriever->get_operators() : *plain_operators;
- fill_binlog_system_columns(out, *c.binlog_schema, c.binlog_tso_cid, c.binlog_lsn_cid,
- c.binlog_op_cid, *c.lsn_ids, operators, c.num_rows);
+ fill_changelog_system_columns(out, *c.changelog_schema, c.changelog_tso_cid,
+ c.changelog_lsn_cid, c.changelog_op_cid, *c.lsn_ids, operators,
+ c.num_rows);
block->swap(out);
return Status::OK();
}
} // namespace
-bool binlog_needs_historical_lookup(const RowsetWriterContext& context) {
- const auto& cfg = context.write_binlog_opt().write_binlog_config();
+bool changelog_needs_historical_lookup(const RowsetWriterContext& context) {
+ const auto& cfg = context.write_changelog_opt().write_changelog_config();
const bool is_source_direct_write =
cfg.source.source_write_type == DataWriteType::TYPE_DIRECT &&
!cfg.source.is_transient_rowset_writer;
@@ -309,14 +317,14 @@ bool binlog_needs_historical_lookup(const RowsetWriterContext& context) {
return is_partial_update || cfg.write_before;
}
-Status RowBinlogDeriveStage::apply(TransformExecContext& ctx, Block* block) const {
- BinlogDeriveContext c;
- RETURN_IF_ERROR(resolve_binlog_context(ctx, block, &c));
+Status ChangelogDeriveStage::apply(TransformExecContext& ctx, Block* block) const {
+ ChangelogDeriveContext c;
+ RETURN_IF_ERROR(resolve_changelog_context(ctx, block, &c));
return derive(ctx, block, c);
}
-Status PlainRowBinlogDeriveStage::derive(TransformExecContext& /*ctx*/, Block* block,
- const BinlogDeriveContext& c) const {
+Status PlainChangelogDeriveStage::derive(TransformExecContext& /*ctx*/, Block* block,
+ const ChangelogDeriveContext& c) const {
DCHECK(!c.write_before);
// No probe: op is APPEND, or DELETE when the row carries a delete sign. DUP
@@ -327,14 +335,14 @@ Status PlainRowBinlogDeriveStage::derive(TransformExecContext& /*ctx*/, Block* b
operators.reserve(c.num_rows);
for (size_t pos = 0; pos < c.num_rows; ++pos) {
bool have_delete_sign = (delete_signs && (*delete_signs)[pos] != 0);
- operators.emplace_back(have_delete_sign ? ROW_BINLOG_DELETE : ROW_BINLOG_APPEND);
+ operators.emplace_back(have_delete_sign ? CHANGELOG_OP_DELETE : CHANGELOG_OP_APPEND);
}
- return emit_binlog_block(c, /*after_src=*/block, /*retriever=*/nullptr, &operators, block);
+ return emit_changelog_block(c, /*after_src=*/block, /*retriever=*/nullptr, &operators, block);
}
-Status MowRowBinlogDeriveStage::derive(TransformExecContext& ctx, Block* block,
- const BinlogDeriveContext& c) const {
- const auto& cfg = ctx.rowset_ctx->write_binlog_opt().write_binlog_config();
+Status MowChangelogDeriveStage::derive(TransformExecContext& ctx, Block* block,
+ const ChangelogDeriveContext& c) const {
+ const auto& cfg = ctx.rowset_ctx->write_changelog_opt().write_changelog_config();
const bool is_source_direct_write =
cfg.source.source_write_type == DataWriteType::TYPE_DIRECT &&
@@ -359,12 +367,12 @@ Status MowRowBinlogDeriveStage::derive(TransformExecContext& ctx, Block* block,
RETURN_IF_ERROR(retriever->materialize_flexible_partial_update(
block, cfg.source.mow_context, &effective_lsn_ids));
- BinlogDeriveContext effective_context = c;
+ ChangelogDeriveContext effective_context = c;
effective_context.num_rows = block->rows();
effective_context.lsn_ids =
std::make_shared<const std::vector<int64_t>>(std::move(effective_lsn_ids));
- return emit_binlog_block(effective_context, block, retriever.get(),
- /*plain_operators=*/nullptr, block);
+ return emit_changelog_block(effective_context, block, retriever.get(),
+ /*plain_operators=*/nullptr, block);
}
if (is_fixed_partial_update) {
@@ -419,7 +427,7 @@ Status MowRowBinlogDeriveStage::derive(TransformExecContext& ctx, Block* block,
after_src = &full_block;
}
- return emit_binlog_block(c, after_src, retriever.get(), /*plain_operators=*/nullptr, block);
+ return emit_changelog_block(c, after_src, retriever.get(), /*plain_operators=*/nullptr, block);
}
} // namespace doris::segment_v2
diff --git a/be/src/storage/transform/row_binlog_derive.h b/be/src/storage/transform/changelog_derive.h
similarity index 70%
rename from be/src/storage/transform/row_binlog_derive.h
rename to be/src/storage/transform/changelog_derive.h
index 5e004f23ae0..db12888427f 100644
--- a/be/src/storage/transform/row_binlog_derive.h
+++ b/be/src/storage/transform/changelog_derive.h
@@ -25,27 +25,27 @@ struct RowsetWriterContext;
namespace segment_v2 {
-// The binlog<Row> derive stages rebuild the load block into a full-width block
-// over the binlog schema -- key + AFTER values, optional __BEFORE__* values, and
+// The changelog derive stages rebuild the load block into a full-width block
+// over the changelog schema -- key + AFTER values, optional BEFORE values, and
// the TSO / LSN / op columns -- so the ordinary segment writers can write
// it like any DUP_KEYS block. build_transform_chain picks Plain (no historical
-// probe) or Mow (with probe) via binlog_needs_historical_lookup().
+// probe) or Mow (with probe) via changelog_needs_historical_lookup().
// Whether the flush needs the historical key probe: a direct partial update or
// a requested BEFORE image. Decided per flush.
-bool binlog_needs_historical_lookup(const RowsetWriterContext& context);
+bool changelog_needs_historical_lookup(const RowsetWriterContext& context);
// Context for each flush that the base stage works out once and passes to
-// derive(): the binlog/source schemas, the consumed LSN range, row count, and
+// derive(): the changelog/source schemas, the consumed LSN range, row count, and
// column layout [keys..., AFTER..., (BEFORE...), TSO, LSN, OP].
-struct BinlogDeriveContext {
- TabletSchemaSPtr binlog_schema;
+struct ChangelogDeriveContext {
+ TabletSchemaSPtr changelog_schema;
TabletSchemaSPtr source_schema;
ConstAllocatedLsnVectorSharedPtr lsn_ids;
size_t num_rows = 0;
- uint32_t binlog_tso_cid = 0;
- uint32_t binlog_lsn_cid = 0;
- uint32_t binlog_op_cid = 0;
+ uint32_t changelog_tso_cid = 0;
+ uint32_t changelog_lsn_cid = 0;
+ uint32_t changelog_op_cid = 0;
uint32_t normal_col_start = 0;
uint32_t before_col_start = 0;
std::vector<uint32_t> normal_source_cids;
@@ -55,36 +55,36 @@ struct BinlogDeriveContext {
// Base for the derive stages: apply() runs the setup steps (schema layout + LSN
// hand-off), then calls derive().
-class RowBinlogDeriveStage : public BlockTransform {
+class ChangelogDeriveStage : public BlockTransform {
public:
Status apply(TransformExecContext& ctx, Block* block) const final;
protected:
virtual Status derive(TransformExecContext& ctx, Block* block,
- const BinlogDeriveContext& c) const = 0;
+ const ChangelogDeriveContext& c) const = 0;
};
// No probe: DUP loads and full-row upserts without a BEFORE image. AFTER is the
// source columns as-is; op is APPEND, or DELETE from the row's own delete sign.
-class PlainRowBinlogDeriveStage : public RowBinlogDeriveStage {
+class PlainChangelogDeriveStage : public ChangelogDeriveStage {
public:
- std::string_view name() const override { return "PlainRowBinlogDerive"; }
+ std::string_view name() const override { return "PlainChangelogDerive"; }
protected:
Status derive(TransformExecContext& ctx, Block* block,
- const BinlogDeriveContext& c) const override;
+ const ChangelogDeriveContext& c) const override;
};
// With a probe: fixed/flexible partial update (rebuild AFTER from history)
// and/or the BEFORE image. It repeats the data chain's key probe on the same
// source block, but never marks the delete bitmap.
-class MowRowBinlogDeriveStage : public RowBinlogDeriveStage {
+class MowChangelogDeriveStage : public ChangelogDeriveStage {
public:
- std::string_view name() const override { return "MowRowBinlogDerive"; }
+ std::string_view name() const override { return "MowChangelogDerive"; }
protected:
Status derive(TransformExecContext& ctx, Block* block,
- const BinlogDeriveContext& c) const override;
+ const ChangelogDeriveContext& c) const override;
};
} // namespace segment_v2
diff --git a/be/src/storage/txn/txn_manager.cpp b/be/src/storage/txn/txn_manager.cpp
index 1c76ce18d77..345247bd674 100644
--- a/be/src/storage/txn/txn_manager.cpp
+++ b/be/src/storage/txn/txn_manager.cpp
@@ -37,7 +37,7 @@
#include "common/logging.h"
#include "common/status.h"
#include "load/delta_writer/delta_writer.h"
-#include "storage/binlog.h"
+#include "storage/changelog.h"
#include "storage/data_dir.h"
#include "storage/olap_common.h"
#include "storage/partial_update_info.h"
@@ -187,11 +187,11 @@ Status TxnManager::prepare_txn(TPartitionId partition_id, TTransactionId transac
return Status::OK();
}
-Status TxnManager::attach_row_binlog_tablet_to_txn(TPartitionId partition_id,
- TTransactionId transaction_id,
- const TabletInfo& base_tablet_info,
- const BaseTabletSPtr& row_binlog_tablet) {
- DCHECK(row_binlog_tablet != nullptr);
+Status TxnManager::attach_changelog_tablet_to_txn(TPartitionId partition_id,
+ TTransactionId transaction_id,
+ const TabletInfo& base_tablet_info,
+ const BaseTabletSPtr& changelog_tablet) {
+ DCHECK(changelog_tablet != nullptr);
TxnKey key(partition_id, transaction_id);
std::lock_guard<std::shared_mutex> txn_lock(_get_txn_lock(transaction_id));
std::lock_guard<std::shared_mutex> txn_map_lock(_get_txn_map_lock(transaction_id));
@@ -200,32 +200,32 @@ Status TxnManager::attach_row_binlog_tablet_to_txn(TPartitionId partition_id,
auto txn_it = txn_tablet_map.find(key);
if (txn_it == txn_tablet_map.end()) {
return Status::Error<TRANSACTION_NOT_EXIST>(
- "failed to attach row-binlog tablet because transaction does not exist. "
+ "failed to attach changelog tablet because transaction does not exist. "
"partition_id: {}, transaction_id: {}, base tablet: {}",
partition_id, transaction_id, base_tablet_info.to_string());
}
auto base_tablet_it = txn_it->second.find(base_tablet_info);
if (base_tablet_it == txn_it->second.end()) {
return Status::Error<TRANSACTION_NOT_EXIST>(
- "failed to attach row-binlog tablet because base tablet transaction does not "
+ "failed to attach changelog tablet because base tablet transaction does not "
"exist. partition_id: {}, transaction_id: {}, base tablet: {}",
partition_id, transaction_id, base_tablet_info.to_string());
}
- auto& attached_tablet = base_tablet_it->second->attach_row_binlog.tablet;
+ auto& attached_tablet = base_tablet_it->second->attach_changelog.tablet;
if (attached_tablet == nullptr) {
- attached_tablet = row_binlog_tablet;
+ attached_tablet = changelog_tablet;
return Status::OK();
}
- if (attached_tablet->tablet_id() != row_binlog_tablet->tablet_id() ||
- attached_tablet->tablet_uid() != row_binlog_tablet->tablet_uid()) {
+ if (attached_tablet->tablet_id() != changelog_tablet->tablet_id() ||
+ attached_tablet->tablet_uid() != changelog_tablet->tablet_uid()) {
return Status::Error<PUSH_TRANSACTION_ALREADY_EXIST>(
- "base tablet transaction already attaches a different row-binlog tablet. "
- "partition_id: {}, transaction_id: {}, base tablet: {}, existing row-binlog "
- "tablet: {}, new row-binlog tablet: {}",
+ "base tablet transaction already attaches a different changelog tablet. "
+ "partition_id: {}, transaction_id: {}, base tablet: {}, existing changelog "
+ "tablet: {}, new changelog tablet: {}",
partition_id, transaction_id, base_tablet_info.to_string(),
attached_tablet->get_tablet_info().to_string(),
- row_binlog_tablet->get_tablet_info().to_string());
+ changelog_tablet->get_tablet_info().to_string());
}
return Status::OK();
}
@@ -235,10 +235,10 @@ Status TxnManager::commit_txn(TPartitionId partition_id, const Tablet& tablet,
const RowsetSharedPtr& rowset_ptr, PendingRowsetGuard guard,
bool is_recovery,
std::shared_ptr<PartialUpdateInfo> partial_update_info,
- const RowBinlogTxnInfo& attach_row_binlog) {
+ const ChangelogTxnInfo& attach_changelog) {
return commit_txn(tablet.data_dir()->get_meta(), partition_id, transaction_id,
tablet.tablet_id(), tablet.tablet_uid(), load_id, rowset_ptr,
- std::move(guard), is_recovery, partial_update_info, attach_row_binlog);
+ std::move(guard), is_recovery, partial_update_info, attach_changelog);
}
Status TxnManager::publish_txn(TPartitionId partition_id, const TabletSharedPtr& tablet,
@@ -326,7 +326,7 @@ Status TxnManager::commit_txn(OlapMeta* meta, TPartitionId partition_id,
const RowsetSharedPtr& rowset_ptr, PendingRowsetGuard guard,
bool is_recovery,
std::shared_ptr<PartialUpdateInfo> partial_update_info,
- const RowBinlogTxnInfo& attach_row_binlog) {
+ const ChangelogTxnInfo& attach_changelog) {
if (partition_id < 1 || transaction_id < 1 || tablet_id < 1) {
LOG(WARNING) << "invalid commit req "
<< " partition_id=" << partition_id << " transaction_id=" << transaction_id
@@ -421,15 +421,14 @@ Status TxnManager::commit_txn(OlapMeta* meta, TPartitionId partition_id,
// it is under a single txn lock
if (!is_recovery) {
std::optional<BinlogFormatPB> binlog_format;
- std::optional<RowsetMetaPB> attach_row_binlog_rowset_meta;
- if (attach_row_binlog.rowset != nullptr) {
- attach_row_binlog_rowset_meta =
- attach_row_binlog.rowset->rowset_meta()->get_rowset_pb();
- binlog_format = BinlogFormatPB::ROW;
+ std::optional<RowsetMetaPB> attach_changelog_rowset_meta;
+ if (attach_changelog.rowset != nullptr) {
+ attach_changelog_rowset_meta = attach_changelog.rowset->rowset_meta()->get_rowset_pb();
+ binlog_format = BinlogFormatPB::CHANGELOG;
}
Status save_status = RowsetMetaManager::save(meta, tablet_uid, rowset_ptr->rowset_id(),
rowset_ptr->rowset_meta()->get_rowset_pb(),
- binlog_format, attach_row_binlog_rowset_meta);
+ binlog_format, attach_changelog_rowset_meta);
DBUG_EXECUTE_IF("TxnManager.RowsetMetaManager.save_wait", {
if (auto wait = dp->param<int>("duration", 0); wait > 0) {
LOG_WARNING("TxnManager.RowsetMetaManager.save_wait")
@@ -478,18 +477,19 @@ Status TxnManager::commit_txn(OlapMeta* meta, TPartitionId partition_id,
{
std::lock_guard<std::shared_mutex> wrlock(_get_txn_map_lock(transaction_id));
auto load_info = std::make_shared<TabletTxnInfo>(load_id, rowset_ptr);
- load_info->attach_row_binlog = attach_row_binlog;
- // resolve the independent binlog tablet in advance for the later publish phase.
- if (load_info->attach_row_binlog.rowset != nullptr &&
- load_info->attach_row_binlog.tablet == nullptr) {
- int64_t binlog_tablet_id =
- load_info->attach_row_binlog.rowset->rowset_meta()->tablet_id();
- load_info->attach_row_binlog.tablet =
- _engine.tablet_manager()->get_tablet(binlog_tablet_id);
- if (load_info->attach_row_binlog.tablet == nullptr) {
+ load_info->attach_changelog = attach_changelog;
+ // resolve the independent changelog tablet in advance for the later publish phase.
+ if (load_info->attach_changelog.rowset != nullptr &&
+ load_info->attach_changelog.tablet == nullptr) {
+ int64_t changelog_tablet_id =
+ load_info->attach_changelog.rowset->rowset_meta()->tablet_id();
+ load_info->attach_changelog.tablet =
+ _engine.tablet_manager()->get_tablet(changelog_tablet_id);
+ if (load_info->attach_changelog.tablet == nullptr) {
return Status::Error<TABLE_NOT_FOUND>(
- "binlog tablet not found when commit txn, binlog_tablet_id={}, txn_id={}",
- binlog_tablet_id, transaction_id);
+ "changelog tablet not found when commit txn, changelog_tablet_id={}, "
+ "txn_id={}",
+ changelog_tablet_id, transaction_id);
}
}
load_info->pending_rs_guard = std::move(guard);
@@ -508,13 +508,13 @@ Status TxnManager::commit_txn(OlapMeta* meta, TPartitionId partition_id,
}
}
- // For binlog<Row> txn, the binlog delete bitmap is only needed in the publish phase to
- // copy delete bitmap deltas onto the independent binlog tablet.
- if (load_info->attach_row_binlog.rowset != nullptr) {
+ // For changelog txn, the changelog delete bitmap is only needed in the publish phase to
+ // copy delete bitmap deltas onto the independent changelog tablet.
+ if (load_info->attach_changelog.rowset != nullptr) {
TabletSharedPtr t = _engine.tablet_manager()->get_tablet(tablet_id, tablet_uid);
if (t != nullptr && t->enable_unique_key_merge_on_write()) {
- load_info->attach_row_binlog.delete_bitmap.reset(
- new DeleteBitmap(load_info->attach_row_binlog.tablet->tablet_id()));
+ load_info->attach_changelog.delete_bitmap.reset(
+ new DeleteBitmap(load_info->attach_changelog.tablet->tablet_id()));
}
}
load_info->commit();
@@ -574,10 +574,10 @@ Status TxnManager::publish_txn(OlapMeta* meta, TPartitionId partition_id,
"tablet={}, commit_tso={}",
partition_id, transaction_id, tablet_info.to_string(), commit_tso);
}
- const auto& attached_row_binlog = tablet_txn_info->attach_row_binlog.rowset;
+ const auto& attached_changelog = tablet_txn_info->attach_changelog.rowset;
const bool requires_commit_tso = rowset->tablet_schema()->is_tso_enabled() ||
- (attached_row_binlog != nullptr &&
- attached_row_binlog->tablet_schema()->is_tso_enabled());
+ (attached_changelog != nullptr &&
+ attached_changelog->tablet_schema()->is_tso_enabled());
// A published TSO-enabled rowset must not retain the pre-publish -1 sentinel. For example,
// publishing version [12-12] without a commit TSO would make COMMIT_TSO_COL unreadable after
// the rowset becomes visible.
@@ -612,9 +612,9 @@ Status TxnManager::publish_txn(OlapMeta* meta, TPartitionId partition_id,
// it maybe a fatal error
rowset->make_visible(version, commit_tso);
- // Make the attached binlog rowset visible together.
- if (attached_row_binlog != nullptr) {
- attached_row_binlog->make_visible(version, commit_tso);
+ // Make the attached changelog rowset visible together.
+ if (attached_changelog != nullptr) {
+ attached_changelog->make_visible(version, commit_tso);
}
DBUG_EXECUTE_IF("TxnManager.publish_txn.random_failed_after_save_rs_meta", {
@@ -655,16 +655,16 @@ Status TxnManager::publish_txn(OlapMeta* meta, TPartitionId partition_id,
RETURN_IF_ERROR(TabletMetaManager::save_delete_bitmap(
tablet->data_dir(), tablet->tablet_id(), tablet_txn_info->delete_bitmap,
version.second));
- if (tablet_txn_info->attach_row_binlog.rowset != nullptr) {
- DCHECK(tablet_txn_info->attach_row_binlog.tablet != nullptr);
- if (tablet_txn_info->attach_row_binlog.delete_bitmap != nullptr) {
- auto binlog_tablet =
- std::static_pointer_cast<Tablet>(tablet_txn_info->attach_row_binlog.tablet);
- binlog_tablet->merge_delete_bitmap(
- *tablet_txn_info->attach_row_binlog.delete_bitmap);
+ if (tablet_txn_info->attach_changelog.rowset != nullptr) {
+ DCHECK(tablet_txn_info->attach_changelog.tablet != nullptr);
+ if (tablet_txn_info->attach_changelog.delete_bitmap != nullptr) {
+ auto changelog_tablet =
+ std::static_pointer_cast<Tablet>(tablet_txn_info->attach_changelog.tablet);
+ changelog_tablet->merge_delete_bitmap(
+ *tablet_txn_info->attach_changelog.delete_bitmap);
RETURN_IF_ERROR(TabletMetaManager::save_delete_bitmap(
- binlog_tablet->data_dir(), binlog_tablet->tablet_id(),
- tablet_txn_info->attach_row_binlog.delete_bitmap, version.second));
+ changelog_tablet->data_dir(), changelog_tablet->tablet_id(),
+ tablet_txn_info->attach_changelog.delete_bitmap, version.second));
}
}
stats->save_meta_time_us = MonotonicMicros() - t3;
@@ -684,18 +684,18 @@ Status TxnManager::publish_txn(OlapMeta* meta, TPartitionId partition_id,
}
}
- std::optional<RowsetMetaPB> attach_row_binlog_rowset_meta;
- if (tablet_txn_info->attach_row_binlog.rowset != nullptr) {
- attach_row_binlog_rowset_meta =
- tablet_txn_info->attach_row_binlog.rowset->rowset_meta()->get_rowset_pb();
- binlog_format = BinlogFormatPB::ROW;
+ std::optional<RowsetMetaPB> attach_changelog_rowset_meta;
+ if (tablet_txn_info->attach_changelog.rowset != nullptr) {
+ attach_changelog_rowset_meta =
+ tablet_txn_info->attach_changelog.rowset->rowset_meta()->get_rowset_pb();
+ binlog_format = BinlogFormatPB::CHANGELOG;
}
/// Step 4: save meta
int64_t t5 = MonotonicMicros();
auto status = RowsetMetaManager::save(meta, tablet_uid, rowset->rowset_id(),
rowset->rowset_meta()->get_rowset_pb(), binlog_format,
- attach_row_binlog_rowset_meta);
+ attach_changelog_rowset_meta);
stats->save_meta_time_us += MonotonicMicros() - t5;
if (!status.ok()) {
status.append(fmt::format(", txn id: {}", transaction_id));
@@ -836,12 +836,12 @@ Status TxnManager::delete_txn(OlapMeta* meta, TPartitionId partition_id,
rowset->rowset_id().to_string(), rowset->version().to_string(),
RowsetStatePB_Name(rowset->rowset_meta_state()));
} else {
- const auto& attach_binlog_rowset = load_info->attach_row_binlog.rowset;
- if (attach_binlog_rowset != nullptr) {
+ const auto& attach_changelog_rowset = load_info->attach_changelog.rowset;
+ if (attach_changelog_rowset != nullptr) {
static_cast<void>(RowsetMetaManager::remove(
- meta, attach_binlog_rowset->rowset_meta()->tablet_uid(),
- attach_binlog_rowset->rowset_id()));
- _engine.add_unused_rowset(attach_binlog_rowset);
+ meta, attach_changelog_rowset->rowset_meta()->tablet_uid(),
+ attach_changelog_rowset->rowset_id()));
+ _engine.add_unused_rowset(attach_changelog_rowset);
}
static_cast<void>(RowsetMetaManager::remove(meta, tablet_uid, rowset->rowset_id()));
#ifndef BE_TEST
@@ -851,9 +851,9 @@ Status TxnManager::delete_txn(OlapMeta* meta, TPartitionId partition_id,
<< " partition_id: " << key.first << ", transaction_id: " << key.second
<< ", tablet: " << tablet_info.to_string() << ", rowset: "
<< (rowset != nullptr ? rowset->rowset_id().to_string() : "0")
- << ", binlog<row> rowset: "
- << (attach_binlog_rowset != nullptr
- ? attach_binlog_rowset->rowset_id().to_string()
+ << ", changelog rowset: "
+ << (attach_changelog_rowset != nullptr
+ ? attach_changelog_rowset->rowset_id().to_string()
: "0");
}
}
@@ -883,7 +883,7 @@ void TxnManager::get_tablet_related_txns(TTabletId tablet_id, TabletUid tablet_u
bool is_related = it.second.find(tablet_info) != it.second.end();
if (!is_related) {
for (const auto& tablet_txn : it.second) {
- const auto& attached_tablet = tablet_txn.second->attach_row_binlog.tablet;
+ const auto& attached_tablet = tablet_txn.second->attach_changelog.tablet;
if (attached_tablet != nullptr &&
attached_tablet->tablet_id() == tablet_info.tablet_id &&
attached_tablet->tablet_uid() == tablet_info.tablet_uid) {
@@ -917,20 +917,20 @@ void TxnManager::force_rollback_tablet_related_txns(OlapMeta* meta, TTabletId ta
if (load_itr != it->second.end()) {
auto& load_info = load_itr->second;
auto& rowset = load_info->rowset;
- const auto& attach_binlog_rowset = load_info->attach_row_binlog.rowset;
+ const auto& attach_changelog_rowset = load_info->attach_changelog.rowset;
if (rowset != nullptr && meta != nullptr) {
LOG(INFO) << " delete transaction from engine "
<< ", tablet: " << tablet_info.to_string()
<< ", rowset id: " << rowset->rowset_id();
- // clean the attached binlog rowset first
- if (attach_binlog_rowset != nullptr) {
+ // clean the attached changelog rowset first
+ if (attach_changelog_rowset != nullptr) {
Status status = RowsetMetaManager::remove(
- meta, attach_binlog_rowset->rowset_meta()->tablet_uid(),
- attach_binlog_rowset->rowset_id());
+ meta, attach_changelog_rowset->rowset_meta()->tablet_uid(),
+ attach_changelog_rowset->rowset_id());
if (!status.ok() && !status.is<META_KEY_NOT_FOUND>()) {
- LOG(WARNING)
- << "failed to remove binlog<row> rowset meta, rowset_id="
- << attach_binlog_rowset->rowset_id() << ", status=" << status;
+ LOG(WARNING) << "failed to remove changelog rowset meta, rowset_id="
+ << attach_changelog_rowset->rowset_id()
+ << ", status=" << status;
}
}
static_cast<void>(
@@ -941,9 +941,9 @@ void TxnManager::force_rollback_tablet_related_txns(OlapMeta* meta, TTabletId ta
<< ", transaction_id: " << it->first.second
<< ", tablet: " << tablet_info.to_string() << ", rowset: "
<< (rowset != nullptr ? rowset->rowset_id().to_string() : "0")
- << ", binlog<row> rowset: "
- << (attach_binlog_rowset != nullptr
- ? attach_binlog_rowset->rowset_id().to_string()
+ << ", changelog rowset: "
+ << (attach_changelog_rowset != nullptr
+ ? attach_changelog_rowset->rowset_id().to_string()
: "0");
it->second.erase(load_itr);
}
diff --git a/be/src/storage/txn/txn_manager.h b/be/src/storage/txn/txn_manager.h
index b0195bb9e35..c6c7c2c6ffe 100644
--- a/be/src/storage/txn/txn_manager.h
+++ b/be/src/storage/txn/txn_manager.h
@@ -62,11 +62,11 @@ enum class TxnState {
};
enum class PublishStatus { INIT = 0, PREPARE = 1, SUCCEED = 2 };
-// The row binlog rowset and its independent binlog tablet, carried through commit and publish.
-struct RowBinlogTxnInfo {
+// The changelog rowset and its independent changelog tablet, carried through commit and publish.
+struct ChangelogTxnInfo {
RowsetSharedPtr rowset;
BaseTabletSPtr tablet;
- // Delete bitmap deltas that should be applied to the independent binlog tablet.
+ // Delete bitmap deltas that should be applied to the independent changelog tablet.
DeleteBitmapPtr delete_bitmap;
};
@@ -80,8 +80,8 @@ struct TxnPublishInfo {
struct TabletTxnInfo {
PUniqueId load_id;
RowsetSharedPtr rowset;
- // The row-binlog tablet is attached while PREPARED; its rowset is filled at commit.
- RowBinlogTxnInfo attach_row_binlog;
+ // The changelog tablet is attached while PREPARED; its rowset is filled at commit.
+ ChangelogTxnInfo attach_changelog;
PendingRowsetGuard pending_rs_guard;
bool unique_key_merge_on_write {false};
DeleteBitmapPtr delete_bitmap;
@@ -172,17 +172,17 @@ public:
TTabletId tablet_id, TabletUid tablet_uid, const PUniqueId& load_id,
bool is_ingest = false);
- // Internal: attach the independent row-binlog tablet to an existing base-tablet transaction
+ // Internal: attach the independent changelog tablet to an existing base-tablet transaction
// before commit, so tablet lifecycle operations can drain the whole group transaction.
- Status attach_row_binlog_tablet_to_txn(TPartitionId partition_id, TTransactionId transaction_id,
- const TabletInfo& base_tablet_info,
- const BaseTabletSPtr& row_binlog_tablet);
+ Status attach_changelog_tablet_to_txn(TPartitionId partition_id, TTransactionId transaction_id,
+ const TabletInfo& base_tablet_info,
+ const BaseTabletSPtr& changelog_tablet);
Status commit_txn(TPartitionId partition_id, const Tablet& tablet,
TTransactionId transaction_id, const PUniqueId& load_id,
const RowsetSharedPtr& rowset_ptr, PendingRowsetGuard guard, bool is_recovery,
std::shared_ptr<PartialUpdateInfo> partial_update_info = nullptr,
- const RowBinlogTxnInfo& attach_row_binlog = {});
+ const ChangelogTxnInfo& attach_changelog = {});
Status publish_txn(TPartitionId partition_id, const TabletSharedPtr& tablet,
TTransactionId transaction_id, const Version& version,
@@ -201,7 +201,7 @@ public:
TTabletId tablet_id, TabletUid tablet_uid, const PUniqueId& load_id,
const RowsetSharedPtr& rowset_ptr, PendingRowsetGuard guard, bool is_recovery,
std::shared_ptr<PartialUpdateInfo> partial_update_info = nullptr,
- const RowBinlogTxnInfo& attach_row_binlog = {});
+ const ChangelogTxnInfo& attach_changelog = {});
// remove a txn from txn manager
// not persist rowset meta because
@@ -226,7 +226,7 @@ public:
TTabletId tablet_id, TabletUid tablet_uid);
// Includes transactions keyed by this tablet and base-tablet transactions that attach this
- // tablet as their independent row-binlog tablet.
+ // tablet as their independent changelog tablet.
void get_tablet_related_txns(TTabletId tablet_id, TabletUid tablet_uid, int64_t* partition_id,
std::set<int64_t>* transaction_ids);
diff --git a/be/src/storage/utils.h b/be/src/storage/utils.h
index a7fcb63f11d..fdeea6cad10 100644
--- a/be/src/storage/utils.h
+++ b/be/src/storage/utils.h
@@ -41,9 +41,12 @@ static const std::string SKIP_BITMAP_COL = "__DORIS_SKIP_BITMAP_COL__";
static const std::string SEQUENCE_COL = "__DORIS_SEQUENCE_COL__";
static const std::string COMMIT_TSO_COL = "__DORIS_COMMIT_TSO_COL__";
static const std::string ROW_LSN_COL = "__DORIS_ROW_LSN_COL__";
-static const std::string BINLOG_TSO_COL = "__DORIS_BINLOG_TSO__";
-static const std::string BINLOG_LSN_COL = "__DORIS_BINLOG_LSN__";
-static const std::string BINLOG_OP_COL = "__DORIS_BINLOG_OP__";
+// ATTN(changelog-rename): formerly "__DORIS_BINLOG_TSO__"
+static const std::string CHANGELOG_TSO_COL = "__DORIS_CHANGELOG_TSO__";
+// ATTN(changelog-rename): formerly "__DORIS_BINLOG_LSN__"
+static const std::string CHANGELOG_LSN_COL = "__DORIS_CHANGELOG_LSN__";
+// ATTN(changelog-rename): formerly "__DORIS_BINLOG_OP__"
+static const std::string CHANGELOG_OP_COL = "__DORIS_CHANGELOG_OP__";
// 用来加速运算
const static int32_t g_power_table[] = {1, 10, 100, 1000, 10000,
diff --git a/be/src/util/thrift_util.cpp b/be/src/util/thrift_util.cpp
index 6177d81770a..83968a010da 100644
--- a/be/src/util/thrift_util.cpp
+++ b/be/src/util/thrift_util.cpp
@@ -179,13 +179,13 @@ bool _has_inverted_index_v1_or_partial_update(TOlapTableSink sink) {
return false;
}
-bool _has_row_binlog(const TOlapTableSink& sink) {
+bool _has_changelog(const TOlapTableSink& sink) {
OlapTableSchemaParam schema;
if (!schema.init(sink.schema).ok()) {
return false;
}
for (const auto* index_schema : schema.indexes()) {
- if (index_schema->row_binlog_id > 0) {
+ if (index_schema->changelog_index_id > 0) {
return true;
}
}
diff --git a/be/src/util/thrift_util.h b/be/src/util/thrift_util.h
index 73c753e2271..2d841867b36 100644
--- a/be/src/util/thrift_util.h
+++ b/be/src/util/thrift_util.h
@@ -178,6 +178,6 @@ bool t_network_address_comparator(const TNetworkAddress& a, const TNetworkAddres
std::string to_string(const TUniqueId& id);
bool _has_inverted_index_v1_or_partial_update(TOlapTableSink sink);
-bool _has_row_binlog(const TOlapTableSink& sink);
+bool _has_changelog(const TOlapTableSink& sink);
} // namespace doris
diff --git a/be/test/cloud/cloud_compaction_test.cpp b/be/test/cloud/cloud_compaction_test.cpp
index db53f73fec9..b762ab853c1 100644
--- a/be/test/cloud/cloud_compaction_test.cpp
+++ b/be/test/cloud/cloud_compaction_test.cpp
@@ -40,7 +40,7 @@
#include "cpp/sync_point.h"
#include "io/fs/s3_file_system.h"
#include "json2pb/json_to_pb.h"
-#include "storage/binlog.h"
+#include "storage/changelog.h"
#include "storage/compaction/cumulative_compaction_time_series_policy.h"
#include "storage/olap_common.h"
#include "storage/rowset/rowset_factory.h"
@@ -249,8 +249,8 @@ TEST_F(CloudCompactionTest, failure_cumu_compaction_tablet_sleep_test) {
ASSERT_EQ(tablets.size(), 0);
}
-TEST_F(CloudCompactionTest, binlog_compaction_max_score_ignores_normal_tablets) {
- auto filter_out = [](CloudTablet* t) { return !t->is_row_binlog_tablet(); };
+TEST_F(CloudCompactionTest, changelog_compaction_max_score_ignores_normal_tablets) {
+ auto filter_out = [](CloudTablet* t) { return !t->is_changelog_tablet(); };
CloudTabletMgr mgr(_engine);
auto normal_meta = std::make_shared<TabletMeta>(*_tablet_meta);
@@ -261,23 +261,23 @@ TEST_F(CloudCompactionTest, binlog_compaction_max_score_ignores_normal_tablets)
normal_tablet->_approximate_cumu_num_deltas = 10;
mgr.put_tablet_for_UT(normal_tablet);
- auto binlog_meta = std::make_shared<TabletMeta>(*_tablet_meta);
- binlog_meta->set_tablet_role(TabletRolePB::TABLET_ROLE_ROW_BINLOG);
- CloudTabletSPtr binlog_tablet = std::make_shared<CloudTablet>(_engine, binlog_meta);
- binlog_tablet->tablet_meta()->_tablet_id = 10002;
- binlog_tablet->tablet_meta()->set_disable_auto_compaction(false);
- binlog_tablet->_approximate_cumu_num_deltas = 7;
- mgr.put_tablet_for_UT(binlog_tablet);
... 40270 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org