You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by "yjhjstz (via GitHub)" <gi...@apache.org> on 2026/09/07 00:45:16 UTC

[PR] fix(orca): merge colid->attno mapping in CStatistics::AppendStats to avoid extended-stats crash [cloudberry]

yjhjstz opened a new pull request, #1966:
URL: https://github.com/apache/cloudberry/pull/1966

   
   When a memo group's cached statistics lack columns needed by a later derivation request (e.g. a ROLLUP expanded into a CTE whose consumers request different column sets), the missing columns are derived separately and merged via CStatistics::AppendStats(). That merge copied histograms and widths only, so the colid -> attno mapping (added for extended statistics) diverged from the histogram map: a column could have a histogram but no attno entry.
   
   ApplyCorrelatedStatsToScaleFactorFilterCalculation() then dereferenced the result of the attno lookup without a null check and crashed the coordinator with SIGSEGV whenever the table had any extended statistics object with stored data and a filter referenced one of the appended columns (e.g. TPC-DS Q67 after CREATE STATISTICS (ndistinct) on item/date_dim). Reproduced on main with a cassert build: colid 11 had a histogram (5 entries) but the attno map only held 4 entries, attnum was NULL at CExtendedStatsProcessor.cpp:280. This is the same class of bug as the CTE remapping fix in 124a048e8c6 (gpdb#16212), which repaired CopyStatsWithRemap() but not AppendStats().
   
   Fix both layers:
   
   * CStatistics::AppendStats() now also merges the colid -> attno mapping (copy-on-write, since the mapping is shared by refcount across stats objects, see ScaleStats()) and adopts the input's extended-stats info when none is set, restoring the invariant that every column with a histogram has an attno entry.
   
   * CExtendedStatsProcessor now null-checks the attno and histogram lookups and skips clauses it cannot translate, falling back to the independence assumption, matching what ApplyCorrelatedStatsToNDistinctCalculation() already does. Also bail out gracefully if no ndistinct item matches the attribute set.
   
   * CExtendedStatsProcessor also skips system columns (attno <= 0): extended statistics never cover them, and feeding a negative attno into CBitSet::ExchangeSet() would convert it to a huge unsigned bit index. The backend rejects these clauses too (dependency_is_compatible_clause()).
   
   Add a regression test with the minimized reproduction: a ROLLUP over a join where the dimension table has ndistinct extended statistics and the filter column is appended to the scan group's stats after the initial derivation. Verified the gporca test passes with both optimizer=on and optimizer=off answer files.
   
   Ported from warehouse-pg/warehouse-pg#251.
   
   
   Fixes #1949 
   
   ### What does this PR do?
   <!-- Brief overview of the changes, including any major features or fixes -->
   
   ### Type of Change
   - [ ] Bug fix (non-breaking change)
   - [ ] New feature (non-breaking change)
   - [ ] Breaking change (fix or feature with breaking changes)
   - [ ] Documentation update
   
   ### Breaking Changes
   <!-- Remove if not applicable. If yes, explain impact and migration path -->
   
   ### Test Plan
   <!-- How did you test these changes? -->
   - [ ] Unit tests added/updated
   - [ ] Integration tests added/updated
   - [ ] Passed `make installcheck`
   - [ ] Passed `make -C src/test installcheck-cbdb-parallel`
   
   ### Impact
   <!-- Remove sections that don't apply -->
   **Performance:**
   <!-- Any performance implications? -->
   
   **User-facing changes:**
   <!-- Any changes visible to users? -->
   
   **Dependencies:**
   <!-- New dependencies or version changes? -->
   
   ### Checklist
   - [ ] Followed [contribution guide](https://cloudberry.apache.org/contribute/code)
   - [ ] Added/updated documentation
   - [ ] Reviewed code for security implications
   - [ ] This PR contains AI-assisted code generation
   - [ ] Requested review from [cloudberry committers](https://github.com/orgs/apache/teams/cloudberry-committers)
   
   ### Additional Context
   <!-- Any other information that would help reviewers? Remove if none -->
   
   ### CI Skip Instructions
   <!--
   To skip CI builds, add the appropriate CI skip identifier to your PR title.
   The identifier must:
   - Be in square brackets []
   - Include the word "ci" and either "skip" or "no"
   - Only use for documentation-only changes or when absolutely necessary
   -->
   
   ---
   <!-- Join our community:
   - Mailing list: [dev@cloudberry.apache.org](https://lists.apache.org/list.html?dev@cloudberry.apache.org) (subscribe: dev-subscribe@cloudberry.apache.org)
   - Discussions: https://github.com/apache/cloudberry/discussions -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] fix(orca): merge colid->attno mapping in CStatistics::AppendStats to avoid extended-stats crash [cloudberry]

Posted by "Alena0704 (via GitHub)" <gi...@apache.org>.
Alena0704 commented on code in PR #1966:
URL: https://github.com/apache/cloudberry/pull/1966#discussion_r4347050021


##########
src/backend/gporca/libnaucrates/src/statistics/CStatistics.cpp:
##########
@@ -552,6 +552,33 @@ CStatistics::AppendStats(CMemoryPool *mp, IStatistics *input_stats)
 	CStatisticsUtils::AddWidthInfo(mp, stats->m_colid_width_mapping,
 								   m_colid_width_mapping);
 	GPOS_CHECK_ABORT;
+
+	// Also merge the colid -> attno mapping so that every column that has a
+	// histogram keeps an attno entry; extended-stats estimation relies on
+	// this invariant. The mapping may be shared with other stats objects
+	// (see ScaleStats()), so merge copy-on-write instead of mutating it.
+	if (0 < stats->m_colid_to_attno_mapping->Size())
+	{
+		UlongToIntMap *attnos_new = GPOS_NEW(mp) UlongToIntMap(mp);
+		AddAttnoInfo(mp, m_colid_to_attno_mapping, attnos_new);
+		AddAttnoInfo(mp, stats->m_colid_to_attno_mapping, attnos_new);
+		m_colid_to_attno_mapping->Release();
+		m_colid_to_attno_mapping = attnos_new;
+	}
+
+	// Attnos are only meaningful relative to m_ext_stats' base relation, so
+	// merging is only valid for stats objects describing the same relation;
+	// all callers merge stats of the same memo group / expression today.
+	GPOS_ASSERT(nullptr == m_ext_stats || nullptr == stats->m_ext_stats ||

Review Comment:
   Could you clarify what guarantees that two non-null m_ext_stats values in AppendStats() refer to the same base relation? The same memo group can contain different expressions, and missing-column statistics are derived through PgexprBestPromise(), so the comment alone doesn’t seem to establish this invariant. Is pointer identity an intentional requirement here, or would MDId()->Equals() better express the intended check?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] fix(orca): merge colid->attno mapping in CStatistics::AppendStats to avoid extended-stats crash [cloudberry]

Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1966:
URL: https://github.com/apache/cloudberry/pull/1966#discussion_r4341825596


##########
src/backend/gporca/libnaucrates/src/statistics/CFilterStatsProcessor.cpp:
##########
@@ -574,8 +602,20 @@ CFilterStatsProcessor::MakeHistHashMapDisjFilter(
 			if (is_colid_present)
 			{
 				// conjunction or disjunction uses only a single column
-				disjunctive_child_col_histogram =
-					child_histograms->Find(&colid)->CopyHistogram();
+				const CHistogram *child_col_histogram =
+					child_histograms->Find(&colid);
+				if (nullptr != child_col_histogram)
+				{
+					disjunctive_child_col_histogram =
+						child_col_histogram->CopyHistogram();
+				}
+				else
+				{
+					// the column has no histogram, so the child left none
+					// behind; treat the child like a multi-column predicate
+					// and only merge its histogram map and scale factor
+					is_colid_present = false;

Review Comment:
   Good catch, and it turns out the consequence is worse than just the scale factor.
   
   The multi-column path at the bottom of the loop is otherwise only reached with `colid == gpos::ulong_max`, which makes `IsNewStatsColumn()` true at the top of the loop, so that path *assumes* the pending per-column state has already been flushed. By falling into it with a real `colid` still set, my patch broke that assumption twice:
   
   1. as you say, a later disjunct on the same column sees `previous_colid == colid`, so `previous_scale_factor` is never appended to `scale_factors` and is overwritten instead of accumulated — the OR estimate becomes branch-order dependent;
   2. the path's `previous_histogram = nullptr;` (a no-op in the original code, since the flush had already run) now drops a live `CHistogram *` — its contribution to the disjunction is lost and the object is leaked.
   
   Fixed in a1af361792b by flushing the pending histogram and scale factor in the fallback and then resetting `colid`:
   
   ```cpp
   if (nullptr != previous_histogram)
   {
       scale_factors->Append(
           GPOS_NEW(mp) CDouble(previous_scale_factor.Get()));
       CStatisticsUtils::UpdateDisjStatistics(
           mp, non_updatable_cols, input_rows, cumulative_rows,
           previous_histogram, disjunctive_result_histograms,
           previous_colid);
       previous_histogram = nullptr;
   }
   is_colid_present = false;
   colid = gpos::ulong_max;
   ```
   
   The `nullptr != previous_histogram` guard is what distinguishes "state pending for this column" from "already flushed at the top of this iteration" — in the latter case `previous_scale_factor` still holds the value that was just appended, so an unconditional append would double-count it.
   
   Verified on a cassert build (ORCA on): `gporca` (which carries this PR's new test), `stats_ext`, `bfv_statistic`, `table_statistics`, `gporca_faults`, `qp_orca_fallback`, `bfv_joins`, `bfv_subquery`, `bfv_partition_plans`, `subselect_gp`, `qp_select`, `select`, `subselect`, `orca_static_pruning` all pass.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] fix(orca): merge colid->attno mapping in CStatistics::AppendStats to avoid extended-stats crash [cloudberry]

Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1966:
URL: https://github.com/apache/cloudberry/pull/1966#discussion_r4124511340


##########
src/backend/gporca/libnaucrates/src/statistics/CFilterStatsProcessor.cpp:
##########
@@ -375,7 +375,16 @@ CFilterStatsProcessor::MakeHistHashMapConjFilter(
 		if (CStatsPred::EsptDisj != child_pred_stats->GetPredStatsType())
 		{
 			GPOS_ASSERT(gpos::ulong_max != colid);
-			hist_before = result_histograms->Find(&colid)->CopyHistogram();
+			const CHistogram *base_histogram = result_histograms->Find(&colid);
+			if (nullptr == base_histogram)
+			{
+				// no histogram for the filter column; estimate the clause
+				// with the default selectivity instead of dereferencing null
+				scale_factors->Append(GPOS_NEW(mp) CDouble(
+					1 / CHistogram::DefaultSelectivity.Get()));
+				continue;

Review Comment:
   fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] fix(orca): merge colid->attno mapping in CStatistics::AppendStats to avoid extended-stats crash [cloudberry]

Posted by "Alena0704 (via GitHub)" <gi...@apache.org>.
Alena0704 commented on code in PR #1966:
URL: https://github.com/apache/cloudberry/pull/1966#discussion_r4335822711


##########
src/backend/gporca/libnaucrates/src/statistics/CFilterStatsProcessor.cpp:
##########
@@ -574,8 +602,20 @@ CFilterStatsProcessor::MakeHistHashMapDisjFilter(
 			if (is_colid_present)
 			{
 				// conjunction or disjunction uses only a single column
-				disjunctive_child_col_histogram =
-					child_histograms->Find(&colid)->CopyHistogram();
+				const CHistogram *child_col_histogram =
+					child_histograms->Find(&colid);
+				if (nullptr != child_col_histogram)
+				{
+					disjunctive_child_col_histogram =
+						child_col_histogram->CopyHistogram();
+				}
+				else
+				{
+					// the column has no histogram, so the child left none
+					// behind; treat the child like a multi-column predicate
+					// and only merge its histogram map and scale factor
+					is_colid_present = false;

Review Comment:
   It looks like `colid = gpos::ulong_max;` is missing here.
   Without this reset, the next nested branch on the same column overwrites the previous branch’s scale factor instead of accumulating it, making OR estimates depend on branch order. Resetting colid preserves both branches’ contributions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] fix(orca): merge colid->attno mapping in CStatistics::AppendStats to avoid extended-stats crash [cloudberry]

Posted by "Alena0704 (via GitHub)" <gi...@apache.org>.
Alena0704 commented on code in PR #1966:
URL: https://github.com/apache/cloudberry/pull/1966#discussion_r4357731259


##########
src/backend/gporca/libnaucrates/src/statistics/CStatistics.cpp:
##########
@@ -552,6 +552,33 @@ CStatistics::AppendStats(CMemoryPool *mp, IStatistics *input_stats)
 	CStatisticsUtils::AddWidthInfo(mp, stats->m_colid_width_mapping,
 								   m_colid_width_mapping);
 	GPOS_CHECK_ABORT;
+
+	// Also merge the colid -> attno mapping so that every column that has a
+	// histogram keeps an attno entry; extended-stats estimation relies on
+	// this invariant. The mapping may be shared with other stats objects
+	// (see ScaleStats()), so merge copy-on-write instead of mutating it.
+	if (0 < stats->m_colid_to_attno_mapping->Size())
+	{
+		UlongToIntMap *attnos_new = GPOS_NEW(mp) UlongToIntMap(mp);
+		AddAttnoInfo(mp, m_colid_to_attno_mapping, attnos_new);
+		AddAttnoInfo(mp, stats->m_colid_to_attno_mapping, attnos_new);
+		m_colid_to_attno_mapping->Release();
+		m_colid_to_attno_mapping = attnos_new;
+	}
+
+	// Attnos are only meaningful relative to m_ext_stats' base relation, so
+	// merging is only valid for stats objects describing the same relation;
+	// all callers merge stats of the same memo group / expression today.
+	GPOS_ASSERT(nullptr == m_ext_stats || nullptr == stats->m_ext_stats ||
+				m_ext_stats == stats->m_ext_stats);

Review Comment:
   would MDId()->Equals() be better?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] fix(orca): merge colid->attno mapping in CStatistics::AppendStats to avoid extended-stats crash [cloudberry]

Posted by "Alena0704 (via GitHub)" <gi...@apache.org>.
Alena0704 commented on code in PR #1966:
URL: https://github.com/apache/cloudberry/pull/1966#discussion_r4363127837


##########
src/backend/gporca/libnaucrates/src/statistics/CStatistics.cpp:
##########
@@ -552,6 +552,33 @@ CStatistics::AppendStats(CMemoryPool *mp, IStatistics *input_stats)
 	CStatisticsUtils::AddWidthInfo(mp, stats->m_colid_width_mapping,
 								   m_colid_width_mapping);
 	GPOS_CHECK_ABORT;
+
+	// Also merge the colid -> attno mapping so that every column that has a
+	// histogram keeps an attno entry; extended-stats estimation relies on
+	// this invariant. The mapping may be shared with other stats objects
+	// (see ScaleStats()), so merge copy-on-write instead of mutating it.
+	if (0 < stats->m_colid_to_attno_mapping->Size())
+	{
+		UlongToIntMap *attnos_new = GPOS_NEW(mp) UlongToIntMap(mp);
+		AddAttnoInfo(mp, m_colid_to_attno_mapping, attnos_new);
+		AddAttnoInfo(mp, stats->m_colid_to_attno_mapping, attnos_new);
+		m_colid_to_attno_mapping->Release();
+		m_colid_to_attno_mapping = attnos_new;
+	}
+
+	// Attnos are only meaningful relative to m_ext_stats' base relation, so
+	// merging is only valid for stats objects describing the same relation;
+	// all callers merge stats of the same memo group / expression today.
+	GPOS_ASSERT(nullptr == m_ext_stats || nullptr == stats->m_ext_stats ||
+				m_ext_stats == stats->m_ext_stats);

Review Comment:
   I checked this, and switching to Equals() would be a style/robustness nit only, with no behavioral difference.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] fix(orca): merge colid->attno mapping in CStatistics::AppendStats to avoid extended-stats crash [cloudberry]

Posted by "leborchuk (via GitHub)" <gi...@apache.org>.
leborchuk commented on code in PR #1966:
URL: https://github.com/apache/cloudberry/pull/1966#discussion_r4117000648


##########
src/backend/gporca/libnaucrates/src/statistics/CFilterStatsProcessor.cpp:
##########
@@ -375,7 +375,16 @@ CFilterStatsProcessor::MakeHistHashMapConjFilter(
 		if (CStatsPred::EsptDisj != child_pred_stats->GetPredStatsType())
 		{
 			GPOS_ASSERT(gpos::ulong_max != colid);
-			hist_before = result_histograms->Find(&colid)->CopyHistogram();
+			const CHistogram *base_histogram = result_histograms->Find(&colid);
+			if (nullptr == base_histogram)
+			{
+				// no histogram for the filter column; estimate the clause
+				// with the default selectivity instead of dereferencing null
+				scale_factors->Append(GPOS_NEW(mp) CDouble(
+					1 / CHistogram::DefaultSelectivity.Get()));
+				continue;

Review Comment:
   Yes, it will be wise to switch to default selectivity here.
   
   Consider to do the same check and switch in case of null histogram for lines 550 and 578.
   
   It could be important for predicates like `WHERE (c2 > 0 OR c1 = 'x')` where second has extended stat



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] fix(orca): merge colid->attno mapping in CStatistics::AppendStats to avoid extended-stats crash [cloudberry]

Posted by "Alena0704 (via GitHub)" <gi...@apache.org>.
Alena0704 commented on code in PR #1966:
URL: https://github.com/apache/cloudberry/pull/1966#discussion_r4347050021


##########
src/backend/gporca/libnaucrates/src/statistics/CStatistics.cpp:
##########
@@ -552,6 +552,33 @@ CStatistics::AppendStats(CMemoryPool *mp, IStatistics *input_stats)
 	CStatisticsUtils::AddWidthInfo(mp, stats->m_colid_width_mapping,
 								   m_colid_width_mapping);
 	GPOS_CHECK_ABORT;
+
+	// Also merge the colid -> attno mapping so that every column that has a
+	// histogram keeps an attno entry; extended-stats estimation relies on
+	// this invariant. The mapping may be shared with other stats objects
+	// (see ScaleStats()), so merge copy-on-write instead of mutating it.
+	if (0 < stats->m_colid_to_attno_mapping->Size())
+	{
+		UlongToIntMap *attnos_new = GPOS_NEW(mp) UlongToIntMap(mp);
+		AddAttnoInfo(mp, m_colid_to_attno_mapping, attnos_new);
+		AddAttnoInfo(mp, stats->m_colid_to_attno_mapping, attnos_new);
+		m_colid_to_attno_mapping->Release();
+		m_colid_to_attno_mapping = attnos_new;
+	}
+
+	// Attnos are only meaningful relative to m_ext_stats' base relation, so
+	// merging is only valid for stats objects describing the same relation;
+	// all callers merge stats of the same memo group / expression today.
+	GPOS_ASSERT(nullptr == m_ext_stats || nullptr == stats->m_ext_stats ||

Review Comment:
   Could you clarify what guarantees that two non-null m_ext_stats values in AppendStats() refer to the same base relation? The same memo group can contain different expressions, and missing-column statistics are derived through PgexprBestPromise(), so the comment alone doesn’t seem to establish this invariant. Is pointer identity an intentional requirement here, or would MDId()->Equals() better express the intended check?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] fix(orca): merge colid->attno mapping in CStatistics::AppendStats to avoid extended-stats crash [cloudberry]

Posted by "yjhjstz (via GitHub) via commits" <co...@cloudberry.apache.org>.
yjhjstz merged PR #1966:
URL: https://github.com/apache/cloudberry/pull/1966


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org