You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by "yjhjstz (via GitHub)" <gi...@apache.org> on 2026/01/11 15:39:56 UTC

[PR] ORCA: Fix detection of mixed storage in partitioned tables [cloudberry]

yjhjstz opened a new pull request, #1524:
URL: https://github.com/apache/cloudberry/pull/1524

   
   
   The storage type detection logic failed to properly identify mixed storage when foreign and non-foreign partitions coexisted, leading to incorrect metadata that could cause issues with scan type selection and query planning.
   
   
   Fixes #ISSUE_Number
   
   ### What does this PR do?
   <!-- Brief overview of the changes, including any major features or fixes -->
   
   ### Type of Change
   - [ ] Bug fix (non-breaking change)
   - [ ] New feature (non-breaking change)
   - [ ] Breaking change (fix or feature with breaking changes)
   - [ ] Documentation update
   
   ### Breaking Changes
   <!-- Remove if not applicable. If yes, explain impact and migration path -->
   
   ### Test Plan
   <!-- How did you test these changes? -->
   - [ ] Unit tests added/updated
   - [ ] Integration tests added/updated
   - [ ] Passed `make installcheck`
   - [ ] Passed `make -C src/test installcheck-cbdb-parallel`
   
   ### Impact
   <!-- Remove sections that don't apply -->
   **Performance:**
   <!-- Any performance implications? -->
   
   **User-facing changes:**
   <!-- Any changes visible to users? -->
   
   **Dependencies:**
   <!-- New dependencies or version changes? -->
   
   ### Checklist
   - [ ] Followed [contribution guide](https://cloudberry.apache.org/contribute/code)
   - [ ] Added/updated documentation
   - [ ] Reviewed code for security implications
   - [ ] Requested review from [cloudberry committers](https://github.com/orgs/apache/teams/cloudberry-committers)
   
   ### Additional Context
   <!-- Any other information that would help reviewers? Remove if none -->
   
   ### CI Skip Instructions
   <!--
   To skip CI builds, add the appropriate CI skip identifier to your PR title.
   The identifier must:
   - Be in square brackets []
   - Include the word "ci" and either "skip" or "no"
   - Only use for documentation-only changes or when absolutely necessary
   -->
   
   ---
   <!-- Join our community:
   - Mailing list: [dev@cloudberry.apache.org](https://lists.apache.org/list.html?dev@cloudberry.apache.org) (subscribe: dev-subscribe@cloudberry.apache.org)
   - Discussions: https://github.com/apache/cloudberry/discussions -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] ORCA: Fix detection of mixed storage in partitioned tables [cloudberry]

Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1524:
URL: https://github.com/apache/cloudberry/pull/1524#discussion_r2726269193


##########
src/backend/gporca/data/dxl/minidump/MixedPartitioned-ForeignHeap-Test.mdp:
##########
@@ -0,0 +1,17 @@
+<?xml version='1.0' encoding='UTF-8'?>
+<dxl:DXLMessage xmlns:dxl="http://greenplum.com/dxl/2010/12/"><dxl:Comment>

Review Comment:
   all the *.[mdp] are `<dxl:DXLMessage xmlns:dxl="http://greenplum.com/dxl/2010/12/"><dxl:Comment>`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] ORCA: Fix detection of mixed storage in partitioned tables [cloudberry]

Posted by "leborchuk (via GitHub)" <gi...@apache.org>.
leborchuk commented on code in PR #1524:
URL: https://github.com/apache/cloudberry/pull/1524#discussion_r2681387965


##########
src/backend/gpopt/translate/CTranslatorRelcacheToDXL.cpp:
##########
@@ -3147,18 +3147,36 @@ CTranslatorRelcacheToDXL::RetrieveStorageTypeForPartitionedTable(Relation rel)
 						"Queries with partitions of greenplum_fdw are not supported"));
 			}
 			GPOS_DELETE(fdw_name_str);
+
+			// Check for mixed storage before continuing
+			// If we already encountered non-foreign partitions, mark as mixed
+			if (rel_storage_type != IMDRelation::ErelstorageSentinel &&
+				rel_storage_type != IMDRelation::ErelstorageForeign)
+			{
+				// Already have non-foreign partition(s), now found foreign → mixed
+				rel_storage_type = IMDRelation::ErelstorageMixedPartitioned;
+			}
+			else if (rel_storage_type == IMDRelation::ErelstorageSentinel)
+			{
+				// First partition is foreign
+				rel_storage_type = IMDRelation::ErelstorageForeign;
+			}
 			continue;
 		}
 		all_foreign = false;
 		if (rel_storage_type == IMDRelation::ErelstorageSentinel)
 		{
 			rel_storage_type = child_storage;
 		}
-
+		else if (rel_storage_type == IMDRelation::ErelstorageForeign)
+		{
+			// Previously had foreign partition(s), now found non-foreign → mixed
+			rel_storage_type = IMDRelation::ErelstorageMixedPartitioned;
+		}
 		// mark any partitioned table with supported partitions of mixed storage types,
 		// this is more conservative for certain skans (eg: we can't do an index scan if any
 		// partition is ao, we must only do a sequential or bitmap scan)
-		if (rel_storage_type != child_storage)
+		else if (rel_storage_type != child_storage)

Review Comment:
   Sorry, but in which cases this logic failed? For me it should always set ErelstorageMixedPartitioned in a case of mixed storage options



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] ORCA: Fix detection of mixed storage in partitioned tables [cloudberry]

Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1524:
URL: https://github.com/apache/cloudberry/pull/1524#discussion_r2683296256


##########
src/backend/gpopt/translate/CTranslatorRelcacheToDXL.cpp:
##########
@@ -3147,18 +3147,36 @@ CTranslatorRelcacheToDXL::RetrieveStorageTypeForPartitionedTable(Relation rel)
 						"Queries with partitions of greenplum_fdw are not supported"));
 			}
 			GPOS_DELETE(fdw_name_str);
+
+			// Check for mixed storage before continuing
+			// If we already encountered non-foreign partitions, mark as mixed
+			if (rel_storage_type != IMDRelation::ErelstorageSentinel &&
+				rel_storage_type != IMDRelation::ErelstorageForeign)
+			{
+				// Already have non-foreign partition(s), now found foreign → mixed
+				rel_storage_type = IMDRelation::ErelstorageMixedPartitioned;
+			}
+			else if (rel_storage_type == IMDRelation::ErelstorageSentinel)
+			{
+				// First partition is foreign
+				rel_storage_type = IMDRelation::ErelstorageForeign;
+			}
 			continue;
 		}
 		all_foreign = false;
 		if (rel_storage_type == IMDRelation::ErelstorageSentinel)
 		{
 			rel_storage_type = child_storage;
 		}
-
+		else if (rel_storage_type == IMDRelation::ErelstorageForeign)
+		{
+			// Previously had foreign partition(s), now found non-foreign → mixed
+			rel_storage_type = IMDRelation::ErelstorageMixedPartitioned;
+		}
 		// mark any partitioned table with supported partitions of mixed storage types,
 		// this is more conservative for certain skans (eg: we can't do an index scan if any
 		// partition is ao, we must only do a sequential or bitmap scan)
-		if (rel_storage_type != child_storage)
+		else if (rel_storage_type != child_storage)

Review Comment:
   ```sql
   create table part (a int, b int) partition by range (b);
   create external table p1_e (a int, b int) location ('file://@hostname@@abs_srcdir@/data/part1.csv') format 'csv';
   create external table p2_e (a int, b int) location ('file://@hostname@@abs_srcdir@/data/part2.csv') format 'csv';
   alter table part attach partition p1_e for values from (0) to (10);
   alter table part attach partition p2_e for values from (10) to (19);
   analyze part;
   create table p3 (a int, b int) distributed by (a);
   create table p4 (a int, b int) distributed by (a);
   alter table part attach partition p3 for values from (20) to (30);
   alter table part attach partition p4 for values from (30) to (40);
   insert into part select i,i from generate_series(25,35)i;
   analyze part;
   ```
   you can test this case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] ORCA: Fix detection of mixed storage in partitioned tables [cloudberry]

Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on code in PR #1524:
URL: https://github.com/apache/cloudberry/pull/1524#discussion_r2726201454


##########
src/backend/gporca/data/dxl/minidump/MixedPartitioned-ForeignHeap-Test.mdp:
##########
@@ -0,0 +1,17 @@
+<?xml version='1.0' encoding='UTF-8'?>
+<dxl:DXLMessage xmlns:dxl="http://greenplum.com/dxl/2010/12/"><dxl:Comment>

Review Comment:
   Any possiblity changing `greenplum.com` to `cloudberry.apache.org`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] ORCA: Fix detection of mixed storage in partitioned tables [cloudberry]

Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on PR #1524:
URL: https://github.com/apache/cloudberry/pull/1524#issuecomment-3791459184

   > purpose
   
   Added minidump to verify: 
   ```shell
   grep -o 'StorageType="[^"]*"' src/backend/gporca/data/dxl/minidump/MixedPartitioned-ForeignHeap-Test.mdp/MixedPartitioned-ForeignHeap-Test.mdp |sort|uniq
   StorageType="Foreign"
   StorageType="Heap"
   StorageType="MixedPartitioned"
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] ORCA: Fix detection of mixed storage in partitioned tables [cloudberry]

Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz merged PR #1524:
URL: https://github.com/apache/cloudberry/pull/1524


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] ORCA: Fix detection of mixed storage in partitioned tables [cloudberry]

Posted by "leborchuk (via GitHub)" <gi...@apache.org>.
leborchuk commented on code in PR #1524:
URL: https://github.com/apache/cloudberry/pull/1524#discussion_r2693358878


##########
src/backend/gpopt/translate/CTranslatorRelcacheToDXL.cpp:
##########
@@ -3147,18 +3147,36 @@ CTranslatorRelcacheToDXL::RetrieveStorageTypeForPartitionedTable(Relation rel)
 						"Queries with partitions of greenplum_fdw are not supported"));
 			}
 			GPOS_DELETE(fdw_name_str);
+
+			// Check for mixed storage before continuing
+			// If we already encountered non-foreign partitions, mark as mixed
+			if (rel_storage_type != IMDRelation::ErelstorageSentinel &&
+				rel_storage_type != IMDRelation::ErelstorageForeign)
+			{
+				// Already have non-foreign partition(s), now found foreign → mixed
+				rel_storage_type = IMDRelation::ErelstorageMixedPartitioned;
+			}
+			else if (rel_storage_type == IMDRelation::ErelstorageSentinel)
+			{
+				// First partition is foreign
+				rel_storage_type = IMDRelation::ErelstorageForeign;
+			}
 			continue;
 		}
 		all_foreign = false;
 		if (rel_storage_type == IMDRelation::ErelstorageSentinel)
 		{
 			rel_storage_type = child_storage;
 		}
-
+		else if (rel_storage_type == IMDRelation::ErelstorageForeign)
+		{
+			// Previously had foreign partition(s), now found non-foreign → mixed
+			rel_storage_type = IMDRelation::ErelstorageMixedPartitioned;
+		}
 		// mark any partitioned table with supported partitions of mixed storage types,
 		// this is more conservative for certain skans (eg: we can't do an index scan if any
 		// partition is ao, we must only do a sequential or bitmap scan)
-		if (rel_storage_type != child_storage)
+		else if (rel_storage_type != child_storage)

Review Comment:
   Quite cumbersome for me logic here. I missed continue in a case of Foreign partition. Yep, now understood all, thank you. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org