You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by "jiaqizho (via GitHub)" <gi...@apache.org> on 2025/03/28 08:22:46 UTC

[PR] [DNM]ORCA: allow partition table use the append node [cloudberry]

jiaqizho opened a new pull request, #1022:
URL: https://github.com/apache/cloudberry/pull/1022

   <!-- Thank you for your contribution to Apache Cloudberry (Incubating)! -->
   
   Fixes #ISSUE_Number
   
   ### What does this PR do?
   <!-- Brief overview of the changes, including any major features or fixes -->
   
   ### Type of Change
   - [ ] Bug fix (non-breaking change)
   - [ ] New feature (non-breaking change)
   - [ ] Breaking change (fix or feature with breaking changes)
   - [ ] Documentation update
   
   ### Breaking Changes
   <!-- Remove if not applicable. If yes, explain impact and migration path -->
   
   ### Test Plan
   <!-- How did you test these changes? -->
   - [ ] Unit tests added/updated
   - [ ] Integration tests added/updated
   - [ ] Passed `make installcheck`
   - [ ] Passed `make -C src/test installcheck-cbdb-parallel`
   
   ### Impact
   <!-- Remove sections that don't apply -->
   **Performance:**
   <!-- Any performance implications? -->
   
   **User-facing changes:**
   <!-- Any changes visible to users? -->
   
   **Dependencies:**
   <!-- New dependencies or version changes? -->
   
   ### Checklist
   - [ ] Followed [contribution guide](https://cloudberry.apache.org/contribute/code)
   - [ ] Added/updated documentation
   - [ ] Reviewed code for security implications
   - [ ] Requested review from [cloudberry committers](https://github.com/orgs/apache/teams/cloudberry-committers)
   
   ### Additional Context
   <!-- Any other information that would help reviewers? Remove if none -->
   
   ### CI Skip Instructions
   <!--
   To skip CI builds, add the appropriate CI skip identifier to your PR title.
   The identifier must:
   - Be in square brackets []
   - Include the word "ci" and either "skip" or "no"
   - Only use for documentation-only changes or when absolutely necessary
   -->
   
   ---
   <!-- Join our community:
   - Mailing list: [dev@cloudberry.apache.org](https://lists.apache.org/list.html?dev@cloudberry.apache.org) (subscribe: dev-subscribe@cloudberry.apache.org)
   - Discussions: https://github.com/apache/cloudberry/discussions -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] ORCA: allow partition table use the append node [cloudberry]

Posted by "zhangyue-hashdata (via GitHub)" <gi...@apache.org>.
zhangyue-hashdata commented on code in PR #1022:
URL: https://github.com/apache/cloudberry/pull/1022#discussion_r2156197767


##########
src/backend/gporca/libgpopt/src/translate/CTranslatorExprToDXL.cpp:
##########
@@ -1429,6 +1434,425 @@ CTranslatorExprToDXL::PdxlnDynamicTableScan(
 	return pdxlnDTS;
 }
 
+//---------------------------------------------------------------------------
+//	@function:
+//		CTranslatorExprToDXL::PdxlnDynamicScanToAppend
+//
+//	@doc:
+//		Create a DXL Append node from an optimizer
+//		dynamic scan or foreign scan node.
+//
+//---------------------------------------------------------------------------
+template <class PhysicalScanType>
+CDXLNode *
+CTranslatorExprToDXL::PdxlnDynamicScanToAppend(CExpression *pexprDTS,
+	CColRefArray *colref_array,
+	CDistributionSpecArray *pdrgpdsBaseTables,
+	CExpression *pexprScalarCond,
+	CDXLPhysicalProperties *dxl_properties) {
+	
+	PhysicalScanType *popDTS =
+		PhysicalScanType::PopConvert(pexprDTS->Pop());
+
+	ULongPtrArray *selector_ids = GPOS_NEW(m_mp) ULongPtrArray(m_mp);
+	CPartitionPropagationSpec *pps_reqd =
+		pexprDTS->Prpp()->Pepp()->PppsRequired();
+	if (pps_reqd->Contains(popDTS->ScanId()))
+	{
+		const CBitSet *bs = pps_reqd->SelectorIds(popDTS->ScanId());
+		CBitSetIter bsi(*bs);
+		for (ULONG ul = 0; bsi.Advance(); ul++)
+		{
+			selector_ids->Append(GPOS_NEW(m_mp) ULONG(bsi.Bit()));
+		}
+	}
+
+	// construct plan costs
+	CDXLPhysicalProperties *pdxlpropDTS = GetProperties(pexprDTS);
+
+	if (nullptr != dxl_properties)
+	{
+		CWStringDynamic *rows_out_str = GPOS_NEW(m_mp) CWStringDynamic(
+			m_mp,
+			dxl_properties->GetDXLOperatorCost()->GetRowsOutStr()->GetBuffer());
+		CWStringDynamic *pstrCost = GPOS_NEW(m_mp)
+			CWStringDynamic(m_mp, dxl_properties->GetDXLOperatorCost()
+									->GetTotalCostStr()
+									->GetBuffer());
+
+		pdxlpropDTS->GetDXLOperatorCost()->SetRows(rows_out_str);
+		pdxlpropDTS->GetDXLOperatorCost()->SetCost(pstrCost);
+		dxl_properties->Release();
+	}
+	GPOS_ASSERT(nullptr != pexprDTS->Prpp());
+
+	// construct projection list for top-level Append node
+	CColRefSet *pcrsOutput = pexprDTS->Prpp()->PcrsRequired();
+	CDXLNode *pdxlnPrLAppend = PdxlnProjList(pcrsOutput, colref_array);
+	CDXLTableDescr *root_dxl_table_descr = MakeDXLTableDescr(
+		popDTS->Ptabdesc(), popDTS->PdrgpcrOutput(), pexprDTS->Prpp());
+
+	// Construct the Append node - even when there is only one child partition.
+	// This is done for two reasons:
+	// * Dynamic partition pruning
+	//   Even if one partition is present in the statically pruned plan, we could
+	//   still dynamically prune it away. This needs an Append node.
+	// * Col mappings issues
+	//   When the first selected child partition's cols have different types/order
+	//   than the root partition, we can no longer re-use the colrefs of the root
+	//   partition, since colrefs are immutable. Thus, we create new colrefs for
+	//   this partition. But, if there is no Append (in case of just one selected
+	//   partition), then we also go through update all references above the DTS
+	//   with the new colrefs. For simplicity, we decided to keep the Append
+	//   around to maintain this projection (mapping) from the old root colrefs
+	//   to the first selected partition colrefs.
+	//
+	// GPDB_12_MERGE_FIXME: An Append on a single TableScan can be removed in
+	// CTranslatorDXLToPlstmt since these points do not apply there.
+	CDXLNode *pdxlnAppend = GPOS_NEW(m_mp) CDXLNode(
+		m_mp,
+		GPOS_NEW(m_mp) CDXLPhysicalAppend(m_mp, false, false, popDTS->ScanId(),
+										root_dxl_table_descr, selector_ids));
+	pdxlnAppend->SetProperties(pdxlpropDTS);
+	pdxlnAppend->AddChild(pdxlnPrLAppend);
+	pdxlnAppend->AddChild(PdxlnFilter(nullptr));
+
+	IMdIdArray *part_mdids = popDTS->GetPartitionMdids();
+	for (ULONG ul = 0; ul < part_mdids->Size(); ++ul)
+	{
+		IMDId *part_mdid = (*part_mdids)[ul];
+		const IMDRelation *part = m_pmda->RetrieveRel(part_mdid);
+
+		CTableDescriptor *part_tabdesc =

Review Comment:
   Is it better `CAutoP<CTableDescriptor> auto_part_tabdesc(part_tabdesc);`?



##########
src/backend/gporca/libnaucrates/src/operators/CDXLPhysicalAppend.cpp:
##########
@@ -113,9 +132,28 @@ CDXLPhysicalAppend::SerializeToDXL(CXMLSerializer *xml_serializer,
 	xml_serializer->AddAttribute(
 		CDXLTokens::GetDXLTokenStr(EdxltokenAppendIsZapped), m_is_zapped);
 
+	if (m_scan_id != gpos::ulong_max)
+	{
+		xml_serializer->AddAttribute(
+			CDXLTokens::GetDXLTokenStr(EdxltokenPartIndexId), m_scan_id);
+
+		CWStringDynamic *serialized_selector_ids =

Review Comment:
   Is it better `CAutoP<CWStringDynamic> serialized_selector_ids(xxx);`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] ORCA: allow partition table use the append node [cloudberry]

Posted by "my-ship-it (via GitHub)" <gi...@apache.org>.
my-ship-it commented on code in PR #1022:
URL: https://github.com/apache/cloudberry/pull/1022#discussion_r2153920844


##########
src/backend/gpopt/translate/CTranslatorQueryToDXL.cpp:
##########
@@ -528,6 +528,11 @@ CTranslatorQueryToDXL::CheckRangeTable(Query *query)
 			GPOS_RAISE(gpdxl::ExmaDXL, gpdxl::ExmiQuery2DXLUnsupportedFeature,
 					   GPOS_WSZ_LIT("TABLESAMPLE in the FROM clause"));
 		}
+
+		if (rte->relkind == 'p' && query->hasRowSecurity && GPOS_FTRACE(EopttraceDisableDynamicTableScan)) {

Review Comment:
   RELKIND_PARTITIONED_TABLE



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] ORCA: allow partition table use the append node [cloudberry]

Posted by "my-ship-it (via GitHub)" <gi...@apache.org>.
my-ship-it merged PR #1022:
URL: https://github.com/apache/cloudberry/pull/1022


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] [DNM]ORCA: allow partition table use the append node [cloudberry]

Posted by "jiaqizho (via GitHub)" <gi...@apache.org>.
jiaqizho commented on PR #1022:
URL: https://github.com/apache/cloudberry/pull/1022#issuecomment-2760533230

   Run CI with `optimizer_disable_dynamic_table_scan` turned on to ensure there are no other issues. 
   
   After that, i will added the test and set `optimizer_disable_dynamic_table_scan` default false.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] ORCA: allow partition table use the append node [cloudberry]

Posted by "jiaqizho (via GitHub)" <gi...@apache.org>.
jiaqizho commented on code in PR #1022:
URL: https://github.com/apache/cloudberry/pull/1022#discussion_r2157054377


##########
src/backend/gporca/libnaucrates/src/operators/CDXLPhysicalAppend.cpp:
##########
@@ -113,9 +132,28 @@ CDXLPhysicalAppend::SerializeToDXL(CXMLSerializer *xml_serializer,
 	xml_serializer->AddAttribute(
 		CDXLTokens::GetDXLTokenStr(EdxltokenAppendIsZapped), m_is_zapped);
 
+	if (m_scan_id != gpos::ulong_max)
+	{
+		xml_serializer->AddAttribute(
+			CDXLTokens::GetDXLTokenStr(EdxltokenPartIndexId), m_scan_id);
+
+		CWStringDynamic *serialized_selector_ids =

Review Comment:
   nope, the autoP in ORCA need inherit the `CRefCount`. `CWStringDynamic` can't inherit the  `CRefCount`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org