You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "anuragmantri (via GitHub)" <gi...@apache.org> on 2026/08/05 05:25:45 UTC
Re: [PR] [SPARK-58111][SQL] Scan and write schema narrowing for column-level UPDATE in DSv2 [spark]
anuragmantri commented on code in PR #55518:
URL: https://github.com/apache/spark/pull/55518#discussion_r3718167431
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteUpdateTable.scala:
##########
@@ -226,4 +389,98 @@ object RewriteUpdateTable extends RewriteRowLevelCommand {
val expandOutput = generateExpandOutput(attrs, outputs)
Expand(outputs, expandOutput, matchedRowsPlan)
}
+
+ /**
+ * Variant of `buildDeletesAndInserts` for the `SupportsColumnUpdates` narrow-scan path.
+ * This variant realigns the assignments to one value per surviving rowAttr padding unassigned
+ * rowAttrs with identity, so the reinsert output arity matches the delete output arity in
+ * the resulting Expand.
+ */
+ private def buildNarrowDeletesAndInserts(
+ matchedRowsPlan: LogicalPlan,
+ assignments: Seq[Assignment],
+ rowIdAttrs: Seq[Attribute]): Expand = {
+
+ val (metadataAttrs, rowAttrs) = matchedRowsPlan.output.partition { attr =>
+ MetadataAttribute.isValid(attr.metadata)
+ }
+ val assignmentMap = AttributeMap(assignments.collect {
+ case a @ Assignment(key: Attribute, _) => key -> a
+ })
+ val reinsertAssignments = rowAttrs.map { attr =>
+ assignmentMap.get(attr) match {
+ case Some(a) => a
+ case None => Assignment(attr, attr)
+ }
+ }
+ val deleteOutput = deltaDeleteOutput(rowAttrs, rowIdAttrs, metadataAttrs)
+ val insertOutput = deltaReinsertOutput(reinsertAssignments, metadataAttrs)
+ val outputs = Seq(deleteOutput, insertOutput)
+ val operationTypeAttr = AttributeReference(OPERATION_COLUMN, IntegerType, nullable = false)()
+ val attrs = operationTypeAttr +: matchedRowsPlan.output
+ val expandOutput = generateExpandOutput(attrs, outputs)
+ Expand(outputs, expandOutput, matchedRowsPlan)
+ }
+
+ /**
+ * Resolves the connector's `requiredDataAttributes()` if the operation opts into column
+ * updates. Returns `Nil` otherwise.
+ */
+ private def resolveConnectorDataAttrs(
+ relation: DataSourceV2Relation,
+ operation: RowLevelOperation): Seq[AttributeReference] = operation match {
+ case scu: SupportsColumnUpdates => resolveRequiredDataAttrs(relation, scu)
+ case _ => Nil
+ }
+
+ /**
+ * Computes the narrow set of data columns that must be present in the scan for a column-update
+ * write: connector-declared attrs, unioned with any table columns referenced by non-identity
+ * assignment RHS expressions, the operation condition, and the table's partition expressions.
+ * Partition-column refs are always kept so downstream rules (V2ScanPartitioningAndOrdering,
+ * GroupBasedRowLevelOperationScanPlanning) can resolve the table's partitioning expressions
+ * against the scan output.
+ */
+ private def computeNarrowReadAttrs(
+ relation: DataSourceV2Relation,
+ connectorDataAttrs: Seq[AttributeReference],
+ assignments: Seq[Assignment],
+ cond: Expression): Seq[AttributeReference] = {
+ val relationSet = relation.outputSet
+ val nonIdentityRhsRefs = assignments.iterator
+ .filterNot(a => a.key.isInstanceOf[Attribute] &&
+ isIdentityAssignment(a.key.asInstanceOf[Attribute], a.value))
+ .flatMap(_.value.references.toSeq)
+ .toSeq
+ val extraRefs = (cond.references.toSeq ++ nonIdentityRhsRefs)
+ .collect { case a: AttributeReference => a }
+ .filter(relationSet.contains)
+ val partitionRefNames = relation.table.partitioning().toImmutableArraySeq
Review Comment:
After speaking to @aokolnychyi, the current unconditional inclusion of partition source columns in the narrow scan is incorrect. It should be connector-driven, not implicit on Spark's side (e.g. only requested when the connector actually needs it for V2ScanPartitioningAndOrdering or write-side clustering).
The tension is that connectors may need a column visible in the scan (for planning/clustering resolution) but not in the write payload. I'm considering three options:
1. Add `scanOnlyDataAttributes()` to the mixin for columns the connector wants in the narrow scan but not in the write payload.
2. Use `requiredDataAttributes()` for everything. Connectors that need scan-side resolution declare it here, but by contract it then also flows into the write row, putting the burden on the connector to ignore/filter it when it's not being updated.
3. Use existing `requiredMetadataAttributes()` (we would need to expand its scope to allow certain data attributes). These don't flow into the narrow write data row, but they do reach the writer via the metadata slot, which conflicts with requiredMetadataAttributes()'s original scope of synthetic columns only.
I'm leaning towards option 1 as it seems cleanest but I'm open to any other suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org