You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 12:21:06 UTC
[I] Revisit HoodieSparkXXDataSourceV2ToV1Fallback [hudi]
hudi-bot opened a new issue, #17008:
URL: https://github.com/apache/hudi/issues/17008
Addressing https://github.com/apache/hudi/pull/12772#discussion_r2074437722
Revisit the classes {{{}HoodieSpark33DataSourceV2ToV1Fallback{}}}, {{{}HoodieSpark34DataSourceV2ToV1Fallback{}}}, {{{}HoodieSpark35DataSourceV2ToV1Fallback{}}}, and {{{}HoodieSpark40DataSourceV2ToV1Fallback{}}}. If there is no difference, we should unify them into a single class in Hudi spark common module.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9404
- Type: Improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Revisit HoodieSparkXXDataSourceV2ToV1Fallback [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #17008:
URL: https://github.com/apache/hudi/issues/17008#issuecomment-5391443617
This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-9404).
**Findings: confirmed not done - and the duplication has grown since this was filed.**
The description lists four classes (Spark 3.3, 3.4, 3.5, 4.0). Today `hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala` resolves **five** by fully-qualified name at `:61-69`:
```scala
"org.apache.spark.sql.hudi.analysis.HoodieSpark42DataSourceV2ToV1Fallback" // :61
"org.apache.spark.sql.hudi.analysis.HoodieSpark41DataSourceV2ToV1Fallback" // :63
"org.apache.spark.sql.hudi.analysis.HoodieSpark40DataSourceV2ToV1Fallback" // :65
"org.apache.spark.sql.hudi.analysis.HoodieSpark35DataSourceV2ToV1Fallback" // :67
"org.apache.spark.sql.hudi.analysis.HoodieSpark34DataSourceV2ToV1Fallback" // :69
```
Spark 4.1 and 4.2 variants were added; none were unified.
Context that matters for the "if there is no difference, unify them" premise: the supported-version matrix is currently widening, not narrowing. Spark 3.3 support was **dropped** by PR #18786 (`facb517ef957`, 2026-05-21) and then **reintroduced** by PR #18956 (`7c934f10f89b`, 2026-06-13). So a unification now has to hold across 3.4 through 4.2, and any future line gets added to whatever shape it lands in.
That argues for doing this sooner rather than later - each new Spark line makes it more expensive. Note the classes are resolved reflectively by FQN string, so a unified class would also let that lookup collapse.
Keeping this open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org