You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2026/04/09 03:34:09 UTC

Re: [PR] [SPARK-55748][SQL] Use `DSv2` for `avro|csv|json|kafka|orc|parquet|text` by default [spark]

LuciferYang commented on PR #54547:
URL: https://github.com/apache/spark/pull/54547#issuecomment-4211290758

   > > @dongjoon-hyun When I previously worked on File Source V2, the DSV2 framework was still relatively immature.
   > > Even today, there are still several missing pieces on the write path, including: • Partitioned writes • Bucketing • Custom partition locations (e.g., ALTER TABLE ... SET LOCATION) • Sort-before-write • Overwrite modes • etc.
   > > Revisiting and completing the migration could be an interesting project now that the DSV2 framework is much more mature. In addition, LLM-assisted development could help accelerate parts of the implementation.
   > 
   > If we don't have yet, can we open tickets for these known limitations of File Source V2 today? So we can have an explicit list to track and work on in order to achieve this goal.
   
   I have created https://issues.apache.org/jira/browse/SPARK-56170, and within the scope of my knowledge of the project, I have also created several tickets and submitted some prs:
   
   
   - https://github.com/apache/spark/pull/54998
   - https://github.com/apache/spark/pull/55034
   - https://github.com/apache/spark/pull/55091
   - https://github.com/apache/spark/pull/55108
   - https://github.com/apache/spark/pull/55127
   - https://github.com/apache/spark/pull/55128
   - https://github.com/apache/spark/pull/55129
   - https://github.com/apache/spark/pull/55138
   - https://github.com/apache/spark/pull/55154
   - https://github.com/apache/spark/pull/55229
   - https://github.com/apache/spark/pull/55231
   - https://github.com/apache/spark/pull/55232


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org