You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "felipepessoto (via GitHub)" <gi...@apache.org> on 2026/05/07 20:54:16 UTC

[I] Is Spark limited to split the Parquet read granularity by Row Group level only? [spark]

felipepessoto opened a new issue, #55747:
URL: https://github.com/apache/spark/issues/55747

   According to some articles I found:
   
   - https://cloudsqale.com/2021/03/19/spark-reading-parquet-why-the-number-of-tasks-can-be-much-larger-than-the-number-of-row-groups/
   - https://www.gresearch.com/news/parquet-files-know-your-scaling-limits/
   
   Seems Spark can only parallelize the reads across row groups. Is this a known limitation?
   
   Is there any way to split it by row or page level?
   
   If a file has a single row group, it means all the tasks, except 1, would be idle? And that one task would read the entire file?
   
   Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [I] Is Spark limited to split the Parquet read granularity by Row Group level only? [spark]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on issue #55747:
URL: https://github.com/apache/spark/issues/55747#issuecomment-4423183270

   @pan3793 is the row count limit available in Spark 4.1?
   
   Is there any plan in Spark to support more granular splits? cc @cloud-fan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [I] Is Spark limited to split the Parquet read granularity by Row Group level only? [spark]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #55747:
URL: https://github.com/apache/spark/issues/55747#issuecomment-4406154035

   yes, it's how it works for now. and I can contribute an extreme bad case https://github.com/apache/parquet-java/issues/3235


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [I] Is Spark limited to split the Parquet read granularity by Row Group level only? [spark]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #55747:
URL: https://github.com/apache/spark/issues/55747#issuecomment-4427827745

   > is the row count limit available in Spark 4.1?
   
   yes, spark 4.1 uses parquet 1.16.0, which includes that feature. you can try it with
   
   ```
   spark-sql --conf spark.hadoop.parquet.block.row.count.limit=100000000
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [I] Is Spark limited to split the Parquet read granularity by Row Group level only? [spark]

Posted by "acruise (via GitHub)" <gi...@apache.org>.
acruise commented on issue #55747:
URL: https://github.com/apache/spark/issues/55747#issuecomment-4682520139

   What would a good design look like? Is there a way to coalesce adjacent Scans in DSv2? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [I] Is Spark limited to split the Parquet read granularity by Row Group level only? [spark]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on issue #55747:
URL: https://github.com/apache/spark/issues/55747#issuecomment-4455404503

   I created an item in JIRA: https://issues.apache.org/jira/browse/SPARK-56871


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [I] Is Spark limited to split the Parquet read granularity by Row Group level only? [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #55747: Is Spark limited to split the Parquet read granularity by Row Group level only?
URL: https://github.com/apache/spark/issues/55747


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org