You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Varun Lakhyani <la...@ce.iitr.ac.in> on 2026/03/17 18:49:09 UTC
[DISCUSS] GSoC 2026 idea vetting: Parallel scan task execution in Iceberg Spark readers
Hey All,
I previously started a discussion on making Spark readers work in parallel
(asynchronously), which is beneficial in cases with large numbers of small
files such as compaction, and I have worked on a POC, high-level design,
implementation, and benchmarking for various scenarios. I presented my
approach and benchmarking results in the Iceberg Spark sync; the recording
may be available in the Iceberg Spark Community Sync Notes [0].
I am planning to submit this work as a GSoC 2026 proposal based on this
idea and was advised to seek formal community vetting on the dev mailing
list.
Previous DISCUSS thread:
https://lists.apache.org/thread/b5jrlyv61lmw867kksw05sot2tro5ybn
Issue:
https://github.com/apache/iceberg/issues/15287
Prototype implementation:
https://github.com/apache/iceberg/pull/15341
Design document and benchmarking details:
https://docs.google.com/document/d/17vBz5t-gSDdmB0S40MYRceyvmcBSzw9Gii-FcU97Lds/edit?usp=sharing
Initial benchmarking shows noticeable improvements for workloads involving
many small files, particularly when IO latency is present (details in the
design document).
Any feedback (+1 / concerns / suggestions) would be appreciated.
I am specifically looking for community consensus on whether this is a
viable direction for Iceberg before formalizing the GSoC proposal. The GSoC
2026 proposal deadline is March 31 - early feedback would be especially
appreciated.
[0] Iceberg Spark Community Sync Notes:
https://docs.google.com/document/d/19nno1RoPznbbxKOZZddZNHHafa7XULjbN6RPExdr2n4/edit?usp=sharing
--
Lakhyani Varun
Indian Institute of Technology Roorkee
Re: [DISCUSS] GSoC 2026 idea vetting: Parallel scan task execution in Iceberg Spark readers
Posted by Péter Váry <pe...@gmail.com>.
+1
On Sat, Mar 21, 2026, 06:18 Neelesh Salian <ne...@gmail.com> wrote:
> +1. Thanks for doing this.
>
> On Tue, Mar 17, 2026 at 11:49 Varun Lakhyani <la...@ce.iitr.ac.in>
> wrote:
>
>> Hey All,
>>
>> I previously started a discussion on making Spark readers work in
>> parallel (asynchronously), which is beneficial in cases with large numbers
>> of small files such as compaction, and I have worked on a POC, high-level
>> design, implementation, and benchmarking for various scenarios. I presented
>> my approach and benchmarking results in the Iceberg Spark sync; the
>> recording may be available in the Iceberg Spark Community Sync Notes [0].
>>
>> I am planning to submit this work as a GSoC 2026 proposal based on this
>> idea and was advised to seek formal community vetting on the dev mailing
>> list.
>>
>> Previous DISCUSS thread:
>> https://lists.apache.org/thread/b5jrlyv61lmw867kksw05sot2tro5ybn
>>
>> Issue:
>> https://github.com/apache/iceberg/issues/15287
>>
>> Prototype implementation:
>> https://github.com/apache/iceberg/pull/15341
>>
>> Design document and benchmarking details:
>>
>> https://docs.google.com/document/d/17vBz5t-gSDdmB0S40MYRceyvmcBSzw9Gii-FcU97Lds/edit?usp=sharing
>>
>> Initial benchmarking shows noticeable improvements for workloads
>> involving many small files, particularly when IO latency is present
>> (details in the design document).
>>
>> Any feedback (+1 / concerns / suggestions) would be appreciated.
>> I am specifically looking for community consensus on whether this is a
>> viable direction for Iceberg before formalizing the GSoC proposal. The GSoC
>> 2026 proposal deadline is March 31 - early feedback would be especially
>> appreciated.
>>
>> [0] Iceberg Spark Community Sync Notes:
>> https://docs.google.com/document/d/19nno1RoPznbbxKOZZddZNHHafa7XULjbN6RPExdr2n4/edit?usp=sharing
>> --
>> Lakhyani Varun
>> Indian Institute of Technology Roorkee
>>
>>
Re: [DISCUSS] GSoC 2026 idea vetting: Parallel scan task execution in Iceberg Spark readers
Posted by Neelesh Salian <ne...@gmail.com>.
+1. Thanks for doing this.
On Tue, Mar 17, 2026 at 11:49 Varun Lakhyani <la...@ce.iitr.ac.in>
wrote:
> Hey All,
>
> I previously started a discussion on making Spark readers work in parallel
> (asynchronously), which is beneficial in cases with large numbers of small
> files such as compaction, and I have worked on a POC, high-level design,
> implementation, and benchmarking for various scenarios. I presented my
> approach and benchmarking results in the Iceberg Spark sync; the recording
> may be available in the Iceberg Spark Community Sync Notes [0].
>
> I am planning to submit this work as a GSoC 2026 proposal based on this
> idea and was advised to seek formal community vetting on the dev mailing
> list.
>
> Previous DISCUSS thread:
> https://lists.apache.org/thread/b5jrlyv61lmw867kksw05sot2tro5ybn
>
> Issue:
> https://github.com/apache/iceberg/issues/15287
>
> Prototype implementation:
> https://github.com/apache/iceberg/pull/15341
>
> Design document and benchmarking details:
>
> https://docs.google.com/document/d/17vBz5t-gSDdmB0S40MYRceyvmcBSzw9Gii-FcU97Lds/edit?usp=sharing
>
> Initial benchmarking shows noticeable improvements for workloads involving
> many small files, particularly when IO latency is present (details in the
> design document).
>
> Any feedback (+1 / concerns / suggestions) would be appreciated.
> I am specifically looking for community consensus on whether this is a
> viable direction for Iceberg before formalizing the GSoC proposal. The GSoC
> 2026 proposal deadline is March 31 - early feedback would be especially
> appreciated.
>
> [0] Iceberg Spark Community Sync Notes:
> https://docs.google.com/document/d/19nno1RoPznbbxKOZZddZNHHafa7XULjbN6RPExdr2n4/edit?usp=sharing
> --
> Lakhyani Varun
> Indian Institute of Technology Roorkee
>
>
Re: [DISCUSS] GSoC 2026 idea vetting: Parallel scan task execution in Iceberg Spark readers
Posted by huaxin gao <hu...@gmail.com>.
+1 from me as well.
On Fri, Mar 20, 2026 at 9:37 PM Holden Karau <ho...@gmail.com> wrote:
> This seems like a great idea to explore.
>
> Twitter: https://twitter.com/holdenkarau
> Fight Health Insurance: https://www.fighthealthinsurance.com/
> <https://www.fighthealthinsurance.com/?q=hk_email>
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> Pronouns: she/her
>
>
> On Fri, Mar 20, 2026 at 7:12 PM Prashant Singh <pr...@gmail.com>
> wrote:
>
>> +1 from me too.
>>
>> On Fri, Mar 20, 2026 at 3:03 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> +1 from me. This looks like a useful improvement, especially for small
>>> files and IO-heavy workloads.
>>>
>>> Yufei
>>>
>>>
>>> On Fri, Mar 20, 2026 at 2:27 PM Anurag Mantripragada <
>>> anuragmantripragada@apache.org> wrote:
>>>
>>>> +1. This will be a good improvement for compaction and other use-cases
>>>> and initial numbers are already promising. I support this project for GSoC
>>>> 2026.
>>>>
>>>> ~ Anurag
>>>>
>>>> On Fri, Mar 20, 2026 at 1:32 PM Russell Spitzer <
>>>> russell.spitzer@gmail.com> wrote:
>>>>
>>>>> +1 I think this would be a great project to work on and I would be
>>>>> glad to support working on it for GSoC 2026
>>>>>
>>>>> On Fri, Mar 20, 2026 at 2:12 PM Varun Lakhyani <
>>>>> lakhyani_vg@ce.iitr.ac.in> wrote:
>>>>>
>>>>>> Benchmarked it against real cloud storage AWS S3 (1000 files - 14.6
>>>>>> Kb each) :
>>>>>>
>>>>>> - Sync time = 219.694 s
>>>>>> - Async time = 51.853 s
>>>>>>
>>>>>> % Improvement = 76.4%
>>>>>> It can be seen as cloud storage has high IO overheads so, async flow
>>>>>> can be beneficial for small files.
>>>>>>
>>>>>> I would really appreciate any feedback on this.
>>>>>>
>>>>>> On Wed, Mar 18, 2026 at 12:19 AM Varun Lakhyani <
>>>>>> lakhyani_vg@ce.iitr.ac.in> wrote:
>>>>>>
>>>>>>> Hey All,
>>>>>>>
>>>>>>> I previously started a discussion on making Spark readers work in
>>>>>>> parallel (asynchronously), which is beneficial in cases with large numbers
>>>>>>> of small files such as compaction, and I have worked on a POC, high-level
>>>>>>> design, implementation, and benchmarking for various scenarios. I presented
>>>>>>> my approach and benchmarking results in the Iceberg Spark sync; the
>>>>>>> recording may be available in the Iceberg Spark Community Sync Notes [0].
>>>>>>>
>>>>>>> I am planning to submit this work as a GSoC 2026 proposal based on
>>>>>>> this idea and was advised to seek formal community vetting on the dev
>>>>>>> mailing list.
>>>>>>>
>>>>>>> Previous DISCUSS thread:
>>>>>>> https://lists.apache.org/thread/b5jrlyv61lmw867kksw05sot2tro5ybn
>>>>>>>
>>>>>>> Issue:
>>>>>>> https://github.com/apache/iceberg/issues/15287
>>>>>>>
>>>>>>> Prototype implementation:
>>>>>>> https://github.com/apache/iceberg/pull/15341
>>>>>>>
>>>>>>> Design document and benchmarking details:
>>>>>>>
>>>>>>> https://docs.google.com/document/d/17vBz5t-gSDdmB0S40MYRceyvmcBSzw9Gii-FcU97Lds/edit?usp=sharing
>>>>>>>
>>>>>>> Initial benchmarking shows noticeable improvements for workloads
>>>>>>> involving many small files, particularly when IO latency is present
>>>>>>> (details in the design document).
>>>>>>>
>>>>>>> Any feedback (+1 / concerns / suggestions) would be appreciated.
>>>>>>> I am specifically looking for community consensus on whether this is
>>>>>>> a viable direction for Iceberg before formalizing the GSoC proposal. The
>>>>>>> GSoC 2026 proposal deadline is March 31 - early feedback would be
>>>>>>> especially appreciated.
>>>>>>>
>>>>>>> [0] Iceberg Spark Community Sync Notes:
>>>>>>> https://docs.google.com/document/d/19nno1RoPznbbxKOZZddZNHHafa7XULjbN6RPExdr2n4/edit?usp=sharing
>>>>>>> --
>>>>>>> Lakhyani Varun
>>>>>>> Indian Institute of Technology Roorkee
>>>>>>>
>>>>>>>
Re: [DISCUSS] GSoC 2026 idea vetting: Parallel scan task execution in Iceberg Spark readers
Posted by Holden Karau <ho...@gmail.com>.
This seems like a great idea to explore.
Twitter: https://twitter.com/holdenkarau
Fight Health Insurance: https://www.fighthealthinsurance.com/
<https://www.fighthealthinsurance.com/?q=hk_email>
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Pronouns: she/her
On Fri, Mar 20, 2026 at 7:12 PM Prashant Singh <pr...@gmail.com>
wrote:
> +1 from me too.
>
> On Fri, Mar 20, 2026 at 3:03 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> +1 from me. This looks like a useful improvement, especially for small
>> files and IO-heavy workloads.
>>
>> Yufei
>>
>>
>> On Fri, Mar 20, 2026 at 2:27 PM Anurag Mantripragada <
>> anuragmantripragada@apache.org> wrote:
>>
>>> +1. This will be a good improvement for compaction and other use-cases
>>> and initial numbers are already promising. I support this project for GSoC
>>> 2026.
>>>
>>> ~ Anurag
>>>
>>> On Fri, Mar 20, 2026 at 1:32 PM Russell Spitzer <
>>> russell.spitzer@gmail.com> wrote:
>>>
>>>> +1 I think this would be a great project to work on and I would be glad
>>>> to support working on it for GSoC 2026
>>>>
>>>> On Fri, Mar 20, 2026 at 2:12 PM Varun Lakhyani <
>>>> lakhyani_vg@ce.iitr.ac.in> wrote:
>>>>
>>>>> Benchmarked it against real cloud storage AWS S3 (1000 files - 14.6 Kb
>>>>> each) :
>>>>>
>>>>> - Sync time = 219.694 s
>>>>> - Async time = 51.853 s
>>>>>
>>>>> % Improvement = 76.4%
>>>>> It can be seen as cloud storage has high IO overheads so, async flow
>>>>> can be beneficial for small files.
>>>>>
>>>>> I would really appreciate any feedback on this.
>>>>>
>>>>> On Wed, Mar 18, 2026 at 12:19 AM Varun Lakhyani <
>>>>> lakhyani_vg@ce.iitr.ac.in> wrote:
>>>>>
>>>>>> Hey All,
>>>>>>
>>>>>> I previously started a discussion on making Spark readers work in
>>>>>> parallel (asynchronously), which is beneficial in cases with large numbers
>>>>>> of small files such as compaction, and I have worked on a POC, high-level
>>>>>> design, implementation, and benchmarking for various scenarios. I presented
>>>>>> my approach and benchmarking results in the Iceberg Spark sync; the
>>>>>> recording may be available in the Iceberg Spark Community Sync Notes [0].
>>>>>>
>>>>>> I am planning to submit this work as a GSoC 2026 proposal based on
>>>>>> this idea and was advised to seek formal community vetting on the dev
>>>>>> mailing list.
>>>>>>
>>>>>> Previous DISCUSS thread:
>>>>>> https://lists.apache.org/thread/b5jrlyv61lmw867kksw05sot2tro5ybn
>>>>>>
>>>>>> Issue:
>>>>>> https://github.com/apache/iceberg/issues/15287
>>>>>>
>>>>>> Prototype implementation:
>>>>>> https://github.com/apache/iceberg/pull/15341
>>>>>>
>>>>>> Design document and benchmarking details:
>>>>>>
>>>>>> https://docs.google.com/document/d/17vBz5t-gSDdmB0S40MYRceyvmcBSzw9Gii-FcU97Lds/edit?usp=sharing
>>>>>>
>>>>>> Initial benchmarking shows noticeable improvements for workloads
>>>>>> involving many small files, particularly when IO latency is present
>>>>>> (details in the design document).
>>>>>>
>>>>>> Any feedback (+1 / concerns / suggestions) would be appreciated.
>>>>>> I am specifically looking for community consensus on whether this is
>>>>>> a viable direction for Iceberg before formalizing the GSoC proposal. The
>>>>>> GSoC 2026 proposal deadline is March 31 - early feedback would be
>>>>>> especially appreciated.
>>>>>>
>>>>>> [0] Iceberg Spark Community Sync Notes:
>>>>>> https://docs.google.com/document/d/19nno1RoPznbbxKOZZddZNHHafa7XULjbN6RPExdr2n4/edit?usp=sharing
>>>>>> --
>>>>>> Lakhyani Varun
>>>>>> Indian Institute of Technology Roorkee
>>>>>>
>>>>>>
Re: [DISCUSS] GSoC 2026 idea vetting: Parallel scan task execution in Iceberg Spark readers
Posted by Prashant Singh <pr...@gmail.com>.
+1 from me too.
On Fri, Mar 20, 2026 at 3:03 PM Yufei Gu <fl...@gmail.com> wrote:
> +1 from me. This looks like a useful improvement, especially for small
> files and IO-heavy workloads.
>
> Yufei
>
>
> On Fri, Mar 20, 2026 at 2:27 PM Anurag Mantripragada <
> anuragmantripragada@apache.org> wrote:
>
>> +1. This will be a good improvement for compaction and other use-cases
>> and initial numbers are already promising. I support this project for GSoC
>> 2026.
>>
>> ~ Anurag
>>
>> On Fri, Mar 20, 2026 at 1:32 PM Russell Spitzer <
>> russell.spitzer@gmail.com> wrote:
>>
>>> +1 I think this would be a great project to work on and I would be glad
>>> to support working on it for GSoC 2026
>>>
>>> On Fri, Mar 20, 2026 at 2:12 PM Varun Lakhyani <
>>> lakhyani_vg@ce.iitr.ac.in> wrote:
>>>
>>>> Benchmarked it against real cloud storage AWS S3 (1000 files - 14.6 Kb
>>>> each) :
>>>>
>>>> - Sync time = 219.694 s
>>>> - Async time = 51.853 s
>>>>
>>>> % Improvement = 76.4%
>>>> It can be seen as cloud storage has high IO overheads so, async flow
>>>> can be beneficial for small files.
>>>>
>>>> I would really appreciate any feedback on this.
>>>>
>>>> On Wed, Mar 18, 2026 at 12:19 AM Varun Lakhyani <
>>>> lakhyani_vg@ce.iitr.ac.in> wrote:
>>>>
>>>>> Hey All,
>>>>>
>>>>> I previously started a discussion on making Spark readers work in
>>>>> parallel (asynchronously), which is beneficial in cases with large numbers
>>>>> of small files such as compaction, and I have worked on a POC, high-level
>>>>> design, implementation, and benchmarking for various scenarios. I presented
>>>>> my approach and benchmarking results in the Iceberg Spark sync; the
>>>>> recording may be available in the Iceberg Spark Community Sync Notes [0].
>>>>>
>>>>> I am planning to submit this work as a GSoC 2026 proposal based on
>>>>> this idea and was advised to seek formal community vetting on the dev
>>>>> mailing list.
>>>>>
>>>>> Previous DISCUSS thread:
>>>>> https://lists.apache.org/thread/b5jrlyv61lmw867kksw05sot2tro5ybn
>>>>>
>>>>> Issue:
>>>>> https://github.com/apache/iceberg/issues/15287
>>>>>
>>>>> Prototype implementation:
>>>>> https://github.com/apache/iceberg/pull/15341
>>>>>
>>>>> Design document and benchmarking details:
>>>>>
>>>>> https://docs.google.com/document/d/17vBz5t-gSDdmB0S40MYRceyvmcBSzw9Gii-FcU97Lds/edit?usp=sharing
>>>>>
>>>>> Initial benchmarking shows noticeable improvements for workloads
>>>>> involving many small files, particularly when IO latency is present
>>>>> (details in the design document).
>>>>>
>>>>> Any feedback (+1 / concerns / suggestions) would be appreciated.
>>>>> I am specifically looking for community consensus on whether this is a
>>>>> viable direction for Iceberg before formalizing the GSoC proposal. The GSoC
>>>>> 2026 proposal deadline is March 31 - early feedback would be especially
>>>>> appreciated.
>>>>>
>>>>> [0] Iceberg Spark Community Sync Notes:
>>>>> https://docs.google.com/document/d/19nno1RoPznbbxKOZZddZNHHafa7XULjbN6RPExdr2n4/edit?usp=sharing
>>>>> --
>>>>> Lakhyani Varun
>>>>> Indian Institute of Technology Roorkee
>>>>>
>>>>>
Re: [DISCUSS] GSoC 2026 idea vetting: Parallel scan task execution in Iceberg Spark readers
Posted by Yufei Gu <fl...@gmail.com>.
+1 from me. This looks like a useful improvement, especially for small
files and IO-heavy workloads.
Yufei
On Fri, Mar 20, 2026 at 2:27 PM Anurag Mantripragada <
anuragmantripragada@apache.org> wrote:
> +1. This will be a good improvement for compaction and other use-cases and
> initial numbers are already promising. I support this project for GSoC 2026.
>
> ~ Anurag
>
> On Fri, Mar 20, 2026 at 1:32 PM Russell Spitzer <ru...@gmail.com>
> wrote:
>
>> +1 I think this would be a great project to work on and I would be glad
>> to support working on it for GSoC 2026
>>
>> On Fri, Mar 20, 2026 at 2:12 PM Varun Lakhyani <la...@ce.iitr.ac.in>
>> wrote:
>>
>>> Benchmarked it against real cloud storage AWS S3 (1000 files - 14.6 Kb
>>> each) :
>>>
>>> - Sync time = 219.694 s
>>> - Async time = 51.853 s
>>>
>>> % Improvement = 76.4%
>>> It can be seen as cloud storage has high IO overheads so, async flow can
>>> be beneficial for small files.
>>>
>>> I would really appreciate any feedback on this.
>>>
>>> On Wed, Mar 18, 2026 at 12:19 AM Varun Lakhyani <
>>> lakhyani_vg@ce.iitr.ac.in> wrote:
>>>
>>>> Hey All,
>>>>
>>>> I previously started a discussion on making Spark readers work in
>>>> parallel (asynchronously), which is beneficial in cases with large numbers
>>>> of small files such as compaction, and I have worked on a POC, high-level
>>>> design, implementation, and benchmarking for various scenarios. I presented
>>>> my approach and benchmarking results in the Iceberg Spark sync; the
>>>> recording may be available in the Iceberg Spark Community Sync Notes [0].
>>>>
>>>> I am planning to submit this work as a GSoC 2026 proposal based on this
>>>> idea and was advised to seek formal community vetting on the dev mailing
>>>> list.
>>>>
>>>> Previous DISCUSS thread:
>>>> https://lists.apache.org/thread/b5jrlyv61lmw867kksw05sot2tro5ybn
>>>>
>>>> Issue:
>>>> https://github.com/apache/iceberg/issues/15287
>>>>
>>>> Prototype implementation:
>>>> https://github.com/apache/iceberg/pull/15341
>>>>
>>>> Design document and benchmarking details:
>>>>
>>>> https://docs.google.com/document/d/17vBz5t-gSDdmB0S40MYRceyvmcBSzw9Gii-FcU97Lds/edit?usp=sharing
>>>>
>>>> Initial benchmarking shows noticeable improvements for workloads
>>>> involving many small files, particularly when IO latency is present
>>>> (details in the design document).
>>>>
>>>> Any feedback (+1 / concerns / suggestions) would be appreciated.
>>>> I am specifically looking for community consensus on whether this is a
>>>> viable direction for Iceberg before formalizing the GSoC proposal. The GSoC
>>>> 2026 proposal deadline is March 31 - early feedback would be especially
>>>> appreciated.
>>>>
>>>> [0] Iceberg Spark Community Sync Notes:
>>>> https://docs.google.com/document/d/19nno1RoPznbbxKOZZddZNHHafa7XULjbN6RPExdr2n4/edit?usp=sharing
>>>> --
>>>> Lakhyani Varun
>>>> Indian Institute of Technology Roorkee
>>>>
>>>>
Re: [DISCUSS] GSoC 2026 idea vetting: Parallel scan task execution in Iceberg Spark readers
Posted by Anurag Mantripragada <an...@apache.org>.
+1. This will be a good improvement for compaction and other use-cases and
initial numbers are already promising. I support this project for GSoC 2026.
~ Anurag
On Fri, Mar 20, 2026 at 1:32 PM Russell Spitzer <ru...@gmail.com>
wrote:
> +1 I think this would be a great project to work on and I would be glad to
> support working on it for GSoC 2026
>
> On Fri, Mar 20, 2026 at 2:12 PM Varun Lakhyani <la...@ce.iitr.ac.in>
> wrote:
>
>> Benchmarked it against real cloud storage AWS S3 (1000 files - 14.6 Kb
>> each) :
>>
>> - Sync time = 219.694 s
>> - Async time = 51.853 s
>>
>> % Improvement = 76.4%
>> It can be seen as cloud storage has high IO overheads so, async flow can
>> be beneficial for small files.
>>
>> I would really appreciate any feedback on this.
>>
>> On Wed, Mar 18, 2026 at 12:19 AM Varun Lakhyani <
>> lakhyani_vg@ce.iitr.ac.in> wrote:
>>
>>> Hey All,
>>>
>>> I previously started a discussion on making Spark readers work in
>>> parallel (asynchronously), which is beneficial in cases with large numbers
>>> of small files such as compaction, and I have worked on a POC, high-level
>>> design, implementation, and benchmarking for various scenarios. I presented
>>> my approach and benchmarking results in the Iceberg Spark sync; the
>>> recording may be available in the Iceberg Spark Community Sync Notes [0].
>>>
>>> I am planning to submit this work as a GSoC 2026 proposal based on this
>>> idea and was advised to seek formal community vetting on the dev mailing
>>> list.
>>>
>>> Previous DISCUSS thread:
>>> https://lists.apache.org/thread/b5jrlyv61lmw867kksw05sot2tro5ybn
>>>
>>> Issue:
>>> https://github.com/apache/iceberg/issues/15287
>>>
>>> Prototype implementation:
>>> https://github.com/apache/iceberg/pull/15341
>>>
>>> Design document and benchmarking details:
>>>
>>> https://docs.google.com/document/d/17vBz5t-gSDdmB0S40MYRceyvmcBSzw9Gii-FcU97Lds/edit?usp=sharing
>>>
>>> Initial benchmarking shows noticeable improvements for workloads
>>> involving many small files, particularly when IO latency is present
>>> (details in the design document).
>>>
>>> Any feedback (+1 / concerns / suggestions) would be appreciated.
>>> I am specifically looking for community consensus on whether this is a
>>> viable direction for Iceberg before formalizing the GSoC proposal. The GSoC
>>> 2026 proposal deadline is March 31 - early feedback would be especially
>>> appreciated.
>>>
>>> [0] Iceberg Spark Community Sync Notes:
>>> https://docs.google.com/document/d/19nno1RoPznbbxKOZZddZNHHafa7XULjbN6RPExdr2n4/edit?usp=sharing
>>> --
>>> Lakhyani Varun
>>> Indian Institute of Technology Roorkee
>>>
>>>
Re: [DISCUSS] GSoC 2026 idea vetting: Parallel scan task execution in Iceberg Spark readers
Posted by Russell Spitzer <ru...@gmail.com>.
+1 I think this would be a great project to work on and I would be glad to
support working on it for GSoC 2026
On Fri, Mar 20, 2026 at 2:12 PM Varun Lakhyani <la...@ce.iitr.ac.in>
wrote:
> Benchmarked it against real cloud storage AWS S3 (1000 files - 14.6 Kb
> each) :
>
> - Sync time = 219.694 s
> - Async time = 51.853 s
>
> % Improvement = 76.4%
> It can be seen as cloud storage has high IO overheads so, async flow can
> be beneficial for small files.
>
> I would really appreciate any feedback on this.
>
> On Wed, Mar 18, 2026 at 12:19 AM Varun Lakhyani <la...@ce.iitr.ac.in>
> wrote:
>
>> Hey All,
>>
>> I previously started a discussion on making Spark readers work in
>> parallel (asynchronously), which is beneficial in cases with large numbers
>> of small files such as compaction, and I have worked on a POC, high-level
>> design, implementation, and benchmarking for various scenarios. I presented
>> my approach and benchmarking results in the Iceberg Spark sync; the
>> recording may be available in the Iceberg Spark Community Sync Notes [0].
>>
>> I am planning to submit this work as a GSoC 2026 proposal based on this
>> idea and was advised to seek formal community vetting on the dev mailing
>> list.
>>
>> Previous DISCUSS thread:
>> https://lists.apache.org/thread/b5jrlyv61lmw867kksw05sot2tro5ybn
>>
>> Issue:
>> https://github.com/apache/iceberg/issues/15287
>>
>> Prototype implementation:
>> https://github.com/apache/iceberg/pull/15341
>>
>> Design document and benchmarking details:
>>
>> https://docs.google.com/document/d/17vBz5t-gSDdmB0S40MYRceyvmcBSzw9Gii-FcU97Lds/edit?usp=sharing
>>
>> Initial benchmarking shows noticeable improvements for workloads
>> involving many small files, particularly when IO latency is present
>> (details in the design document).
>>
>> Any feedback (+1 / concerns / suggestions) would be appreciated.
>> I am specifically looking for community consensus on whether this is a
>> viable direction for Iceberg before formalizing the GSoC proposal. The GSoC
>> 2026 proposal deadline is March 31 - early feedback would be especially
>> appreciated.
>>
>> [0] Iceberg Spark Community Sync Notes:
>> https://docs.google.com/document/d/19nno1RoPznbbxKOZZddZNHHafa7XULjbN6RPExdr2n4/edit?usp=sharing
>> --
>> Lakhyani Varun
>> Indian Institute of Technology Roorkee
>>
>>
Re: [DISCUSS] GSoC 2026 idea vetting: Parallel scan task execution in Iceberg Spark readers
Posted by Varun Lakhyani <la...@ce.iitr.ac.in>.
Benchmarked it against real cloud storage AWS S3 (1000 files - 14.6 Kb
each) :
- Sync time = 219.694 s
- Async time = 51.853 s
% Improvement = 76.4%
It can be seen as cloud storage has high IO overheads so, async flow can be
beneficial for small files.
I would really appreciate any feedback on this.
On Wed, Mar 18, 2026 at 12:19 AM Varun Lakhyani <la...@ce.iitr.ac.in>
wrote:
> Hey All,
>
> I previously started a discussion on making Spark readers work in parallel
> (asynchronously), which is beneficial in cases with large numbers of small
> files such as compaction, and I have worked on a POC, high-level design,
> implementation, and benchmarking for various scenarios. I presented my
> approach and benchmarking results in the Iceberg Spark sync; the recording
> may be available in the Iceberg Spark Community Sync Notes [0].
>
> I am planning to submit this work as a GSoC 2026 proposal based on this
> idea and was advised to seek formal community vetting on the dev mailing
> list.
>
> Previous DISCUSS thread:
> https://lists.apache.org/thread/b5jrlyv61lmw867kksw05sot2tro5ybn
>
> Issue:
> https://github.com/apache/iceberg/issues/15287
>
> Prototype implementation:
> https://github.com/apache/iceberg/pull/15341
>
> Design document and benchmarking details:
>
> https://docs.google.com/document/d/17vBz5t-gSDdmB0S40MYRceyvmcBSzw9Gii-FcU97Lds/edit?usp=sharing
>
> Initial benchmarking shows noticeable improvements for workloads involving
> many small files, particularly when IO latency is present (details in the
> design document).
>
> Any feedback (+1 / concerns / suggestions) would be appreciated.
> I am specifically looking for community consensus on whether this is a
> viable direction for Iceberg before formalizing the GSoC proposal. The GSoC
> 2026 proposal deadline is March 31 - early feedback would be especially
> appreciated.
>
> [0] Iceberg Spark Community Sync Notes:
> https://docs.google.com/document/d/19nno1RoPznbbxKOZZddZNHHafa7XULjbN6RPExdr2n4/edit?usp=sharing
> --
> Lakhyani Varun
> Indian Institute of Technology Roorkee
>
>
Re: [DISCUSS] GSoC 2026 idea vetting: Parallel scan task execution in Iceberg Spark readers
Posted by Varun Lakhyani <la...@ce.iitr.ac.in>.
Thanks a lot to everyone, I am selected for GSoC 2026 with Apache Software
Foundation as an org and Apache Iceberg as a project.
I will be working on this idea with Russell Spitzer mentoring me and will
keep the community posted.
On Tue, Mar 31, 2026 at 7:40 PM Varun Lakhyani <la...@ce.iitr.ac.in>
wrote:
> Thanks a lot - Russell Spitzer has kindly agreed to mentor this project.
>
> I would also like to confirm if there are any formal steps required for
> mentor registration on the GSoC/ASF side for such vetted ideas. If so, I
> would appreciate any guidance to ensure everything is in order before the
> deadline.
>
> On Tue, Mar 31, 2026 at 6:13 PM Varun Lakhyani <la...@ce.iitr.ac.in>
> wrote:
>
>> Thanks a lot to everyone for vetting this proposal.
>>
>> Today's the deadline for submitting proposals and I have mine ready but I
>> got to know that before 1st April (Tomorrow) all proposals must have
>> accepted mentors.
>> I am looking for a community member whom I list as mentor and If possible
>> I would need someone to register as a mentor for ASF organization and
>> approve my proposal.
>> I have applied using email id: lakhiyanivarun14@gmail.com
>> Proposal that I will be submitting:
>> https://docs.google.com/document/d/1ZEgzQj1cxt1fQLXh7auZE7E1xCDGkTNt7dNpA0PaG7U/edit?usp=sharing
>> :
>>
>> I have already reached out to Russell Spitzer and Peter Vary directly,
>> but given the tight timeline I wanted to flag this to the broader community
>> as well.
>> Thanks and Apologies for last minute request
>>
>> On Wed, Mar 18, 2026 at 12:19 AM Varun Lakhyani <
>> lakhyani_vg@ce.iitr.ac.in> wrote:
>>
>>> Hey All,
>>>
>>> I previously started a discussion on making Spark readers work in
>>> parallel (asynchronously), which is beneficial in cases with large numbers
>>> of small files such as compaction, and I have worked on a POC, high-level
>>> design, implementation, and benchmarking for various scenarios. I presented
>>> my approach and benchmarking results in the Iceberg Spark sync; the
>>> recording may be available in the Iceberg Spark Community Sync Notes [0].
>>>
>>> I am planning to submit this work as a GSoC 2026 proposal based on this
>>> idea and was advised to seek formal community vetting on the dev mailing
>>> list.
>>>
>>> Previous DISCUSS thread:
>>> https://lists.apache.org/thread/b5jrlyv61lmw867kksw05sot2tro5ybn
>>>
>>> Issue:
>>> https://github.com/apache/iceberg/issues/15287
>>>
>>> Prototype implementation:
>>> https://github.com/apache/iceberg/pull/15341
>>>
>>> Design document and benchmarking details:
>>>
>>> https://docs.google.com/document/d/17vBz5t-gSDdmB0S40MYRceyvmcBSzw9Gii-FcU97Lds/edit?usp=sharing
>>>
>>> Initial benchmarking shows noticeable improvements for workloads
>>> involving many small files, particularly when IO latency is present
>>> (details in the design document).
>>>
>>> Any feedback (+1 / concerns / suggestions) would be appreciated.
>>> I am specifically looking for community consensus on whether this is a
>>> viable direction for Iceberg before formalizing the GSoC proposal. The GSoC
>>> 2026 proposal deadline is March 31 - early feedback would be especially
>>> appreciated.
>>>
>>> [0] Iceberg Spark Community Sync Notes:
>>> https://docs.google.com/document/d/19nno1RoPznbbxKOZZddZNHHafa7XULjbN6RPExdr2n4/edit?usp=sharing
>>> --
>>> Lakhyani Varun
>>> Indian Institute of Technology Roorkee
>>>
>>>
Re: [DISCUSS] GSoC 2026 idea vetting: Parallel scan task execution in Iceberg Spark readers
Posted by Varun Lakhyani <la...@ce.iitr.ac.in>.
Thanks a lot - Russell Spitzer has kindly agreed to mentor this project.
I would also like to confirm if there are any formal steps required for
mentor registration on the GSoC/ASF side for such vetted ideas. If so, I
would appreciate any guidance to ensure everything is in order before the
deadline.
On Tue, Mar 31, 2026 at 6:13 PM Varun Lakhyani <la...@ce.iitr.ac.in>
wrote:
> Thanks a lot to everyone for vetting this proposal.
>
> Today's the deadline for submitting proposals and I have mine ready but I
> got to know that before 1st April (Tomorrow) all proposals must have
> accepted mentors.
> I am looking for a community member whom I list as mentor and If possible
> I would need someone to register as a mentor for ASF organization and
> approve my proposal.
> I have applied using email id: lakhiyanivarun14@gmail.com
> Proposal that I will be submitting:
> https://docs.google.com/document/d/1ZEgzQj1cxt1fQLXh7auZE7E1xCDGkTNt7dNpA0PaG7U/edit?usp=sharing
> :
>
> I have already reached out to Russell Spitzer and Peter Vary directly, but
> given the tight timeline I wanted to flag this to the broader community as
> well.
> Thanks and Apologies for last minute request
>
> On Wed, Mar 18, 2026 at 12:19 AM Varun Lakhyani <la...@ce.iitr.ac.in>
> wrote:
>
>> Hey All,
>>
>> I previously started a discussion on making Spark readers work in
>> parallel (asynchronously), which is beneficial in cases with large numbers
>> of small files such as compaction, and I have worked on a POC, high-level
>> design, implementation, and benchmarking for various scenarios. I presented
>> my approach and benchmarking results in the Iceberg Spark sync; the
>> recording may be available in the Iceberg Spark Community Sync Notes [0].
>>
>> I am planning to submit this work as a GSoC 2026 proposal based on this
>> idea and was advised to seek formal community vetting on the dev mailing
>> list.
>>
>> Previous DISCUSS thread:
>> https://lists.apache.org/thread/b5jrlyv61lmw867kksw05sot2tro5ybn
>>
>> Issue:
>> https://github.com/apache/iceberg/issues/15287
>>
>> Prototype implementation:
>> https://github.com/apache/iceberg/pull/15341
>>
>> Design document and benchmarking details:
>>
>> https://docs.google.com/document/d/17vBz5t-gSDdmB0S40MYRceyvmcBSzw9Gii-FcU97Lds/edit?usp=sharing
>>
>> Initial benchmarking shows noticeable improvements for workloads
>> involving many small files, particularly when IO latency is present
>> (details in the design document).
>>
>> Any feedback (+1 / concerns / suggestions) would be appreciated.
>> I am specifically looking for community consensus on whether this is a
>> viable direction for Iceberg before formalizing the GSoC proposal. The GSoC
>> 2026 proposal deadline is March 31 - early feedback would be especially
>> appreciated.
>>
>> [0] Iceberg Spark Community Sync Notes:
>> https://docs.google.com/document/d/19nno1RoPznbbxKOZZddZNHHafa7XULjbN6RPExdr2n4/edit?usp=sharing
>> --
>> Lakhyani Varun
>> Indian Institute of Technology Roorkee
>>
>>
Re: [DISCUSS] GSoC 2026 idea vetting: Parallel scan task execution in Iceberg Spark readers
Posted by Varun Lakhyani <la...@ce.iitr.ac.in>.
Thanks a lot to everyone for vetting this proposal.
Today's the deadline for submitting proposals and I have mine ready but I
got to know that before 1st April (Tomorrow) all proposals must have
accepted mentors.
I am looking for a community member whom I list as mentor and If possible I
would need someone to register as a mentor for ASF organization and approve
my proposal.
I have applied using email id: lakhiyanivarun14@gmail.com
Proposal that I will be submitting:
https://docs.google.com/document/d/1ZEgzQj1cxt1fQLXh7auZE7E1xCDGkTNt7dNpA0PaG7U/edit?usp=sharing
:
I have already reached out to Russell Spitzer and Peter Vary directly, but
given the tight timeline I wanted to flag this to the broader community as
well.
Thanks and Apologies for last minute request
On Wed, Mar 18, 2026 at 12:19 AM Varun Lakhyani <la...@ce.iitr.ac.in>
wrote:
> Hey All,
>
> I previously started a discussion on making Spark readers work in parallel
> (asynchronously), which is beneficial in cases with large numbers of small
> files such as compaction, and I have worked on a POC, high-level design,
> implementation, and benchmarking for various scenarios. I presented my
> approach and benchmarking results in the Iceberg Spark sync; the recording
> may be available in the Iceberg Spark Community Sync Notes [0].
>
> I am planning to submit this work as a GSoC 2026 proposal based on this
> idea and was advised to seek formal community vetting on the dev mailing
> list.
>
> Previous DISCUSS thread:
> https://lists.apache.org/thread/b5jrlyv61lmw867kksw05sot2tro5ybn
>
> Issue:
> https://github.com/apache/iceberg/issues/15287
>
> Prototype implementation:
> https://github.com/apache/iceberg/pull/15341
>
> Design document and benchmarking details:
>
> https://docs.google.com/document/d/17vBz5t-gSDdmB0S40MYRceyvmcBSzw9Gii-FcU97Lds/edit?usp=sharing
>
> Initial benchmarking shows noticeable improvements for workloads involving
> many small files, particularly when IO latency is present (details in the
> design document).
>
> Any feedback (+1 / concerns / suggestions) would be appreciated.
> I am specifically looking for community consensus on whether this is a
> viable direction for Iceberg before formalizing the GSoC proposal. The GSoC
> 2026 proposal deadline is March 31 - early feedback would be especially
> appreciated.
>
> [0] Iceberg Spark Community Sync Notes:
> https://docs.google.com/document/d/19nno1RoPznbbxKOZZddZNHHafa7XULjbN6RPExdr2n4/edit?usp=sharing
> --
> Lakhyani Varun
> Indian Institute of Technology Roorkee
>
>