Re: Question on Large-Scale Data Processing and File Scan Parallelism in Texera

[Chen Li <ch...@apache.org>]

Thank you, Jay,  It will be great to have your optimization integrated into
the system.

Chen

On Mon, Aug 31, 2026 at 3:38 PM Jaykumar Rajeshbhai Tandel <jr...@asu.edu>
wrote:

> Hi Dr. Li and Rahul,
>
> Yes, we observed the same single-worker limitation on the CSV reader. In
> our case, though, it was not the bottleneck. For UDF-centric workloads, we
> found the dominant costs were the distribution of work to downstream UDF
> workers and the per-tuple build/read/serialize for the Python UDF.
>
> For reference, our setup: two neural-network UDFs on CPU over a 2.25 GB
> dataset, on a machine with 48 cores and 125 GB of memory. The best observed
> end-to-end time was 11 minutes 8 seconds. The two UDFs and a link to the
> dataset are attached.
>
> Two changes made the largest difference:
>
> *Use more workers.* Each Texera worker runs as a separate process, so UDF
> work parallelizes well when the machine has cores to spare. We used 20
> workers for each UDF.
>
> *Use a smaller data transfer batch size.* A batch only ships once a
> buffer fills, and each sending worker keeps a separate buffer per receiving
> worker, so a large batch size can delay downstream workers considerably. We
> saw the best result at a 400 batch size if parallelization is needed.
>
> Rahul, with 3 workers and 27M records, this is worth checking. Texera does
> not yet allocate resources for UDFs automatically, so worker counts and
> batch size have to be set manually, based on observations and the resources
> available at your end.
>
> We are currently working on making that overlap independent of the batch
> size, so that pipelining also holds at larger batch sizes in multi-UDF
> workflows. We plan to raise a PR for the parallelining issue by Sept 9,
> and one for the per-tuple cost by Sept 15, so the wider community can
> benefit.
> Please feel free to reach out if you have more queries.
>
> Thanks and Regards,
> Jay.
>
>  creditcard_x16.csv
> <https://drive.google.com/open?id=12n5TRN64sJDFtsa-3igbGguQqYopo1AY>
>
>  UDF Code
> <https://drive.google.com/open?id=1jgBexhIhxnoxx0GC_ZPMwI2eJ-ThtsDPnJfYjvQgrxg>
>
> On Sun, Aug 30, 2026 at 9:27 PM Chen Li <ch...@apache.org> wrote:
>
>> Rahul,
>>
>> Glad to know you were able to upload a 1GB file and process it.  We know
>> the system has a lot of room for performance improvement.  You are very
>> welcome to do some investigation and identify bottlenecks.
>>
>> I am including two colleagues from ASU, Jaykumar Rajeshbhai Tandel <
>> jrtandel@asu.edu> and Jia Zou <ji...@asu.edu>.  Recently they are
>> investigating related issues, and may be able to chime in to
>> provide insights.
>>
>> We will be happy to work with you on an optimization project.
>>
>> Chen Li
>> PPMC
>> Apache Texera (Incubating)
>>
>> On Sun, Aug 30, 2026 at 8:14 AM Rahul P <ra...@gmail.com> wrote:
>>
>>> Hi  Chen Li / Team
>>>
>>> Thanks for your valuable feedback.
>>>
>>> I have installed Texera on a remote Linux server and created the 1 GB
>>> test file on that server. Since I am accessing the Texera UI from my
>>> Windows machine, I manually used Texera's multipart upload API from the
>>> Linux server to upload the file as a Texera dataset.
>>>
>>> During the upload, I initially encountered a file size limit issue,
>>> which I resolved by increasing the configured limit. After that, the file
>>> was uploaded successfully.
>>>
>>> During execution, I observed that the *File Scan operator was reading
>>> the file in small chunks and writing the data in Iceberg format*. I
>>> increased the chunk size and restarted the execution.
>>>
>>> However, the *Python UDF operator*, which reads one line at a time,
>>> parses it as a CSV row, converts it into six structured fields, and outputs
>>> the tuple, took around *11 hours* to process the 27 million records.
>>>
>>> I am using *3 worker nodes*, and the Python UDF was also running with *3
>>> workers*.
>>>
>>> Is this expected behavior for Texera, or am I missing any configuration
>>> or optimization that could improve the Python UDF performance?
>>>
>>> [image: image.png]
>>>
>>>
>>> Thanks & Regards
>>> Rahul
>>>
>>> On Fri, Aug 28, 2026 at 11:17 AM Chen Li <ch...@apache.org> wrote:
>>>
>>>> Rahul,
>>>>
>>>> Sorry for the late reply.  Currently the CSVFileScan operator doesn't
>>>> support multiple workers to run in parallel.  So reading a large file
>>>> may
>>>> take time.  If you are interested in developing a parallel reader, you
>>>> are
>>>> very welcome to do so and the team can provide support.
>>>>
>>>> The Python UDF operator allows a user to specify the number of workers.
>>>>
>>>> The backend engine is doing in-memory computation.  Data transfer
>>>> between
>>>> operators can be pipelined.
>>>>
>>>> The backend storage layer supports large files using solutions such as
>>>> MinIO, LakeFS, and Iceberg.
>>>>
>>>> The system had done a lot of optimization in uploading large files
>>>> efficiently (e.g., in the range of TBs).
>>>>
>>>> Our team will be happy to answer questions related to your particular
>>>> use
>>>> case.
>>>>
>>>> Thank you,
>>>>
>>>> Chen Li
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Aug 27, 2026 at 9:17 PM Rahul P <ra...@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi Team
>>>> >
>>>> > Can I please get an update on this.
>>>> >
>>>> > Thanks & Regards
>>>> > Rahul
>>>> >
>>>> > On Thu, Aug 27, 2026 at 12:09 PM Rahul P <ra...@gmail.com>
>>>> wrote:
>>>> >
>>>> > > Hi Team,
>>>> > >
>>>> > > I have installed Texera and am able to execute workflows
>>>> successfully. I
>>>> > > would like to understand how large-scale data processing is handled
>>>> in
>>>> > > Texera.
>>>> > >
>>>> > > If the input data size is very large, will Texera process it
>>>> reliably
>>>> > > without crashing, or are there any known limitations related to file
>>>> > size,
>>>> > > memory, or cluster resources?
>>>> > >
>>>> > > I also could not find any option to configure parallelism for the
>>>> file
>>>> > > fetch/file scan operator. How can parallelism be configured for this
>>>> > > operator?
>>>> > >
>>>> > > Thanks & Regards
>>>> > > Rahul
>>>> > >
>>>> >
>>>>
>>>

Re: Question on Large-Scale Data Processing and File Scan Parallelism in Texera

[Jaykumar Rajeshbhai Tandel <jr...@asu.edu>]

Hi Rahul and Dr. Li,

I had a discussion with Professor Zou. The current implementation is
affecting the round-robin partitioner. What we're proposing instead is a
new, separate partitioner. I'll isolate the current changes so they don't
impact round-robin and create a separate PR within the next two weeks.

This has taken a bit longer as I've been working on another SIGMOD
submission, but I'll have this wrapped up within that timeline.

Thanks and Regards,
Jay

On Mon, Sep 14, 2026 at 9:03 PM Rahul P <ra...@gmail.com> wrote:

> Hi
>
> When is the optimization expected to be integrated?
>
> Thanks & Regards
> Rahul
>
>
> On Tue, 1 Sept, 2026, 08:13 Chen Li, <ch...@apache.org> wrote:
>
>> Thank you, Jay,  It will be great to have your optimization integrated
>> into
>> the system.
>>
>> Chen
>>
>> On Mon, Aug 31, 2026 at 3:38 PM Jaykumar Rajeshbhai Tandel <
>> jrtandel@asu.edu>
>> wrote:
>>
>> > Hi Dr. Li and Rahul,
>> >
>> > Yes, we observed the same single-worker limitation on the CSV reader. In
>> > our case, though, it was not the bottleneck. For UDF-centric workloads,
>> we
>> > found the dominant costs were the distribution of work to downstream UDF
>> > workers and the per-tuple build/read/serialize for the Python UDF.
>> >
>> > For reference, our setup: two neural-network UDFs on CPU over a 2.25 GB
>> > dataset, on a machine with 48 cores and 125 GB of memory. The best
>> observed
>> > end-to-end time was 11 minutes 8 seconds. The two UDFs and a link to the
>> > dataset are attached.
>> >
>> > Two changes made the largest difference:
>> >
>> > *Use more workers.* Each Texera worker runs as a separate process, so
>> UDF
>> > work parallelizes well when the machine has cores to spare. We used 20
>> > workers for each UDF.
>> >
>> > *Use a smaller data transfer batch size.* A batch only ships once a
>> > buffer fills, and each sending worker keeps a separate buffer per
>> receiving
>> > worker, so a large batch size can delay downstream workers
>> considerably. We
>> > saw the best result at a 400 batch size if parallelization is needed.
>> >
>> > Rahul, with 3 workers and 27M records, this is worth checking. Texera
>> does
>> > not yet allocate resources for UDFs automatically, so worker counts and
>> > batch size have to be set manually, based on observations and the
>> resources
>> > available at your end.
>> >
>> > We are currently working on making that overlap independent of the batch
>> > size, so that pipelining also holds at larger batch sizes in multi-UDF
>> > workflows. We plan to raise a PR for the parallelining issue by Sept 9,
>> > and one for the per-tuple cost by Sept 15, so the wider community can
>> > benefit.
>> > Please feel free to reach out if you have more queries.
>> >
>> > Thanks and Regards,
>> > Jay.
>> >
>> >  creditcard_x16.csv
>> > <https://drive.google.com/open?id=12n5TRN64sJDFtsa-3igbGguQqYopo1AY>
>> >
>> >  UDF Code
>> > <
>> https://drive.google.com/open?id=1jgBexhIhxnoxx0GC_ZPMwI2eJ-ThtsDPnJfYjvQgrxg
>> >
>> >
>> > On Sun, Aug 30, 2026 at 9:27 PM Chen Li <ch...@apache.org> wrote:
>> >
>> >> Rahul,
>> >>
>> >> Glad to know you were able to upload a 1GB file and process it.  We
>> know
>> >> the system has a lot of room for performance improvement.  You are very
>> >> welcome to do some investigation and identify bottlenecks.
>> >>
>> >> I am including two colleagues from ASU, Jaykumar Rajeshbhai Tandel <
>> >> jrtandel@asu.edu> and Jia Zou <ji...@asu.edu>.  Recently they are
>> >> investigating related issues, and may be able to chime in to
>> >> provide insights.
>> >>
>> >> We will be happy to work with you on an optimization project.
>> >>
>> >> Chen Li
>> >> PPMC
>> >> Apache Texera (Incubating)
>> >>
>> >> On Sun, Aug 30, 2026 at 8:14 AM Rahul P <ra...@gmail.com>
>> wrote:
>> >>
>> >>> Hi  Chen Li / Team
>> >>>
>> >>> Thanks for your valuable feedback.
>> >>>
>> >>> I have installed Texera on a remote Linux server and created the 1 GB
>> >>> test file on that server. Since I am accessing the Texera UI from my
>> >>> Windows machine, I manually used Texera's multipart upload API from
>> the
>> >>> Linux server to upload the file as a Texera dataset.
>> >>>
>> >>> During the upload, I initially encountered a file size limit issue,
>> >>> which I resolved by increasing the configured limit. After that, the
>> file
>> >>> was uploaded successfully.
>> >>>
>> >>> During execution, I observed that the *File Scan operator was reading
>> >>> the file in small chunks and writing the data in Iceberg format*. I
>> >>> increased the chunk size and restarted the execution.
>> >>>
>> >>> However, the *Python UDF operator*, which reads one line at a time,
>> >>> parses it as a CSV row, converts it into six structured fields, and
>> outputs
>> >>> the tuple, took around *11 hours* to process the 27 million records.
>> >>>
>> >>> I am using *3 worker nodes*, and the Python UDF was also running with
>> *3
>> >>> workers*.
>> >>>
>> >>> Is this expected behavior for Texera, or am I missing any
>> configuration
>> >>> or optimization that could improve the Python UDF performance?
>> >>>
>> >>> [image: image.png]
>> >>>
>> >>>
>> >>> Thanks & Regards
>> >>> Rahul
>> >>>
>> >>> On Fri, Aug 28, 2026 at 11:17 AM Chen Li <ch...@apache.org> wrote:
>> >>>
>> >>>> Rahul,
>> >>>>
>> >>>> Sorry for the late reply.  Currently the CSVFileScan operator doesn't
>> >>>> support multiple workers to run in parallel.  So reading a large file
>> >>>> may
>> >>>> take time.  If you are interested in developing a parallel reader,
>> you
>> >>>> are
>> >>>> very welcome to do so and the team can provide support.
>> >>>>
>> >>>> The Python UDF operator allows a user to specify the number of
>> workers.
>> >>>>
>> >>>> The backend engine is doing in-memory computation.  Data transfer
>> >>>> between
>> >>>> operators can be pipelined.
>> >>>>
>> >>>> The backend storage layer supports large files using solutions such
>> as
>> >>>> MinIO, LakeFS, and Iceberg.
>> >>>>
>> >>>> The system had done a lot of optimization in uploading large files
>> >>>> efficiently (e.g., in the range of TBs).
>> >>>>
>> >>>> Our team will be happy to answer questions related to your particular
>> >>>> use
>> >>>> case.
>> >>>>
>> >>>> Thank you,
>> >>>>
>> >>>> Chen Li
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Thu, Aug 27, 2026 at 9:17 PM Rahul P <ra...@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>> > Hi Team
>> >>>> >
>> >>>> > Can I please get an update on this.
>> >>>> >
>> >>>> > Thanks & Regards
>> >>>> > Rahul
>> >>>> >
>> >>>> > On Thu, Aug 27, 2026 at 12:09 PM Rahul P <rahuledavalath@gmail.com
>> >
>> >>>> wrote:
>> >>>> >
>> >>>> > > Hi Team,
>> >>>> > >
>> >>>> > > I have installed Texera and am able to execute workflows
>> >>>> successfully. I
>> >>>> > > would like to understand how large-scale data processing is
>> handled
>> >>>> in
>> >>>> > > Texera.
>> >>>> > >
>> >>>> > > If the input data size is very large, will Texera process it
>> >>>> reliably
>> >>>> > > without crashing, or are there any known limitations related to
>> file
>> >>>> > size,
>> >>>> > > memory, or cluster resources?
>> >>>> > >
>> >>>> > > I also could not find any option to configure parallelism for the
>> >>>> file
>> >>>> > > fetch/file scan operator. How can parallelism be configured for
>> this
>> >>>> > > operator?
>> >>>> > >
>> >>>> > > Thanks & Regards
>> >>>> > > Rahul
>> >>>> > >
>> >>>> >
>> >>>>
>> >>>
>>
>

Re: Question on Large-Scale Data Processing and File Scan Parallelism in Texera

[Rahul P <ra...@gmail.com>]

Hi

When is the optimization expected to be integrated?

Thanks & Regards
Rahul


On Tue, 1 Sept, 2026, 08:13 Chen Li, <ch...@apache.org> wrote:

> Thank you, Jay,  It will be great to have your optimization integrated into
> the system.
>
> Chen
>
> On Mon, Aug 31, 2026 at 3:38 PM Jaykumar Rajeshbhai Tandel <
> jrtandel@asu.edu>
> wrote:
>
> > Hi Dr. Li and Rahul,
> >
> > Yes, we observed the same single-worker limitation on the CSV reader. In
> > our case, though, it was not the bottleneck. For UDF-centric workloads,
> we
> > found the dominant costs were the distribution of work to downstream UDF
> > workers and the per-tuple build/read/serialize for the Python UDF.
> >
> > For reference, our setup: two neural-network UDFs on CPU over a 2.25 GB
> > dataset, on a machine with 48 cores and 125 GB of memory. The best
> observed
> > end-to-end time was 11 minutes 8 seconds. The two UDFs and a link to the
> > dataset are attached.
> >
> > Two changes made the largest difference:
> >
> > *Use more workers.* Each Texera worker runs as a separate process, so UDF
> > work parallelizes well when the machine has cores to spare. We used 20
> > workers for each UDF.
> >
> > *Use a smaller data transfer batch size.* A batch only ships once a
> > buffer fills, and each sending worker keeps a separate buffer per
> receiving
> > worker, so a large batch size can delay downstream workers considerably.
> We
> > saw the best result at a 400 batch size if parallelization is needed.
> >
> > Rahul, with 3 workers and 27M records, this is worth checking. Texera
> does
> > not yet allocate resources for UDFs automatically, so worker counts and
> > batch size have to be set manually, based on observations and the
> resources
> > available at your end.
> >
> > We are currently working on making that overlap independent of the batch
> > size, so that pipelining also holds at larger batch sizes in multi-UDF
> > workflows. We plan to raise a PR for the parallelining issue by Sept 9,
> > and one for the per-tuple cost by Sept 15, so the wider community can
> > benefit.
> > Please feel free to reach out if you have more queries.
> >
> > Thanks and Regards,
> > Jay.
> >
> >  creditcard_x16.csv
> > <https://drive.google.com/open?id=12n5TRN64sJDFtsa-3igbGguQqYopo1AY>
> >
> >  UDF Code
> > <
> https://drive.google.com/open?id=1jgBexhIhxnoxx0GC_ZPMwI2eJ-ThtsDPnJfYjvQgrxg
> >
> >
> > On Sun, Aug 30, 2026 at 9:27 PM Chen Li <ch...@apache.org> wrote:
> >
> >> Rahul,
> >>
> >> Glad to know you were able to upload a 1GB file and process it.  We know
> >> the system has a lot of room for performance improvement.  You are very
> >> welcome to do some investigation and identify bottlenecks.
> >>
> >> I am including two colleagues from ASU, Jaykumar Rajeshbhai Tandel <
> >> jrtandel@asu.edu> and Jia Zou <ji...@asu.edu>.  Recently they are
> >> investigating related issues, and may be able to chime in to
> >> provide insights.
> >>
> >> We will be happy to work with you on an optimization project.
> >>
> >> Chen Li
> >> PPMC
> >> Apache Texera (Incubating)
> >>
> >> On Sun, Aug 30, 2026 at 8:14 AM Rahul P <ra...@gmail.com>
> wrote:
> >>
> >>> Hi  Chen Li / Team
> >>>
> >>> Thanks for your valuable feedback.
> >>>
> >>> I have installed Texera on a remote Linux server and created the 1 GB
> >>> test file on that server. Since I am accessing the Texera UI from my
> >>> Windows machine, I manually used Texera's multipart upload API from the
> >>> Linux server to upload the file as a Texera dataset.
> >>>
> >>> During the upload, I initially encountered a file size limit issue,
> >>> which I resolved by increasing the configured limit. After that, the
> file
> >>> was uploaded successfully.
> >>>
> >>> During execution, I observed that the *File Scan operator was reading
> >>> the file in small chunks and writing the data in Iceberg format*. I
> >>> increased the chunk size and restarted the execution.
> >>>
> >>> However, the *Python UDF operator*, which reads one line at a time,
> >>> parses it as a CSV row, converts it into six structured fields, and
> outputs
> >>> the tuple, took around *11 hours* to process the 27 million records.
> >>>
> >>> I am using *3 worker nodes*, and the Python UDF was also running with
> *3
> >>> workers*.
> >>>
> >>> Is this expected behavior for Texera, or am I missing any configuration
> >>> or optimization that could improve the Python UDF performance?
> >>>
> >>> [image: image.png]
> >>>
> >>>
> >>> Thanks & Regards
> >>> Rahul
> >>>
> >>> On Fri, Aug 28, 2026 at 11:17 AM Chen Li <ch...@apache.org> wrote:
> >>>
> >>>> Rahul,
> >>>>
> >>>> Sorry for the late reply.  Currently the CSVFileScan operator doesn't
> >>>> support multiple workers to run in parallel.  So reading a large file
> >>>> may
> >>>> take time.  If you are interested in developing a parallel reader, you
> >>>> are
> >>>> very welcome to do so and the team can provide support.
> >>>>
> >>>> The Python UDF operator allows a user to specify the number of
> workers.
> >>>>
> >>>> The backend engine is doing in-memory computation.  Data transfer
> >>>> between
> >>>> operators can be pipelined.
> >>>>
> >>>> The backend storage layer supports large files using solutions such as
> >>>> MinIO, LakeFS, and Iceberg.
> >>>>
> >>>> The system had done a lot of optimization in uploading large files
> >>>> efficiently (e.g., in the range of TBs).
> >>>>
> >>>> Our team will be happy to answer questions related to your particular
> >>>> use
> >>>> case.
> >>>>
> >>>> Thank you,
> >>>>
> >>>> Chen Li
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Aug 27, 2026 at 9:17 PM Rahul P <ra...@gmail.com>
> >>>> wrote:
> >>>>
> >>>> > Hi Team
> >>>> >
> >>>> > Can I please get an update on this.
> >>>> >
> >>>> > Thanks & Regards
> >>>> > Rahul
> >>>> >
> >>>> > On Thu, Aug 27, 2026 at 12:09 PM Rahul P <ra...@gmail.com>
> >>>> wrote:
> >>>> >
> >>>> > > Hi Team,
> >>>> > >
> >>>> > > I have installed Texera and am able to execute workflows
> >>>> successfully. I
> >>>> > > would like to understand how large-scale data processing is
> handled
> >>>> in
> >>>> > > Texera.
> >>>> > >
> >>>> > > If the input data size is very large, will Texera process it
> >>>> reliably
> >>>> > > without crashing, or are there any known limitations related to
> file
> >>>> > size,
> >>>> > > memory, or cluster resources?
> >>>> > >
> >>>> > > I also could not find any option to configure parallelism for the
> >>>> file
> >>>> > > fetch/file scan operator. How can parallelism be configured for
> this
> >>>> > > operator?
> >>>> > >
> >>>> > > Thanks & Regards
> >>>> > > Rahul
> >>>> > >
> >>>> >
> >>>>
> >>>
>