You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@texera.apache.org by "kunwp1 (via GitHub)" <gi...@apache.org> on 2026/04/08 04:37:15 UTC
[D] Supporting Directory-Based Dataset Access in UDFs [texera]
GitHub user kunwp1 created a discussion: Supporting Directory-Based Dataset Access in UDFs
## The Problem
Currently, Texera's `DatasetFileDocument` API allows users to fetch individual files as streams. However, some bioinformatics and data science libraries require a local filesystem directory path to function.
For example, R’s `read10X(data.dir=...)` and Python’s `scanpy.read_10x_mtx()` expect a folder containing a specific set of files (e.g., `matrix.mtx.gz`, `barcodes.tsv.gz`). These functions cannot operate on isolated file streams; they need a physical directory handle.
## Root Cause
Texera's backend (LakeFS) stores data as flat objects using path-separator names. Consequently, there is no native "folder" object to pass to a UDF. Our current Python SDK and backend endpoints lack a mechanism to materialize a specific path prefix (a "folder") as a local directory on a computing unit.
---
## Design Choice 1: Client-Side File-by-File Materialization
Introduce a `DatasetFolderDocument` class in the Python SDK that simulates a directory by downloading all objects with a matching prefix.
```python
with DatasetFolderDocument("/bob@texera.com/myDataset/v1/Counts") as local_path:
adata = sc.read_10x_mtx(local_path)
```
Workflow:
1. SDK queries the existing file-tree API to enumerate all files under the specified prefix.
2. SDK downloads each file individually via presigned URLs.
3. SDK recreates the sub-directory structure in a local `/tmp` directory.
4. The context manager returns the path and handles cleanup on exit.
Pros:
- Requires no backend changes; purely an SDK-level implementation.
Cons:
- N HTTP round-trips.
- Potential for partial state failures (e.g., download fails halfway through).
- Temp disk usage on the worker is proportional to folder size.
## Design Choice 2: Server-Side Archiving
Add a backend REST endpoint that accepts a folder path and streams a single archive (e.g., ZIP or Tar) containing the requested files. The user experience is same as Design Choice 1.
```python
with DatasetFolderDocument("/bob@texera.com/myDataset/v1/Counts") as local_path:
adata = sc.read_10x_mtx(local_path)
```
Workflow:
1. SDK makes a single request to the new endpoint.
2. The backend filters the LakeFS objects and streams a ZIP on-the-fly.
3. The SDK downloads and extracts this single archive to a local temp directory.
Pros:
- Reduces N HTTP round-trips to 1
- Atomic download
Cons:
- Requires a new backend endpoint
- Temp disk usage on the worker is proportional to folder size.
GitHub link: https://github.com/apache/texera/discussions/4352
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Supporting Directory-Based Dataset Access in UDFs
@xuang7 Can you chime in based on your recent experience of migrating a few full stacks to the platform? I will chime in later.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484299
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Supporting Directory-Based Dataset Access in UDFs
Thanks for starting this discussion. one quick clarification question, do we always have to send a copy of the files to the CU? I thought we are attaching a volume instead.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484151
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
GitHub user aglinxinyuan added a comment to the discussion: Supporting Directory-Based Dataset Access in UDFs
Can we utilize our existing data processing interface to support this? I.e. use binary tuple as a single file, and use table of binary tuples as a directory.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484307
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica edited a comment on the discussion: Supporting Directory-Based Dataset Access in UDFs
Before providing more technical input (later), I want to say that this "file-path interface within a dataset" feature has been requested by several other users in the past.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484407
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Supporting Directory-Based Dataset Access in UDFs
on the high level, I think mounting is a more feasible option. related topic: we also would need to be able solve different env management that @SarahAsad23 is exploring. I can see the relationship: python virtual env is also just a directory, and CU just need a path of the python callable, which has all the dependencies available in the directory.
The mounting approach may not be that hard: instead of download the files to the CU's local volume, download it to a shared volume instead, and mount it to CU. Then in python, the Document API can be resolved and point to a local path.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484338
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
GitHub user aglinxinyuan edited a comment on the discussion: Supporting Directory-Based Dataset Access in UDFs
Can we utilize our existing data processing interface to support this? I.e. use binary tuple as a single file, and use table of binary tuples as a directory. I don't like the idea of bypassing the assumption that data always comes from upstream operators. If we allow any operator to read the data from filesystem, any operator can become a source operator and it's no longer a data flow.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484307
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "kunwp1 (via GitHub)" <gi...@apache.org>.
GitHub user kunwp1 edited a comment on the discussion: Supporting Directory-Based Dataset Access in UDFs
I know it's doable but our current implementation is not very easy and straightforward to read a list of files (can be thousands) from a folder in the dataset.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484392
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "kunwp1 (via GitHub)" <gi...@apache.org>.
GitHub user kunwp1 added a comment to the discussion: Supporting Directory-Based Dataset Access in UDFs
The R function (`Read10X`) she is using requires "local directory path" so the data processing interface won't work in this case. User needs a way to read a folder from the dataset.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484348
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Supporting Directory-Based Dataset Access in UDFs
whether it is a directory or files, the underneath technique should be the same.
tuple(contains a URL field of `file_or_dir_path`) -> Python UDF
-> Python Document API (`url`)
-> make the data available on local fs ( by fetching, or attaching volume of the data)
-> library can use it.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484435
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "kunwp1 (via GitHub)" <gi...@apache.org>.
GitHub user kunwp1 added a comment to the discussion: Supporting Directory-Based Dataset Access in UDFs
I know it's doable but currently it's not very easy and straightforward to read a list of files (can be thousands) from a folder in the dataset.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484392
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
GitHub user aglinxinyuan added a comment to the discussion: Supporting Directory-Based Dataset Access in UDFs
It's accurate.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16499362
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Supporting Directory-Based Dataset Access in UDFs
We will have an offline discussion and report here.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16490510
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Supporting Directory-Based Dataset Access in UDFs
second this. sorry I did not think about this before. I also agree data should be kept within the data interface. BigObject should not bypass the interface.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484356
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
GitHub user aglinxinyuan edited a comment on the discussion: Supporting Directory-Based Dataset Access in UDFs
We can still let data come from our data interface, and after that let it become a local directory path. I.e convert a table to a local directory path in UDF.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484362
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Supporting Directory-Based Dataset Access in UDFs
Before providing more technical input (later), I want to say that this "file interface within a dataset" feature has been requested by several other users in the past.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484407
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "xuang7 (via GitHub)" <gi...@apache.org>.
GitHub user xuang7 added a comment to the discussion: Supporting Directory-Based Dataset Access in UDFs
Recently, I have been working on integrating brain imaging visualization tools into the platform. These tools typically expect a folder containing a specific set of files, such as a manifest, coordinates, cell metadata, and other supporting files.
Since we do not currently have a directory-access API, the current workaround is essentially a file-by-file presigned URL approach at the frontend level. For each file, the viewer has to:
1. call `/api/dataset/presign-download?...` to get a presigned URL
2. fetch the actual file content from MinIO using that URL
This is effectively similar to Design Choice 1, except that it is currently happening at the frontend level. It works as a workaround, but it may become inefficient when a dataset contains a large number of files. A directory-access API would simplify this significantly and would be very helpful.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484470
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Supporting Directory-Based Dataset Access in UDFs
we can provide Document API to expose a directory.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484414
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "kunwp1 (via GitHub)" <gi...@apache.org>.
GitHub user kunwp1 added a comment to the discussion: Supporting Directory-Based Dataset Access in UDFs
The CU has a local volume, but it starts empty. Our data currently lives in Object Storage rather than on a physical hard drive. We can have two ways to get it to the CU:
- The Download Approach: We grab the files from the cloud and save a copy onto the CU’s local volume. It's simple and reliable, which is why I'm leaning toward it.
- The Mounting Approach: This is likely what you are thinking of: making the cloud storage appear as a local folder. I haven't explored much on this but it seems much harder to set up.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484226
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "xuang7 (via GitHub)" <gi...@apache.org>.
GitHub user xuang7 edited a comment on the discussion: Supporting Directory-Based Dataset Access in UDFs
Recently, I have been working on integrating brain imaging visualization tools into the platform. These tools typically expect a folder containing a specific set of files, such as a manifest, coordinates, cell metadata, and other supporting files.
Since we do not currently have a directory-access API, the current workaround is essentially a file-by-file presigned URL approach at the frontend level. For each file, the viewer has to:
1. call `/api/dataset/presign-download?...` to get a presigned URL
2. fetch the actual file content from MinIO using that URL
This is effectively similar to Design Choice 1, except that it is currently happening at the frontend level. It works as a workaround since my use case can lazy-load files on demand, but it may become inefficient when a dataset contains a large number of files. A directory-access API would simplify this significantly and would be very helpful for tools that require the full directory.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484470
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "kunwp1 (via GitHub)" <gi...@apache.org>.
GitHub user kunwp1 added a comment to the discussion: Supporting Directory-Based Dataset Access in UDFs
Following our offline discussion, we've decided to integrate new operators being developed by @aglinxinyuan with the Python UDF. The proposed workflow is as follows:
- Dataset Selector (Source): This operator accepts a dataset URI (e.g., /ownerEmail/datasetName/versionName) and flattens its structure into a table of file URIs. For instance, a nested structure like /folder1/file1 and /folder1/folder2/file2 will be returned as individual string rows.
- Text to File Scan: It's a downstream operator that resolves the URIs into file contents. Users can toggle an option to include the original URI as an attribute, resulting in tuples of (file_uri, file_content).
- Python UDF: This operator consumes these tuples, providing users with the raw paths and data.
We initially considered using `io.BytesIO` to provide a folder-like interface within the UDF. However, it seems like `io.ByteIO` can only mimic a single file, not a folder-like file system. So, the responsibility for reconstructing or mimicking a file-tree structure (e.g., `tempfile`) will rest with the UDF logic itself.
Please feel free to add your comments or suggestions on this approach.
CC: @aglinxinyuan @chenlica @xuang7
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16495961
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Supporting Directory-Based Dataset Access in UDFs
@aglinxinyuan Please confirm if this summary is consistent with your solution and implemented operators.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16498598
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "kunwp1 (via GitHub)" <gi...@apache.org>.
GitHub user kunwp1 edited a comment on the discussion: Supporting Directory-Based Dataset Access in UDFs
I know it's doable but our current implementation is not very easy and straightforward to read a folder that contains list of files (can be thousands) from a dataset.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484392
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
GitHub user Yicong-Huang added a comment to the discussion: Supporting Directory-Based Dataset Access in UDFs
we can make it work. it is doable. data comes from tuples, underneath it is written to local fs and let libraries use it.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484370
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Supporting Directory-Based Dataset Access in UDFs [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
GitHub user aglinxinyuan added a comment to the discussion: Supporting Directory-Based Dataset Access in UDFs
We can still let data come from our data interface, and after that let it become a local directory path.
GitHub link: https://github.com/apache/texera/discussions/4352#discussioncomment-16484362
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org