You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@opendal.apache.org by "cbisht31 (via GitHub)" <gi...@apache.org> on 2025/04/17 05:20:05 UTC
Native Pagination Support for File Listing in OpenDAL
GitHub user cbisht31 created a discussion: Native Pagination Support for File Listing in OpenDAL
In order to iterate over large number of files in a directory, it becomes challenging to handle large datasets efficiently. Does openDAL supports pagination or similar mechanism to make this efficient ?
GitHub link: https://github.com/apache/opendal/discussions/6040
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Native Pagination Support for File Listing in OpenDAL
Posted by "cbisht31 (via GitHub)" <gi...@apache.org>.
GitHub user cbisht31 added a comment to the discussion: Native Pagination Support for File Listing in OpenDAL
Thank you! I'll create an issue for the same.
GitHub link: https://github.com/apache/opendal/discussions/6040#discussioncomment-12863074
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Native Pagination Support for File Listing in OpenDAL
Posted by "cbisht31 (via GitHub)" <gi...@apache.org>.
GitHub user cbisht31 edited a comment on the discussion: Native Pagination Support for File Listing in OpenDAL
Thank you for the clarification regarding OpenDAL's internal pagination.
I'm using S3 and wanted to know how can I add offset and limit functionality to this code. The API should support pagination for the frontend. For example, I want to skip the first offset files and return only limit files in the response. Does OpenDAL provide a way to achieve this directly?
Also, is there a way to get the total count of files in a directory without listing all of them? For example, does OpenDAL provide a count function or a similar method to retrieve the total number of files in a directory? Here is the code snippet where I wish to add:
`async def list_files_with_metainfo(operator: opendal.Operator, subpath: str) -> List[dict]:
"""
List files at the given subpath and retrieve full metainfo for each file.
"""
metainfo_list = []
try:
entries = await operator.list(subpath)
async for entry in entries:
name = entry.name if hasattr(entry, 'name') else str(entry)
full_path = entry.path
is_dir = False
error = None
try:
stat_info = await operator.stat(full_path)
if hasattr(stat_info, 'mode'):
is_dir = stat_info.mode.is_dir()
except Exception as e:
error = str(e)
file_info = {
"name": name,
"path": full_path,
"is_dir": is_dir,
"error": error
}
metainfo_list.append(file_info)
except Exception as e:
print(f"Error: {e}")
return metainfo_list`
GitHub link: https://github.com/apache/opendal/discussions/6040#discussioncomment-12862971
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Native Pagination Support for File Listing in OpenDAL
Posted by "cbisht31 (via GitHub)" <gi...@apache.org>.
GitHub user cbisht31 edited a comment on the discussion: Native Pagination Support for File Listing in OpenDAL
Thank you for the clarification regarding OpenDAL's internal pagination.
I'm using S3 and wanted to know how can I add offset and limit functionality to this code. The API should support pagination for the frontend. For example, I want to skip the first offset files and return only limit files in the response. Does OpenDAL provide a way to achieve this directly?
Also, is there a way to get the total count of files in a directory without listing all of them? For example, does OpenDAL provide a count function or a similar method to retrieve the total number of files in a directory? Here is the code snippet where I wish to add:
```
async def list_files_with_metainfo(operator: opendal.Operator, subpath: str) -> List[dict]:
"""
List files at the given subpath and retrieve full metainfo for each file.
"""
metainfo_list = []
try:
entries = await operator.list(subpath)
async for entry in entries:
name = entry.name if hasattr(entry, 'name') else str(entry)
full_path = entry.path
is_dir = False
error = None
try:
stat_info = await operator.stat(full_path)
if hasattr(stat_info, 'mode'):
is_dir = stat_info.mode.is_dir()
except Exception as e:
error = str(e)
file_info = {
"name": name,
"path": full_path,
"is_dir": is_dir,
"error": error
}
metainfo_list.append(file_info)
except Exception as e:
print(f"Error: {e}")
return metainfo_list
```
GitHub link: https://github.com/apache/opendal/discussions/6040#discussioncomment-12862971
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Native Pagination Support for File Listing in OpenDAL
Posted by "cbisht31 (via GitHub)" <gi...@apache.org>.
GitHub user cbisht31 added a comment to the discussion: Native Pagination Support for File Listing in OpenDAL
Thank you for the clarification regarding OpenDAL's internal pagination.
I'm using S3 and wanted to know how can I add offset and limit functionality to this code. The API should support pagination for the frontend. For example, I want to skip the first offset files and return only limit files in the response. Does OpenDAL provide a way to achieve this directly?
Also, is there a way to get the total count of files in a directory without listing all of them? For example, does OpenDAL provide a count function or a similar method to retrieve the total number of files in a directory? Here is the code snippet where I wish to add:
`async def list_files_with_metainfo(operator: opendal.Operator, subpath: str) -> List[dict]:
"""
List files at the given subpath and retrieve full metainfo for each file.
"""
metainfo_list = []
try:
entries = await operator.list(subpath)
async for entry in entries:
name = entry.name if hasattr(entry, 'name') else str(entry)
full_path = entry.path
is_dir = False
error = None
try:
stat_info = await operator.stat(full_path)
if hasattr(stat_info, 'mode'):
is_dir = stat_info.mode.is_dir()
except Exception as e:
error = str(e)
file_info = {
"name": name,
"path": full_path,
"is_dir": is_dir,
"error": error
}
metainfo_list.append(file_info)
except Exception as e:
print(f"Error: {e}")
return metainfo_list`
GitHub link: https://github.com/apache/opendal/discussions/6040#discussioncomment-12862971
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Native Pagination Support for File Listing in OpenDAL
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo added a comment to the discussion: Native Pagination Support for File Listing in OpenDAL
Hi, thank you for using OpenDAL.
OpenDAL uses pagination internally; for example, with S3, it lists 1,000 entries per request.
Would you like to share more details about this? Which service are you using? What pagination you expect?
GitHub link: https://github.com/apache/opendal/discussions/6040#discussioncomment-12862698
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Native Pagination Support for File Listing in OpenDAL
Posted by "cbisht31 (via GitHub)" <gi...@apache.org>.
GitHub user cbisht31 edited a comment on the discussion: Native Pagination Support for File Listing in OpenDAL
Thank you for the clarification regarding OpenDAL's internal pagination.
I'm using S3 and wanted to know how can I add offset and limit functionality to this code. The API should support pagination for the frontend. For example, I want to skip the first offset files and return only limit files in the response. Does OpenDAL provide a way to achieve this directly?
Also, is there a way to get the total count of files in a directory without listing all of them? For example, does OpenDAL provide a count function or a similar method to retrieve the total number of files in a directory? Here is the code snippet where I wish to add:
async def list_files_with_metainfo(operator: opendal.Operator, subpath: str) -> List[dict]:
"""
List files at the given subpath and retrieve full metainfo for each file.
"""
metainfo_list = []
try:
entries = await operator.list(subpath)
async for entry in entries:
name = entry.name if hasattr(entry, 'name') else str(entry)
full_path = entry.path
is_dir = False
error = None
try:
stat_info = await operator.stat(full_path)
if hasattr(stat_info, 'mode'):
is_dir = stat_info.mode.is_dir()
except Exception as e:
error = str(e)
file_info = {
"name": name,
"path": full_path,
"is_dir": is_dir,
"error": error
}
metainfo_list.append(file_info)
except Exception as e:
print(f"Error: {e}")
return metainfo_list
GitHub link: https://github.com/apache/opendal/discussions/6040#discussioncomment-12862971
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Native Pagination Support for File Listing in OpenDAL
Posted by "cbisht31 (via GitHub)" <gi...@apache.org>.
GitHub user cbisht31 added a comment to the discussion: Native Pagination Support for File Listing in OpenDAL
Hi @Xuanwo,
Hope you doing well, just wanted check whether `start-after` feature is there on python yet?
Thanks!
GitHub link: https://github.com/apache/opendal/discussions/6040#discussioncomment-14694829
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Native Pagination Support for File Listing in OpenDAL
Posted by "cbisht31 (via GitHub)" <gi...@apache.org>.
GitHub user cbisht31 edited a comment on the discussion: Native Pagination Support for File Listing in OpenDAL
Thank you for the clarification regarding OpenDAL's internal pagination.
I'm using S3 and wanted to know how can I add offset and limit functionality to this code. The API should support pagination for the frontend. For example, I want to skip the first offset files and return only limit files in the response. Does OpenDAL provide a way to achieve this directly?
Also, is there a way to get the total count of files in a directory without listing all of them? For example, does OpenDAL provide a count function or a similar method to retrieve the total number of files in a directory? Here is the code snippet where I wish to add:
async def list_files_with_metainfo(operator: opendal.Operator, subpath: str) -> List[dict]:
"""
List files at the given subpath and retrieve full metainfo for each file.
"""
metainfo_list = []
try:
entries = await operator.list(subpath)
async for entry in entries:
name = entry.name if hasattr(entry, 'name') else str(entry)
full_path = entry.path
is_dir = False
error = None
try:
stat_info = await operator.stat(full_path)
if hasattr(stat_info, 'mode'):
is_dir = stat_info.mode.is_dir()
except Exception as e:
error = str(e)
file_info = {
"name": name,
"path": full_path,
"is_dir": is_dir,
"error": error
}
metainfo_list.append(file_info)
except Exception as e:
print(f"Error: {e}")
return metainfo_list
GitHub link: https://github.com/apache/opendal/discussions/6040#discussioncomment-12862971
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Native Pagination Support for File Listing in OpenDAL
Posted by "cbisht31 (via GitHub)" <gi...@apache.org>.
GitHub user cbisht31 edited a comment on the discussion: Native Pagination Support for File Listing in OpenDAL
Thank you for the clarification regarding OpenDAL's internal pagination.
I'm using S3 and wanted to know how can I add offset and limit functionality to this code. The API should support pagination for the frontend. For example, I want to skip the first offset files and return only limit files in the response. Does OpenDAL provide a way to achieve this directly?
Also, is there a way to get the total count of files in a directory without listing all of them? For example, does OpenDAL provide a count function or a similar method to retrieve the total number of files in a directory? Here is the code snippet where I wish to add:
`async def list_files_with_metainfo(operator: opendal.Operator, subpath: str) -> List[dict]:
"""
List files at the given subpath and retrieve full metainfo for each file.
"""
metainfo_list = []
try:
entries = await operator.list(subpath)
async for entry in entries:
name = entry.name if hasattr(entry, 'name') else str(entry)
full_path = entry.path
is_dir = False
error = None
try:
stat_info = await operator.stat(full_path)
if hasattr(stat_info, 'mode'):
is_dir = stat_info.mode.is_dir()
except Exception as e:
error = str(e)
file_info = {
"name": name,
"path": full_path,
"is_dir": is_dir,
"error": error
}
metainfo_list.append(file_info)
except Exception as e:
print(f"Error: {e}")
return metainfo_list`
GitHub link: https://github.com/apache/opendal/discussions/6040#discussioncomment-12862971
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Native Pagination Support for File Listing in OpenDAL
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo added a comment to the discussion: Native Pagination Support for File Listing in OpenDAL
> I'm using S3 and wanted to know how can I add offset and limit functionality to this code. The API should support pagination for the frontend.
opendal rust core provides an option called `start-after`, users can start list from offset in this way:
```rust
let lister = op.lister_with(path).start_after(offset).await?;
```
But this feature seems not exported to python yet. Let's create an issue to track it.
> Also, is there a way to get the total count of files in a directory without listing all of them?
Sadly, object storage services like s3 doesn't provide feature like this. We will need to list them all to get the count.
GitHub link: https://github.com/apache/opendal/discussions/6040#discussioncomment-12863006
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org