You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org> on 2025/11/26 11:38:28 UTC
[PR] Ray data dev [paimon]
XiaoHongbo-Hope opened a new pull request, #6686:
URL: https://github.com/apache/paimon/pull/6686
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #xxx
<!-- What is the purpose of the change -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] support paimon as ray datasource [paimon]
Posted by "chenghuichen (via GitHub)" <gi...@apache.org>.
chenghuichen commented on code in PR #6686:
URL: https://github.com/apache/paimon/pull/6686#discussion_r2577711522
##########
paimon-python/pypaimon/read/table_read.py:
##########
@@ -126,10 +126,24 @@ def to_duckdb(self, splits: List[Split], table_name: str,
con.register(table_name, self.to_arrow(splits))
return con
- def to_ray(self, splits: List[Split]) -> "ray.data.dataset.Dataset":
+ def to_ray(self, splits: List[Split], use_distributed_read: bool = True) -> "ray.data.dataset.Dataset":
Review Comment:
Suggestion: Replace `use_distributed_read: bool` with `parallelism: int =1 `. This way, parallelism=1 indicates a single-task read, and other values can be passed directly to `ray.data.read_datasource(datasource, parallelism)`. `parallelism` is more practical, instead of just serving as a binary switch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] support paimon as ray datasource [paimon]
Posted by "chenghuichen (via GitHub)" <gi...@apache.org>.
chenghuichen commented on PR #6686:
URL: https://github.com/apache/paimon/pull/6686#issuecomment-3604778463
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] support paimon as ray datasource [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #6686:
URL: https://github.com/apache/paimon/pull/6686#issuecomment-3601608548
> It would be great if test cases for PrimaryKey tables could be added as well.
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] support paimon as ray datasource [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #6686:
URL: https://github.com/apache/paimon/pull/6686#issuecomment-3601607273
> It would be great if test cases for PrimaryKey tables could be added as well.
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] support paimon as ray datasource [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #6686:
URL: https://github.com/apache/paimon/pull/6686
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] support paimon as ray datasource [paimon]
Posted by "chenghuichen (via GitHub)" <gi...@apache.org>.
chenghuichen commented on PR #6686:
URL: https://github.com/apache/paimon/pull/6686#issuecomment-3597275143
It would be great if test cases for PrimaryKey tables could be added as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] support paimon as ray datasource [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #6686:
URL: https://github.com/apache/paimon/pull/6686#discussion_r2580804276
##########
paimon-python/pypaimon/read/table_read.py:
##########
@@ -126,10 +126,24 @@ def to_duckdb(self, splits: List[Split], table_name: str,
con.register(table_name, self.to_arrow(splits))
return con
- def to_ray(self, splits: List[Split]) -> "ray.data.dataset.Dataset":
+ def to_ray(self, splits: List[Split], use_distributed_read: bool = True) -> "ray.data.dataset.Dataset":
Review Comment:
> Suggestion: Replace `use_distributed_read: bool` with `parallelism: int =1 `. This way, parallelism=1 indicates a single-task read, and other values can be passed directly to `ray.data.read_datasource(datasource, parallelism)`. `parallelism` is more useful, instead of just serving as a binary switch.
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] support paimon as ray datasource [paimon]
Posted by "chenghuichen (via GitHub)" <gi...@apache.org>.
chenghuichen commented on code in PR #6686:
URL: https://github.com/apache/paimon/pull/6686#discussion_r2577711522
##########
paimon-python/pypaimon/read/table_read.py:
##########
@@ -126,10 +126,24 @@ def to_duckdb(self, splits: List[Split], table_name: str,
con.register(table_name, self.to_arrow(splits))
return con
- def to_ray(self, splits: List[Split]) -> "ray.data.dataset.Dataset":
+ def to_ray(self, splits: List[Split], use_distributed_read: bool = True) -> "ray.data.dataset.Dataset":
Review Comment:
Suggestion: Replace `use_distributed_read: bool` with `parallelism: int =1 `. This way, parallelism=1 indicates a single-task read, and other values can be passed directly to `ray.data.read_datasource(datasource, parallelism)`. `parallelism` is more useful, instead of just serving as a binary switch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ray data dev [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope closed pull request #6686: Ray data dev
URL: https://github.com/apache/paimon/pull/6686
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] support paimon as ray datasource [paimon]
Posted by "chenghuichen (via GitHub)" <gi...@apache.org>.
chenghuichen commented on code in PR #6686:
URL: https://github.com/apache/paimon/pull/6686#discussion_r2577711522
##########
paimon-python/pypaimon/read/table_read.py:
##########
@@ -126,10 +126,24 @@ def to_duckdb(self, splits: List[Split], table_name: str,
con.register(table_name, self.to_arrow(splits))
return con
- def to_ray(self, splits: List[Split]) -> "ray.data.dataset.Dataset":
+ def to_ray(self, splits: List[Split], use_distributed_read: bool = True) -> "ray.data.dataset.Dataset":
Review Comment:
Suggestion: Replace `use_distributed_read: bool` with `parallelism: int =1 `. This way, parallelism=1 indicates a single-task read, and other values can be passed directly to `ray.data.read_datasource(datasource, parallelism)`. `parallelism` is more intuitive and practical, instead of just serving as a binary switch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] support paimon as ray datasource [paimon]
Posted by "chenghuichen (via GitHub)" <gi...@apache.org>.
chenghuichen commented on code in PR #6686:
URL: https://github.com/apache/paimon/pull/6686#discussion_r2577711522
##########
paimon-python/pypaimon/read/table_read.py:
##########
@@ -126,10 +126,24 @@ def to_duckdb(self, splits: List[Split], table_name: str,
con.register(table_name, self.to_arrow(splits))
return con
- def to_ray(self, splits: List[Split]) -> "ray.data.dataset.Dataset":
+ def to_ray(self, splits: List[Split], use_distributed_read: bool = True) -> "ray.data.dataset.Dataset":
Review Comment:
Suggestion: Replace `use_distributed_read: bool` with `parallelism: int =1 `. This way, parallelism=1 indicates a single-task read, and other values can be passed directly to `ray.data.read_datasource(datasource, parallelism)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org