You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "yuzelin (via GitHub)" <gi...@apache.org> on 2024/09/04 06:15:46 UTC
[PR] Add TableRead#create_parallel_reader for reading multiple splits parallely [paimon-python]
yuzelin opened a new pull request, #11:
URL: https://github.com/apache/paimon-python/pull/11
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #xxx
<!-- What is the purpose of the change -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TableRead#create_parallel_reader for reading multiple splits parallely [paimon-python]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #11:
URL: https://github.com/apache/paimon-python/pull/11#discussion_r1745178786
##########
paimon_python_api/table_read.py:
##########
@@ -19,11 +19,21 @@
from abc import ABC, abstractmethod
from pyarrow import RecordBatchReader
from paimon_python_api.split import Split
+from paimon_python_api.table_scan import Plan
+from typing import List
class TableRead(ABC):
"""To read data from data splits."""
@abstractmethod
- def create_reader(self, split: Split) -> RecordBatchReader:
+ def create_reader(self, splits: List[Split]) -> RecordBatchReader:
"""Return a reader containing batches of pyarrow format."""
+
+ @abstractmethod
+ def create_reader_from_split(self, split: Split) -> RecordBatchReader:
+ """Reader that only read one split."""
+
+ def create_reader_from_plan(self, plan: Plan) -> RecordBatchReader:
Review Comment:
remove this method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TableRead supports reading multiple splits parallely [paimon-python]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #11:
URL: https://github.com/apache/paimon-python/pull/11#issuecomment-2342572920
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TableRead supports reading multiple splits parallely [paimon-python]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #11:
URL: https://github.com/apache/paimon-python/pull/11
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TableRead#create_parallel_reader for reading multiple splits parallely [paimon-python]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #11:
URL: https://github.com/apache/paimon-python/pull/11#discussion_r1745177868
##########
java_based_implementation/api_impl.py:
##########
@@ -113,24 +117,43 @@ def to_j_split(self):
class TableRead(table_read.TableRead):
- def __init__(self, j_table_read, j_row_type):
+ def __init__(self, j_table_read, j_row_type, catalog_options):
self._j_table_read = j_table_read
- self._j_bytes_reader = get_gateway().jvm.InvocationUtil.createBytesReader(
- j_table_read, j_row_type)
+ self._j_row_type = j_row_type
+ self._catalog_options = catalog_options
+ self._j_bytes_reader = None
self._arrow_schema = None
- def create_reader(self, split: Split):
- self._j_bytes_reader.setSplit(split.to_j_split())
- # get schema
+ def create_reader(self, splits):
+ self._init()
+ j_splits = list(map(lambda s: s.to_j_split(), splits))
+ self._j_bytes_reader.setSplits(j_splits)
+ batch_iterator = self._batch_generator()
+ return RecordBatchReader.from_batches(self._arrow_schema, batch_iterator)
+
+ def create_reader_from_split(self, split: Split):
Review Comment:
remove this method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org