You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "discivigour (via GitHub)" <gi...@apache.org> on 2026/07/07 04:36:42 UTC

[PR] [python] Resolve BlobView blobs with upstream FileIO [paimon]

discivigour opened a new pull request, #8488:
URL: https://github.com/apache/paimon/pull/8488

   ### Purpose
   
     Fix PyPaimon default BlobView reads to resolve upstream blob data with
     the upstream table FileIO instead of the downstream table FileIO.
   
     Previously, BlobView fields were first converted from `BlobViewStruct`
     bytes to `BlobDescriptor` bytes, then handled by the same descriptor
     resolution path as local `blob-descriptor-field` columns. That path
     uses the current table FileIO, which is wrong for BlobView values
     pointing to blobs in the source/upstream table.
   
     This patch:
     - records the upstream FileIO while resolving BlobView fields;
     - keeps ordinary `blob-descriptor-field` columns resolved with the
     current table FileIO;
     - resolves BlobView descriptor bytes with the recorded upstream FileIO;
     - keeps BlobView default reads on the concurrent blob read path when
     `blob_parallelism > 1`;
     - changes descriptor-byte resolution to build a file-backed `UriReader`
     from the passed FileIO context.
   
     ### Tests
   
     - `python -m py_compile pypaimon/read/reader/
     blob_descriptor_convert_reader.py pypaimon/utils/blob_view_lookup.py
     pypaimon/table/row/blob.py pypaimon/tests/blob_table_test.py`
     - `python -m unittest
     pypaimon.tests.blob_table_test.DedicatedFormatWriterTest.test_blob_view
     _fields_resolve_upstream_blob
     pypaimon.tests.blob_table_test.DedicatedFormatWriterTest.test_blob_view
     _default_read_uses_upstream_file_io`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [python] Resolve BlobView blobs with upstream FileIO [paimon]

Posted by "discivigour (via GitHub)" <gi...@apache.org>.
discivigour commented on code in PR #8488:
URL: https://github.com/apache/paimon/pull/8488#discussion_r3534249668


##########
paimon-python/pypaimon/read/reader/blob_descriptor_convert_reader.py:
##########
@@ -187,6 +189,39 @@ def _resolve_descriptor_fields(self, batch):
                 pyarrow.field(field_name, pyarrow.large_binary(), nullable=True),
                 pyarrow.array(converted_values, type=pyarrow.large_binary()),
             )
+
+        view_file_ios = view_file_ios or {}
+        for field_name in self._view_fields:
+            field_file_ios = view_file_ios.get(field_name)
+            if field_name not in batch.schema.names or field_file_ios is None:
+                continue
+            values = [self._normalize_blob_to_bytes(v) for v in batch.column(field_name).to_pylist()]
+            blobs_by_file_io = {}
+            converted_values = []
+
+            for idx, value in enumerate(values):
+                file_io = field_file_ios[idx] or self._table.file_io
+                blob = Blob.from_bytes(value, file_io)

Review Comment:
   I have resolved this problem in another pr #8489



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [python] Resolve BlobView blobs with upstream FileIO [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8488:
URL: https://github.com/apache/paimon/pull/8488#discussion_r3534151575


##########
paimon-python/pypaimon/read/reader/blob_descriptor_convert_reader.py:
##########
@@ -187,6 +189,39 @@ def _resolve_descriptor_fields(self, batch):
                 pyarrow.field(field_name, pyarrow.large_binary(), nullable=True),
                 pyarrow.array(converted_values, type=pyarrow.large_binary()),
             )
+
+        view_file_ios = view_file_ios or {}
+        for field_name in self._view_fields:
+            field_file_ios = view_file_ios.get(field_name)
+            if field_name not in batch.schema.names or field_file_ios is None:
+                continue
+            values = [self._normalize_blob_to_bytes(v) for v in batch.column(field_name).to_pylist()]
+            blobs_by_file_io = {}
+            converted_values = []
+
+            for idx, value in enumerate(values):
+                file_io = field_file_ios[idx] or self._table.file_io
+                blob = Blob.from_bytes(value, file_io)

Review Comment:
   This still goes through `Blob.from_bytes`, whose descriptor branch calls `file_io.uri_reader_factory.create(descriptor.uri)` instead of reading through the `file_io` object passed here. For BlobView fields that means the default non-parallel path can still rebuild a reader and lose the upstream FileIO context/credentials (for example `RESTTokenFileIO`), while the parallel path below uses `file_io.read_blobs_concurrent(...)` and does use the cached upstream FileIO. Could we construct the view-field `BlobRef` from this `file_io` directly here (or merge the `Blob.from_bytes` fix first) and add a test with an upstream FileIO whose factory must not be used?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [python] Resolve BlobView blobs with upstream FileIO [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8488:
URL: https://github.com/apache/paimon/pull/8488


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [python] Resolve BlobView blobs with upstream FileIO [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8488:
URL: https://github.com/apache/paimon/pull/8488#issuecomment-4901048458

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org