You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org> on 2026/01/08 02:19:37 UTC
[PR] [python] Fix IndexError when reading manifest with empty _MIN_VALUES [paimon]
XiaoHongbo-Hope opened a new pull request, #6971:
URL: https://github.com/apache/paimon/pull/6971
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close https://github.com/apache/paimon/issues/6962
<!-- What is the purpose of the change -->
### Tests
`BinaryRowTest`
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Fix IndexError when reading manifest with empty _MIN_VALUES [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #6971:
URL: https://github.com/apache/paimon/pull/6971#issuecomment-3721621984
Why Python does not produce empty _MIN_VALUES? Different to Java?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Fix IndexError when reading manifest with empty _MIN_VALUES [paimon]
Posted by "klboke (via GitHub)" <gi...@apache.org>.
klboke commented on PR #6971:
URL: https://github.com/apache/paimon/pull/6971#issuecomment-3721686813
After the repair, a new problem appeared, and the error message is as follows:
```python
INFO:pypaimon.catalog.rest.rest_token_file_io:end refresh data token for identifier [Identifier(database='adn', object='wide_table_200cols', branch=None)] expiresAtMillis [1767884499000]
2026-01-08 11:01:44,750 - paimon_dataset.py:189 - ERROR - Error reading table using Paimon API: '_VALUE_STATS_COLS'
Traceback (most recent call last):
File "/Users/kl/worknamespace/infra-mono/pkg/tap_common_io/taptap_io/paimon_dataset.py", line 190, in _paimon_table_to_data_files
raise e
File "/Users/kl/worknamespace/infra-mono/pkg/tap_common_io/taptap_io/paimon_dataset.py", line 178, in _paimon_table_to_data_files
splits = scan.plan().splits()
File "/Users/kl/worknamespace/infra-mono/pkg/tap_common_io/.venv-py/lib/python3.10/site-packages/pypaimon/read/table_scan.py", line 45, in plan
return self.starting_scanner.scan()
File "/Users/kl/worknamespace/infra-mono/pkg/tap_common_io/.venv-py/lib/python3.10/site-packages/pypaimon/read/scanner/full_starting_scanner.py", line 77, in scan
file_entries = self.plan_files()
File "/Users/kl/worknamespace/infra-mono/pkg/tap_common_io/.venv-py/lib/python3.10/site-packages/pypaimon/read/scanner/full_starting_scanner.py", line 95, in plan_files
return self.read_manifest_entries(manifest_files)
File "/Users/kl/worknamespace/infra-mono/pkg/tap_common_io/.venv-py/lib/python3.10/site-packages/pypaimon/read/scanner/full_starting_scanner.py", line 102, in read_manifest_entries
return self.manifest_file_manager.read_entries_parallel(manifest_files,
File "/Users/kl/worknamespace/infra-mono/pkg/tap_common_io/.venv-py/lib/python3.10/site-packages/pypaimon/manifest/manifest_file_manager.py", line 57, in read_entries_parallel
for entries in future_results:
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
yield _result_or_cancel(fs.pop())
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
return fut.result(timeout)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/kl/worknamespace/infra-mono/pkg/tap_common_io/.venv-py/lib/python3.10/site-packages/pypaimon/manifest/manifest_file_manager.py", line 51, in _process_single_manifest
return self.read(manifest_file.file_name, manifest_entry_filter, drop_stats)
File "/Users/kl/worknamespace/infra-mono/pkg/tap_common_io/.venv-py/lib/python3.10/site-packages/pypaimon/manifest/manifest_file_manager.py", line 90, in read
fields = self._get_value_stats_fields(file_dict, schema_fields)
File "/Users/kl/worknamespace/infra-mono/pkg/tap_common_io/.venv-py/lib/python3.10/site-packages/pypaimon/manifest/manifest_file_manager.py", line 134, in _get_value_stats_fields
if file_dict['_VALUE_STATS_COLS'] is None:
KeyError: '_VALUE_STATS_COLS'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Fix IndexError when reading manifest with empty _MIN_VALUES [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #6971:
URL: https://github.com/apache/paimon/pull/6971#issuecomment-3721701559
> After the repair, a new problem appeared, and the error message is as follows:
>
> ```python
> INFO:pypaimon.catalog.rest.rest_token_file_io:end refresh data token for identifier [Identifier(database='adn', object='wide_table_200cols', branch=None)] expiresAtMillis [1767884499000]
> 2026-01-08 11:01:44,750 - paimon_dataset.py:189 - ERROR - Error reading table using Paimon API: '_VALUE_STATS_COLS'
> Traceback (most recent call last):
> File "/Users/kl/worknamespace/infra-mono/pkg/tap_common_io/taptap_io/paimon_dataset.py", line 190, in _paimon_table_to_data_files
> raise e
> File "/Users/kl/worknamespace/infra-mono/pkg/tap_common_io/taptap_io/paimon_dataset.py", line 178, in _paimon_table_to_data_files
> splits = scan.plan().splits()
> File "/Users/kl/worknamespace/infra-mono/pkg/tap_common_io/.venv-py/lib/python3.10/site-packages/pypaimon/read/table_scan.py", line 45, in plan
> return self.starting_scanner.scan()
> File "/Users/kl/worknamespace/infra-mono/pkg/tap_common_io/.venv-py/lib/python3.10/site-packages/pypaimon/read/scanner/full_starting_scanner.py", line 77, in scan
> file_entries = self.plan_files()
> File "/Users/kl/worknamespace/infra-mono/pkg/tap_common_io/.venv-py/lib/python3.10/site-packages/pypaimon/read/scanner/full_starting_scanner.py", line 95, in plan_files
> return self.read_manifest_entries(manifest_files)
> File "/Users/kl/worknamespace/infra-mono/pkg/tap_common_io/.venv-py/lib/python3.10/site-packages/pypaimon/read/scanner/full_starting_scanner.py", line 102, in read_manifest_entries
> return self.manifest_file_manager.read_entries_parallel(manifest_files,
> File "/Users/kl/worknamespace/infra-mono/pkg/tap_common_io/.venv-py/lib/python3.10/site-packages/pypaimon/manifest/manifest_file_manager.py", line 57, in read_entries_parallel
> for entries in future_results:
> File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
> yield _result_or_cancel(fs.pop())
> File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
> return fut.result(timeout)
> File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 458, in result
> return self.__get_result()
> File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
> raise self._exception
> File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/thread.py", line 58, in run
> result = self.fn(*self.args, **self.kwargs)
> File "/Users/kl/worknamespace/infra-mono/pkg/tap_common_io/.venv-py/lib/python3.10/site-packages/pypaimon/manifest/manifest_file_manager.py", line 51, in _process_single_manifest
> return self.read(manifest_file.file_name, manifest_entry_filter, drop_stats)
> File "/Users/kl/worknamespace/infra-mono/pkg/tap_common_io/.venv-py/lib/python3.10/site-packages/pypaimon/manifest/manifest_file_manager.py", line 90, in read
> fields = self._get_value_stats_fields(file_dict, schema_fields)
> File "/Users/kl/worknamespace/infra-mono/pkg/tap_common_io/.venv-py/lib/python3.10/site-packages/pypaimon/manifest/manifest_file_manager.py", line 134, in _get_value_stats_fields
> if file_dict['_VALUE_STATS_COLS'] is None:
> KeyError: '_VALUE_STATS_COLS'
> ```
👌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python/hotfix] Fix IndexError when reading manifest with empty _MIN_VALUES [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope closed pull request #6971: [python/hotfix] Fix IndexError when reading manifest with empty _MIN_VALUES
URL: https://github.com/apache/paimon/pull/6971
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python/hotfix] Fix IndexError when reading manifest with empty _MIN_VALUES [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #6971:
URL: https://github.com/apache/paimon/pull/6971#issuecomment-3732236400
> Why Python does not produce empty _MIN_VALUES? Different to Java?
Actually, Python and Java produce the same result for empty stats - both serialize to **12 bytes**, which I added an assertion to show it. I searched all the git history, did not find the root cause of the 0 bytes MIN_VALUES. The data is written by older version pypaimon too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org