You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Fokko (via GitHub)" <gi...@apache.org> on 2023/06/06 09:16:46 UTC
[GitHub] [iceberg] Fokko opened a new pull request, #7778: Python: Add support for sequence number inheritance
Fokko opened a new pull request, #7778:
URL: https://github.com/apache/iceberg/pull/7778
When a manifest-entry doesn't have any sequence-number, it will inherit it from the manifest file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on a diff in pull request #7778: Python: Add support for sequence number inheritance
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #7778:
URL: https://github.com/apache/iceberg/pull/7778#discussion_r1220280670
##########
python/pyiceberg/manifest.py:
##########
@@ -266,11 +267,25 @@ def __init__(self, *data: Any, **named_data: Any) -> None:
def fetch_manifest_entry(self, io: FileIO) -> List[ManifestEntry]:
file = io.new_input(self.manifest_path)
- return list(read_manifest_entry(file))
+ return [_inherit_sequence_number(entry, self) for entry in read_manifest_entry(file)]
Review Comment:
Hmm. There were two paths for fetching the manifest entries, both through the `files` and `live_files`. I've removed the `files` and `live_files` to do everything through the ManifestList. Otherwise, we had to change the `files` and `live_files signatures by adding the ManifestList for doing the inheritance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #7778: Python: Add support for sequence number inheritance
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #7778:
URL: https://github.com/apache/iceberg/pull/7778#discussion_r1219844965
##########
python/pyiceberg/manifest.py:
##########
@@ -266,11 +267,25 @@ def __init__(self, *data: Any, **named_data: Any) -> None:
def fetch_manifest_entry(self, io: FileIO) -> List[ManifestEntry]:
file = io.new_input(self.manifest_path)
- return list(read_manifest_entry(file))
+ return [_inherit_sequence_number(entry, self) for entry in read_manifest_entry(file)]
Review Comment:
Why not do this in `read_manifest_entry`? I think it would make sense to pass the sequence number into that method so that there isn't a way for people to read without sequence numbers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #7778: Python: Add support for sequence number inheritance
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #7778:
URL: https://github.com/apache/iceberg/pull/7778#discussion_r1219862150
##########
python/tests/utils/test_manifest.py:
##########
@@ -232,8 +244,8 @@ def test_read_manifest_v2(generated_manifest_file_file_v2: str) -> None:
assert manifest_list.manifest_length == 7989
assert manifest_list.partition_spec_id == 0
assert manifest_list.content == ManifestContent.DELETES
- assert manifest_list.sequence_number is None # inheritance
- assert manifest_list.min_sequence_number is None # inheritance
+ assert manifest_list.sequence_number == 3 # inherited
+ assert manifest_list.min_sequence_number == 3 # inherited
Review Comment:
The manifest list's sequence numbers aren't inherited. They are always statically set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko merged pull request #7778: Python: Add support for sequence number inheritance
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko merged PR #7778:
URL: https://github.com/apache/iceberg/pull/7778
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #7778: Python: Add support for sequence number inheritance
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #7778:
URL: https://github.com/apache/iceberg/pull/7778#discussion_r1219865568
##########
python/tests/utils/test_manifest.py:
##########
@@ -253,3 +265,14 @@ def test_read_manifest_v2(generated_manifest_file_file_v2: str) -> None:
assert partition.contains_nan is False
assert partition.lower_bound == b"\x01\x00\x00\x00"
assert partition.upper_bound == b"\x02\x00\x00\x00"
+
+ entries = manifest_list.fetch_manifest_entry(io)
Review Comment:
How is the manifest list written? It looks like the location is static in the test fixtures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #7778: Python: Add support for sequence number inheritance
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #7778:
URL: https://github.com/apache/iceberg/pull/7778#discussion_r1219847972
##########
python/pyiceberg/manifest.py:
##########
@@ -283,7 +298,49 @@ def files(input_file: InputFile) -> Iterator[DataFile]:
def read_manifest_list(input_file: InputFile) -> Iterator[ManifestFile]:
+ """
+ Reads the manifests from the manifest list
+
+ Args:
+ input_file: The input file where the stream can be read from
+
+ Returns:
+ An iterator of ManifestFiles that are part of the list
+ """
with AvroFile[ManifestFile](
- input_file, MANIFEST_FILE_SCHEMA, {-1: ManifestFile, 508: PartitionFieldSummary}, {517: ManifestContent}
+ input_file,
+ MANIFEST_FILE_SCHEMA,
+ read_types={-1: ManifestFile, 508: PartitionFieldSummary},
+ read_enums={517: ManifestContent},
) as reader:
yield from reader
+
+
+def _inherit_sequence_number(entry: ManifestEntry, manifest: ManifestFile) -> ManifestEntry:
+ """Inherits the sequence numbers
+
+ More information in the spec: https://iceberg.apache.org/spec/#sequence-number-inheritance
+
+ Args:
+ entry: The manifest entry that has null sequence numbers
+ manifest: The manifest that has a sequence number
+
+ Returns:
+ The manifest entry with the sequence numbers set
+ """
+ # The snapshot_id is required in V1, inherit with V2 when null
+ if entry.snapshot_id is None:
+ entry.snapshot_id = manifest.added_snapshot_id
+
+ # in v1 tables, the data sequence number is not persisted and can be safely defaulted to 0
+ # in v2 tables, the data sequence number should be inherited iff the entry status is ADDED
+ if entry.sequence_number is None and (manifest.sequence_number == 0 or entry.status == ManifestEntryStatus.ADDED):
+ entry.sequence_number = manifest.sequence_number
Review Comment:
We may want to rename this to `data_sequence_number` since that was recently done in the JVM implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on a diff in pull request #7778: Python: Add support for sequence number inheritance
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #7778:
URL: https://github.com/apache/iceberg/pull/7778#discussion_r1220252532
##########
python/pyiceberg/manifest.py:
##########
@@ -266,11 +267,25 @@ def __init__(self, *data: Any, **named_data: Any) -> None:
def fetch_manifest_entry(self, io: FileIO) -> List[ManifestEntry]:
file = io.new_input(self.manifest_path)
- return list(read_manifest_entry(file))
+ return [_inherit_sequence_number(entry, self) for entry in read_manifest_entry(file)]
Review Comment:
Then we have to pass the `ManifestFile` through, but you're right. In hindsight, the `read_manifest_entry` should have been marked private.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on a diff in pull request #7778: Python: Add support for sequence number inheritance
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #7778:
URL: https://github.com/apache/iceberg/pull/7778#discussion_r1220253741
##########
python/pyiceberg/manifest.py:
##########
@@ -283,7 +298,49 @@ def files(input_file: InputFile) -> Iterator[DataFile]:
def read_manifest_list(input_file: InputFile) -> Iterator[ManifestFile]:
+ """
+ Reads the manifests from the manifest list
+
+ Args:
+ input_file: The input file where the stream can be read from
+
+ Returns:
+ An iterator of ManifestFiles that are part of the list
+ """
with AvroFile[ManifestFile](
- input_file, MANIFEST_FILE_SCHEMA, {-1: ManifestFile, 508: PartitionFieldSummary}, {517: ManifestContent}
+ input_file,
+ MANIFEST_FILE_SCHEMA,
+ read_types={-1: ManifestFile, 508: PartitionFieldSummary},
+ read_enums={517: ManifestContent},
) as reader:
yield from reader
+
+
+def _inherit_sequence_number(entry: ManifestEntry, manifest: ManifestFile) -> ManifestEntry:
+ """Inherits the sequence numbers
+
+ More information in the spec: https://iceberg.apache.org/spec/#sequence-number-inheritance
+
+ Args:
+ entry: The manifest entry that has null sequence numbers
+ manifest: The manifest that has a sequence number
+
+ Returns:
+ The manifest entry with the sequence numbers set
+ """
+ # The snapshot_id is required in V1, inherit with V2 when null
+ if entry.snapshot_id is None:
+ entry.snapshot_id = manifest.added_snapshot_id
+
+ # in v1 tables, the data sequence number is not persisted and can be safely defaulted to 0
+ # in v2 tables, the data sequence number should be inherited iff the entry status is ADDED
+ if entry.sequence_number is None and (manifest.sequence_number == 0 or entry.status == ManifestEntryStatus.ADDED):
+ entry.sequence_number = manifest.sequence_number
Review Comment:
I missed that change, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on a diff in pull request #7778: Python: Add support for sequence number inheritance
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #7778:
URL: https://github.com/apache/iceberg/pull/7778#discussion_r1220255517
##########
python/tests/utils/test_manifest.py:
##########
@@ -253,3 +265,14 @@ def test_read_manifest_v2(generated_manifest_file_file_v2: str) -> None:
assert partition.contains_nan is False
assert partition.lower_bound == b"\x01\x00\x00\x00"
assert partition.upper_bound == b"\x02\x00\x00\x00"
+
+ entries = manifest_list.fetch_manifest_entry(io)
Review Comment:
It creates a temporary directory:
```
@pytest.fixture(scope="session")
def generated_manifest_file_file_v2(
avro_schema_manifest_file_v2: Dict[str, Any], generated_manifest_entry_file: str
) -> Generator[str, None, None]:
from fastavro import parse_schema, writer
parsed_schema = parse_schema(avro_schema_manifest_file_v2)
# Make sure that a valid manifest_path is set
manifest_file_records_v2[0]["manifest_path"] = generated_manifest_entry_file
with TemporaryDirectory() as tmpdir:
tmp_avro_file = tmpdir + "/manifest.avro"
with open(tmp_avro_file, "wb") as out:
writer(out, parsed_schema, manifest_file_records_v2)
yield tmp_avro_file
```
##########
python/tests/utils/test_manifest.py:
##########
@@ -253,3 +265,14 @@ def test_read_manifest_v2(generated_manifest_file_file_v2: str) -> None:
assert partition.contains_nan is False
assert partition.lower_bound == b"\x01\x00\x00\x00"
assert partition.upper_bound == b"\x02\x00\x00\x00"
+
+ entries = manifest_list.fetch_manifest_entry(io)
Review Comment:
It creates a temporary directory:
```python
@pytest.fixture(scope="session")
def generated_manifest_file_file_v2(
avro_schema_manifest_file_v2: Dict[str, Any], generated_manifest_entry_file: str
) -> Generator[str, None, None]:
from fastavro import parse_schema, writer
parsed_schema = parse_schema(avro_schema_manifest_file_v2)
# Make sure that a valid manifest_path is set
manifest_file_records_v2[0]["manifest_path"] = generated_manifest_entry_file
with TemporaryDirectory() as tmpdir:
tmp_avro_file = tmpdir + "/manifest.avro"
with open(tmp_avro_file, "wb") as out:
writer(out, parsed_schema, manifest_file_records_v2)
yield tmp_avro_file
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on a diff in pull request #7778: Python: Add support for sequence number inheritance
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #7778:
URL: https://github.com/apache/iceberg/pull/7778#discussion_r1220280670
##########
python/pyiceberg/manifest.py:
##########
@@ -266,11 +267,25 @@ def __init__(self, *data: Any, **named_data: Any) -> None:
def fetch_manifest_entry(self, io: FileIO) -> List[ManifestEntry]:
file = io.new_input(self.manifest_path)
- return list(read_manifest_entry(file))
+ return [_inherit_sequence_number(entry, self) for entry in read_manifest_entry(file)]
Review Comment:
Hmm. There were two paths for fetching the manifest entries, both through the `files` and `live_files`. I've removed the `files` and `live_files` to do everything through the ManifestList. Otherwise, we had to change the `files` and `live_files` signatures by adding the ManifestList for doing the inheritance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org