You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "rambleraptor (via GitHub)" <gi...@apache.org> on 2025/12/08 22:56:25 UTC

[PR] Write Deletion Vectors [iceberg-python]

rambleraptor opened a new pull request, #2822:
URL: https://github.com/apache/iceberg-python/pull/2822

   <!--
   Thanks for opening a pull request!
   -->
   
   <!-- In the case this PR will resolve an issue, please replace ${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
   Part of #2261
   
   # Rationale for this change
   This adds a PuffinWriter for writing deletion vectors.
   
   Right now, it's just the writer class + some round trip tests (where we read + write the same file) to sanity check that the PuffinWriter works as expected. Writing Puffin files is very complex, so I wanted to make sure we all agreed on the writing semantics before using this elsewhere.
   
   Let me know your thoughts on this (or if it's too granular)
   
   ## Are these changes tested?
   Unit tests included
   
   ## Are there any user-facing changes?
   
   <!-- In the case of user-facing changes, please add the changelog label. -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#issuecomment-4139211666

   This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#discussion_r2605347832


##########
pyiceberg/table/puffin.py:
##########
@@ -114,3 +147,97 @@ def __init__(self, puffin: bytes) -> None:
 
     def to_vector(self) -> dict[str, "pa.ChunkedArray"]:
         return {path: _bitmaps_to_chunked_array(bitmaps) for path, bitmaps in self._deletion_vectors.items()}
+
+
+class PuffinWriter:
+    _blobs: list[PuffinBlobMetadata]
+    _blob_payloads: list[bytes]
+
+    def __init__(self) -> None:
+        self._blobs = []
+        self._blob_payloads = []
+
+    def add(
+        self,
+        positions: Iterable[int],
+        referenced_data_file: str,
+    ) -> None:
+        # 1. Create bitmaps from positions
+        bitmaps: dict[int, BitMap] = {}
+        cardinality = 0
+        for pos in positions:
+            cardinality += 1

Review Comment:
   cardinality could be incorrect with same positions passed in



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#discussion_r2605347832


##########
pyiceberg/table/puffin.py:
##########
@@ -114,3 +147,97 @@ def __init__(self, puffin: bytes) -> None:
 
     def to_vector(self) -> dict[str, "pa.ChunkedArray"]:
         return {path: _bitmaps_to_chunked_array(bitmaps) for path, bitmaps in self._deletion_vectors.items()}
+
+
+class PuffinWriter:
+    _blobs: list[PuffinBlobMetadata]
+    _blob_payloads: list[bytes]
+
+    def __init__(self) -> None:
+        self._blobs = []
+        self._blob_payloads = []
+
+    def add(
+        self,
+        positions: Iterable[int],
+        referenced_data_file: str,
+    ) -> None:
+        # 1. Create bitmaps from positions
+        bitmaps: dict[int, BitMap] = {}
+        cardinality = 0
+        for pos in positions:
+            cardinality += 1

Review Comment:
   cardinality could be incorrect with same positions passed in we can probably use the pyroaring stats to get this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "moomindani (via GitHub)" <gi...@apache.org>.
moomindani commented on PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#issuecomment-4627143323

   Thanks @glesperance for the candid update, and totally understand stepping back.
   
   @rambleraptor — gentle nudge on my note above: are you still planning to drive this, or is it open for someone to carry forward? Happy to pick it up (crediting you both) if you're tied up — just don't want to step on your effort.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "glesperance (via GitHub)" <gi...@apache.org>.
glesperance commented on PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#issuecomment-3639829500

   Hey @rambleraptor, I was working on a DV implementation before discovering this PR. Since review is already underway, I'd rather contribute here than duplicate effort.
   
   I've added a Spark interoperability test: glesperance@80c611c4
   
   This verifies pyiceberg can read Spark-written DVs. Combined with your existing round-trip tests, this confirms format compatibility... ie if the same reader handles both, Spark can read ours too.
   
   This may be redundant with your existing .bin fixture tests, though I believe those test the raw bitmap format rather than full Puffin DVs with the Java wrapper (length + magic + CRC). Let me know if I'm wrong on that.
   
   Happy to PR to your fork if you think it's pertinent -or- feel free to cherry pick the commit as you see fit.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on code in PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#discussion_r2612504183


##########
pyiceberg/table/puffin.py:
##########
@@ -114,3 +147,97 @@ def __init__(self, puffin: bytes) -> None:
 
     def to_vector(self) -> dict[str, "pa.ChunkedArray"]:
         return {path: _bitmaps_to_chunked_array(bitmaps) for path, bitmaps in self._deletion_vectors.items()}
+
+
+class PuffinWriter:
+    _blobs: list[PuffinBlobMetadata]
+    _blob_payloads: list[bytes]
+
+    def __init__(self) -> None:
+        self._blobs = []
+        self._blob_payloads = []
+
+    def add(
+        self,
+        positions: Iterable[int],
+        referenced_data_file: str,
+    ) -> None:
+        # 1. Create bitmaps from positions
+        bitmaps: dict[int, BitMap] = {}
+        cardinality = 0
+        for pos in positions:
+            cardinality += 1
+            key = pos >> 32
+            low_bits = pos & 0xFFFFFFFF
+            if key not in bitmaps:
+                bitmaps[key] = BitMap()
+            bitmaps[key].add(low_bits)
+
+        # 2. Serialize bitmaps for the vector payload
+        vector_payload = _serialize_bitmaps(bitmaps)
+
+        # 3. Construct the full blob payload for deletion-vector-v1
+        with io.BytesIO() as blob_payload_buffer:
+            # Magic bytes for DV
+            blob_payload_buffer.write(DELETION_VECTOR_MAGIC)
+            # The vector itself
+            blob_payload_buffer.write(vector_payload)
+
+            # The content for CRC calculation
+            crc_content = blob_payload_buffer.getvalue()
+            crc32 = zlib.crc32(crc_content)
+
+            # The full blob to be stored in the Puffin file
+            with io.BytesIO() as full_blob_buffer:
+                # Combined length of the vector and magic bytes stored as 4 bytes, big-endian
+                full_blob_buffer.write(len(crc_content).to_bytes(4, "big"))
+                # The content (magic + vector)
+                full_blob_buffer.write(crc_content)
+                # A CRC-32 checksum of the magic bytes and serialized vector as 4 bytes, big-endian
+                full_blob_buffer.write(crc32.to_bytes(4, "big"))
+
+                self._blob_payloads.append(full_blob_buffer.getvalue())
+
+        # 4. Create blob metadata
+        properties = {PROPERTY_REFERENCED_DATA_FILE: referenced_data_file, "cardinality": str(cardinality)}
+
+        self._blobs.append(
+            PuffinBlobMetadata(
+                type="deletion-vector-v1",
+                fields=[],

Review Comment:
   Yep, added that in. Great find!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on code in PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#discussion_r2604965779


##########
pyiceberg/table/puffin.py:
##########
@@ -114,3 +147,97 @@ def __init__(self, puffin: bytes) -> None:
 
     def to_vector(self) -> dict[str, "pa.ChunkedArray"]:
         return {path: _bitmaps_to_chunked_array(bitmaps) for path, bitmaps in self._deletion_vectors.items()}
+
+
+class PuffinWriter:
+    _blobs: list[PuffinBlobMetadata]
+    _blob_payloads: list[bytes]
+
+    def __init__(self) -> None:
+        self._blobs = []
+        self._blob_payloads = []
+
+    def add(
+        self,
+        positions: Iterable[int],
+        referenced_data_file: str,
+    ) -> None:
+        # 1. Create bitmaps from positions
+        bitmaps: dict[int, BitMap] = {}
+        cardinality = 0
+        for pos in positions:
+            cardinality += 1
+            key = pos >> 32
+            low_bits = pos & 0xFFFFFFFF
+            if key not in bitmaps:
+                bitmaps[key] = BitMap()
+            bitmaps[key].add(low_bits)
+
+        # 2. Serialize bitmaps for the vector payload
+        vector_payload = _serialize_bitmaps(bitmaps)
+
+        # 3. Construct the full blob payload for deletion-vector-v1
+        with io.BytesIO() as blob_payload_buffer:
+            # Magic bytes for DV
+            blob_payload_buffer.write(DELETION_VECTOR_MAGIC)
+            # The vector itself
+            blob_payload_buffer.write(vector_payload)
+
+            # The content for CRC calculation
+            crc_content = blob_payload_buffer.getvalue()
+            crc32 = zlib.crc32(crc_content)
+
+            # The full blob to be stored in the Puffin file
+            with io.BytesIO() as full_blob_buffer:
+                # Combined length of the vector and magic bytes stored as 4 bytes, big-endian
+                full_blob_buffer.write(len(crc_content).to_bytes(4, "big"))
+                # The content (magic + vector)
+                full_blob_buffer.write(crc_content)
+                # A CRC-32 checksum of the magic bytes and serialized vector as 4 bytes, big-endian
+                full_blob_buffer.write(crc32.to_bytes(4, "big"))
+
+                self._blob_payloads.append(full_blob_buffer.getvalue())
+
+        # 4. Create blob metadata
+        properties = {PROPERTY_REFERENCED_DATA_FILE: referenced_data_file, "cardinality": str(cardinality)}
+
+        self._blobs.append(
+            PuffinBlobMetadata(
+                type="deletion-vector-v1",
+                fields=[],

Review Comment:
   Does this field correspond to `BlobMetadata.inputFields` in Iceberg Java? Java implementation writes 2147483645 (Integer.MAX_VALUE - 2): https://github.com/apache/iceberg/blob/c6ba7a4c48cf1280eeff35829492fd7e6a5f6787/core/src/main/java/org/apache/iceberg/deletes/BaseDVFileWriter.java#L158



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "glesperance (via GitHub)" <gi...@apache.org>.
glesperance commented on PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#issuecomment-3661206114

   Really excited to see this moving forward.
   
   @rambleraptor Thanks for the opportunity to contribute and for handling the updates. On full DV support, have you started on the delete/manifest writers for v3 and the MOR logic?
   
   I’ve got a working PoC with some tests but it’ll certainly need more polish before it's PR-ready.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "moomindani (via GitHub)" <gi...@apache.org>.
moomindani commented on PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#issuecomment-4664913707

   Picked this up as #3474 — it keeps your commits as-is (@rambleraptor, @glesperance) and adds unit tests for a couple of agreed review items (@ebyhr's field id, and the DV blob framing) that weren't yet covered by a test. Continuing the deletion-vector write work from there. Thanks both!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "moomindani (via GitHub)" <gi...@apache.org>.
moomindani commented on PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#issuecomment-4602378202

   Hi @rambleraptor @glesperance — I'd like to help get deletion-vector write support across the line.
   
   To check it's still viable, I **locally** applied this PR's `PuffinWriter` changes on top of the current `main` (no push here): they apply cleanly with no conflicts, and `tests/table/test_puffin.py` passes (7/7).
   
   The DV-write pieces seem to have stalled on inactivity rather than merit — this PR, #2193 (roaring bitmap), and #2180 (v3 metadata write) were all auto-closed by the stale bot, and @kevinjqliu had said "let's get this in" on #2193.
   
   @rambleraptor @glesperance are either of you still planning to drive this? I don't want to duplicate your effort. If it's open, I'd be glad to carry it forward — e.g. open a fresh PR that continues this `PuffinWriter` work (crediting you both), then layer the v3 delete/manifest writers + MoR on top.
   
   (Tracking issue for context: #2261.)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on code in PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#discussion_r2604965779


##########
pyiceberg/table/puffin.py:
##########
@@ -114,3 +147,97 @@ def __init__(self, puffin: bytes) -> None:
 
     def to_vector(self) -> dict[str, "pa.ChunkedArray"]:
         return {path: _bitmaps_to_chunked_array(bitmaps) for path, bitmaps in self._deletion_vectors.items()}
+
+
+class PuffinWriter:
+    _blobs: list[PuffinBlobMetadata]
+    _blob_payloads: list[bytes]
+
+    def __init__(self) -> None:
+        self._blobs = []
+        self._blob_payloads = []
+
+    def add(
+        self,
+        positions: Iterable[int],
+        referenced_data_file: str,
+    ) -> None:
+        # 1. Create bitmaps from positions
+        bitmaps: dict[int, BitMap] = {}
+        cardinality = 0
+        for pos in positions:
+            cardinality += 1
+            key = pos >> 32
+            low_bits = pos & 0xFFFFFFFF
+            if key not in bitmaps:
+                bitmaps[key] = BitMap()
+            bitmaps[key].add(low_bits)
+
+        # 2. Serialize bitmaps for the vector payload
+        vector_payload = _serialize_bitmaps(bitmaps)
+
+        # 3. Construct the full blob payload for deletion-vector-v1
+        with io.BytesIO() as blob_payload_buffer:
+            # Magic bytes for DV
+            blob_payload_buffer.write(DELETION_VECTOR_MAGIC)
+            # The vector itself
+            blob_payload_buffer.write(vector_payload)
+
+            # The content for CRC calculation
+            crc_content = blob_payload_buffer.getvalue()
+            crc32 = zlib.crc32(crc_content)
+
+            # The full blob to be stored in the Puffin file
+            with io.BytesIO() as full_blob_buffer:
+                # Combined length of the vector and magic bytes stored as 4 bytes, big-endian
+                full_blob_buffer.write(len(crc_content).to_bytes(4, "big"))
+                # The content (magic + vector)
+                full_blob_buffer.write(crc_content)
+                # A CRC-32 checksum of the magic bytes and serialized vector as 4 bytes, big-endian
+                full_blob_buffer.write(crc32.to_bytes(4, "big"))
+
+                self._blob_payloads.append(full_blob_buffer.getvalue())
+
+        # 4. Create blob metadata
+        properties = {PROPERTY_REFERENCED_DATA_FILE: referenced_data_file, "cardinality": str(cardinality)}
+
+        self._blobs.append(
+            PuffinBlobMetadata(
+                type="deletion-vector-v1",
+                fields=[],

Review Comment:
   Does this `fields` correspond to `BlobMetadata.inputFields` in Iceberg Java? Java implementation writes 2147483645 (Integer.MAX_VALUE - 2): https://github.com/apache/iceberg/blob/c6ba7a4c48cf1280eeff35829492fd7e6a5f6787/core/src/main/java/org/apache/iceberg/deletes/BaseDVFileWriter.java#L158



##########
pyiceberg/table/puffin.py:
##########
@@ -114,3 +147,97 @@ def __init__(self, puffin: bytes) -> None:
 
     def to_vector(self) -> dict[str, "pa.ChunkedArray"]:
         return {path: _bitmaps_to_chunked_array(bitmaps) for path, bitmaps in self._deletion_vectors.items()}
+
+
+class PuffinWriter:
+    _blobs: list[PuffinBlobMetadata]
+    _blob_payloads: list[bytes]
+
+    def __init__(self) -> None:
+        self._blobs = []
+        self._blob_payloads = []
+
+    def add(
+        self,
+        positions: Iterable[int],
+        referenced_data_file: str,
+    ) -> None:
+        # 1. Create bitmaps from positions
+        bitmaps: dict[int, BitMap] = {}
+        cardinality = 0
+        for pos in positions:
+            cardinality += 1
+            key = pos >> 32
+            low_bits = pos & 0xFFFFFFFF
+            if key not in bitmaps:
+                bitmaps[key] = BitMap()
+            bitmaps[key].add(low_bits)
+
+        # 2. Serialize bitmaps for the vector payload
+        vector_payload = _serialize_bitmaps(bitmaps)
+
+        # 3. Construct the full blob payload for deletion-vector-v1
+        with io.BytesIO() as blob_payload_buffer:
+            # Magic bytes for DV
+            blob_payload_buffer.write(DELETION_VECTOR_MAGIC)
+            # The vector itself
+            blob_payload_buffer.write(vector_payload)
+
+            # The content for CRC calculation
+            crc_content = blob_payload_buffer.getvalue()
+            crc32 = zlib.crc32(crc_content)
+
+            # The full blob to be stored in the Puffin file
+            with io.BytesIO() as full_blob_buffer:
+                # Combined length of the vector and magic bytes stored as 4 bytes, big-endian
+                full_blob_buffer.write(len(crc_content).to_bytes(4, "big"))
+                # The content (magic + vector)
+                full_blob_buffer.write(crc_content)
+                # A CRC-32 checksum of the magic bytes and serialized vector as 4 bytes, big-endian
+                full_blob_buffer.write(crc32.to_bytes(4, "big"))
+
+                self._blob_payloads.append(full_blob_buffer.getvalue())
+
+        # 4. Create blob metadata
+        properties = {PROPERTY_REFERENCED_DATA_FILE: referenced_data_file, "cardinality": str(cardinality)}
+
+        self._blobs.append(
+            PuffinBlobMetadata(
+                type="deletion-vector-v1",
+                fields=[],

Review Comment:
   Does this `fields` correspond to `BlobMetadata.inputFields` in Iceberg Java? Java implementation writes 2147483645 (Integer.MAX_VALUE - 2) for DV: https://github.com/apache/iceberg/blob/c6ba7a4c48cf1280eeff35829492fd7e6a5f6787/core/src/main/java/org/apache/iceberg/deletes/BaseDVFileWriter.java#L158



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#discussion_r2605022493


##########
pyiceberg/table/puffin.py:
##########
@@ -114,3 +147,97 @@ def __init__(self, puffin: bytes) -> None:
 
     def to_vector(self) -> dict[str, "pa.ChunkedArray"]:
         return {path: _bitmaps_to_chunked_array(bitmaps) for path, bitmaps in self._deletion_vectors.items()}
+
+
+class PuffinWriter:
+    _blobs: list[PuffinBlobMetadata]
+    _blob_payloads: list[bytes]
+
+    def __init__(self) -> None:
+        self._blobs = []
+        self._blob_payloads = []
+
+    def add(
+        self,
+        positions: Iterable[int],
+        referenced_data_file: str,
+    ) -> None:
+        # 1. Create bitmaps from positions
+        bitmaps: dict[int, BitMap] = {}
+        cardinality = 0
+        for pos in positions:
+            cardinality += 1
+            key = pos >> 32
+            low_bits = pos & 0xFFFFFFFF
+            if key not in bitmaps:
+                bitmaps[key] = BitMap()
+            bitmaps[key].add(low_bits)
+
+        # 2. Serialize bitmaps for the vector payload
+        vector_payload = _serialize_bitmaps(bitmaps)
+
+        # 3. Construct the full blob payload for deletion-vector-v1
+        with io.BytesIO() as blob_payload_buffer:
+            # Magic bytes for DV
+            blob_payload_buffer.write(DELETION_VECTOR_MAGIC)
+            # The vector itself
+            blob_payload_buffer.write(vector_payload)
+
+            # The content for CRC calculation
+            crc_content = blob_payload_buffer.getvalue()
+            crc32 = zlib.crc32(crc_content)
+
+            # The full blob to be stored in the Puffin file
+            with io.BytesIO() as full_blob_buffer:
+                # Combined length of the vector and magic bytes stored as 4 bytes, big-endian
+                full_blob_buffer.write(len(crc_content).to_bytes(4, "big"))
+                # The content (magic + vector)
+                full_blob_buffer.write(crc_content)
+                # A CRC-32 checksum of the magic bytes and serialized vector as 4 bytes, big-endian
+                full_blob_buffer.write(crc32.to_bytes(4, "big"))
+
+                self._blob_payloads.append(full_blob_buffer.getvalue())
+
+        # 4. Create blob metadata
+        properties = {PROPERTY_REFERENCED_DATA_FILE: referenced_data_file, "cardinality": str(cardinality)}
+
+        self._blobs.append(
+            PuffinBlobMetadata(
+                type="deletion-vector-v1",
+                fields=[],

Review Comment:
   with field ids be set later too



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on code in PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#discussion_r2612481186


##########
tests/table/test_puffin.py:
##########
@@ -71,3 +71,78 @@ def test_map_high_vals() -> None:
 
     with pytest.raises(ValueError, match="Key 4022190063 is too large, max 2147483647 to maintain compatibility with Java impl"):
         _ = _deserialize_bitmap(puffin)
+
+
+def test_puffin_round_trip() -> None:
+    # Define some deletion positions for multiple files
+    deletions1 = [10, 20, 30]

Review Comment:
   The Caller should provide a merged blob. I've changed this method to `set_blob` to make it more apparent.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on code in PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#discussion_r2612468858


##########
pyiceberg/table/puffin.py:
##########
@@ -114,3 +147,97 @@ def __init__(self, puffin: bytes) -> None:
 
     def to_vector(self) -> dict[str, "pa.ChunkedArray"]:
         return {path: _bitmaps_to_chunked_array(bitmaps) for path, bitmaps in self._deletion_vectors.items()}
+
+
+class PuffinWriter:
+    _blobs: list[PuffinBlobMetadata]
+    _blob_payloads: list[bytes]
+
+    def __init__(self) -> None:
+        self._blobs = []
+        self._blob_payloads = []
+
+    def add(
+        self,
+        positions: Iterable[int],
+        referenced_data_file: str,
+    ) -> None:
+        # 1. Create bitmaps from positions
+        bitmaps: dict[int, BitMap] = {}
+        cardinality = 0
+        for pos in positions:
+            cardinality += 1

Review Comment:
   I think I made the proper change, let me know if you're thinking differently.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on code in PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#discussion_r2604941480


##########
pyiceberg/table/puffin.py:
##########
@@ -114,3 +147,97 @@ def __init__(self, puffin: bytes) -> None:
 
     def to_vector(self) -> dict[str, "pa.ChunkedArray"]:
         return {path: _bitmaps_to_chunked_array(bitmaps) for path, bitmaps in self._deletion_vectors.items()}
+
+
+class PuffinWriter:

Review Comment:
   Iceberg spec recommends writing `created-by` property in Puffin files: https://iceberg.apache.org/puffin-spec/#deletion-vector-v1-blob-type. Could you please write the property? 



##########
tests/table/test_puffin.py:
##########
@@ -71,3 +71,78 @@ def test_map_high_vals() -> None:
 
     with pytest.raises(ValueError, match="Key 4022190063 is too large, max 2147483647 to maintain compatibility with Java impl"):
         _ = _deserialize_bitmap(puffin)
+
+
+def test_puffin_round_trip() -> None:
+    # Define some deletion positions for multiple files
+    deletions1 = [10, 20, 30]

Review Comment:
   Iceberg deletion vectors expect 1:1 for DV:path. The current PuffinWriter may violate the expectation. 
   
   For instance, the following code writes 2 blobs without merging them. Do you fix this behavior as a follow-up, or the caller should provide the merged DV? 
   ```py
       deletions1a = [10, 20]
       deletions1b = [30]
   ...
       writer.add(positions=deletions1a, referenced_data_file=file1_path)
       writer.add(positions=deletions1b, referenced_data_file=file1_path)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#discussion_r2605022493


##########
pyiceberg/table/puffin.py:
##########
@@ -114,3 +147,97 @@ def __init__(self, puffin: bytes) -> None:
 
     def to_vector(self) -> dict[str, "pa.ChunkedArray"]:
         return {path: _bitmaps_to_chunked_array(bitmaps) for path, bitmaps in self._deletion_vectors.items()}
+
+
+class PuffinWriter:
+    _blobs: list[PuffinBlobMetadata]
+    _blob_payloads: list[bytes]
+
+    def __init__(self) -> None:
+        self._blobs = []
+        self._blob_payloads = []
+
+    def add(
+        self,
+        positions: Iterable[int],
+        referenced_data_file: str,
+    ) -> None:
+        # 1. Create bitmaps from positions
+        bitmaps: dict[int, BitMap] = {}
+        cardinality = 0
+        for pos in positions:
+            cardinality += 1
+            key = pos >> 32
+            low_bits = pos & 0xFFFFFFFF
+            if key not in bitmaps:
+                bitmaps[key] = BitMap()
+            bitmaps[key].add(low_bits)
+
+        # 2. Serialize bitmaps for the vector payload
+        vector_payload = _serialize_bitmaps(bitmaps)
+
+        # 3. Construct the full blob payload for deletion-vector-v1
+        with io.BytesIO() as blob_payload_buffer:
+            # Magic bytes for DV
+            blob_payload_buffer.write(DELETION_VECTOR_MAGIC)
+            # The vector itself
+            blob_payload_buffer.write(vector_payload)
+
+            # The content for CRC calculation
+            crc_content = blob_payload_buffer.getvalue()
+            crc32 = zlib.crc32(crc_content)
+
+            # The full blob to be stored in the Puffin file
+            with io.BytesIO() as full_blob_buffer:
+                # Combined length of the vector and magic bytes stored as 4 bytes, big-endian
+                full_blob_buffer.write(len(crc_content).to_bytes(4, "big"))
+                # The content (magic + vector)
+                full_blob_buffer.write(crc_content)
+                # A CRC-32 checksum of the magic bytes and serialized vector as 4 bytes, big-endian
+                full_blob_buffer.write(crc32.to_bytes(4, "big"))
+
+                self._blob_payloads.append(full_blob_buffer.getvalue())
+
+        # 4. Create blob metadata
+        properties = {PROPERTY_REFERENCED_DATA_FILE: referenced_data_file, "cardinality": str(cardinality)}
+
+        self._blobs.append(
+            PuffinBlobMetadata(
+                type="deletion-vector-v1",
+                fields=[],

Review Comment:
   with field ids be set later too



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "glesperance (via GitHub)" <gi...@apache.org>.
glesperance commented on PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#issuecomment-4625496614

   @moomindani I largely stepped back from the initiative because progress on the pieces I was interested in seemed quite slow from my perspective.
   
   My reading of the repository activity is that development is increasingly happening in iceberg-rust, and my assumption is that this is where a lot of future Iceberg-related work will land.
   
   I may be missing some context, or perhaps decisions are being discussed elsewhere, but from the repository alone it's difficult for me to understand why the PRs were closed and what the current roadmap is.
   
   The move towards v3 has been ongoing for over a year now (https://github.com/apache/iceberg-python/issues/1818), so I'll let others form their own view on the pace of that effort.
   
   Personally, the current rate of progress gives me some concern, and it's caused me to evaluate other options for my long-term data lake format choice.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on code in PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#discussion_r2612324590


##########
pyiceberg/table/puffin.py:
##########
@@ -114,3 +147,97 @@ def __init__(self, puffin: bytes) -> None:
 
     def to_vector(self) -> dict[str, "pa.ChunkedArray"]:
         return {path: _bitmaps_to_chunked_array(bitmaps) for path, bitmaps in self._deletion_vectors.items()}
+
+
+class PuffinWriter:
+    _blobs: list[PuffinBlobMetadata]
+    _blob_payloads: list[bytes]
+
+    def __init__(self) -> None:
+        self._blobs = []
+        self._blob_payloads = []
+
+    def add(
+        self,
+        positions: Iterable[int],
+        referenced_data_file: str,
+    ) -> None:
+        # 1. Create bitmaps from positions
+        bitmaps: dict[int, BitMap] = {}
+        cardinality = 0
+        for pos in positions:
+            cardinality += 1
+            key = pos >> 32
+            low_bits = pos & 0xFFFFFFFF
+            if key not in bitmaps:
+                bitmaps[key] = BitMap()
+            bitmaps[key].add(low_bits)
+
+        # 2. Serialize bitmaps for the vector payload
+        vector_payload = _serialize_bitmaps(bitmaps)
+
+        # 3. Construct the full blob payload for deletion-vector-v1
+        with io.BytesIO() as blob_payload_buffer:
+            # Magic bytes for DV
+            blob_payload_buffer.write(DELETION_VECTOR_MAGIC)
+            # The vector itself
+            blob_payload_buffer.write(vector_payload)
+
+            # The content for CRC calculation
+            crc_content = blob_payload_buffer.getvalue()
+            crc32 = zlib.crc32(crc_content)
+
+            # The full blob to be stored in the Puffin file
+            with io.BytesIO() as full_blob_buffer:
+                # Combined length of the vector and magic bytes stored as 4 bytes, big-endian
+                full_blob_buffer.write(len(crc_content).to_bytes(4, "big"))
+                # The content (magic + vector)
+                full_blob_buffer.write(crc_content)
+                # A CRC-32 checksum of the magic bytes and serialized vector as 4 bytes, big-endian
+                full_blob_buffer.write(crc32.to_bytes(4, "big"))
+
+                self._blob_payloads.append(full_blob_buffer.getvalue())
+
+        # 4. Create blob metadata
+        properties = {PROPERTY_REFERENCED_DATA_FILE: referenced_data_file, "cardinality": str(cardinality)}
+
+        self._blobs.append(
+            PuffinBlobMetadata(
+                type="deletion-vector-v1",
+                fields=[],
+                snapshot_id=-1,
+                sequence_number=-1,
+                offset=0,  # Will be set later
+                length=0,  # Will be set later
+                properties=properties,
+                compression_codec=None,  # Explicitly None
+            )
+        )
+
+    def finish(self) -> bytes:
+        with io.BytesIO() as out:
+            payload_buffer = io.BytesIO()
+            for blob_payload in self._blob_payloads:
+                payload_buffer.write(blob_payload)
+
+            updated_blobs_metadata: list[PuffinBlobMetadata] = []
+            current_offset = 4  # Start after file magic (4 bytes)
+            for i, blob_payload in enumerate(self._blob_payloads):
+                original_metadata_dict = self._blobs[i].model_dump(by_alias=True, exclude_none=True)
+                original_metadata_dict["offset"] = current_offset
+                original_metadata_dict["length"] = len(blob_payload)
+                updated_blobs_metadata.append(PuffinBlobMetadata(**original_metadata_dict))
+                current_offset += len(blob_payload)
+
+            footer = Footer(blobs=updated_blobs_metadata)
+            footer_payload_bytes = footer.model_dump_json(by_alias=True, exclude_none=True).encode("utf-8")
+
+            # Final assembly
+            out.write(MAGIC_BYTES)
+            out.write(payload_buffer.getvalue())
+            out.write(footer_payload_bytes)

Review Comment:
   Yep, looks like I missed one of the magics. Added, thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#issuecomment-3644095413

   @glesperance Thanks so much! I patched in your commit and I'll push it up along with my changes. Your name should appear in the commit log + PR. Let me know if you don't see it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#issuecomment-3644420430

   PR comments have been addressed.
   
   @geruh it looks like your work on DeleteFileIndexes will be very useful for determing offsets + lengths on the blobs!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #2822:
URL: https://github.com/apache/iceberg-python/pull/2822#discussion_r2605149847


##########
pyiceberg/table/puffin.py:
##########
@@ -114,3 +147,97 @@ def __init__(self, puffin: bytes) -> None:
 
     def to_vector(self) -> dict[str, "pa.ChunkedArray"]:
         return {path: _bitmaps_to_chunked_array(bitmaps) for path, bitmaps in self._deletion_vectors.items()}
+
+
+class PuffinWriter:
+    _blobs: list[PuffinBlobMetadata]
+    _blob_payloads: list[bytes]
+
+    def __init__(self) -> None:
+        self._blobs = []
+        self._blob_payloads = []
+
+    def add(
+        self,
+        positions: Iterable[int],
+        referenced_data_file: str,
+    ) -> None:
+        # 1. Create bitmaps from positions
+        bitmaps: dict[int, BitMap] = {}
+        cardinality = 0
+        for pos in positions:
+            cardinality += 1
+            key = pos >> 32
+            low_bits = pos & 0xFFFFFFFF
+            if key not in bitmaps:
+                bitmaps[key] = BitMap()
+            bitmaps[key].add(low_bits)
+
+        # 2. Serialize bitmaps for the vector payload
+        vector_payload = _serialize_bitmaps(bitmaps)
+
+        # 3. Construct the full blob payload for deletion-vector-v1
+        with io.BytesIO() as blob_payload_buffer:
+            # Magic bytes for DV
+            blob_payload_buffer.write(DELETION_VECTOR_MAGIC)
+            # The vector itself
+            blob_payload_buffer.write(vector_payload)
+
+            # The content for CRC calculation
+            crc_content = blob_payload_buffer.getvalue()
+            crc32 = zlib.crc32(crc_content)
+
+            # The full blob to be stored in the Puffin file
+            with io.BytesIO() as full_blob_buffer:
+                # Combined length of the vector and magic bytes stored as 4 bytes, big-endian
+                full_blob_buffer.write(len(crc_content).to_bytes(4, "big"))
+                # The content (magic + vector)
+                full_blob_buffer.write(crc_content)
+                # A CRC-32 checksum of the magic bytes and serialized vector as 4 bytes, big-endian
+                full_blob_buffer.write(crc32.to_bytes(4, "big"))
+
+                self._blob_payloads.append(full_blob_buffer.getvalue())
+
+        # 4. Create blob metadata
+        properties = {PROPERTY_REFERENCED_DATA_FILE: referenced_data_file, "cardinality": str(cardinality)}
+
+        self._blobs.append(
+            PuffinBlobMetadata(
+                type="deletion-vector-v1",
+                fields=[],
+                snapshot_id=-1,
+                sequence_number=-1,
+                offset=0,  # Will be set later
+                length=0,  # Will be set later
+                properties=properties,
+                compression_codec=None,  # Explicitly None
+            )
+        )
+
+    def finish(self) -> bytes:
+        with io.BytesIO() as out:
+            payload_buffer = io.BytesIO()
+            for blob_payload in self._blob_payloads:
+                payload_buffer.write(blob_payload)
+
+            updated_blobs_metadata: list[PuffinBlobMetadata] = []
+            current_offset = 4  # Start after file magic (4 bytes)
+            for i, blob_payload in enumerate(self._blob_payloads):
+                original_metadata_dict = self._blobs[i].model_dump(by_alias=True, exclude_none=True)
+                original_metadata_dict["offset"] = current_offset
+                original_metadata_dict["length"] = len(blob_payload)
+                updated_blobs_metadata.append(PuffinBlobMetadata(**original_metadata_dict))
+                current_offset += len(blob_payload)
+
+            footer = Footer(blobs=updated_blobs_metadata)
+            footer_payload_bytes = footer.model_dump_json(by_alias=True, exclude_none=True).encode("utf-8")
+
+            # Final assembly
+            out.write(MAGIC_BYTES)
+            out.write(payload_buffer.getvalue())
+            out.write(footer_payload_bytes)

Review Comment:
   We need to write `MAGIC_BYTES` before the payload as well
   
   https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/puffin/PuffinWriter.java#L152



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Write Deletion Vectors [iceberg-python]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #2822: Write Deletion Vectors
URL: https://github.com/apache/iceberg-python/pull/2822


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org