You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "geruh (via GitHub)" <gi...@apache.org> on 2026/01/25 06:06:29 UTC

[PR] Remove sortedcontainers dependency [iceberg-python]

geruh opened a new pull request, #2947:
URL: https://github.com/apache/iceberg-python/pull/2947

   closes #2945 
   
   # Rationale for this change
   
   This PR removes the `SortedContainers` dependency. Looking at the behavior of sorted containers we can simplify the logic for merging manigests and collecting the results while maintaining identical behavior.
   
   **What the logic today was doing:**
   1. Submit all manifest merge tasks to thread pool (pallelism starts with executor)
   2. Collect futures as they complete using `as_completed()` which is out of order
   3. Store completed futures in a SortedList to maintain order by submission
   4. Extract all results from the sorted futures
   5. Flatten and return
   
   **What we do now:**
   1. Submit all manifest merge tasks to thread pool (pallelism starts with executor)
   2. Iterate through futures in submission order, calling .result() on each
   3. Flatten and return
   
   This shows we must collect the results before the next step. So we can iterate futures directly and call `.result()` in order. This blocks the main thread until each future completes, but doesn't block worker threads and they all continue running in parallel.
   
   ## Are these changes tested?
   
   All existing tests pass.
   
   ## Are there any user-facing changes?
   
   No
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Remove sortedcontainers dependency [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu merged PR #2947:
URL: https://github.com/apache/iceberg-python/pull/2947


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Remove sortedcontainers dependency [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2947:
URL: https://github.com/apache/iceberg-python/pull/2947#discussion_r2725659572


##########
pyiceberg/table/update/snapshot.py:
##########
@@ -792,14 +788,7 @@ def merge_bin(manifest_bin: list[ManifestFile]) -> list[ManifestFile]:
 
         executor = ExecutorFactory.get_or_create()
         futures = [executor.submit(merge_bin, b) for b in bins]
-
-        # for consistent ordering, we need to maintain future order
-        futures_index = {f: i for i, f in enumerate(futures)}
-        completed_futures: SortedList[Future[list[ManifestFile]]] = SortedList(iterable=[], key=lambda f: futures_index[f])
-        for future in concurrent.futures.as_completed(futures):
-            completed_futures.add(future)
-
-        bin_results: list[list[ManifestFile]] = [f.result() for f in completed_futures if f.result()]
+        bin_results: list[list[ManifestFile]] = [r for f in futures if (r := f.result())]

Review Comment:
   nice! 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Remove sortedcontainers dependency [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #2947:
URL: https://github.com/apache/iceberg-python/pull/2947#issuecomment-3797206166

   Sweet!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Remove sortedcontainers dependency [iceberg-python]

Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #2947:
URL: https://github.com/apache/iceberg-python/pull/2947#discussion_r2724966166


##########
pyiceberg/table/update/snapshot.py:
##########
@@ -792,14 +788,7 @@ def merge_bin(manifest_bin: list[ManifestFile]) -> list[ManifestFile]:
 
         executor = ExecutorFactory.get_or_create()
         futures = [executor.submit(merge_bin, b) for b in bins]
-
-        # for consistent ordering, we need to maintain future order
-        futures_index = {f: i for i, f in enumerate(futures)}
-        completed_futures: SortedList[Future[list[ManifestFile]]] = SortedList(iterable=[], key=lambda f: futures_index[f])
-        for future in concurrent.futures.as_completed(futures):
-            completed_futures.add(future)
-
-        bin_results: list[list[ManifestFile]] = [f.result() for f in completed_futures if f.result()]
+        bin_results: list[list[ManifestFile]] = [r for f in futures if (r := f.result())]

Review Comment:
   We can use a walrus operator to avoid calling `.result()` twice per future



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org