You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "sdd (via GitHub)" <gi...@apache.org> on 2024/05/13 06:52:41 UTC

[PR] Concurrent table scans [iceberg-rust]

sdd opened a new pull request, #373:
URL: https://github.com/apache/iceberg-rust/pull/373

   This is a bit of an experiment to see how things could look if we tried to:
   
   - process Manifest Lists and Manifest Files concurrently rather than sequentially
   - process the stream of file scan tasks concurrently, streaming record batches from multiple files at the same time.
   
   I'd like to add some unit tests to confirm that this behaves as expected beyond the existing tests that we have for TableScan, and add an integration / performance test that can quantify any performance improvements (or regressions 😅 ) that we get from these changes.
   
   Let me know what you all think.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#issuecomment-2265398860

   > Now that caching of the retrieval and parsing of the `Manifest` and `ManifestList`s are taking place, the first run in the performance test will take the same time as above but all of the subsequent runs that are averaged to get an average time for each test are much, much faster, reducing the average time from 500ms to about 1.5ms. 🤯
   
   Great!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1676960798


##########
crates/iceberg/src/scan.rs:
##########
@@ -389,12 +333,158 @@ impl FileScanStreamContext {
             file_io,
             bound_filter,
             case_sensitive,
+            sender,
+            manifest_evaluator_cache: ManifestEvaluatorCache::new(),
+            partition_filter_cache: PartitionFilterCache::new(),
+            expression_evaluator_cache: Arc::new(Mutex::new(ExpressionEvaluatorCache::new())),
         })
     }
 
-    /// Returns a reference to the [`BoundPredicate`] filter.
-    fn bound_filter(&self) -> Option<&BoundPredicate> {
-        self.bound_filter.as_ref()
+    async fn run(&mut self, manifest_list: ManifestList) -> Result<()> {
+        let file_io = self.file_io.clone();
+        let sender = self.sender.clone();
+
+        // This whole Vec-and-for-loop approach feels sub-optimally structured.
+        // I've tried structuring this in multiple ways but run into
+        // issues with ownership. Ideally I'd like to structure this
+        // with a functional programming approach: extracting
+        // sections 1, 2, and 3 out into different methods on Self,
+        // and then use some iterator combinators to orchestrate it all.
+        // Section 1 is pretty trivially refactorable into a static method
+        // that can be used in a closure that can be used with Iterator::filter.
+        // Similarly, section 3 seems easily factor-able into a method that can
+        // be passed into Iterator::map.
+        // Section 2 turns out trickier - we want to exit the entire `run` method early
+        // if the eval fails, and filter out any manifest_files from the iterator / stream
+        // if the eval succeeds but returns true. We bump into ownership issues due
+        // to needing to pass mut self as the caches need to be able to mutate.
+        // Section 3 runs into ownership issues when trying to refactor its closure to be
+        // a static or non-static method.
+
+        // 1
+        let filtered_manifest_files = manifest_list
+            .entries()
+            .iter()
+            .filter(Self::reject_unsupported_content_types);
+
+        // 2
+        let mut filtered_manifest_files2 = vec![];
+        for manifest_file in filtered_manifest_files {
+            if !self.apply_evaluator(manifest_file)? {
+                continue;
+            }
+
+            filtered_manifest_files2.push(manifest_file);
+        }
+
+        // 3
+        let filtered_manifest_files = filtered_manifest_files2.into_iter().map(|manifest_file| {
+            Ok(ManifestFileProcessingContext {
+                manifest_file,
+                file_io: file_io.clone(),
+                sender: sender.clone(),
+                filter: &self.bound_filter,
+                expression_evaluator_cache: self.expression_evaluator_cache.clone(),
+            })
+        });
+
+        futures::stream::iter(filtered_manifest_files)
+            .try_for_each_concurrent(

Review Comment:
   Aah. You're right. `try_for_each_concurrent` is concurrent but not parallel. What I have should still be an improvement on what we have already, as we'll at least be able to have multiple fileIO file requests in-flight at the same time, but this is not fully parallelised.
   
   I'll work on making this parallel as well as concurrent. Some initial experimentation shows that it is tricky to please the borrow checker, but I'll keep trying.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#issuecomment-2271017314

   Sorry guys, been off-grid for a few days. I've addressed all of your suggestions @liurenjie1024, thanks.
   
   > > Thanks @sdd! That's awesome! BTW, can we integrate the performance test within the repository so that it will be helpful for performance improvement in the future? cc @liurenjie1024 @Xuanwo @sdd
   > 
   > I agree that a benchmark would be helpful for validating this, but I would prefer to do this after integrating with a sql engine like datafusion, which makes maintaince easier.
   
   @liurenjie1024  Why don't we just bring in the performance tests as they are / with whatever changes are needed from review, and we can always delete them and add different ones once we have the SQL engine in place? It's not much effort to delete them when they've been superseded but it is very useful to have them in place now whilst a lot of performance-related changes are being done. I'm happy to get the perf tests in my other PR into shape to merge as @ZENOTME suggests, even if they will be replaced at some point in the future.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "marvinlanhenke (via GitHub)" <gi...@apache.org>.
marvinlanhenke commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1600144851


##########
crates/iceberg/src/scan.rs:
##########
@@ -302,13 +262,147 @@ impl TableScan {
 
         arrow_reader_builder.build().read(self.plan_files().await?)
     }
+}
+
+#[derive(Debug)]
+struct ConcurrentFileScanStreamContext {
+    context: FileScanStreamContext,
+    sender: Sender<Result<FileScanTask>>,
+    manifest_evaluator_cache: ManifestEvaluatorCache,
+    partition_filter_cache: PartitionFilterCache,
+}
+
+impl ConcurrentFileScanStreamContext {
+    fn new(context: FileScanStreamContext, sender: Sender<Result<FileScanTask>>) -> Self {
+        ConcurrentFileScanStreamContext {
+            context,
+            sender,
+            manifest_evaluator_cache: ManifestEvaluatorCache::new(),
+            partition_filter_cache: PartitionFilterCache::new(),
+        }
+    }
+
+    async fn run(&mut self, manifest_list: ManifestList) -> Result<()> {
+        let file_io = self.context.file_io.clone();
+        let sender = self.sender.clone();
+
+        // This whole Vec-and-for-loop approach feels sub-optimally structured.
+        // I've tried structuring this in multiple ways but run into
+        // issues with ownership. Ideally I'd like to structure this
+        // with a functional programming approach: extracting
+        // sections 1, 2, and 3 out into different methods on Self,
+        // and then use some iterator combinators to orchestrate it all.
+        // Section 1 is pretty trivially refactorable into a static method
+        // that can be used in a closure that can be used with Iterator::filter.
+        // Similarly, section 3 seems easily factor-able into a method that can
+        // be passed into Iterator::map.
+        // Section 2 turns out trickier - we want to exit the entire `run` method early
+        // if the eval fails, and filter out any manifest_files from the iterator / stream
+        // if the eval succeeds but returns true. We bump into ownership issues due
+        // to needing to pass mut self as the caches need to be able to mutate.
+        // Section 3 runs into ownership issues when trying to refactor its closure to be
+        // a static or non-static method.
+
+        // 1
+        let filtered_manifest_files = manifest_list
+            .entries()
+            .iter()
+            .filter(Self::reject_unsupported_content_types);
+
+        // 2
+        let mut filtered_manifest_files2 = vec![];
+        for manifest_file in filtered_manifest_files {
+            if !self.apply_evaluator(manifest_file)? {
+                continue;
+            }
+
+            filtered_manifest_files2.push(manifest_file);
+        }
+
+        // 3
+        let filtered_manifest_files = filtered_manifest_files2
+            .into_iter()
+            .map(|manifest_file| Ok((manifest_file, file_io.clone(), sender.clone())));
+
+        futures::stream::iter(filtered_manifest_files)
+            .try_for_each_concurrent(
+                CONCURRENCY_LIMIT_MANIFEST_FILES,
+                Self::process_manifest_file,
+            )
+            .await

Review Comment:
   thanks for explaining



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1690017774


##########
crates/iceberg/src/scan.rs:
##########
@@ -197,134 +193,103 @@ impl<'a> TableScanBuilder<'a> {
             field_ids.push(field_id);
         }
 
-        Ok(TableScan {
+        let snapshot_bound_predicate = if let Some(ref predicates) = self.filter {
+            Some(predicates.bind(schema.clone(), true)?)
+        } else {
+            None
+        };
+
+        let plan_context = PlanContext {
             snapshot,
-            file_io: self.table.file_io().clone(),
             table_metadata: self.table.metadata_ref(),
-            column_names: self.column_names,
-            field_ids,
-            bound_predicates,
-            schema,
-            batch_size: self.batch_size,
+            snapshot_schema: schema,
             case_sensitive: self.case_sensitive,
-            filter: self.filter.map(Arc::new),
+            predicate: self.filter.map(Arc::new),
+            snapshot_bound_predicate: snapshot_bound_predicate.map(Arc::new),
+            file_io: self.table.file_io().clone(),
+            field_ids: Arc::new(field_ids),
+            partition_filter_cache: Arc::new(PartitionFilterCache::new()),
+            manifest_evaluator_cache: Arc::new(ManifestEvaluatorCache::new()),
+            expression_evaluator_cache: Arc::new(ExpressionEvaluatorCache::new()),
+        };
+
+        Ok(TableScan {
+            batch_size: self.batch_size,
+            column_names: self.column_names,
+            file_io: self.table.file_io().clone(),
+            plan_context,
         })
     }
 }
 
 /// Table scan.
 #[derive(Debug)]
 pub struct TableScan {
-    snapshot: SnapshotRef,
-    table_metadata: TableMetadataRef,
+    plan_context: PlanContext,
+    batch_size: Option<usize>,
     file_io: FileIO,
     column_names: Vec<String>,
-    field_ids: Vec<i32>,
-    bound_predicates: Option<BoundPredicate>,
-    schema: SchemaRef,
-    batch_size: Option<usize>,
+}
+
+/// PlanContext wraps a [`SnapshotRef`] alongside all the other
+/// objects that are required to perform a scan file plan.
+#[derive(Debug)]
+struct PlanContext {
+    snapshot: SnapshotRef,
+
+    table_metadata: TableMetadataRef,
+    snapshot_schema: SchemaRef,
     case_sensitive: bool,
-    filter: Option<Arc<Predicate>>,
+    predicate: Option<Arc<Predicate>>,
+    snapshot_bound_predicate: Option<Arc<BoundPredicate>>,
+    file_io: FileIO,
+    field_ids: Arc<Vec<i32>>,
+
+    partition_filter_cache: Arc<PartitionFilterCache>,
+    manifest_evaluator_cache: Arc<ManifestEvaluatorCache>,
+    expression_evaluator_cache: Arc<ExpressionEvaluatorCache>,
 }
 
 impl TableScan {
     /// Returns a stream of [`FileScanTask`]s.
     pub async fn plan_files(&self) -> Result<FileScanTaskStream> {
-        let context = FileScanStreamContext::new(
-            self.schema.clone(),
-            self.snapshot.clone(),
-            self.table_metadata.clone(),
-            self.file_io.clone(),
-            self.filter.clone(),
-            self.case_sensitive,
-        )?;
-
-        let mut partition_filter_cache = PartitionFilterCache::new();
-        let mut manifest_evaluator_cache = ManifestEvaluatorCache::new();
-        let mut expression_evaluator_cache = ExpressionEvaluatorCache::new();
-
-        let field_ids = self.field_ids.clone();
-        let bound_predicates = self.bound_predicates.clone();
-
-        Ok(try_stream! {
-            let manifest_list = context
-                .snapshot
-                .load_manifest_list(&context.file_io, &context.table_metadata)
-                .await?;
-
-            for entry in manifest_list.entries() {
-                if !Self::content_type_is_data(entry) {
-                    continue;
-                }
-
-                let partition_spec_id = entry.partition_spec_id;
-
-                let partition_filter = partition_filter_cache.get(
-                    partition_spec_id,
-                    &context,
-                )?;
-
-                if let Some(partition_filter) = partition_filter {
-                    let manifest_evaluator = manifest_evaluator_cache.get(
-                        partition_spec_id,
-                        partition_filter,
-                    );
-
-                    if !manifest_evaluator.eval(entry)? {
-                        continue;
-                    }
-                }
+        // used to stream ManifestEntryContexts between stages of the file plan operation
+        let (manifest_entry_ctx_tx, manifest_entry_ctx_rx) =
+            channel(CONCURRENCY_LIMIT_MANIFEST_FILES);
+        // used to stream the results back to the caller
+        let (file_scan_task_tx, file_scan_task_rx) = channel(CONCURRENCY_LIMIT_MANIFEST_ENTRIES);
+
+        let manifest_list = self.plan_context.get_manifest_list().await?;
+
+        // get the [`ManifestFile`]s from the [`ManifestList`], filtering out any
+        // whose content type is not Data or whose partitions cannot match this
+        // scan's filter
+        let manifest_file_contexts = self
+            .plan_context
+            .build_manifest_file_contexts(manifest_list, manifest_entry_ctx_tx)?;
+
+        // Concurrently load all [`Manifest`]s and stream their [`ManifestEntry`]s
+        futures::stream::iter(manifest_file_contexts)
+            .try_for_each_concurrent(CONCURRENCY_LIMIT_MANIFEST_FILES, |ctx| async move {
+                ctx.fetch_manifest_and_stream_manifest_entries().await
+            })
+            .await?;
+
+        // Process the [`ManifestEntry`] stream in parallel
+        manifest_entry_ctx_rx
+            .map(|me_ctx| Ok((me_ctx, file_scan_task_tx.clone())))
+            .try_for_each_concurrent(
+                CONCURRENCY_LIMIT_MANIFEST_ENTRIES,
+                |(manifest_entry_context, tx)| async move {
+                    crate::runtime::spawn(async move {
+                        Self::process_manifest_entry(manifest_entry_context, tx).await

Review Comment:
   As a user, I prefer to manage logic like this myself because I have my own pipeline with specific resource allocation and concurrency limits. So I think @liurenjie1024's concern is true. Maybe we can make those behavior optional and disable by default. Users who want iceberg-rust to do this can enable with some config or flags.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1598826218


##########
crates/iceberg/src/scan.rs:
##########
@@ -302,13 +262,147 @@ impl TableScan {
 
         arrow_reader_builder.build().read(self.plan_files().await?)
     }
+}
+
+#[derive(Debug)]
+struct ConcurrentFileScanStreamContext {

Review Comment:
   Yeah, completely agree. Kept them separate for now as it made it easier to have the old non-concurrent version in the code base at the same time. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1706317094


##########
crates/iceberg/src/scan.rs:
##########
@@ -55,17 +55,23 @@ pub struct TableScanBuilder<'a> {
     batch_size: Option<usize>,
     case_sensitive: bool,
     filter: Option<Predicate>,
+    concurrency_limit_manifest_files: usize,
+    concurrency_limit_manifest_entries: usize,
 }
 
 impl<'a> TableScanBuilder<'a> {
     pub(crate) fn new(table: &'a Table) -> Self {
+        let num_cpus = num_cpus::get();

Review Comment:
   Thanks for reporting, I'll create an issue for this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1598867675


##########
crates/iceberg/src/scan.rs:
##########
@@ -302,13 +262,147 @@ impl TableScan {
 
         arrow_reader_builder.build().read(self.plan_files().await?)
     }
+}
+
+#[derive(Debug)]
+struct ConcurrentFileScanStreamContext {
+    context: FileScanStreamContext,
+    sender: Sender<Result<FileScanTask>>,
+    manifest_evaluator_cache: ManifestEvaluatorCache,
+    partition_filter_cache: PartitionFilterCache,
+}
+
+impl ConcurrentFileScanStreamContext {
+    fn new(context: FileScanStreamContext, sender: Sender<Result<FileScanTask>>) -> Self {
+        ConcurrentFileScanStreamContext {
+            context,
+            sender,
+            manifest_evaluator_cache: ManifestEvaluatorCache::new(),
+            partition_filter_cache: PartitionFilterCache::new(),
+        }
+    }
+
+    async fn run(&mut self, manifest_list: ManifestList) -> Result<()> {
+        let file_io = self.context.file_io.clone();
+        let sender = self.sender.clone();
+
+        // This whole Vec-and-for-loop approach feels sub-optimally structured.
+        // I've tried structuring this in multiple ways but run into
+        // issues with ownership. Ideally I'd like to structure this
+        // with a functional programming approach: extracting
+        // sections 1, 2, and 3 out into different methods on Self,
+        // and then use some iterator combinators to orchestrate it all.
+        // Section 1 is pretty trivially refactorable into a static method
+        // that can be used in a closure that can be used with Iterator::filter.
+        // Similarly, section 3 seems easily factor-able into a method that can
+        // be passed into Iterator::map.
+        // Section 2 turns out trickier - we want to exit the entire `run` method early
+        // if the eval fails, and filter out any manifest_files from the iterator / stream
+        // if the eval succeeds but returns true. We bump into ownership issues due
+        // to needing to pass mut self as the caches need to be able to mutate.
+        // Section 3 runs into ownership issues when trying to refactor its closure to be
+        // a static or non-static method.
+
+        // 1
+        let filtered_manifest_files = manifest_list
+            .entries()
+            .iter()
+            .filter(Self::reject_unsupported_content_types);
+
+        // 2
+        let mut filtered_manifest_files2 = vec![];
+        for manifest_file in filtered_manifest_files {
+            if !self.apply_evaluator(manifest_file)? {
+                continue;
+            }
+
+            filtered_manifest_files2.push(manifest_file);
+        }
+
+        // 3
+        let filtered_manifest_files = filtered_manifest_files2
+            .into_iter()
+            .map(|manifest_file| Ok((manifest_file, file_io.clone(), sender.clone())));
+
+        futures::stream::iter(filtered_manifest_files)
+            .try_for_each_concurrent(
+                CONCURRENCY_LIMIT_MANIFEST_FILES,
+                Self::process_manifest_file,
+            )
+            .await

Review Comment:
   Yes, everything proceeds fully concurrently - whatever manifest is ready first will start yielding tasks up to the stream first



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1690195558


##########
crates/iceberg/src/expr/predicate.rs:
##########
@@ -668,6 +668,10 @@ pub enum BoundPredicate {
     Set(SetExpression<BoundReference>),
 }
 
+/// Newtype to prevent accidentally confusing predicates that are bound to a partition with ones that are bound to a schema.
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+pub struct PartitionBoundPredicate(pub(crate) BoundPredicate);

Review Comment:
   Fair enough. Happy to remove. I only added because I got so annoyed after spending hours tracking down a bug that was because I had confused the two that I never wanted to have that pain again, but since then I've been more consistent with the variable names. 😁



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#issuecomment-2111009920

   I've updated this to ditch the concurrency when processing `ManifestEntry` items within a single `Manifest`, producing them asynchronously but sequentially instead. I've kept the limited concurrency when processing `ManifestFile`s within the scan's snapshot's `ManifestList`.
   
   I've kept the approach of using an mpsc channel with a spawned task, with that task using `try_for_each_concurrent` to achieve the concurrency. This is because without the channel and spawned task, we'd need to use an async closure, which is unstable rust. With the spawned task we only need to use an async block, which is in stable rust.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "SteveLauC (via GitHub)" <gi...@apache.org>.
SteveLauC commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1706281350


##########
crates/iceberg/src/scan.rs:
##########
@@ -55,17 +55,23 @@ pub struct TableScanBuilder<'a> {
     batch_size: Option<usize>,
     case_sensitive: bool,
     filter: Option<Predicate>,
+    concurrency_limit_manifest_files: usize,
+    concurrency_limit_manifest_entries: usize,
 }
 
 impl<'a> TableScanBuilder<'a> {
     pub(crate) fn new(table: &'a Table) -> Self {
+        let num_cpus = num_cpus::get();

Review Comment:
   Not a reivew comment, just a glimpse of this PR, we can replace this with `std::thread::available_parallelism()`, `num_cpus` is almost unmaintained (as it no longer needs to), and has issues that are solved by the std counterpart, e.g., this one: [std::thread::available_parallelism() return 0](https://github.com/rust-lang/rust/issues/115868).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1691246764


##########
crates/iceberg/src/scan.rs:
##########
@@ -197,134 +193,103 @@ impl<'a> TableScanBuilder<'a> {
             field_ids.push(field_id);
         }
 
-        Ok(TableScan {
+        let snapshot_bound_predicate = if let Some(ref predicates) = self.filter {
+            Some(predicates.bind(schema.clone(), true)?)
+        } else {
+            None
+        };
+
+        let plan_context = PlanContext {
             snapshot,
-            file_io: self.table.file_io().clone(),
             table_metadata: self.table.metadata_ref(),
-            column_names: self.column_names,
-            field_ids,
-            bound_predicates,
-            schema,
-            batch_size: self.batch_size,
+            snapshot_schema: schema,
             case_sensitive: self.case_sensitive,
-            filter: self.filter.map(Arc::new),
+            predicate: self.filter.map(Arc::new),
+            snapshot_bound_predicate: snapshot_bound_predicate.map(Arc::new),
+            file_io: self.table.file_io().clone(),
+            field_ids: Arc::new(field_ids),
+            partition_filter_cache: Arc::new(PartitionFilterCache::new()),
+            manifest_evaluator_cache: Arc::new(ManifestEvaluatorCache::new()),
+            expression_evaluator_cache: Arc::new(ExpressionEvaluatorCache::new()),
+        };
+
+        Ok(TableScan {
+            batch_size: self.batch_size,
+            column_names: self.column_names,
+            file_io: self.table.file_io().clone(),
+            plan_context,
         })
     }
 }
 
 /// Table scan.
 #[derive(Debug)]
 pub struct TableScan {
-    snapshot: SnapshotRef,
-    table_metadata: TableMetadataRef,
+    plan_context: PlanContext,
+    batch_size: Option<usize>,
     file_io: FileIO,
     column_names: Vec<String>,
-    field_ids: Vec<i32>,
-    bound_predicates: Option<BoundPredicate>,
-    schema: SchemaRef,
-    batch_size: Option<usize>,
+}
+
+/// PlanContext wraps a [`SnapshotRef`] alongside all the other
+/// objects that are required to perform a scan file plan.
+#[derive(Debug)]
+struct PlanContext {
+    snapshot: SnapshotRef,
+
+    table_metadata: TableMetadataRef,
+    snapshot_schema: SchemaRef,
     case_sensitive: bool,
-    filter: Option<Arc<Predicate>>,
+    predicate: Option<Arc<Predicate>>,
+    snapshot_bound_predicate: Option<Arc<BoundPredicate>>,
+    file_io: FileIO,
+    field_ids: Arc<Vec<i32>>,
+
+    partition_filter_cache: Arc<PartitionFilterCache>,
+    manifest_evaluator_cache: Arc<ManifestEvaluatorCache>,
+    expression_evaluator_cache: Arc<ExpressionEvaluatorCache>,
 }
 
 impl TableScan {
     /// Returns a stream of [`FileScanTask`]s.
     pub async fn plan_files(&self) -> Result<FileScanTaskStream> {
-        let context = FileScanStreamContext::new(
-            self.schema.clone(),
-            self.snapshot.clone(),
-            self.table_metadata.clone(),
-            self.file_io.clone(),
-            self.filter.clone(),
-            self.case_sensitive,
-        )?;
-
-        let mut partition_filter_cache = PartitionFilterCache::new();
-        let mut manifest_evaluator_cache = ManifestEvaluatorCache::new();
-        let mut expression_evaluator_cache = ExpressionEvaluatorCache::new();
-
-        let field_ids = self.field_ids.clone();
-        let bound_predicates = self.bound_predicates.clone();
-
-        Ok(try_stream! {
-            let manifest_list = context
-                .snapshot
-                .load_manifest_list(&context.file_io, &context.table_metadata)
-                .await?;
-
-            for entry in manifest_list.entries() {
-                if !Self::content_type_is_data(entry) {
-                    continue;
-                }
-
-                let partition_spec_id = entry.partition_spec_id;
-
-                let partition_filter = partition_filter_cache.get(
-                    partition_spec_id,
-                    &context,
-                )?;
-
-                if let Some(partition_filter) = partition_filter {
-                    let manifest_evaluator = manifest_evaluator_cache.get(
-                        partition_spec_id,
-                        partition_filter,
-                    );
-
-                    if !manifest_evaluator.eval(entry)? {
-                        continue;
-                    }
-                }
+        // used to stream ManifestEntryContexts between stages of the file plan operation
+        let (manifest_entry_ctx_tx, manifest_entry_ctx_rx) =
+            channel(CONCURRENCY_LIMIT_MANIFEST_FILES);
+        // used to stream the results back to the caller
+        let (file_scan_task_tx, file_scan_task_rx) = channel(CONCURRENCY_LIMIT_MANIFEST_ENTRIES);
+
+        let manifest_list = self.plan_context.get_manifest_list().await?;
+
+        // get the [`ManifestFile`]s from the [`ManifestList`], filtering out any
+        // whose content type is not Data or whose partitions cannot match this
+        // scan's filter
+        let manifest_file_contexts = self
+            .plan_context
+            .build_manifest_file_contexts(manifest_list, manifest_entry_ctx_tx)?;
+
+        // Concurrently load all [`Manifest`]s and stream their [`ManifestEntry`]s
+        futures::stream::iter(manifest_file_contexts)
+            .try_for_each_concurrent(CONCURRENCY_LIMIT_MANIFEST_FILES, |ctx| async move {
+                ctx.fetch_manifest_and_stream_manifest_entries().await
+            })
+            .await?;
+
+        // Process the [`ManifestEntry`] stream in parallel
+        manifest_entry_ctx_rx
+            .map(|me_ctx| Ok((me_ctx, file_scan_task_tx.clone())))
+            .try_for_each_concurrent(
+                CONCURRENCY_LIMIT_MANIFEST_ENTRIES,
+                |(manifest_entry_context, tx)| async move {
+                    crate::runtime::spawn(async move {
+                        Self::process_manifest_entry(manifest_entry_context, tx).await

Review Comment:
   Yeah you're right, `with_no_concurrency()` is a bit pointless when you can just set `with_max_concurrency(1)`! 😂 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1683284743


##########
crates/iceberg/src/scan.rs:
##########
@@ -389,12 +333,158 @@ impl FileScanStreamContext {
             file_io,
             bound_filter,
             case_sensitive,
+            sender,
+            manifest_evaluator_cache: ManifestEvaluatorCache::new(),
+            partition_filter_cache: PartitionFilterCache::new(),
+            expression_evaluator_cache: Arc::new(Mutex::new(ExpressionEvaluatorCache::new())),
         })
     }
 
-    /// Returns a reference to the [`BoundPredicate`] filter.
-    fn bound_filter(&self) -> Option<&BoundPredicate> {
-        self.bound_filter.as_ref()
+    async fn run(&mut self, manifest_list: ManifestList) -> Result<()> {
+        let file_io = self.file_io.clone();
+        let sender = self.sender.clone();
+
+        // This whole Vec-and-for-loop approach feels sub-optimally structured.
+        // I've tried structuring this in multiple ways but run into
+        // issues with ownership. Ideally I'd like to structure this
+        // with a functional programming approach: extracting
+        // sections 1, 2, and 3 out into different methods on Self,
+        // and then use some iterator combinators to orchestrate it all.
+        // Section 1 is pretty trivially refactorable into a static method
+        // that can be used in a closure that can be used with Iterator::filter.
+        // Similarly, section 3 seems easily factor-able into a method that can
+        // be passed into Iterator::map.
+        // Section 2 turns out trickier - we want to exit the entire `run` method early
+        // if the eval fails, and filter out any manifest_files from the iterator / stream
+        // if the eval succeeds but returns true. We bump into ownership issues due
+        // to needing to pass mut self as the caches need to be able to mutate.
+        // Section 3 runs into ownership issues when trying to refactor its closure to be
+        // a static or non-static method.
+
+        // 1
+        let filtered_manifest_files = manifest_list
+            .entries()
+            .iter()
+            .filter(Self::reject_unsupported_content_types);
+
+        // 2
+        let mut filtered_manifest_files2 = vec![];
+        for manifest_file in filtered_manifest_files {
+            if !self.apply_evaluator(manifest_file)? {
+                continue;
+            }
+
+            filtered_manifest_files2.push(manifest_file);
+        }
+
+        // 3
+        let filtered_manifest_files = filtered_manifest_files2.into_iter().map(|manifest_file| {
+            Ok(ManifestFileProcessingContext {
+                manifest_file,
+                file_io: file_io.clone(),
+                sender: sender.clone(),
+                filter: &self.bound_filter,
+                expression_evaluator_cache: self.expression_evaluator_cache.clone(),
+            })
+        });
+
+        futures::stream::iter(filtered_manifest_files)
+            .try_for_each_concurrent(

Review Comment:
   I've resolved that issue now @liurenjie1024 - I'm happy with this now and I'd love to get feedback from you, plus @odysa / @Xuanwo / @ZENOTME / @marvinlanhenke as this is almost a complete rewrite to ensure that everything was able to be passed between threads.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#issuecomment-2237194797

   Thanks @odysa - I must be going crazy, I thought I tried that already but you were right, that worked! 👍🏼 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1690020806


##########
crates/iceberg/src/expr/predicate.rs:
##########
@@ -668,6 +668,10 @@ pub enum BoundPredicate {
     Set(SetExpression<BoundReference>),
 }
 
+/// Newtype to prevent accidentally confusing predicates that are bound to a partition with ones that are bound to a schema.
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+pub struct PartitionBoundPredicate(pub(crate) BoundPredicate);

Review Comment:
   Yes, I don't think the confusion is significant enough to require a new-type.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#issuecomment-2267424040

   > Thanks @sdd! That's awesome! BTW, can we integrate the performance test within the repository so that it will be helpful for performance improvement in the future? cc @liurenjie1024 @Xuanwo @sdd
   
   I agree that a benchmark would be helpful for validating this, but I would prefer to do this after integrating with a sql engine like datafusion, which makes maintaince easier.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1641879767


##########
crates/iceberg/src/scan.rs:
##########
@@ -389,12 +333,158 @@ impl FileScanStreamContext {
             file_io,
             bound_filter,
             case_sensitive,
+            sender,
+            manifest_evaluator_cache: ManifestEvaluatorCache::new(),
+            partition_filter_cache: PartitionFilterCache::new(),
+            expression_evaluator_cache: Arc::new(Mutex::new(ExpressionEvaluatorCache::new())),
         })
     }
 
-    /// Returns a reference to the [`BoundPredicate`] filter.
-    fn bound_filter(&self) -> Option<&BoundPredicate> {
-        self.bound_filter.as_ref()
+    async fn run(&mut self, manifest_list: ManifestList) -> Result<()> {

Review Comment:
   ```suggestion
       async fn run(mut self, manifest_list: ManifestList) -> Result<()> {
   ```
   
   It would be unsound to reuse this object.



##########
crates/iceberg/src/scan.rs:
##########
@@ -389,12 +333,158 @@ impl FileScanStreamContext {
             file_io,
             bound_filter,
             case_sensitive,
+            sender,
+            manifest_evaluator_cache: ManifestEvaluatorCache::new(),
+            partition_filter_cache: PartitionFilterCache::new(),
+            expression_evaluator_cache: Arc::new(Mutex::new(ExpressionEvaluatorCache::new())),
         })
     }
 
-    /// Returns a reference to the [`BoundPredicate`] filter.
-    fn bound_filter(&self) -> Option<&BoundPredicate> {
-        self.bound_filter.as_ref()
+    async fn run(&mut self, manifest_list: ManifestList) -> Result<()> {
+        let file_io = self.file_io.clone();
+        let sender = self.sender.clone();
+
+        // This whole Vec-and-for-loop approach feels sub-optimally structured.
+        // I've tried structuring this in multiple ways but run into
+        // issues with ownership. Ideally I'd like to structure this
+        // with a functional programming approach: extracting
+        // sections 1, 2, and 3 out into different methods on Self,
+        // and then use some iterator combinators to orchestrate it all.
+        // Section 1 is pretty trivially refactorable into a static method
+        // that can be used in a closure that can be used with Iterator::filter.
+        // Similarly, section 3 seems easily factor-able into a method that can
+        // be passed into Iterator::map.
+        // Section 2 turns out trickier - we want to exit the entire `run` method early
+        // if the eval fails, and filter out any manifest_files from the iterator / stream
+        // if the eval succeeds but returns true. We bump into ownership issues due
+        // to needing to pass mut self as the caches need to be able to mutate.
+        // Section 3 runs into ownership issues when trying to refactor its closure to be
+        // a static or non-static method.
+
+        // 1
+        let filtered_manifest_files = manifest_list
+            .entries()
+            .iter()
+            .filter(Self::reject_unsupported_content_types);
+
+        // 2
+        let mut filtered_manifest_files2 = vec![];
+        for manifest_file in filtered_manifest_files {
+            if !self.apply_evaluator(manifest_file)? {
+                continue;
+            }
+
+            filtered_manifest_files2.push(manifest_file);
+        }
+
+        // 3
+        let filtered_manifest_files = filtered_manifest_files2.into_iter().map(|manifest_file| {
+            Ok(ManifestFileProcessingContext {
+                manifest_file,
+                file_io: file_io.clone(),
+                sender: sender.clone(),
+                filter: &self.bound_filter,
+                expression_evaluator_cache: self.expression_evaluator_cache.clone(),
+            })
+        });
+
+        futures::stream::iter(filtered_manifest_files)
+            .try_for_each_concurrent(

Review Comment:
   I think this method in fact made io concurrent, but not computing concurrent, e.g. the processing of manifest file is not paralleled.



##########
crates/iceberg/src/scan.rs:
##########
@@ -189,66 +195,20 @@ impl TableScan {
             self.case_sensitive,
         )?;
 
-        let mut partition_filter_cache = PartitionFilterCache::new();
-        let mut manifest_evaluator_cache = ManifestEvaluatorCache::new();
-
-        Ok(try_stream! {
-            let manifest_list = context
-                .snapshot
-                .load_manifest_list(&context.file_io, &context.table_metadata)
-                .await?;
-
-            for entry in manifest_list.entries() {
-                if !Self::content_type_is_data(entry) {
-                    continue;
-                }
+        let (sender, receiver) = channel(CHANNEL_BUFFER_SIZE);
 
-                let partition_spec_id = entry.partition_spec_id;
+        let manifest_list = context
+            .snapshot
+            .load_manifest_list(&context.file_io, &context.table_metadata)
+            .await?;
 
-                let partition_filter = partition_filter_cache.get(
-                    partition_spec_id,
-                    &context,
-                )?;
+        spawn(async move {

Review Comment:
   This still introduced dependency of tokio.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1677771995


##########
crates/iceberg/src/scan.rs:
##########
@@ -389,12 +333,158 @@ impl FileScanStreamContext {
             file_io,
             bound_filter,
             case_sensitive,
+            sender,
+            manifest_evaluator_cache: ManifestEvaluatorCache::new(),
+            partition_filter_cache: PartitionFilterCache::new(),
+            expression_evaluator_cache: Arc::new(Mutex::new(ExpressionEvaluatorCache::new())),
         })
     }
 
-    /// Returns a reference to the [`BoundPredicate`] filter.
-    fn bound_filter(&self) -> Option<&BoundPredicate> {
-        self.bound_filter.as_ref()
+    async fn run(&mut self, manifest_list: ManifestList) -> Result<()> {
+        let file_io = self.file_io.clone();
+        let sender = self.sender.clone();
+
+        // This whole Vec-and-for-loop approach feels sub-optimally structured.
+        // I've tried structuring this in multiple ways but run into
+        // issues with ownership. Ideally I'd like to structure this
+        // with a functional programming approach: extracting
+        // sections 1, 2, and 3 out into different methods on Self,
+        // and then use some iterator combinators to orchestrate it all.
+        // Section 1 is pretty trivially refactorable into a static method
+        // that can be used in a closure that can be used with Iterator::filter.
+        // Similarly, section 3 seems easily factor-able into a method that can
+        // be passed into Iterator::map.
+        // Section 2 turns out trickier - we want to exit the entire `run` method early
+        // if the eval fails, and filter out any manifest_files from the iterator / stream
+        // if the eval succeeds but returns true. We bump into ownership issues due
+        // to needing to pass mut self as the caches need to be able to mutate.
+        // Section 3 runs into ownership issues when trying to refactor its closure to be
+        // a static or non-static method.
+
+        // 1
+        let filtered_manifest_files = manifest_list
+            .entries()
+            .iter()
+            .filter(Self::reject_unsupported_content_types);
+
+        // 2
+        let mut filtered_manifest_files2 = vec![];
+        for manifest_file in filtered_manifest_files {
+            if !self.apply_evaluator(manifest_file)? {
+                continue;
+            }
+
+            filtered_manifest_files2.push(manifest_file);
+        }
+
+        // 3
+        let filtered_manifest_files = filtered_manifest_files2.into_iter().map(|manifest_file| {
+            Ok(ManifestFileProcessingContext {
+                manifest_file,
+                file_io: file_io.clone(),
+                sender: sender.clone(),
+                filter: &self.bound_filter,
+                expression_evaluator_cache: self.expression_evaluator_cache.clone(),
+            })
+        });
+
+        futures::stream::iter(filtered_manifest_files)
+            .try_for_each_concurrent(

Review Comment:
   Thanks, feel free to ping me if you think it's ready for review.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1598841588


##########
crates/iceberg/src/scan.rs:
##########
@@ -189,66 +195,20 @@ impl TableScan {
             self.case_sensitive,
         )?;
 
-        let mut partition_filter_cache = PartitionFilterCache::new();
-        let mut manifest_evaluator_cache = ManifestEvaluatorCache::new();
-
-        Ok(try_stream! {
-            let manifest_list = context
-                .snapshot
-                .load_manifest_list(&context.file_io, &context.table_metadata)
-                .await?;
-
-            for entry in manifest_list.entries() {
-                if !Self::content_type_is_data(entry) {
-                    continue;
-                }
+        let (sender, receiver) = channel(CHANNEL_BUFFER_SIZE);
 
-                let partition_spec_id = entry.partition_spec_id;
+        let manifest_list = context
+            .snapshot
+            .load_manifest_list(&context.file_io, &context.table_metadata)
+            .await?;
 
-                let partition_filter = partition_filter_cache.get(
-                    partition_spec_id,
-                    &context,
-                )?;
+        spawn(async move {

Review Comment:
   I went with using an mpsc channel to avoid having 2 nested try_for_each_concurrent macros. On reflection the inner try_for_each_concurrent is probably unnecessary, since there's only one async operation needed for each manifest file, despite that manifest file producing n DataFileTasks. So I can probably ditch the channel too. The channel in reader.rs is also overkill as I've only got a single try_for_each_concurrent there.
   
   I just wanted to get the ball rolling on this while we're waiting for the filtering code to get signed off, it's fun writing this kind of stuff 😁



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1690213759


##########
crates/iceberg/src/scan.rs:
##########
@@ -197,134 +193,103 @@ impl<'a> TableScanBuilder<'a> {
             field_ids.push(field_id);
         }
 
-        Ok(TableScan {
+        let snapshot_bound_predicate = if let Some(ref predicates) = self.filter {
+            Some(predicates.bind(schema.clone(), true)?)
+        } else {
+            None
+        };
+
+        let plan_context = PlanContext {
             snapshot,
-            file_io: self.table.file_io().clone(),
             table_metadata: self.table.metadata_ref(),
-            column_names: self.column_names,
-            field_ids,
-            bound_predicates,
-            schema,
-            batch_size: self.batch_size,
+            snapshot_schema: schema,
             case_sensitive: self.case_sensitive,
-            filter: self.filter.map(Arc::new),
+            predicate: self.filter.map(Arc::new),
+            snapshot_bound_predicate: snapshot_bound_predicate.map(Arc::new),
+            file_io: self.table.file_io().clone(),
+            field_ids: Arc::new(field_ids),
+            partition_filter_cache: Arc::new(PartitionFilterCache::new()),
+            manifest_evaluator_cache: Arc::new(ManifestEvaluatorCache::new()),
+            expression_evaluator_cache: Arc::new(ExpressionEvaluatorCache::new()),
+        };
+
+        Ok(TableScan {
+            batch_size: self.batch_size,
+            column_names: self.column_names,
+            file_io: self.table.file_io().clone(),
+            plan_context,
         })
     }
 }
 
 /// Table scan.
 #[derive(Debug)]
 pub struct TableScan {
-    snapshot: SnapshotRef,
-    table_metadata: TableMetadataRef,
+    plan_context: PlanContext,
+    batch_size: Option<usize>,
     file_io: FileIO,
     column_names: Vec<String>,
-    field_ids: Vec<i32>,
-    bound_predicates: Option<BoundPredicate>,
-    schema: SchemaRef,
-    batch_size: Option<usize>,
+}
+
+/// PlanContext wraps a [`SnapshotRef`] alongside all the other
+/// objects that are required to perform a scan file plan.
+#[derive(Debug)]
+struct PlanContext {
+    snapshot: SnapshotRef,
+
+    table_metadata: TableMetadataRef,
+    snapshot_schema: SchemaRef,
     case_sensitive: bool,
-    filter: Option<Arc<Predicate>>,
+    predicate: Option<Arc<Predicate>>,
+    snapshot_bound_predicate: Option<Arc<BoundPredicate>>,
+    file_io: FileIO,
+    field_ids: Arc<Vec<i32>>,
+
+    partition_filter_cache: Arc<PartitionFilterCache>,
+    manifest_evaluator_cache: Arc<ManifestEvaluatorCache>,
+    expression_evaluator_cache: Arc<ExpressionEvaluatorCache>,
 }
 
 impl TableScan {
     /// Returns a stream of [`FileScanTask`]s.
     pub async fn plan_files(&self) -> Result<FileScanTaskStream> {
-        let context = FileScanStreamContext::new(
-            self.schema.clone(),
-            self.snapshot.clone(),
-            self.table_metadata.clone(),
-            self.file_io.clone(),
-            self.filter.clone(),
-            self.case_sensitive,
-        )?;
-
-        let mut partition_filter_cache = PartitionFilterCache::new();
-        let mut manifest_evaluator_cache = ManifestEvaluatorCache::new();
-        let mut expression_evaluator_cache = ExpressionEvaluatorCache::new();
-
-        let field_ids = self.field_ids.clone();
-        let bound_predicates = self.bound_predicates.clone();
-
-        Ok(try_stream! {
-            let manifest_list = context
-                .snapshot
-                .load_manifest_list(&context.file_io, &context.table_metadata)
-                .await?;
-
-            for entry in manifest_list.entries() {
-                if !Self::content_type_is_data(entry) {
-                    continue;
-                }
-
-                let partition_spec_id = entry.partition_spec_id;
-
-                let partition_filter = partition_filter_cache.get(
-                    partition_spec_id,
-                    &context,
-                )?;
-
-                if let Some(partition_filter) = partition_filter {
-                    let manifest_evaluator = manifest_evaluator_cache.get(
-                        partition_spec_id,
-                        partition_filter,
-                    );
-
-                    if !manifest_evaluator.eval(entry)? {
-                        continue;
-                    }
-                }
+        // used to stream ManifestEntryContexts between stages of the file plan operation
+        let (manifest_entry_ctx_tx, manifest_entry_ctx_rx) =
+            channel(CONCURRENCY_LIMIT_MANIFEST_FILES);
+        // used to stream the results back to the caller
+        let (file_scan_task_tx, file_scan_task_rx) = channel(CONCURRENCY_LIMIT_MANIFEST_ENTRIES);
+
+        let manifest_list = self.plan_context.get_manifest_list().await?;
+
+        // get the [`ManifestFile`]s from the [`ManifestList`], filtering out any
+        // whose content type is not Data or whose partitions cannot match this
+        // scan's filter
+        let manifest_file_contexts = self
+            .plan_context
+            .build_manifest_file_contexts(manifest_list, manifest_entry_ctx_tx)?;
+
+        // Concurrently load all [`Manifest`]s and stream their [`ManifestEntry`]s
+        futures::stream::iter(manifest_file_contexts)
+            .try_for_each_concurrent(CONCURRENCY_LIMIT_MANIFEST_FILES, |ctx| async move {
+                ctx.fetch_manifest_and_stream_manifest_entries().await
+            })
+            .await?;
+
+        // Process the [`ManifestEntry`] stream in parallel
+        manifest_entry_ctx_rx
+            .map(|me_ctx| Ok((me_ctx, file_scan_task_tx.clone())))
+            .try_for_each_concurrent(
+                CONCURRENCY_LIMIT_MANIFEST_ENTRIES,
+                |(manifest_entry_context, tx)| async move {
+                    crate::runtime::spawn(async move {
+                        Self::process_manifest_entry(manifest_entry_context, tx).await

Review Comment:
   Regarding the large number of tasks - there are a large number generated over the duration of the plan, but only a fixed small number that are in existence at one time. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#issuecomment-2259443025

   PTAL @liurenjie1024 and @Xuanwo: I've addressed your helpful feedback and I think this is now ready. Performance tests of this on my performance testing branch show clear gains when scanning snapshots whose manifest list contains multiple manifests. I'm really happy with this now!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#issuecomment-2271899629

   @liurenjie1024 @Xuanwo are we able to merge this now?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#issuecomment-2262683987

   > PTAL @liurenjie1024 and @Xuanwo: I've addressed your helpful feedback and I think this is now ready. Performance tests of this on my performance testing branch show clear gains when scanning snapshots whose manifest list contains multiple manifests. I'm really happy with this now!
   
   I'm curious about the performance gain from this. Can you share the related data?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#issuecomment-2272420118

   > @liurenjie1024 @Xuanwo are we able to merge this now?
   
   Done, thanks @sdd for this pr!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 merged PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1701569163


##########
crates/iceberg/src/scan.rs:
##########
@@ -111,6 +135,44 @@ impl<'a> TableScanBuilder<'a> {
         self
     }
 
+    /// Provides a TableScanConfig to use as the config for this TableScan
+    pub fn with_config(mut self, table_scan_config: TableScanConfig) -> Self {
+        self.table_scan_config = table_scan_config;
+        self
+    }
+
+    /// Sets the concurrency limit for both manifest files and manifest
+    /// entries for this scan
+    pub fn with_concurrency_limit(mut self, limit: usize) -> Self {
+        self.table_scan_config = TableScanConfig {
+            concurrency_limit_manifest_files: limit,
+            concurrency_limit_manifest_entries: limit,
+        };
+        self
+    }
+
+    /// sets the manifest file concurrency limit for this scan
+    pub fn with_manifest_file_concurrency_limit(mut self, limit: usize) -> Self {
+        self.table_scan_config = TableScanConfig {
+            concurrency_limit_manifest_files: limit,
+            concurrency_limit_manifest_entries: self
+                .table_scan_config
+                .concurrency_limit_manifest_entries,
+        };
+        self
+    }
+
+    /// sets the manifest entry concurrency limit for this scan
+    pub fn with_manifest_entru_concurrency_limit(mut self, limit: usize) -> Self {

Review Comment:
   ```suggestion
       pub fn with_manifest_entry_concurrency_limit(mut self, limit: usize) -> Self {
   ```



##########
crates/iceberg/src/scan.rs:
##########
@@ -111,6 +135,44 @@ impl<'a> TableScanBuilder<'a> {
         self
     }
 
+    /// Provides a TableScanConfig to use as the config for this TableScan
+    pub fn with_config(mut self, table_scan_config: TableScanConfig) -> Self {
+        self.table_scan_config = table_scan_config;
+        self
+    }
+
+    /// Sets the concurrency limit for both manifest files and manifest
+    /// entries for this scan
+    pub fn with_concurrency_limit(mut self, limit: usize) -> Self {
+        self.table_scan_config = TableScanConfig {
+            concurrency_limit_manifest_files: limit,
+            concurrency_limit_manifest_entries: limit,
+        };
+        self
+    }
+
+    /// sets the manifest file concurrency limit for this scan

Review Comment:
   ```suggestion
       /// Sets the manifest file concurrency limit for this scan
   ```



##########
crates/iceberg/src/scan.rs:
##########
@@ -111,6 +135,44 @@ impl<'a> TableScanBuilder<'a> {
         self
     }
 
+    /// Provides a TableScanConfig to use as the config for this TableScan
+    pub fn with_config(mut self, table_scan_config: TableScanConfig) -> Self {
+        self.table_scan_config = table_scan_config;
+        self
+    }
+
+    /// Sets the concurrency limit for both manifest files and manifest
+    /// entries for this scan
+    pub fn with_concurrency_limit(mut self, limit: usize) -> Self {
+        self.table_scan_config = TableScanConfig {
+            concurrency_limit_manifest_files: limit,
+            concurrency_limit_manifest_entries: limit,
+        };
+        self
+    }
+
+    /// sets the manifest file concurrency limit for this scan
+    pub fn with_manifest_file_concurrency_limit(mut self, limit: usize) -> Self {
+        self.table_scan_config = TableScanConfig {
+            concurrency_limit_manifest_files: limit,
+            concurrency_limit_manifest_entries: self
+                .table_scan_config
+                .concurrency_limit_manifest_entries,
+        };
+        self
+    }
+
+    /// sets the manifest entry concurrency limit for this scan

Review Comment:
   ```suggestion
       /// Sets the manifest entry concurrency limit for this scan
   ```



##########
crates/iceberg/src/scan.rs:
##########
@@ -46,6 +46,28 @@ pub type FileScanTaskStream = BoxStream<'static, Result<FileScanTask>>;
 /// A stream of arrow [`RecordBatch`]es.
 pub type ArrowRecordBatchStream = BoxStream<'static, Result<RecordBatch>>;
 
+/// Configuration settings for a TableScan
+#[derive(Debug)]
+pub struct TableScanConfig {

Review Comment:
   How about just putting these fields in `TableScanBuilder`? I think they are similar to other parts such as `batch_size`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#issuecomment-2267362756

   > Hey @ZENOTME! Sure. If you check out my `perf-suite` branch from [my other PR](https://github.com/apache/iceberg-rust/pull/497), this is branched off main and can be used to get a baseline of `plan_files` performance without the changes in this PR. You'll need to run `cargo install just` if you don't have `just` already installed, and then `just perf-run`. This should create the docker setup for the performance testing environment, download some test data fro the NYC Taxi dataset, create an Iceberg table inside the perf testing env, insert the test data into the table, and then run the perf tests.
   > 
   > The perf tests themselves consist of four scenarios that each execute `plan_files` in different scenarios:
   > 
   > * a file plan for a scan that reads **all** rows from **all** data files in the current snapshot (ie no filter);
   > * a file plan for a scan that reads **some** rows from **all** data files in the current snapshot (`passenger_count` = 1);
   > * a file plan for a scan that reads **all** rows from **one** data file in the current snapshot (filter: `tpep_pickup_datetime` = '2024-02-01')
   > * a file plan for a scan that reads **some** rows from **one** data file in the current snapshot (filter: `tpep_pickup_datetime` = '2024-02-01' AND `passenger_count` = 1);
   > 
   > The benchmarks get ran using criterion. Here's an example of the test output when I ran it just now on my M1 Pro Macbook:
   > 
   > <img alt="image" width="681" src="https://private-user-images.githubusercontent.com/323439/354498323-3d521b80-b3ab-4421-8889-e39cc1a0ec91.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI3NTIyOTQsIm5iZiI6MTcyMjc1MTk5NCwicGF0aCI6Ii8zMjM0MzkvMzU0NDk4MzIzLTNkNTIxYjgwLWIzYWItNDQyMS04ODg5LWUzOWNjMWEwZWM5MS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODA0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgwNFQwNjEzMTRaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1jMzY3NDhkNTFmMjQ3ZDQyNWE0Yjk0NzY3NzVlNjJjODY4NTcxZTY3MTg4NmNlMjA3MGY1ZDRkMzlmNjJiMDk5JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.bi41R1qt3HLdJjxiQPDf5U1F3WMhEdby2bEpZDTdIgw">
   > Now, if you cherry-pick the commit from this concurrent table scan branch into the `perf-suite` branch:
   > 
   > ```shell
   > git cherry-pick 8eef484094dbb7c55ac3181bbd552090308591c9
   > ```
   > 
   > And re-run the performance tests, you'll see something similar to the following:
   > 
   > <img alt="image" width="689" src="https://private-user-images.githubusercontent.com/323439/354499847-28e41e05-b1e2-48a9-a26c-20ca940e688a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI3NTIyOTQsIm5iZiI6MTcyMjc1MTk5NCwicGF0aCI6Ii8zMjM0MzkvMzU0NDk5ODQ3LTI4ZTQxZTA1LWIxZTItNDhhOS1hMjZjLTIwY2E5NDBlNjg4YS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODA0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgwNFQwNjEzMTRaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT02MWYzOTQxYmMyMDEyZjk5NWUwZTE2NzFjODRkYTkwYzA5MzAyMzM1NjlhOWZlYzgxOGIxODA0Y2I5YTE3YWYwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.WLrE2Rybmfyg4Qn3GmIeIuDmXLfESuXpM-UTvQ26LH4">
   > As you can see the times come down from around 1.5s to around **0.5s**, a big improvement!
   > 
   > In fact, if you then cherry-pick the commit from my [follow-on PR with the object cache](https://github.com/apache/iceberg-rust/pull/512):
   > 
   > ```shell
   > git cherry-pick c696a3f9a7dacf822e6d5bc76f13d24e0b50ee31
   > ```
   > 
   > and re-run the performance tests again, you'll see something even more remarkable:
   > 
   > <img alt="image" width="578" src="https://private-user-images.githubusercontent.com/323439/354501480-12e91179-2ea3-4dad-9581-d7698b80b653.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI3NTIyOTQsIm5iZiI6MTcyMjc1MTk5NCwicGF0aCI6Ii8zMjM0MzkvMzU0NTAxNDgwLTEyZTkxMTc5LTJlYTMtNGRhZC05NTgxLWQ3Njk4YjgwYjY1My5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODA0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgwNFQwNjEzMTRaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1hNmI0MDg4OGI4YWQ3N2FiNjFjNTljODgzMzI2MjA2MWJlYjBhYzE3MmI0M2Q1NmMzNWI1MjczZmFlN2E3MTM1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.WrzRg34n0axA3cur2dUnbvSf1tWBD81aYRaI3HKgCZE">
   > Now that caching of the retrieval and parsing of the `Manifest` and `ManifestList`s are taking place, the first run in the performance test will take the same time as above but all of the subsequent runs that are averaged to get an average time for each test are much, much faster, reducing the average time from 500ms to about 1.5ms. 🤯
   
   Thanks @sdd! That's awesome! BTW, can we integrate the performance test within the repository so that it will be helpful for performance improvement in the future? cc @liurenjie1024 @Xuanwo @sdd 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1697662591


##########
crates/iceberg/src/scan.rs:
##########
@@ -197,134 +193,103 @@ impl<'a> TableScanBuilder<'a> {
             field_ids.push(field_id);
         }
 
-        Ok(TableScan {
+        let snapshot_bound_predicate = if let Some(ref predicates) = self.filter {
+            Some(predicates.bind(schema.clone(), true)?)
+        } else {
+            None
+        };
+
+        let plan_context = PlanContext {
             snapshot,
-            file_io: self.table.file_io().clone(),
             table_metadata: self.table.metadata_ref(),
-            column_names: self.column_names,
-            field_ids,
-            bound_predicates,
-            schema,
-            batch_size: self.batch_size,
+            snapshot_schema: schema,
             case_sensitive: self.case_sensitive,
-            filter: self.filter.map(Arc::new),
+            predicate: self.filter.map(Arc::new),
+            snapshot_bound_predicate: snapshot_bound_predicate.map(Arc::new),
+            file_io: self.table.file_io().clone(),
+            field_ids: Arc::new(field_ids),
+            partition_filter_cache: Arc::new(PartitionFilterCache::new()),
+            manifest_evaluator_cache: Arc::new(ManifestEvaluatorCache::new()),
+            expression_evaluator_cache: Arc::new(ExpressionEvaluatorCache::new()),
+        };
+
+        Ok(TableScan {
+            batch_size: self.batch_size,
+            column_names: self.column_names,
+            file_io: self.table.file_io().clone(),
+            plan_context,
         })
     }
 }
 
 /// Table scan.
 #[derive(Debug)]
 pub struct TableScan {
-    snapshot: SnapshotRef,
-    table_metadata: TableMetadataRef,
+    plan_context: PlanContext,
+    batch_size: Option<usize>,
     file_io: FileIO,
     column_names: Vec<String>,
-    field_ids: Vec<i32>,
-    bound_predicates: Option<BoundPredicate>,
-    schema: SchemaRef,
-    batch_size: Option<usize>,
+}
+
+/// PlanContext wraps a [`SnapshotRef`] alongside all the other
+/// objects that are required to perform a scan file plan.
+#[derive(Debug)]
+struct PlanContext {
+    snapshot: SnapshotRef,
+
+    table_metadata: TableMetadataRef,
+    snapshot_schema: SchemaRef,
     case_sensitive: bool,
-    filter: Option<Arc<Predicate>>,
+    predicate: Option<Arc<Predicate>>,
+    snapshot_bound_predicate: Option<Arc<BoundPredicate>>,
+    file_io: FileIO,
+    field_ids: Arc<Vec<i32>>,
+
+    partition_filter_cache: Arc<PartitionFilterCache>,
+    manifest_evaluator_cache: Arc<ManifestEvaluatorCache>,
+    expression_evaluator_cache: Arc<ExpressionEvaluatorCache>,
 }
 
 impl TableScan {
     /// Returns a stream of [`FileScanTask`]s.
     pub async fn plan_files(&self) -> Result<FileScanTaskStream> {
-        let context = FileScanStreamContext::new(
-            self.schema.clone(),
-            self.snapshot.clone(),
-            self.table_metadata.clone(),
-            self.file_io.clone(),
-            self.filter.clone(),
-            self.case_sensitive,
-        )?;
-
-        let mut partition_filter_cache = PartitionFilterCache::new();
-        let mut manifest_evaluator_cache = ManifestEvaluatorCache::new();
-        let mut expression_evaluator_cache = ExpressionEvaluatorCache::new();
-
-        let field_ids = self.field_ids.clone();
-        let bound_predicates = self.bound_predicates.clone();
-
-        Ok(try_stream! {
-            let manifest_list = context
-                .snapshot
-                .load_manifest_list(&context.file_io, &context.table_metadata)
-                .await?;
-
-            for entry in manifest_list.entries() {
-                if !Self::content_type_is_data(entry) {
-                    continue;
-                }
-
-                let partition_spec_id = entry.partition_spec_id;
-
-                let partition_filter = partition_filter_cache.get(
-                    partition_spec_id,
-                    &context,
-                )?;
-
-                if let Some(partition_filter) = partition_filter {
-                    let manifest_evaluator = manifest_evaluator_cache.get(
-                        partition_spec_id,
-                        partition_filter,
-                    );
-
-                    if !manifest_evaluator.eval(entry)? {
-                        continue;
-                    }
-                }
+        // used to stream ManifestEntryContexts between stages of the file plan operation
+        let (manifest_entry_ctx_tx, manifest_entry_ctx_rx) =
+            channel(CONCURRENCY_LIMIT_MANIFEST_FILES);

Review Comment:
   Done - let me know if you're happy with the approach



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "marvinlanhenke (via GitHub)" <gi...@apache.org>.
marvinlanhenke commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1598053461


##########
crates/iceberg/src/scan.rs:
##########
@@ -189,66 +195,20 @@ impl TableScan {
             self.case_sensitive,
         )?;
 
-        let mut partition_filter_cache = PartitionFilterCache::new();
-        let mut manifest_evaluator_cache = ManifestEvaluatorCache::new();
-
-        Ok(try_stream! {
-            let manifest_list = context
-                .snapshot
-                .load_manifest_list(&context.file_io, &context.table_metadata)
-                .await?;
-
-            for entry in manifest_list.entries() {
-                if !Self::content_type_is_data(entry) {
-                    continue;
-                }
+        let (sender, receiver) = channel(CHANNEL_BUFFER_SIZE);
 
-                let partition_spec_id = entry.partition_spec_id;
+        let manifest_list = context
+            .snapshot
+            .load_manifest_list(&context.file_io, &context.table_metadata)
+            .await?;
 
-                let partition_filter = partition_filter_cache.get(
-                    partition_spec_id,
-                    &context,
-                )?;
+        spawn(async move {

Review Comment:
   do we need to `spawn` here or is the `try_for_each_concurrent` in `run(...)` already enough?



##########
crates/iceberg/src/scan.rs:
##########
@@ -302,13 +262,147 @@ impl TableScan {
 
         arrow_reader_builder.build().read(self.plan_files().await?)
     }
+}
+
+#[derive(Debug)]
+struct ConcurrentFileScanStreamContext {
+    context: FileScanStreamContext,
+    sender: Sender<Result<FileScanTask>>,
+    manifest_evaluator_cache: ManifestEvaluatorCache,
+    partition_filter_cache: PartitionFilterCache,
+}
+
+impl ConcurrentFileScanStreamContext {
+    fn new(context: FileScanStreamContext, sender: Sender<Result<FileScanTask>>) -> Self {
+        ConcurrentFileScanStreamContext {
+            context,
+            sender,
+            manifest_evaluator_cache: ManifestEvaluatorCache::new(),
+            partition_filter_cache: PartitionFilterCache::new(),
+        }
+    }
+
+    async fn run(&mut self, manifest_list: ManifestList) -> Result<()> {
+        let file_io = self.context.file_io.clone();
+        let sender = self.sender.clone();
+
+        // This whole Vec-and-for-loop approach feels sub-optimally structured.
+        // I've tried structuring this in multiple ways but run into
+        // issues with ownership. Ideally I'd like to structure this
+        // with a functional programming approach: extracting
+        // sections 1, 2, and 3 out into different methods on Self,
+        // and then use some iterator combinators to orchestrate it all.
+        // Section 1 is pretty trivially refactorable into a static method
+        // that can be used in a closure that can be used with Iterator::filter.
+        // Similarly, section 3 seems easily factor-able into a method that can
+        // be passed into Iterator::map.
+        // Section 2 turns out trickier - we want to exit the entire `run` method early
+        // if the eval fails, and filter out any manifest_files from the iterator / stream
+        // if the eval succeeds but returns true. We bump into ownership issues due
+        // to needing to pass mut self as the caches need to be able to mutate.
+        // Section 3 runs into ownership issues when trying to refactor its closure to be
+        // a static or non-static method.
+
+        // 1
+        let filtered_manifest_files = manifest_list
+            .entries()
+            .iter()
+            .filter(Self::reject_unsupported_content_types);
+
+        // 2
+        let mut filtered_manifest_files2 = vec![];
+        for manifest_file in filtered_manifest_files {
+            if !self.apply_evaluator(manifest_file)? {
+                continue;
+            }
+
+            filtered_manifest_files2.push(manifest_file);
+        }
+
+        // 3
+        let filtered_manifest_files = filtered_manifest_files2
+            .into_iter()
+            .map(|manifest_file| Ok((manifest_file, file_io.clone(), sender.clone())));
+
+        futures::stream::iter(filtered_manifest_files)
+            .try_for_each_concurrent(
+                CONCURRENCY_LIMIT_MANIFEST_FILES,
+                Self::process_manifest_file,
+            )
+            .await
+    }
+
+    fn reject_unsupported_content_types(manifest_file: &&ManifestFile) -> bool {
+        SUPPORTED_MANIFEST_FILE_CONTENT_TYPES.contains(&manifest_file.content)
+    }
+
+    fn apply_evaluator(&mut self, manifest_file: &ManifestFile) -> Result<bool> {
+        let partition_spec_id = manifest_file.partition_spec_id;
+
+        let partition_filter = self
+            .partition_filter_cache
+            .get(partition_spec_id, &self.context)?;
+
+        if let Some(partition_filter) = partition_filter {
+            let manifest_evaluator = self
+                .manifest_evaluator_cache
+                .get(partition_spec_id, partition_filter);
+
+            if !manifest_evaluator.eval(manifest_file)? {
+                return Ok(false);
+            }
+        }
+
+        Ok(true)
+    }
+
+    async fn process_manifest_file(
+        manifest_and_file_io_and_sender: (&ManifestFile, FileIO, Sender<Result<FileScanTask>>),
+    ) -> Result<()> {
+        let (manifest_file, file_io, sender) = manifest_and_file_io_and_sender;
+
+        let manifest = manifest_file.load_manifest(&file_io).await?;
+
+        let manifest_entries = manifest
+            .entries()
+            .iter()
+            .filter(|x| x.is_alive())
+            .map(|manifest_entry| Ok((manifest_entry, sender.clone())));
+
+        futures::stream::iter(manifest_entries)
+            .try_for_each_concurrent(
+                CONCURRENCY_LIMIT_MANIFEST_ENTRIES,
+                Self::process_manifest_entry,
+            )
+            .await

Review Comment:
   we had a discussion on this in #124 (in case you missed it) to not go overboard with the task spawning



##########
crates/iceberg/src/scan.rs:
##########
@@ -302,13 +262,147 @@ impl TableScan {
 
         arrow_reader_builder.build().read(self.plan_files().await?)
     }
+}
+
+#[derive(Debug)]
+struct ConcurrentFileScanStreamContext {

Review Comment:
   I think once we have a runtime and the async approach is approved, we can get rid of the `FileScanStreamContext` and merge the struct into `ConcurrentFileScanStreamContext`



##########
crates/iceberg/src/scan.rs:
##########
@@ -302,13 +262,147 @@ impl TableScan {
 
         arrow_reader_builder.build().read(self.plan_files().await?)
     }
+}
+
+#[derive(Debug)]
+struct ConcurrentFileScanStreamContext {
+    context: FileScanStreamContext,
+    sender: Sender<Result<FileScanTask>>,
+    manifest_evaluator_cache: ManifestEvaluatorCache,
+    partition_filter_cache: PartitionFilterCache,
+}
+
+impl ConcurrentFileScanStreamContext {
+    fn new(context: FileScanStreamContext, sender: Sender<Result<FileScanTask>>) -> Self {
+        ConcurrentFileScanStreamContext {
+            context,
+            sender,
+            manifest_evaluator_cache: ManifestEvaluatorCache::new(),
+            partition_filter_cache: PartitionFilterCache::new(),
+        }
+    }
+
+    async fn run(&mut self, manifest_list: ManifestList) -> Result<()> {
+        let file_io = self.context.file_io.clone();
+        let sender = self.sender.clone();
+
+        // This whole Vec-and-for-loop approach feels sub-optimally structured.
+        // I've tried structuring this in multiple ways but run into
+        // issues with ownership. Ideally I'd like to structure this
+        // with a functional programming approach: extracting
+        // sections 1, 2, and 3 out into different methods on Self,
+        // and then use some iterator combinators to orchestrate it all.
+        // Section 1 is pretty trivially refactorable into a static method
+        // that can be used in a closure that can be used with Iterator::filter.
+        // Similarly, section 3 seems easily factor-able into a method that can
+        // be passed into Iterator::map.
+        // Section 2 turns out trickier - we want to exit the entire `run` method early
+        // if the eval fails, and filter out any manifest_files from the iterator / stream
+        // if the eval succeeds but returns true. We bump into ownership issues due
+        // to needing to pass mut self as the caches need to be able to mutate.
+        // Section 3 runs into ownership issues when trying to refactor its closure to be
+        // a static or non-static method.
+
+        // 1
+        let filtered_manifest_files = manifest_list
+            .entries()
+            .iter()
+            .filter(Self::reject_unsupported_content_types);
+
+        // 2
+        let mut filtered_manifest_files2 = vec![];
+        for manifest_file in filtered_manifest_files {
+            if !self.apply_evaluator(manifest_file)? {
+                continue;
+            }
+
+            filtered_manifest_files2.push(manifest_file);
+        }
+
+        // 3
+        let filtered_manifest_files = filtered_manifest_files2
+            .into_iter()
+            .map(|manifest_file| Ok((manifest_file, file_io.clone(), sender.clone())));
+
+        futures::stream::iter(filtered_manifest_files)
+            .try_for_each_concurrent(
+                CONCURRENCY_LIMIT_MANIFEST_FILES,
+                Self::process_manifest_file,
+            )
+            .await

Review Comment:
   will this yield the first FileScanTask when its available - or do we have to iterate over the complete stream, before we can return any result?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1609368600


##########
crates/iceberg/src/scan.rs:
##########
@@ -189,66 +195,20 @@ impl TableScan {
             self.case_sensitive,
         )?;
 
-        let mut partition_filter_cache = PartitionFilterCache::new();
-        let mut manifest_evaluator_cache = ManifestEvaluatorCache::new();
-
-        Ok(try_stream! {
-            let manifest_list = context
-                .snapshot
-                .load_manifest_list(&context.file_io, &context.table_metadata)
-                .await?;
-
-            for entry in manifest_list.entries() {
-                if !Self::content_type_is_data(entry) {
-                    continue;
-                }
+        let (sender, receiver) = channel(CHANNEL_BUFFER_SIZE);
 
-                let partition_spec_id = entry.partition_spec_id;
+        let manifest_list = context
+            .snapshot
+            .load_manifest_list(&context.file_io, &context.table_metadata)
+            .await?;
 
-                let partition_filter = partition_filter_cache.get(
-                    partition_spec_id,
-                    &context,
-                )?;
+        spawn(async move {

Review Comment:
   I reverted reader.rs to be essentially the same as before, without any concurrent processing of batches from within the same file. I removed the nested try_for_each_concurrent from scan.rs but kept the mpsc channel.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1689787900


##########
crates/iceberg/src/scan.rs:
##########
@@ -197,134 +193,103 @@ impl<'a> TableScanBuilder<'a> {
             field_ids.push(field_id);
         }
 
-        Ok(TableScan {
+        let snapshot_bound_predicate = if let Some(ref predicates) = self.filter {
+            Some(predicates.bind(schema.clone(), true)?)
+        } else {
+            None
+        };
+
+        let plan_context = PlanContext {
             snapshot,
-            file_io: self.table.file_io().clone(),
             table_metadata: self.table.metadata_ref(),
-            column_names: self.column_names,
-            field_ids,
-            bound_predicates,
-            schema,
-            batch_size: self.batch_size,
+            snapshot_schema: schema,
             case_sensitive: self.case_sensitive,
-            filter: self.filter.map(Arc::new),
+            predicate: self.filter.map(Arc::new),
+            snapshot_bound_predicate: snapshot_bound_predicate.map(Arc::new),
+            file_io: self.table.file_io().clone(),
+            field_ids: Arc::new(field_ids),
+            partition_filter_cache: Arc::new(PartitionFilterCache::new()),
+            manifest_evaluator_cache: Arc::new(ManifestEvaluatorCache::new()),
+            expression_evaluator_cache: Arc::new(ExpressionEvaluatorCache::new()),
+        };
+
+        Ok(TableScan {
+            batch_size: self.batch_size,
+            column_names: self.column_names,
+            file_io: self.table.file_io().clone(),
+            plan_context,
         })
     }
 }
 
 /// Table scan.
 #[derive(Debug)]
 pub struct TableScan {
-    snapshot: SnapshotRef,
-    table_metadata: TableMetadataRef,
+    plan_context: PlanContext,
+    batch_size: Option<usize>,
     file_io: FileIO,
     column_names: Vec<String>,
-    field_ids: Vec<i32>,
-    bound_predicates: Option<BoundPredicate>,
-    schema: SchemaRef,
-    batch_size: Option<usize>,
+}
+
+/// PlanContext wraps a [`SnapshotRef`] alongside all the other
+/// objects that are required to perform a scan file plan.
+#[derive(Debug)]
+struct PlanContext {
+    snapshot: SnapshotRef,
+
+    table_metadata: TableMetadataRef,
+    snapshot_schema: SchemaRef,
     case_sensitive: bool,
-    filter: Option<Arc<Predicate>>,
+    predicate: Option<Arc<Predicate>>,
+    snapshot_bound_predicate: Option<Arc<BoundPredicate>>,
+    file_io: FileIO,
+    field_ids: Arc<Vec<i32>>,
+
+    partition_filter_cache: Arc<PartitionFilterCache>,
+    manifest_evaluator_cache: Arc<ManifestEvaluatorCache>,
+    expression_evaluator_cache: Arc<ExpressionEvaluatorCache>,
 }
 
 impl TableScan {
     /// Returns a stream of [`FileScanTask`]s.
     pub async fn plan_files(&self) -> Result<FileScanTaskStream> {
-        let context = FileScanStreamContext::new(
-            self.schema.clone(),
-            self.snapshot.clone(),
-            self.table_metadata.clone(),
-            self.file_io.clone(),
-            self.filter.clone(),
-            self.case_sensitive,
-        )?;
-
-        let mut partition_filter_cache = PartitionFilterCache::new();
-        let mut manifest_evaluator_cache = ManifestEvaluatorCache::new();
-        let mut expression_evaluator_cache = ExpressionEvaluatorCache::new();
-
-        let field_ids = self.field_ids.clone();
-        let bound_predicates = self.bound_predicates.clone();
-
-        Ok(try_stream! {
-            let manifest_list = context
-                .snapshot
-                .load_manifest_list(&context.file_io, &context.table_metadata)
-                .await?;
-
-            for entry in manifest_list.entries() {
-                if !Self::content_type_is_data(entry) {
-                    continue;
-                }
-
-                let partition_spec_id = entry.partition_spec_id;
-
-                let partition_filter = partition_filter_cache.get(
-                    partition_spec_id,
-                    &context,
-                )?;
-
-                if let Some(partition_filter) = partition_filter {
-                    let manifest_evaluator = manifest_evaluator_cache.get(
-                        partition_spec_id,
-                        partition_filter,
-                    );
-
-                    if !manifest_evaluator.eval(entry)? {
-                        continue;
-                    }
-                }
+        // used to stream ManifestEntryContexts between stages of the file plan operation
+        let (manifest_entry_ctx_tx, manifest_entry_ctx_rx) =
+            channel(CONCURRENCY_LIMIT_MANIFEST_FILES);
+        // used to stream the results back to the caller
+        let (file_scan_task_tx, file_scan_task_rx) = channel(CONCURRENCY_LIMIT_MANIFEST_ENTRIES);

Review Comment:
   Dittol



##########
crates/iceberg/src/scan.rs:
##########
@@ -197,134 +193,103 @@ impl<'a> TableScanBuilder<'a> {
             field_ids.push(field_id);
         }
 
-        Ok(TableScan {
+        let snapshot_bound_predicate = if let Some(ref predicates) = self.filter {
+            Some(predicates.bind(schema.clone(), true)?)
+        } else {
+            None
+        };
+
+        let plan_context = PlanContext {
             snapshot,
-            file_io: self.table.file_io().clone(),
             table_metadata: self.table.metadata_ref(),
-            column_names: self.column_names,
-            field_ids,
-            bound_predicates,
-            schema,
-            batch_size: self.batch_size,
+            snapshot_schema: schema,
             case_sensitive: self.case_sensitive,
-            filter: self.filter.map(Arc::new),
+            predicate: self.filter.map(Arc::new),
+            snapshot_bound_predicate: snapshot_bound_predicate.map(Arc::new),
+            file_io: self.table.file_io().clone(),
+            field_ids: Arc::new(field_ids),
+            partition_filter_cache: Arc::new(PartitionFilterCache::new()),
+            manifest_evaluator_cache: Arc::new(ManifestEvaluatorCache::new()),
+            expression_evaluator_cache: Arc::new(ExpressionEvaluatorCache::new()),
+        };
+
+        Ok(TableScan {
+            batch_size: self.batch_size,
+            column_names: self.column_names,
+            file_io: self.table.file_io().clone(),
+            plan_context,
         })
     }
 }
 
 /// Table scan.
 #[derive(Debug)]
 pub struct TableScan {
-    snapshot: SnapshotRef,
-    table_metadata: TableMetadataRef,
+    plan_context: PlanContext,
+    batch_size: Option<usize>,
     file_io: FileIO,
     column_names: Vec<String>,
-    field_ids: Vec<i32>,
-    bound_predicates: Option<BoundPredicate>,
-    schema: SchemaRef,
-    batch_size: Option<usize>,
+}
+
+/// PlanContext wraps a [`SnapshotRef`] alongside all the other
+/// objects that are required to perform a scan file plan.
+#[derive(Debug)]
+struct PlanContext {
+    snapshot: SnapshotRef,
+
+    table_metadata: TableMetadataRef,
+    snapshot_schema: SchemaRef,
     case_sensitive: bool,
-    filter: Option<Arc<Predicate>>,
+    predicate: Option<Arc<Predicate>>,
+    snapshot_bound_predicate: Option<Arc<BoundPredicate>>,
+    file_io: FileIO,
+    field_ids: Arc<Vec<i32>>,
+
+    partition_filter_cache: Arc<PartitionFilterCache>,
+    manifest_evaluator_cache: Arc<ManifestEvaluatorCache>,
+    expression_evaluator_cache: Arc<ExpressionEvaluatorCache>,
 }
 
 impl TableScan {
     /// Returns a stream of [`FileScanTask`]s.
     pub async fn plan_files(&self) -> Result<FileScanTaskStream> {
-        let context = FileScanStreamContext::new(
-            self.schema.clone(),
-            self.snapshot.clone(),
-            self.table_metadata.clone(),
-            self.file_io.clone(),
-            self.filter.clone(),
-            self.case_sensitive,
-        )?;
-
-        let mut partition_filter_cache = PartitionFilterCache::new();
-        let mut manifest_evaluator_cache = ManifestEvaluatorCache::new();
-        let mut expression_evaluator_cache = ExpressionEvaluatorCache::new();
-
-        let field_ids = self.field_ids.clone();
-        let bound_predicates = self.bound_predicates.clone();
-
-        Ok(try_stream! {
-            let manifest_list = context
-                .snapshot
-                .load_manifest_list(&context.file_io, &context.table_metadata)
-                .await?;
-
-            for entry in manifest_list.entries() {
-                if !Self::content_type_is_data(entry) {
-                    continue;
-                }
-
-                let partition_spec_id = entry.partition_spec_id;
-
-                let partition_filter = partition_filter_cache.get(
-                    partition_spec_id,
-                    &context,
-                )?;
-
-                if let Some(partition_filter) = partition_filter {
-                    let manifest_evaluator = manifest_evaluator_cache.get(
-                        partition_spec_id,
-                        partition_filter,
-                    );
-
-                    if !manifest_evaluator.eval(entry)? {
-                        continue;
-                    }
-                }
+        // used to stream ManifestEntryContexts between stages of the file plan operation
+        let (manifest_entry_ctx_tx, manifest_entry_ctx_rx) =
+            channel(CONCURRENCY_LIMIT_MANIFEST_FILES);

Review Comment:
   How about making this a config of table scan with default value?



##########
crates/iceberg/src/spec/manifest_list.rs:
##########
@@ -79,6 +79,11 @@ impl ManifestList {
     pub fn entries(&self) -> &[ManifestFile] {
         &self.entries
     }
+
+    /// Take ownership of the entries in the manifest list, consuming it
+    pub fn consume_entries(self) -> Box<dyn Iterator<Item = ManifestFile>> {

Review Comment:
   Why not just `impl IntoIterator`?



##########
crates/iceberg/src/expr/predicate.rs:
##########
@@ -668,6 +668,10 @@ pub enum BoundPredicate {
     Set(SetExpression<BoundReference>),
 }
 
+/// Newtype to prevent accidentally confusing predicates that are bound to a partition with ones that are bound to a schema.
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+pub struct PartitionBoundPredicate(pub(crate) BoundPredicate);

Review Comment:
   Hmm, I have concerns adding this struct, since it's essentially the same as `BoundPredicate`, and I guess using variable name would  be enough? cc @Xuanwo @Fokko WDYT?



##########
crates/iceberg/src/scan.rs:
##########
@@ -197,134 +193,103 @@ impl<'a> TableScanBuilder<'a> {
             field_ids.push(field_id);
         }
 
-        Ok(TableScan {
+        let snapshot_bound_predicate = if let Some(ref predicates) = self.filter {
+            Some(predicates.bind(schema.clone(), true)?)
+        } else {
+            None
+        };
+
+        let plan_context = PlanContext {
             snapshot,
-            file_io: self.table.file_io().clone(),
             table_metadata: self.table.metadata_ref(),
-            column_names: self.column_names,
-            field_ids,
-            bound_predicates,
-            schema,
-            batch_size: self.batch_size,
+            snapshot_schema: schema,
             case_sensitive: self.case_sensitive,
-            filter: self.filter.map(Arc::new),
+            predicate: self.filter.map(Arc::new),
+            snapshot_bound_predicate: snapshot_bound_predicate.map(Arc::new),
+            file_io: self.table.file_io().clone(),
+            field_ids: Arc::new(field_ids),
+            partition_filter_cache: Arc::new(PartitionFilterCache::new()),
+            manifest_evaluator_cache: Arc::new(ManifestEvaluatorCache::new()),
+            expression_evaluator_cache: Arc::new(ExpressionEvaluatorCache::new()),
+        };
+
+        Ok(TableScan {
+            batch_size: self.batch_size,
+            column_names: self.column_names,
+            file_io: self.table.file_io().clone(),
+            plan_context,
         })
     }
 }
 
 /// Table scan.
 #[derive(Debug)]
 pub struct TableScan {
-    snapshot: SnapshotRef,
-    table_metadata: TableMetadataRef,
+    plan_context: PlanContext,
+    batch_size: Option<usize>,
     file_io: FileIO,
     column_names: Vec<String>,
-    field_ids: Vec<i32>,
-    bound_predicates: Option<BoundPredicate>,
-    schema: SchemaRef,
-    batch_size: Option<usize>,
+}
+
+/// PlanContext wraps a [`SnapshotRef`] alongside all the other
+/// objects that are required to perform a scan file plan.
+#[derive(Debug)]
+struct PlanContext {
+    snapshot: SnapshotRef,
+
+    table_metadata: TableMetadataRef,
+    snapshot_schema: SchemaRef,
     case_sensitive: bool,
-    filter: Option<Arc<Predicate>>,
+    predicate: Option<Arc<Predicate>>,
+    snapshot_bound_predicate: Option<Arc<BoundPredicate>>,
+    file_io: FileIO,
+    field_ids: Arc<Vec<i32>>,
+
+    partition_filter_cache: Arc<PartitionFilterCache>,
+    manifest_evaluator_cache: Arc<ManifestEvaluatorCache>,
+    expression_evaluator_cache: Arc<ExpressionEvaluatorCache>,
 }
 
 impl TableScan {
     /// Returns a stream of [`FileScanTask`]s.
     pub async fn plan_files(&self) -> Result<FileScanTaskStream> {
-        let context = FileScanStreamContext::new(
-            self.schema.clone(),
-            self.snapshot.clone(),
-            self.table_metadata.clone(),
-            self.file_io.clone(),
-            self.filter.clone(),
-            self.case_sensitive,
-        )?;
-
-        let mut partition_filter_cache = PartitionFilterCache::new();
-        let mut manifest_evaluator_cache = ManifestEvaluatorCache::new();
-        let mut expression_evaluator_cache = ExpressionEvaluatorCache::new();
-
-        let field_ids = self.field_ids.clone();
-        let bound_predicates = self.bound_predicates.clone();
-
-        Ok(try_stream! {
-            let manifest_list = context
-                .snapshot
-                .load_manifest_list(&context.file_io, &context.table_metadata)
-                .await?;
-
-            for entry in manifest_list.entries() {
-                if !Self::content_type_is_data(entry) {
-                    continue;
-                }
-
-                let partition_spec_id = entry.partition_spec_id;
-
-                let partition_filter = partition_filter_cache.get(
-                    partition_spec_id,
-                    &context,
-                )?;
-
-                if let Some(partition_filter) = partition_filter {
-                    let manifest_evaluator = manifest_evaluator_cache.get(
-                        partition_spec_id,
-                        partition_filter,
-                    );
-
-                    if !manifest_evaluator.eval(entry)? {
-                        continue;
-                    }
-                }
+        // used to stream ManifestEntryContexts between stages of the file plan operation
+        let (manifest_entry_ctx_tx, manifest_entry_ctx_rx) =
+            channel(CONCURRENCY_LIMIT_MANIFEST_FILES);
+        // used to stream the results back to the caller
+        let (file_scan_task_tx, file_scan_task_rx) = channel(CONCURRENCY_LIMIT_MANIFEST_ENTRIES);
+
+        let manifest_list = self.plan_context.get_manifest_list().await?;
+
+        // get the [`ManifestFile`]s from the [`ManifestList`], filtering out any
+        // whose content type is not Data or whose partitions cannot match this
+        // scan's filter
+        let manifest_file_contexts = self
+            .plan_context
+            .build_manifest_file_contexts(manifest_list, manifest_entry_ctx_tx)?;
+
+        // Concurrently load all [`Manifest`]s and stream their [`ManifestEntry`]s
+        futures::stream::iter(manifest_file_contexts)
+            .try_for_each_concurrent(CONCURRENCY_LIMIT_MANIFEST_FILES, |ctx| async move {
+                ctx.fetch_manifest_and_stream_manifest_entries().await
+            })
+            .await?;
+
+        // Process the [`ManifestEntry`] stream in parallel
+        manifest_entry_ctx_rx
+            .map(|me_ctx| Ok((me_ctx, file_scan_task_tx.clone())))
+            .try_for_each_concurrent(
+                CONCURRENCY_LIMIT_MANIFEST_ENTRIES,
+                |(manifest_entry_context, tx)| async move {
+                    crate::runtime::spawn(async move {
+                        Self::process_manifest_entry(manifest_entry_context, tx).await

Review Comment:
   To be honest, I have some concerns about generating one task for each manifest entry. Though future is cheap, this means to have hundreds of thousands of `Futures`.  But this is just some concern, maybe we can refactor this later if we actually meet some problems. cc @Xuanwo WDYT?



##########
crates/iceberg/src/scan.rs:
##########
@@ -338,157 +303,463 @@ impl TableScan {
         arrow_reader_builder.build().read(self.plan_files().await?)
     }
 
-    /// Checks whether the [`ManifestContentType`] is `Data` or not.
-    fn content_type_is_data(entry: &ManifestFile) -> bool {
-        if let ManifestContentType::Data = entry.content {
-            return true;
-        }
-        false
-    }
-
     /// Returns a reference to the column names of the table scan.
     pub fn column_names(&self) -> &[String] {
         &self.column_names
     }
+    /// Returns a reference to the snapshot of the table scan.
+    pub fn snapshot(&self) -> &SnapshotRef {
+        &self.plan_context.snapshot
+    }
+
+    async fn process_manifest_entry(
+        manifest_entry_context: ManifestEntryContext,
+        mut file_scan_task_tx: Sender<Result<FileScanTask>>,
+    ) -> Result<()> {
+        // skip processing this manifest entry if it has been marked as deleted
+        if !manifest_entry_context.manifest_entry.is_alive() {
+            return Ok(());
+        }
+
+        // abort the plan if we encounter a manifest entry whose data file's
+        // content type is currently unsupported
+        if manifest_entry_context.manifest_entry.content_type() != DataContentType::Data {
+            return Err(Error::new(
+                ErrorKind::FeatureUnsupported,
+                "Only Data files currently supported",
+            ));
+        }
+
+        if let Some(ref snapshot_bound_predicate) = manifest_entry_context.snapshot_bound_predicate
+        {
+            let Some(ref partition_bound_predicate) =
+                manifest_entry_context.partition_bound_predicate
+            else {
+                panic!("partition_bound_predicate expected to always be present if snapshot_bound_predicate is present");

Review Comment:
   I'm not a big fan of this approach, how about make the field in `ManifestEntryContext` as `Option<(BoundPredicate, PartitionBoundPredicae)>`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "odysa (via GitHub)" <gi...@apache.org>.
odysa commented on PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#issuecomment-2236795680

   @sdd `JoinHandle` in `Tokio` and `async-std` have different return types.
   In [Tokio](https://docs.rs/tokio/latest/src/tokio/runtime/task/join.rs.html#324)
   ```rs
   impl<T> Future for JoinHandle<T> {
       type Output = super::Result<T>;
   ```
   In [Async-std](https://github.com/async-rs/async-std/blob/b7be2183d55a6bd62907cac0b10732f377d39fe2/src/task/join_handle.rs#L79)
   ```rs
   impl<T> Future for JoinHandle<T> {
       type Output = T;
   ```
   `crate::runtime::spawn` is consistent with async-std, so you may remove the `?` to resolve this issue.
   
   However, we can still make it to behave like Tokio apis. We may need to discuss it further. cc @liurenjie1024 @Xuanwo  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1690704340


##########
crates/iceberg/src/scan.rs:
##########
@@ -197,134 +193,103 @@ impl<'a> TableScanBuilder<'a> {
             field_ids.push(field_id);
         }
 
-        Ok(TableScan {
+        let snapshot_bound_predicate = if let Some(ref predicates) = self.filter {
+            Some(predicates.bind(schema.clone(), true)?)
+        } else {
+            None
+        };
+
+        let plan_context = PlanContext {
             snapshot,
-            file_io: self.table.file_io().clone(),
             table_metadata: self.table.metadata_ref(),
-            column_names: self.column_names,
-            field_ids,
-            bound_predicates,
-            schema,
-            batch_size: self.batch_size,
+            snapshot_schema: schema,
             case_sensitive: self.case_sensitive,
-            filter: self.filter.map(Arc::new),
+            predicate: self.filter.map(Arc::new),
+            snapshot_bound_predicate: snapshot_bound_predicate.map(Arc::new),
+            file_io: self.table.file_io().clone(),
+            field_ids: Arc::new(field_ids),
+            partition_filter_cache: Arc::new(PartitionFilterCache::new()),
+            manifest_evaluator_cache: Arc::new(ManifestEvaluatorCache::new()),
+            expression_evaluator_cache: Arc::new(ExpressionEvaluatorCache::new()),
+        };
+
+        Ok(TableScan {
+            batch_size: self.batch_size,
+            column_names: self.column_names,
+            file_io: self.table.file_io().clone(),
+            plan_context,
         })
     }
 }
 
 /// Table scan.
 #[derive(Debug)]
 pub struct TableScan {
-    snapshot: SnapshotRef,
-    table_metadata: TableMetadataRef,
+    plan_context: PlanContext,
+    batch_size: Option<usize>,
     file_io: FileIO,
     column_names: Vec<String>,
-    field_ids: Vec<i32>,
-    bound_predicates: Option<BoundPredicate>,
-    schema: SchemaRef,
-    batch_size: Option<usize>,
+}
+
+/// PlanContext wraps a [`SnapshotRef`] alongside all the other
+/// objects that are required to perform a scan file plan.
+#[derive(Debug)]
+struct PlanContext {
+    snapshot: SnapshotRef,
+
+    table_metadata: TableMetadataRef,
+    snapshot_schema: SchemaRef,
     case_sensitive: bool,
-    filter: Option<Arc<Predicate>>,
+    predicate: Option<Arc<Predicate>>,
+    snapshot_bound_predicate: Option<Arc<BoundPredicate>>,
+    file_io: FileIO,
+    field_ids: Arc<Vec<i32>>,
+
+    partition_filter_cache: Arc<PartitionFilterCache>,
+    manifest_evaluator_cache: Arc<ManifestEvaluatorCache>,
+    expression_evaluator_cache: Arc<ExpressionEvaluatorCache>,
 }
 
 impl TableScan {
     /// Returns a stream of [`FileScanTask`]s.
     pub async fn plan_files(&self) -> Result<FileScanTaskStream> {
-        let context = FileScanStreamContext::new(
-            self.schema.clone(),
-            self.snapshot.clone(),
-            self.table_metadata.clone(),
-            self.file_io.clone(),
-            self.filter.clone(),
-            self.case_sensitive,
-        )?;
-
-        let mut partition_filter_cache = PartitionFilterCache::new();
-        let mut manifest_evaluator_cache = ManifestEvaluatorCache::new();
-        let mut expression_evaluator_cache = ExpressionEvaluatorCache::new();
-
-        let field_ids = self.field_ids.clone();
-        let bound_predicates = self.bound_predicates.clone();
-
-        Ok(try_stream! {
-            let manifest_list = context
-                .snapshot
-                .load_manifest_list(&context.file_io, &context.table_metadata)
-                .await?;
-
-            for entry in manifest_list.entries() {
-                if !Self::content_type_is_data(entry) {
-                    continue;
-                }
-
-                let partition_spec_id = entry.partition_spec_id;
-
-                let partition_filter = partition_filter_cache.get(
-                    partition_spec_id,
-                    &context,
-                )?;
-
-                if let Some(partition_filter) = partition_filter {
-                    let manifest_evaluator = manifest_evaluator_cache.get(
-                        partition_spec_id,
-                        partition_filter,
-                    );
-
-                    if !manifest_evaluator.eval(entry)? {
-                        continue;
-                    }
-                }
+        // used to stream ManifestEntryContexts between stages of the file plan operation
+        let (manifest_entry_ctx_tx, manifest_entry_ctx_rx) =
+            channel(CONCURRENCY_LIMIT_MANIFEST_FILES);
+        // used to stream the results back to the caller
+        let (file_scan_task_tx, file_scan_task_rx) = channel(CONCURRENCY_LIMIT_MANIFEST_ENTRIES);
+
+        let manifest_list = self.plan_context.get_manifest_list().await?;
+
+        // get the [`ManifestFile`]s from the [`ManifestList`], filtering out any
+        // whose content type is not Data or whose partitions cannot match this
+        // scan's filter
+        let manifest_file_contexts = self
+            .plan_context
+            .build_manifest_file_contexts(manifest_list, manifest_entry_ctx_tx)?;
+
+        // Concurrently load all [`Manifest`]s and stream their [`ManifestEntry`]s
+        futures::stream::iter(manifest_file_contexts)
+            .try_for_each_concurrent(CONCURRENCY_LIMIT_MANIFEST_FILES, |ctx| async move {
+                ctx.fetch_manifest_and_stream_manifest_entries().await
+            })
+            .await?;
+
+        // Process the [`ManifestEntry`] stream in parallel
+        manifest_entry_ctx_rx
+            .map(|me_ctx| Ok((me_ctx, file_scan_task_tx.clone())))
+            .try_for_each_concurrent(
+                CONCURRENCY_LIMIT_MANIFEST_ENTRIES,
+                |(manifest_entry_context, tx)| async move {
+                    crate::runtime::spawn(async move {
+                        Self::process_manifest_entry(manifest_entry_context, tx).await

Review Comment:
   I prefer to have concurrent as default and configurable, in fact, @Xuanwo as a database developer is an advanced user to me. 
   
   > In terms of having non-concurrent as the default option, my preference would be for the library to be able to choose the optimally performant (probably the number of cores?) as the default rather than no concurrncy.
   
   This sound an interesting idea to me since the actual parallelism is determined by number of threads of async runtime the user uses. Given user always has the option to change it, I think both solutions: a default value or number of cores are good to me.
   
   > Regarding the large number of tasks - there are a large number generated over the duration of the plan, but only a fixed small number that are in existence at one time.
   
   Sounds reasonable to me.
   
   > If I make the concurrncy limits configurable in the builder as @liurenjie1024 suggested above, then we could have a with_no_concurrency method on the builder. 
   
   If we allow user to set concurrency limits, user just need to set it to 1 to execute it without concurrency?  Having another `with_no_concurrency` somehow seems counterintuitive to me. Or do you mean to maintain another code path without concurrency? If so, I think we should avoid that given it introduces extra maintain effort.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1697765089


##########
crates/iceberg/src/expr/predicate.rs:
##########
@@ -668,6 +668,10 @@ pub enum BoundPredicate {
     Set(SetExpression<BoundReference>),
 }
 
+/// Newtype to prevent accidentally confusing predicates that are bound to a partition with ones that are bound to a schema.
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+pub struct PartitionBoundPredicate(pub(crate) BoundPredicate);

Review Comment:
   I've removed it. It was ugly anyway 😅 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#issuecomment-2264974929

   > Hey @ZENOTME! Sure. If you check out my `perf-suite` branch from [my other PR](https://github.com/apache/iceberg-rust/pull/497), this is branched off main and can be used to get a baseline of `plan_files` performance without the changes in this PR. You'll need to run `cargo install just` if you don't have `just` already installed, and then `just perf-run`. This should create the docker setup for the performance testing environment, download some test data fro the NYC Taxi dataset, create an Iceberg table inside the perf testing env, insert the test data into the table, and then run the perf tests.
   > 
   > The perf tests themselves consist of four scenarios that each execute `plan_files` in different scenarios:
   > 
   > * a file plan for a scan that reads **all** rows from **all** data files in the current snapshot (ie no filter);
   > * a file plan for a scan that reads **some** rows from **all** data files in the current snapshot (`passenger_count` = 1);
   > * a file plan for a scan that reads **all** rows from **one** data file in the current snapshot (filter: `tpep_pickup_datetime` = '2024-02-01')
   > * a file plan for a scan that reads **some** rows from **one** data file in the current snapshot (filter: `tpep_pickup_datetime` = '2024-02-01' AND `passenger_count` = 1);
   > 
   > The benchmarks get ran using criterion. Here's an example of the test output when I ran it just now on my M1 Pro Macbook:
   > 
   > <img alt="image" width="681" src="https://private-user-images.githubusercontent.com/323439/354498323-3d521b80-b3ab-4421-8889-e39cc1a0ec91.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI1OTE1NzgsIm5iZiI6MTcyMjU5MTI3OCwicGF0aCI6Ii8zMjM0MzkvMzU0NDk4MzIzLTNkNTIxYjgwLWIzYWItNDQyMS04ODg5LWUzOWNjMWEwZWM5MS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODAyJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgwMlQwOTM0MzhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT05ZjY1YTJlMWY5MmM4MjYxMDM1MDc2YmI1M2E1NzcxM2U2MWYzMzQ2ZDA0MWE2NjFmYmY4ZGNhNDM2NzRjMGM0JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.VNpmn5XuoJI--VVxypJOyKHZh1JR3kqkjhBQhuRhZps">
   > Now, if you cherry-pick the commit from this concurrent table scan branch into the `perf-suite` branch:
   > 
   > ```shell
   > git cherry-pick 8eef484094dbb7c55ac3181bbd552090308591c9
   > ```
   > 
   > And re-run the performance tests, you'll see something similar to the following:
   > 
   > <img alt="image" width="689" src="https://private-user-images.githubusercontent.com/323439/354499847-28e41e05-b1e2-48a9-a26c-20ca940e688a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI1OTE1NzgsIm5iZiI6MTcyMjU5MTI3OCwicGF0aCI6Ii8zMjM0MzkvMzU0NDk5ODQ3LTI4ZTQxZTA1LWIxZTItNDhhOS1hMjZjLTIwY2E5NDBlNjg4YS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODAyJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgwMlQwOTM0MzhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT00NDU4Zjc5MmVlYzVhNzJkZmRkM2Y3ZmY3ZWJjYmJmNGY5ODI0OGE1ZWM4ZmRhNmJkZGNiMDZmYzk4YzU2ZDAyJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.cHYHn1aK6dHlbI9_uWiRoqcNXp8nBPKqxLcsVHyeQhI">
   > As you can see the times come down from around 1.5s to around **0.5s**, a big improvement!
   > 
   > In fact, if you then cherry-pick the commit from my [follow-on PR with the object cache](https://github.com/apache/iceberg-rust/pull/512):
   > 
   > ```shell
   > git cherry-pick c696a3f9a7dacf822e6d5bc76f13d24e0b50ee31
   > ```
   > 
   > and re-run the performance tests again, you'll see something even more remarkable:
   > 
   > <img alt="image" width="578" src="https://private-user-images.githubusercontent.com/323439/354501480-12e91179-2ea3-4dad-9581-d7698b80b653.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI1OTE1NzgsIm5iZiI6MTcyMjU5MTI3OCwicGF0aCI6Ii8zMjM0MzkvMzU0NTAxNDgwLTEyZTkxMTc5LTJlYTMtNGRhZC05NTgxLWQ3Njk4YjgwYjY1My5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODAyJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgwMlQwOTM0MzhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1lYTE5YWY4ZGNjM2EzYTRhOGYzNzgyZTkyMDhkYTgwMDNiNGZiZDFlZDVhOTk1MDMzMWNhYzMwZTMyMTk5ODY1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.VZgK87tZQkAESZ7JeVZ8A9XAWXUe-6ICVkhSRqtTDD0">
   > Now that caching of the retrieval and parsing of the `Manifest` and `ManifestList`s are taking place, the first run in the performance test will take the same time as above but all of the subsequent runs that are averaged to get an average time for each test are much, much faster, reducing the average time from 500ms to about 1.5ms. 🤯
   
   Good work!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1639394777


##########
crates/iceberg/src/scan.rs:
##########
@@ -302,13 +262,147 @@ impl TableScan {
 
         arrow_reader_builder.build().read(self.plan_files().await?)
     }
+}
+
+#[derive(Debug)]
+struct ConcurrentFileScanStreamContext {
+    context: FileScanStreamContext,
+    sender: Sender<Result<FileScanTask>>,
+    manifest_evaluator_cache: ManifestEvaluatorCache,
+    partition_filter_cache: PartitionFilterCache,
+}
+
+impl ConcurrentFileScanStreamContext {
+    fn new(context: FileScanStreamContext, sender: Sender<Result<FileScanTask>>) -> Self {
+        ConcurrentFileScanStreamContext {
+            context,
+            sender,
+            manifest_evaluator_cache: ManifestEvaluatorCache::new(),
+            partition_filter_cache: PartitionFilterCache::new(),
+        }
+    }
+
+    async fn run(&mut self, manifest_list: ManifestList) -> Result<()> {
+        let file_io = self.context.file_io.clone();
+        let sender = self.sender.clone();
+
+        // This whole Vec-and-for-loop approach feels sub-optimally structured.
+        // I've tried structuring this in multiple ways but run into
+        // issues with ownership. Ideally I'd like to structure this
+        // with a functional programming approach: extracting
+        // sections 1, 2, and 3 out into different methods on Self,
+        // and then use some iterator combinators to orchestrate it all.
+        // Section 1 is pretty trivially refactorable into a static method
+        // that can be used in a closure that can be used with Iterator::filter.
+        // Similarly, section 3 seems easily factor-able into a method that can
+        // be passed into Iterator::map.
+        // Section 2 turns out trickier - we want to exit the entire `run` method early
+        // if the eval fails, and filter out any manifest_files from the iterator / stream
+        // if the eval succeeds but returns true. We bump into ownership issues due
+        // to needing to pass mut self as the caches need to be able to mutate.
+        // Section 3 runs into ownership issues when trying to refactor its closure to be
+        // a static or non-static method.
+
+        // 1
+        let filtered_manifest_files = manifest_list
+            .entries()
+            .iter()
+            .filter(Self::reject_unsupported_content_types);
+
+        // 2
+        let mut filtered_manifest_files2 = vec![];
+        for manifest_file in filtered_manifest_files {
+            if !self.apply_evaluator(manifest_file)? {
+                continue;
+            }
+
+            filtered_manifest_files2.push(manifest_file);
+        }
+
+        // 3
+        let filtered_manifest_files = filtered_manifest_files2
+            .into_iter()
+            .map(|manifest_file| Ok((manifest_file, file_io.clone(), sender.clone())));
+
+        futures::stream::iter(filtered_manifest_files)
+            .try_for_each_concurrent(
+                CONCURRENCY_LIMIT_MANIFEST_FILES,
+                Self::process_manifest_file,
+            )
+            .await
+    }
+
+    fn reject_unsupported_content_types(manifest_file: &&ManifestFile) -> bool {
+        SUPPORTED_MANIFEST_FILE_CONTENT_TYPES.contains(&manifest_file.content)
+    }
+
+    fn apply_evaluator(&mut self, manifest_file: &ManifestFile) -> Result<bool> {
+        let partition_spec_id = manifest_file.partition_spec_id;
+
+        let partition_filter = self
+            .partition_filter_cache
+            .get(partition_spec_id, &self.context)?;
+
+        if let Some(partition_filter) = partition_filter {
+            let manifest_evaluator = self
+                .manifest_evaluator_cache
+                .get(partition_spec_id, partition_filter);
+
+            if !manifest_evaluator.eval(manifest_file)? {
+                return Ok(false);
+            }
+        }
+
+        Ok(true)
+    }
+
+    async fn process_manifest_file(
+        manifest_and_file_io_and_sender: (&ManifestFile, FileIO, Sender<Result<FileScanTask>>),
+    ) -> Result<()> {
+        let (manifest_file, file_io, sender) = manifest_and_file_io_and_sender;
+
+        let manifest = manifest_file.load_manifest(&file_io).await?;
+
+        let manifest_entries = manifest
+            .entries()
+            .iter()
+            .filter(|x| x.is_alive())
+            .map(|manifest_entry| Ok((manifest_entry, sender.clone())));
+
+        futures::stream::iter(manifest_entries)
+            .try_for_each_concurrent(
+                CONCURRENCY_LIMIT_MANIFEST_ENTRIES,
+                Self::process_manifest_entry,
+            )
+            .await

Review Comment:
   Task spawning concurrency is limited by the `CONCURRENCY_LIMIT_MANIFEST_ENTRIES` constant, which I'm happy to adjust in this PR to suit



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "xxchan (via GitHub)" <gi...@apache.org>.
xxchan commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1950284548


##########
crates/iceberg/src/scan.rs:
##########
@@ -199,134 +219,119 @@ impl<'a> TableScanBuilder<'a> {
             field_ids.push(field_id);
         }
 
-        Ok(TableScan {
+        let snapshot_bound_predicate = if let Some(ref predicates) = self.filter {
+            Some(predicates.bind(schema.clone(), true)?)
+        } else {
+            None
+        };
+
+        let plan_context = PlanContext {
             snapshot,
-            file_io: self.table.file_io().clone(),
             table_metadata: self.table.metadata_ref(),
-            column_names: self.column_names,
-            field_ids,
-            bound_predicates,
-            schema,
-            batch_size: self.batch_size,
+            snapshot_schema: schema,
             case_sensitive: self.case_sensitive,
-            filter: self.filter.map(Arc::new),
+            predicate: self.filter.map(Arc::new),
+            snapshot_bound_predicate: snapshot_bound_predicate.map(Arc::new),

Review Comment:
   May I ask what does the "snapshot" mean here in `snapshot_bound_predicate`? :eyes: 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#issuecomment-2264717700

   Hey @ZENOTME! Sure. If you check out my `perf-suite` branch from [my other PR](https://github.com/apache/iceberg-rust/pull/497), this is branched off main and can be used to get a baseline of `plan_files` performance without the changes in this PR. You'll need to run `cargo install just` if you don't have `just` already installed, and then `just perf-run`. This should create the docker setup for the performance testing environment, download some test data fro the NYC Taxi dataset, create an Iceberg table inside the perf testing env, insert the test data into the table, and then run the perf tests.
   
   The perf tests themselves consist of four scenarios that each execute `plan_files` in different scenarios:
   * a file plan for a scan that reads **all** rows from **all** data files in the current snapshot (ie no filter);
   * a file plan for a scan that reads **some** rows from **all** data files in the current snapshot (`passenger_count` = 1);
   * a file plan for a scan that reads **all** rows from **one** data file in the current snapshot (filter: `tpep_pickup_datetime` = '2024-02-01')
   * a file plan for a scan that reads **some** rows from **one** data file in the current snapshot (filter: `tpep_pickup_datetime` = '2024-02-01' AND `passenger_count` = 1);
   
   The benchmarks get ran using criterion. Here's an example of the test output when I ran it just now on my M1 Pro Macbook:
   
   <img width="681" alt="image" src="https://github.com/user-attachments/assets/3d521b80-b3ab-4421-8889-e39cc1a0ec91">
   
   Now, if you cherry-pick the commit from this concurrent table scan branch into the `perf-suite` branch:
   
   ```bash
   git cherry-pick 8eef484094dbb7c55ac3181bbd552090308591c9
   ```
   
   And re-run the performance tests, you'll see something similar to the following:
   
   <img width="689" alt="image" src="https://github.com/user-attachments/assets/28e41e05-b1e2-48a9-a26c-20ca940e688a">
   
   As you can see the times come down from around 1.5s to around **0.5s**, a big improvement!
   
   In fact, if you then cherry-pick the commit from my [follow-on PR with the object cache](https://github.com/apache/iceberg-rust/pull/512):
   
   ```bash
   git cherry-pick c696a3f9a7dacf822e6d5bc76f13d24e0b50ee31
   ```
   
   
   and re-run the performance tests again, you'll see something even more remarkable:
   
   <img width="578" alt="image" src="https://github.com/user-attachments/assets/12e91179-2ea3-4dad-9581-d7698b80b653">
   
   Now that the retrieval and parsing of the `Manifest` and `ManifestList`s are taking place, the first run in the performance test will take the same time as above but all of the subsequent runs that are averaged to get an average time for each test are much, much faster, reducing the average time from 500ms to about 1.5ms. 🤯 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1679853016


##########
crates/iceberg/src/scan.rs:
##########
@@ -389,12 +333,158 @@ impl FileScanStreamContext {
             file_io,
             bound_filter,
             case_sensitive,
+            sender,
+            manifest_evaluator_cache: ManifestEvaluatorCache::new(),
+            partition_filter_cache: PartitionFilterCache::new(),
+            expression_evaluator_cache: Arc::new(Mutex::new(ExpressionEvaluatorCache::new())),
         })
     }
 
-    /// Returns a reference to the [`BoundPredicate`] filter.
-    fn bound_filter(&self) -> Option<&BoundPredicate> {
-        self.bound_filter.as_ref()
+    async fn run(&mut self, manifest_list: ManifestList) -> Result<()> {
+        let file_io = self.file_io.clone();
+        let sender = self.sender.clone();
+
+        // This whole Vec-and-for-loop approach feels sub-optimally structured.
+        // I've tried structuring this in multiple ways but run into
+        // issues with ownership. Ideally I'd like to structure this
+        // with a functional programming approach: extracting
+        // sections 1, 2, and 3 out into different methods on Self,
+        // and then use some iterator combinators to orchestrate it all.
+        // Section 1 is pretty trivially refactorable into a static method
+        // that can be used in a closure that can be used with Iterator::filter.
+        // Similarly, section 3 seems easily factor-able into a method that can
+        // be passed into Iterator::map.
+        // Section 2 turns out trickier - we want to exit the entire `run` method early
+        // if the eval fails, and filter out any manifest_files from the iterator / stream
+        // if the eval succeeds but returns true. We bump into ownership issues due
+        // to needing to pass mut self as the caches need to be able to mutate.
+        // Section 3 runs into ownership issues when trying to refactor its closure to be
+        // a static or non-static method.
+
+        // 1
+        let filtered_manifest_files = manifest_list
+            .entries()
+            .iter()
+            .filter(Self::reject_unsupported_content_types);
+
+        // 2
+        let mut filtered_manifest_files2 = vec![];
+        for manifest_file in filtered_manifest_files {
+            if !self.apply_evaluator(manifest_file)? {
+                continue;
+            }
+
+            filtered_manifest_files2.push(manifest_file);
+        }
+
+        // 3
+        let filtered_manifest_files = filtered_manifest_files2.into_iter().map(|manifest_file| {
+            Ok(ManifestFileProcessingContext {
+                manifest_file,
+                file_io: file_io.clone(),
+                sender: sender.clone(),
+                filter: &self.bound_filter,
+                expression_evaluator_cache: self.expression_evaluator_cache.clone(),
+            })
+        });
+
+        futures::stream::iter(filtered_manifest_files)
+            .try_for_each_concurrent(

Review Comment:
   Thanks. I'm not happy with it now that you pointed out the issue with `try_for_each_concurrent`. I'm in the middle of refactoring it and should have something quite soon.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1697764804


##########
crates/iceberg/src/spec/manifest_list.rs:
##########
@@ -79,6 +79,11 @@ impl ManifestList {
     pub fn entries(&self) -> &[ManifestFile] {
         &self.entries
     }
+
+    /// Take ownership of the entries in the manifest list, consuming it
+    pub fn consume_entries(self) -> Box<dyn Iterator<Item = ManifestFile>> {

Review Comment:
   Done, thanks!



##########
crates/iceberg/src/scan.rs:
##########
@@ -338,157 +303,463 @@ impl TableScan {
         arrow_reader_builder.build().read(self.plan_files().await?)
     }
 
-    /// Checks whether the [`ManifestContentType`] is `Data` or not.
-    fn content_type_is_data(entry: &ManifestFile) -> bool {
-        if let ManifestContentType::Data = entry.content {
-            return true;
-        }
-        false
-    }
-
     /// Returns a reference to the column names of the table scan.
     pub fn column_names(&self) -> &[String] {
         &self.column_names
     }
+    /// Returns a reference to the snapshot of the table scan.
+    pub fn snapshot(&self) -> &SnapshotRef {
+        &self.plan_context.snapshot
+    }
+
+    async fn process_manifest_entry(
+        manifest_entry_context: ManifestEntryContext,
+        mut file_scan_task_tx: Sender<Result<FileScanTask>>,
+    ) -> Result<()> {
+        // skip processing this manifest entry if it has been marked as deleted
+        if !manifest_entry_context.manifest_entry.is_alive() {
+            return Ok(());
+        }
+
+        // abort the plan if we encounter a manifest entry whose data file's
+        // content type is currently unsupported
+        if manifest_entry_context.manifest_entry.content_type() != DataContentType::Data {
+            return Err(Error::new(
+                ErrorKind::FeatureUnsupported,
+                "Only Data files currently supported",
+            ));
+        }
+
+        if let Some(ref snapshot_bound_predicate) = manifest_entry_context.snapshot_bound_predicate
+        {
+            let Some(ref partition_bound_predicate) =
+                manifest_entry_context.partition_bound_predicate
+            else {
+                panic!("partition_bound_predicate expected to always be present if snapshot_bound_predicate is present");

Review Comment:
   Good suggestion - done!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1682309951


##########
crates/iceberg/src/scan.rs:
##########
@@ -389,12 +333,158 @@ impl FileScanStreamContext {
             file_io,
             bound_filter,
             case_sensitive,
+            sender,
+            manifest_evaluator_cache: ManifestEvaluatorCache::new(),
+            partition_filter_cache: PartitionFilterCache::new(),
+            expression_evaluator_cache: Arc::new(Mutex::new(ExpressionEvaluatorCache::new())),
         })
     }
 
-    /// Returns a reference to the [`BoundPredicate`] filter.
-    fn bound_filter(&self) -> Option<&BoundPredicate> {
-        self.bound_filter.as_ref()
+    async fn run(&mut self, manifest_list: ManifestList) -> Result<()> {
+        let file_io = self.file_io.clone();
+        let sender = self.sender.clone();
+
+        // This whole Vec-and-for-loop approach feels sub-optimally structured.
+        // I've tried structuring this in multiple ways but run into
+        // issues with ownership. Ideally I'd like to structure this
+        // with a functional programming approach: extracting
+        // sections 1, 2, and 3 out into different methods on Self,
+        // and then use some iterator combinators to orchestrate it all.
+        // Section 1 is pretty trivially refactorable into a static method
+        // that can be used in a closure that can be used with Iterator::filter.
+        // Similarly, section 3 seems easily factor-able into a method that can
+        // be passed into Iterator::map.
+        // Section 2 turns out trickier - we want to exit the entire `run` method early
+        // if the eval fails, and filter out any manifest_files from the iterator / stream
+        // if the eval succeeds but returns true. We bump into ownership issues due
+        // to needing to pass mut self as the caches need to be able to mutate.
+        // Section 3 runs into ownership issues when trying to refactor its closure to be
+        // a static or non-static method.
+
+        // 1
+        let filtered_manifest_files = manifest_list
+            .entries()
+            .iter()
+            .filter(Self::reject_unsupported_content_types);
+
+        // 2
+        let mut filtered_manifest_files2 = vec![];
+        for manifest_file in filtered_manifest_files {
+            if !self.apply_evaluator(manifest_file)? {
+                continue;
+            }
+
+            filtered_manifest_files2.push(manifest_file);
+        }
+
+        // 3
+        let filtered_manifest_files = filtered_manifest_files2.into_iter().map(|manifest_file| {
+            Ok(ManifestFileProcessingContext {
+                manifest_file,
+                file_io: file_io.clone(),
+                sender: sender.clone(),
+                filter: &self.bound_filter,
+                expression_evaluator_cache: self.expression_evaluator_cache.clone(),
+            })
+        });
+
+        futures::stream::iter(filtered_manifest_files)
+            .try_for_each_concurrent(

Review Comment:
   Hey @liurenjie1024 - I think I'm happy with this now. Let me know what you think.
   
   There is one issue remaining. It seems that our `crate::runtime::spawn` does not behave exactly the same as `tokio::spawn` for tasks that are passed an async function that return a `Result`? I could not get this to work with `crate::runtime::spawn` but it does with `tokio::spawn`. @odysa could you help?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1639361290


##########
crates/iceberg/src/scan.rs:
##########
@@ -302,13 +262,147 @@ impl TableScan {
 
         arrow_reader_builder.build().read(self.plan_files().await?)
     }
+}
+
+#[derive(Debug)]
+struct ConcurrentFileScanStreamContext {

Review Comment:
   I've refactored these into a single context. Now that we are using the `ExpressionEvaluator` though, I ended up creating another context object that gets passed to the manifest file closure. This was just a tuple before but clippy complained when I added one more item to it :-)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1690211456


##########
crates/iceberg/src/scan.rs:
##########
@@ -197,134 +193,103 @@ impl<'a> TableScanBuilder<'a> {
             field_ids.push(field_id);
         }
 
-        Ok(TableScan {
+        let snapshot_bound_predicate = if let Some(ref predicates) = self.filter {
+            Some(predicates.bind(schema.clone(), true)?)
+        } else {
+            None
+        };
+
+        let plan_context = PlanContext {
             snapshot,
-            file_io: self.table.file_io().clone(),
             table_metadata: self.table.metadata_ref(),
-            column_names: self.column_names,
-            field_ids,
-            bound_predicates,
-            schema,
-            batch_size: self.batch_size,
+            snapshot_schema: schema,
             case_sensitive: self.case_sensitive,
-            filter: self.filter.map(Arc::new),
+            predicate: self.filter.map(Arc::new),
+            snapshot_bound_predicate: snapshot_bound_predicate.map(Arc::new),
+            file_io: self.table.file_io().clone(),
+            field_ids: Arc::new(field_ids),
+            partition_filter_cache: Arc::new(PartitionFilterCache::new()),
+            manifest_evaluator_cache: Arc::new(ManifestEvaluatorCache::new()),
+            expression_evaluator_cache: Arc::new(ExpressionEvaluatorCache::new()),
+        };
+
+        Ok(TableScan {
+            batch_size: self.batch_size,
+            column_names: self.column_names,
+            file_io: self.table.file_io().clone(),
+            plan_context,
         })
     }
 }
 
 /// Table scan.
 #[derive(Debug)]
 pub struct TableScan {
-    snapshot: SnapshotRef,
-    table_metadata: TableMetadataRef,
+    plan_context: PlanContext,
+    batch_size: Option<usize>,
     file_io: FileIO,
     column_names: Vec<String>,
-    field_ids: Vec<i32>,
-    bound_predicates: Option<BoundPredicate>,
-    schema: SchemaRef,
-    batch_size: Option<usize>,
+}
+
+/// PlanContext wraps a [`SnapshotRef`] alongside all the other
+/// objects that are required to perform a scan file plan.
+#[derive(Debug)]
+struct PlanContext {
+    snapshot: SnapshotRef,
+
+    table_metadata: TableMetadataRef,
+    snapshot_schema: SchemaRef,
     case_sensitive: bool,
-    filter: Option<Arc<Predicate>>,
+    predicate: Option<Arc<Predicate>>,
+    snapshot_bound_predicate: Option<Arc<BoundPredicate>>,
+    file_io: FileIO,
+    field_ids: Arc<Vec<i32>>,
+
+    partition_filter_cache: Arc<PartitionFilterCache>,
+    manifest_evaluator_cache: Arc<ManifestEvaluatorCache>,
+    expression_evaluator_cache: Arc<ExpressionEvaluatorCache>,
 }
 
 impl TableScan {
     /// Returns a stream of [`FileScanTask`]s.
     pub async fn plan_files(&self) -> Result<FileScanTaskStream> {
-        let context = FileScanStreamContext::new(
-            self.schema.clone(),
-            self.snapshot.clone(),
-            self.table_metadata.clone(),
-            self.file_io.clone(),
-            self.filter.clone(),
-            self.case_sensitive,
-        )?;
-
-        let mut partition_filter_cache = PartitionFilterCache::new();
-        let mut manifest_evaluator_cache = ManifestEvaluatorCache::new();
-        let mut expression_evaluator_cache = ExpressionEvaluatorCache::new();
-
-        let field_ids = self.field_ids.clone();
-        let bound_predicates = self.bound_predicates.clone();
-
-        Ok(try_stream! {
-            let manifest_list = context
-                .snapshot
-                .load_manifest_list(&context.file_io, &context.table_metadata)
-                .await?;
-
-            for entry in manifest_list.entries() {
-                if !Self::content_type_is_data(entry) {
-                    continue;
-                }
-
-                let partition_spec_id = entry.partition_spec_id;
-
-                let partition_filter = partition_filter_cache.get(
-                    partition_spec_id,
-                    &context,
-                )?;
-
-                if let Some(partition_filter) = partition_filter {
-                    let manifest_evaluator = manifest_evaluator_cache.get(
-                        partition_spec_id,
-                        partition_filter,
-                    );
-
-                    if !manifest_evaluator.eval(entry)? {
-                        continue;
-                    }
-                }
+        // used to stream ManifestEntryContexts between stages of the file plan operation
+        let (manifest_entry_ctx_tx, manifest_entry_ctx_rx) =
+            channel(CONCURRENCY_LIMIT_MANIFEST_FILES);
+        // used to stream the results back to the caller
+        let (file_scan_task_tx, file_scan_task_rx) = channel(CONCURRENCY_LIMIT_MANIFEST_ENTRIES);
+
+        let manifest_list = self.plan_context.get_manifest_list().await?;
+
+        // get the [`ManifestFile`]s from the [`ManifestList`], filtering out any
+        // whose content type is not Data or whose partitions cannot match this
+        // scan's filter
+        let manifest_file_contexts = self
+            .plan_context
+            .build_manifest_file_contexts(manifest_list, manifest_entry_ctx_tx)?;
+
+        // Concurrently load all [`Manifest`]s and stream their [`ManifestEntry`]s
+        futures::stream::iter(manifest_file_contexts)
+            .try_for_each_concurrent(CONCURRENCY_LIMIT_MANIFEST_FILES, |ctx| async move {
+                ctx.fetch_manifest_and_stream_manifest_entries().await
+            })
+            .await?;
+
+        // Process the [`ManifestEntry`] stream in parallel
+        manifest_entry_ctx_rx
+            .map(|me_ctx| Ok((me_ctx, file_scan_task_tx.clone())))
+            .try_for_each_concurrent(
+                CONCURRENCY_LIMIT_MANIFEST_ENTRIES,
+                |(manifest_entry_context, tx)| async move {
+                    crate::runtime::spawn(async move {
+                        Self::process_manifest_entry(manifest_entry_context, tx).await

Review Comment:
   In terms of having non-concurrent as the default option, my preference would be for the library to be able to choose the optimally performant (probably the number of cores?) as the default rather than no concurrncy. No concurrncy feels like a more advanced, non-default choice. @liurenjie1024 what are your thoughts? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1690197679


##########
crates/iceberg/src/scan.rs:
##########
@@ -197,134 +193,103 @@ impl<'a> TableScanBuilder<'a> {
             field_ids.push(field_id);
         }
 
-        Ok(TableScan {
+        let snapshot_bound_predicate = if let Some(ref predicates) = self.filter {
+            Some(predicates.bind(schema.clone(), true)?)
+        } else {
+            None
+        };
+
+        let plan_context = PlanContext {
             snapshot,
-            file_io: self.table.file_io().clone(),
             table_metadata: self.table.metadata_ref(),
-            column_names: self.column_names,
-            field_ids,
-            bound_predicates,
-            schema,
-            batch_size: self.batch_size,
+            snapshot_schema: schema,
             case_sensitive: self.case_sensitive,
-            filter: self.filter.map(Arc::new),
+            predicate: self.filter.map(Arc::new),
+            snapshot_bound_predicate: snapshot_bound_predicate.map(Arc::new),
+            file_io: self.table.file_io().clone(),
+            field_ids: Arc::new(field_ids),
+            partition_filter_cache: Arc::new(PartitionFilterCache::new()),
+            manifest_evaluator_cache: Arc::new(ManifestEvaluatorCache::new()),
+            expression_evaluator_cache: Arc::new(ExpressionEvaluatorCache::new()),
+        };
+
+        Ok(TableScan {
+            batch_size: self.batch_size,
+            column_names: self.column_names,
+            file_io: self.table.file_io().clone(),
+            plan_context,
         })
     }
 }
 
 /// Table scan.
 #[derive(Debug)]
 pub struct TableScan {
-    snapshot: SnapshotRef,
-    table_metadata: TableMetadataRef,
+    plan_context: PlanContext,
+    batch_size: Option<usize>,
     file_io: FileIO,
     column_names: Vec<String>,
-    field_ids: Vec<i32>,
-    bound_predicates: Option<BoundPredicate>,
-    schema: SchemaRef,
-    batch_size: Option<usize>,
+}
+
+/// PlanContext wraps a [`SnapshotRef`] alongside all the other
+/// objects that are required to perform a scan file plan.
+#[derive(Debug)]
+struct PlanContext {
+    snapshot: SnapshotRef,
+
+    table_metadata: TableMetadataRef,
+    snapshot_schema: SchemaRef,
     case_sensitive: bool,
-    filter: Option<Arc<Predicate>>,
+    predicate: Option<Arc<Predicate>>,
+    snapshot_bound_predicate: Option<Arc<BoundPredicate>>,
+    file_io: FileIO,
+    field_ids: Arc<Vec<i32>>,
+
+    partition_filter_cache: Arc<PartitionFilterCache>,
+    manifest_evaluator_cache: Arc<ManifestEvaluatorCache>,
+    expression_evaluator_cache: Arc<ExpressionEvaluatorCache>,
 }
 
 impl TableScan {
     /// Returns a stream of [`FileScanTask`]s.
     pub async fn plan_files(&self) -> Result<FileScanTaskStream> {
-        let context = FileScanStreamContext::new(
-            self.schema.clone(),
-            self.snapshot.clone(),
-            self.table_metadata.clone(),
-            self.file_io.clone(),
-            self.filter.clone(),
-            self.case_sensitive,
-        )?;
-
-        let mut partition_filter_cache = PartitionFilterCache::new();
-        let mut manifest_evaluator_cache = ManifestEvaluatorCache::new();
-        let mut expression_evaluator_cache = ExpressionEvaluatorCache::new();
-
-        let field_ids = self.field_ids.clone();
-        let bound_predicates = self.bound_predicates.clone();
-
-        Ok(try_stream! {
-            let manifest_list = context
-                .snapshot
-                .load_manifest_list(&context.file_io, &context.table_metadata)
-                .await?;
-
-            for entry in manifest_list.entries() {
-                if !Self::content_type_is_data(entry) {
-                    continue;
-                }
-
-                let partition_spec_id = entry.partition_spec_id;
-
-                let partition_filter = partition_filter_cache.get(
-                    partition_spec_id,
-                    &context,
-                )?;
-
-                if let Some(partition_filter) = partition_filter {
-                    let manifest_evaluator = manifest_evaluator_cache.get(
-                        partition_spec_id,
-                        partition_filter,
-                    );
-
-                    if !manifest_evaluator.eval(entry)? {
-                        continue;
-                    }
-                }
+        // used to stream ManifestEntryContexts between stages of the file plan operation
+        let (manifest_entry_ctx_tx, manifest_entry_ctx_rx) =
+            channel(CONCURRENCY_LIMIT_MANIFEST_FILES);
+        // used to stream the results back to the caller
+        let (file_scan_task_tx, file_scan_task_rx) = channel(CONCURRENCY_LIMIT_MANIFEST_ENTRIES);
+
+        let manifest_list = self.plan_context.get_manifest_list().await?;
+
+        // get the [`ManifestFile`]s from the [`ManifestList`], filtering out any
+        // whose content type is not Data or whose partitions cannot match this
+        // scan's filter
+        let manifest_file_contexts = self
+            .plan_context
+            .build_manifest_file_contexts(manifest_list, manifest_entry_ctx_tx)?;
+
+        // Concurrently load all [`Manifest`]s and stream their [`ManifestEntry`]s
+        futures::stream::iter(manifest_file_contexts)
+            .try_for_each_concurrent(CONCURRENCY_LIMIT_MANIFEST_FILES, |ctx| async move {
+                ctx.fetch_manifest_and_stream_manifest_entries().await
+            })
+            .await?;
+
+        // Process the [`ManifestEntry`] stream in parallel
+        manifest_entry_ctx_rx
+            .map(|me_ctx| Ok((me_ctx, file_scan_task_tx.clone())))
+            .try_for_each_concurrent(
+                CONCURRENCY_LIMIT_MANIFEST_ENTRIES,
+                |(manifest_entry_context, tx)| async move {
+                    crate::runtime::spawn(async move {
+                        Self::process_manifest_entry(manifest_entry_context, tx).await

Review Comment:
   If I make the concurrncy limits configurable in the builder as @liurenjie1024 suggested above, then we could have a `with_no_concurrency` method on the builder. Would this be acceptable? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Concurrent table scans [iceberg-rust]

Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on code in PR #373:
URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1690208650


##########
crates/iceberg/src/scan.rs:
##########
@@ -197,134 +193,103 @@ impl<'a> TableScanBuilder<'a> {
             field_ids.push(field_id);
         }
 
-        Ok(TableScan {
+        let snapshot_bound_predicate = if let Some(ref predicates) = self.filter {
+            Some(predicates.bind(schema.clone(), true)?)
+        } else {
+            None
+        };
+
+        let plan_context = PlanContext {
             snapshot,
-            file_io: self.table.file_io().clone(),
             table_metadata: self.table.metadata_ref(),
-            column_names: self.column_names,
-            field_ids,
-            bound_predicates,
-            schema,
-            batch_size: self.batch_size,
+            snapshot_schema: schema,
             case_sensitive: self.case_sensitive,
-            filter: self.filter.map(Arc::new),
+            predicate: self.filter.map(Arc::new),
+            snapshot_bound_predicate: snapshot_bound_predicate.map(Arc::new),
+            file_io: self.table.file_io().clone(),
+            field_ids: Arc::new(field_ids),
+            partition_filter_cache: Arc::new(PartitionFilterCache::new()),
+            manifest_evaluator_cache: Arc::new(ManifestEvaluatorCache::new()),
+            expression_evaluator_cache: Arc::new(ExpressionEvaluatorCache::new()),
+        };
+
+        Ok(TableScan {
+            batch_size: self.batch_size,
+            column_names: self.column_names,
+            file_io: self.table.file_io().clone(),
+            plan_context,
         })
     }
 }
 
 /// Table scan.
 #[derive(Debug)]
 pub struct TableScan {
-    snapshot: SnapshotRef,
-    table_metadata: TableMetadataRef,
+    plan_context: PlanContext,
+    batch_size: Option<usize>,
     file_io: FileIO,
     column_names: Vec<String>,
-    field_ids: Vec<i32>,
-    bound_predicates: Option<BoundPredicate>,
-    schema: SchemaRef,
-    batch_size: Option<usize>,
+}
+
+/// PlanContext wraps a [`SnapshotRef`] alongside all the other
+/// objects that are required to perform a scan file plan.
+#[derive(Debug)]
+struct PlanContext {
+    snapshot: SnapshotRef,
+
+    table_metadata: TableMetadataRef,
+    snapshot_schema: SchemaRef,
     case_sensitive: bool,
-    filter: Option<Arc<Predicate>>,
+    predicate: Option<Arc<Predicate>>,
+    snapshot_bound_predicate: Option<Arc<BoundPredicate>>,
+    file_io: FileIO,
+    field_ids: Arc<Vec<i32>>,
+
+    partition_filter_cache: Arc<PartitionFilterCache>,
+    manifest_evaluator_cache: Arc<ManifestEvaluatorCache>,
+    expression_evaluator_cache: Arc<ExpressionEvaluatorCache>,
 }
 
 impl TableScan {
     /// Returns a stream of [`FileScanTask`]s.
     pub async fn plan_files(&self) -> Result<FileScanTaskStream> {
-        let context = FileScanStreamContext::new(
-            self.schema.clone(),
-            self.snapshot.clone(),
-            self.table_metadata.clone(),
-            self.file_io.clone(),
-            self.filter.clone(),
-            self.case_sensitive,
-        )?;
-
-        let mut partition_filter_cache = PartitionFilterCache::new();
-        let mut manifest_evaluator_cache = ManifestEvaluatorCache::new();
-        let mut expression_evaluator_cache = ExpressionEvaluatorCache::new();
-
-        let field_ids = self.field_ids.clone();
-        let bound_predicates = self.bound_predicates.clone();
-
-        Ok(try_stream! {
-            let manifest_list = context
-                .snapshot
-                .load_manifest_list(&context.file_io, &context.table_metadata)
-                .await?;
-
-            for entry in manifest_list.entries() {
-                if !Self::content_type_is_data(entry) {
-                    continue;
-                }
-
-                let partition_spec_id = entry.partition_spec_id;
-
-                let partition_filter = partition_filter_cache.get(
-                    partition_spec_id,
-                    &context,
-                )?;
-
-                if let Some(partition_filter) = partition_filter {
-                    let manifest_evaluator = manifest_evaluator_cache.get(
-                        partition_spec_id,
-                        partition_filter,
-                    );
-
-                    if !manifest_evaluator.eval(entry)? {
-                        continue;
-                    }
-                }
+        // used to stream ManifestEntryContexts between stages of the file plan operation
+        let (manifest_entry_ctx_tx, manifest_entry_ctx_rx) =
+            channel(CONCURRENCY_LIMIT_MANIFEST_FILES);

Review Comment:
   Absolutely, I'd considered doing that, I think it's a good idea. Will add



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org