You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "itsjunetime (via GitHub)" <gi...@apache.org> on 2024/10/21 20:35:08 UTC

[PR] Add `ParquetObjectReader::with_runtime` [arrow-rs]

itsjunetime opened a new pull request, #6612:
URL: https://github.com/apache/arrow-rs/pull/6612

   # Which issue does this PR close?
   
   Closes #6248
   
   # What changes are included in this PR?
   
   This PR works on top of #6249 to add a test for the new `with_runtime` fn, as well as changing the signature of the `spawn` function slightly to avoid an extra re-boxing when a runtime is set.
   
   This also fixes a few things that `clippy` was complaining about.
   
   # Rationale for this change
   
   See #6248 for the API addition.
   
   With regard to the test, I felt like this is really the only test we'd want for this feature - we just want to make sure that the runtime is actually being used by `ParquetObjectReader`. We can't make any guarantees about how it actually performs or would work if there's another runtime being used for CPU-bound operations, so all we really want to test is if it is used.
   
   # Are there any user-facing changes?
   
   No
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `ParquetObjectReader::with_runtime` [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #6612:
URL: https://github.com/apache/arrow-rs/pull/6612#discussion_r1809509415


##########
parquet/src/arrow/async_reader/store.rs:
##########
@@ -180,10 +224,39 @@ mod tests {
                 let err = e.to_string();
                 assert!(
                     err.contains("not found: No such file or directory (os error 2)"),
-                    "{}",
-                    err
+                    "{err}",
                 );
             }
         }
     }
+
+    #[tokio::test]
+    // We need to mark this with the `target_has_atomic` because the spawned_tasks_count() fn is
+    // only available for that cfg
+    #[cfg(all(target_has_atomic = "64", tokio_unstable))]

Review Comment:
   I wonder if we could instead create a runtime with IO / blocking threads disabled and use that to determine that the IO was spawned to a different runtime?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `ParquetObjectReader::with_runtime` [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6612:
URL: https://github.com/apache/arrow-rs/pull/6612#discussion_r1821354383


##########
parquet/src/arrow/async_reader/store.rs:
##########
@@ -180,10 +231,69 @@ mod tests {
                 let err = e.to_string();
                 assert!(
                     err.contains("not found: No such file or directory (os error 2)"),
-                    "{}",
-                    err
+                    "{err}",
                 );
             }
         }
     }
+
+    #[tokio::test]
+    // We need to mark this with the `target_has_atomic` because the spawned_tasks_count() fn is
+    // only available for that cfg
+    async fn test_runtime_is_used() {
+        let num_actions = Arc::new(AtomicUsize::new(0));
+
+        let (a1, a2) = (num_actions.clone(), num_actions.clone());
+        let rt = tokio::runtime::Builder::new_multi_thread()
+            .on_thread_park(move || {

Review Comment:
   Yeah, my alternate was https://github.com/apache/arrow-rs/pull/6612#discussion_r1818853937 which may be too big of a 🔨 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `ParquetObjectReader::with_runtime` [arrow-rs]

Posted by "itsjunetime (via GitHub)" <gi...@apache.org>.
itsjunetime commented on code in PR #6612:
URL: https://github.com/apache/arrow-rs/pull/6612#discussion_r1821265417


##########
parquet/src/arrow/async_reader/store.rs:
##########
@@ -180,10 +231,69 @@ mod tests {
                 let err = e.to_string();
                 assert!(
                     err.contains("not found: No such file or directory (os error 2)"),
-                    "{}",
-                    err
+                    "{err}",
                 );
             }
         }
     }
+
+    #[tokio::test]
+    // We need to mark this with the `target_has_atomic` because the spawned_tasks_count() fn is
+    // only available for that cfg
+    async fn test_runtime_is_used() {
+        let num_actions = Arc::new(AtomicUsize::new(0));
+
+        let (a1, a2) = (num_actions.clone(), num_actions.clone());
+        let rt = tokio::runtime::Builder::new_multi_thread()
+            .on_thread_park(move || {

Review Comment:
   I think all this assumes is that there are wait points where threads get parked to wait for IO and that this runtime's threads aren't saturated. I assume that if they were saturated with tasks, the threads would never be parked as they would be continually polling other tasks, but I think it's safe to assume that if a runtime is only working on one task, and that task has to do IO, the main thread of that runtime is going to be parked at some point to wait for IO.
   
   It's a janky test, I just couldn't think of a better way to determine that the runtime was definitely being used when we used `ParquetObjectReader` as an `AsyncFileReader` without assuming a lot more about how the `spawn` function should work.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `ParquetObjectReader::with_runtime` [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #6612:
URL: https://github.com/apache/arrow-rs/pull/6612#discussion_r1822415739


##########
parquet/src/arrow/async_reader/store.rs:
##########
@@ -180,10 +237,86 @@ mod tests {
                 let err = e.to_string();
                 assert!(
                     err.contains("not found: No such file or directory (os error 2)"),
-                    "{}",
-                    err
+                    "{err}",
                 );
             }
         }
     }
+
+    #[tokio::test]
+    async fn test_runtime_is_used() {
+        let num_actions = Arc::new(AtomicUsize::new(0));
+
+        let (a1, a2) = (num_actions.clone(), num_actions.clone());
+        let rt = tokio::runtime::Builder::new_multi_thread()
+            .on_thread_park(move || {
+                a1.fetch_add(1, Ordering::Relaxed);
+            })
+            .on_thread_unpark(move || {
+                a2.fetch_add(1, Ordering::Relaxed);
+            })
+            .build()
+            .unwrap();
+
+        let (meta, store) = get_meta_store().await;
+
+        let initial_actions = num_actions.load(Ordering::Relaxed);
+
+        let reader = ParquetObjectReader::new(store, meta).with_runtime(rt.handle().clone());
+
+        let builder = ParquetRecordBatchStreamBuilder::new(reader).await.unwrap();
+        let batches: Vec<_> = builder.build().unwrap().try_collect().await.unwrap();
+
+        // Just copied these assert_eqs from the `test_simple` above
+        assert_eq!(batches.len(), 1);
+        assert_eq!(batches[0].num_rows(), 8);
+
+        assert!(num_actions.load(Ordering::Relaxed) - initial_actions > 0);
+
+        // Runtimes have to be dropped in blocking contexts, so we need to move this one to a new
+        // blocking thread to drop it.
+        tokio::runtime::Handle::current().spawn_blocking(move || drop(rt));
+    }
+
+    /// Unit test that `ParquetObjectReader::spawn`spawns on the provided runtime
+    #[tokio::test]
+    async fn test_runtime_thread_id_different() {
+        let rt = tokio::runtime::Builder::new_multi_thread()
+            .worker_threads(1)
+            .build()
+            .unwrap();
+
+        let (meta, store) = get_meta_store().await;
+
+        let reader = ParquetObjectReader::new(store, meta).with_runtime(rt.handle().clone());
+
+        let current_id = std::thread::current().id();
+
+        let other_id = reader
+            .spawn(|_, _| async move { Ok::<_, Infallible>(std::thread::current().id()) }.boxed())

Review Comment:
   ```suggestion
               .spawn(|_, _| async move { Ok::<_, ParquetError>(std::thread::current().id()) }.boxed())
   ```
   
   Would remove the need for the `std::convert::Infallible` conversion



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `ParquetObjectReader::with_runtime` [arrow-rs]

Posted by "itsjunetime (via GitHub)" <gi...@apache.org>.
itsjunetime commented on code in PR #6612:
URL: https://github.com/apache/arrow-rs/pull/6612#discussion_r1815630967


##########
parquet/src/arrow/async_reader/store.rs:
##########
@@ -180,10 +224,39 @@ mod tests {
                 let err = e.to_string();
                 assert!(
                     err.contains("not found: No such file or directory (os error 2)"),
-                    "{}",
-                    err
+                    "{err}",
                 );
             }
         }
     }
+
+    #[tokio::test]
+    // We need to mark this with the `target_has_atomic` because the spawned_tasks_count() fn is
+    // only available for that cfg
+    #[cfg(all(target_has_atomic = "64", tokio_unstable))]

Review Comment:
   I don't think that would work. I'm not certain why, but `ParquetObjectReader` seems to work fine regardless of whether or not IO is 'enabled' on its runtime. I was able to change the tests so they don't rely on `tokio_unstable` anymore and (I think) still show what we want them to show, so I'll push that in a minute.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `ParquetObjectReader::with_runtime` [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6612:
URL: https://github.com/apache/arrow-rs/pull/6612#issuecomment-2452962890

   Thanks again @itsjunetime and @tustvold 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `ParquetObjectReader::with_runtime` [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6612:
URL: https://github.com/apache/arrow-rs/pull/6612#discussion_r1826547937


##########
parquet/src/arrow/async_reader/store.rs:
##########
@@ -180,10 +237,86 @@ mod tests {
                 let err = e.to_string();
                 assert!(
                     err.contains("not found: No such file or directory (os error 2)"),
-                    "{}",
-                    err
+                    "{err}",
                 );
             }
         }
     }
+
+    #[tokio::test]
+    async fn test_runtime_is_used() {
+        let num_actions = Arc::new(AtomicUsize::new(0));
+
+        let (a1, a2) = (num_actions.clone(), num_actions.clone());
+        let rt = tokio::runtime::Builder::new_multi_thread()
+            .on_thread_park(move || {
+                a1.fetch_add(1, Ordering::Relaxed);
+            })
+            .on_thread_unpark(move || {
+                a2.fetch_add(1, Ordering::Relaxed);
+            })
+            .build()
+            .unwrap();
+
+        let (meta, store) = get_meta_store().await;
+
+        let initial_actions = num_actions.load(Ordering::Relaxed);
+
+        let reader = ParquetObjectReader::new(store, meta).with_runtime(rt.handle().clone());
+
+        let builder = ParquetRecordBatchStreamBuilder::new(reader).await.unwrap();
+        let batches: Vec<_> = builder.build().unwrap().try_collect().await.unwrap();
+
+        // Just copied these assert_eqs from the `test_simple` above
+        assert_eq!(batches.len(), 1);
+        assert_eq!(batches[0].num_rows(), 8);
+
+        assert!(num_actions.load(Ordering::Relaxed) - initial_actions > 0);
+
+        // Runtimes have to be dropped in blocking contexts, so we need to move this one to a new
+        // blocking thread to drop it.
+        tokio::runtime::Handle::current().spawn_blocking(move || drop(rt));
+    }
+
+    /// Unit test that `ParquetObjectReader::spawn`spawns on the provided runtime
+    #[tokio::test]
+    async fn test_runtime_thread_id_different() {
+        let rt = tokio::runtime::Builder::new_multi_thread()
+            .worker_threads(1)
+            .build()
+            .unwrap();
+
+        let (meta, store) = get_meta_store().await;
+
+        let reader = ParquetObjectReader::new(store, meta).with_runtime(rt.handle().clone());
+
+        let current_id = std::thread::current().id();
+
+        let other_id = reader
+            .spawn(|_, _| async move { Ok::<_, Infallible>(std::thread::current().id()) }.boxed())

Review Comment:
   I had this repo checked out and in the editor, so I just made this change to accelerate getting this PR in in 8d24cd79cb
   
   It results in a nice simplification



##########
parquet/src/errors.rs:
##########
@@ -107,6 +107,13 @@ impl From<str::Utf8Error> for ParquetError {
     }
 }
 
+#[cfg(test)]
+impl From<std::convert::Infallible> for ParquetError {

Review Comment:
   Removed in 8d24cd79cb



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `ParquetObjectReader::with_runtime` [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #6612:
URL: https://github.com/apache/arrow-rs/pull/6612#discussion_r1822413359


##########
parquet/src/errors.rs:
##########
@@ -107,6 +107,13 @@ impl From<str::Utf8Error> for ParquetError {
     }
 }
 
+#[cfg(test)]
+impl From<std::convert::Infallible> for ParquetError {

Review Comment:
   > aka to transform a Result<.., Infallible> to Result<.., Error> type expected by an API)
   
   Right but this is a little funky, because it then makes code look more fallible than it is. Often you can use an infallible version of the API, i.e. `into()` instead of `try_into()`, but sometimes you do have to either `unwrap()` or `let _ = ...`
   
   FWIW Rust 1.82 gives us a very nice way to handle this, but I'm not sure whether our MSRV policy covers tests.
   
   ```
   let Ok(value) = expression();
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `ParquetObjectReader::with_runtime` [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #6612:
URL: https://github.com/apache/arrow-rs/pull/6612


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `ParquetObjectReader::with_runtime` [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6612:
URL: https://github.com/apache/arrow-rs/pull/6612#discussion_r1818843309


##########
parquet/src/arrow/async_reader/store.rs:
##########
@@ -180,10 +231,69 @@ mod tests {
                 let err = e.to_string();
                 assert!(
                     err.contains("not found: No such file or directory (os error 2)"),
-                    "{}",
-                    err
+                    "{err}",
                 );
             }
         }
     }
+
+    #[tokio::test]
+    // We need to mark this with the `target_has_atomic` because the spawned_tasks_count() fn is
+    // only available for that cfg
+    async fn test_runtime_is_used() {
+        let num_actions = Arc::new(AtomicUsize::new(0));
+
+        let (a1, a2) = (num_actions.clone(), num_actions.clone());
+        let rt = tokio::runtime::Builder::new_multi_thread()
+            .on_thread_park(move || {

Review Comment:
   TIL -- https://docs.rs/tokio/latest/tokio/runtime/struct.Builder.html#method.on_thread_park
   
   I wonder if this is assuming too much about how tokio / the reader operates. For example, does it assume that only one thread is used to run the tasks from the parquet reader?



##########
parquet/src/arrow/async_reader/store.rs:
##########
@@ -180,10 +231,69 @@ mod tests {
                 let err = e.to_string();
                 assert!(
                     err.contains("not found: No such file or directory (os error 2)"),
-                    "{}",
-                    err
+                    "{err}",
                 );
             }
         }
     }
+
+    #[tokio::test]
+    // We need to mark this with the `target_has_atomic` because the spawned_tasks_count() fn is
+    // only available for that cfg
+    async fn test_runtime_is_used() {

Review Comment:
   An alternate testing strategy that might be easier to verify / less dependent on the tokio runtime behavior could be:
   1. Create a new runtime 
   2. Get a handle and register it with the parquet reader
   3. Shutdown the runtime
   4. Try and use the parquet reader 
   
   The test could be marked as `#should_panic` that the runtime was shutdown
   
   



##########
parquet/src/arrow/async_reader/store.rs:
##########
@@ -180,10 +231,69 @@ mod tests {
                 let err = e.to_string();
                 assert!(
                     err.contains("not found: No such file or directory (os error 2)"),
-                    "{}",
-                    err
+                    "{err}",
                 );
             }
         }
     }
+
+    #[tokio::test]
+    // We need to mark this with the `target_has_atomic` because the spawned_tasks_count() fn is
+    // only available for that cfg
+    async fn test_runtime_is_used() {
+        let num_actions = Arc::new(AtomicUsize::new(0));
+
+        let (a1, a2) = (num_actions.clone(), num_actions.clone());
+        let rt = tokio::runtime::Builder::new_multi_thread()
+            .on_thread_park(move || {
+                a1.fetch_add(1, Ordering::Relaxed);
+            })
+            .on_thread_unpark(move || {
+                a2.fetch_add(1, Ordering::Relaxed);
+            })
+            .build()
+            .unwrap();
+
+        let (meta, store) = get_meta_store().await;
+
+        let initial_actions = num_actions.load(Ordering::Relaxed);
+
+        let reader = ParquetObjectReader::new(store, meta).with_runtime(rt.handle().clone());
+
+        let builder = ParquetRecordBatchStreamBuilder::new(reader).await.unwrap();
+        let batches: Vec<_> = builder.build().unwrap().try_collect().await.unwrap();
+
+        // Just copied these assert_eqs from the `test_timple` above
+        assert_eq!(batches.len(), 1);
+        assert_eq!(batches[0].num_rows(), 8);
+
+        assert!(num_actions.load(Ordering::Relaxed) - initial_actions > 0);
+
+        // Runtimes have to be dropped in blocking contexts, so we need to move this one to a new
+        // blocking thread to drop it.
+        tokio::runtime::Handle::current().spawn_blocking(move || drop(rt));
+    }
+
+    #[tokio::test]
+    async fn test_runtime_thread_id_different() {

Review Comment:
   ```suggestion
       /// Unit test that `ParquetObjectReader::spawn`spawns on the provided runtime
       #[tokio::test]
       async fn test_runtime_thread_id_different() {
   ```
   
   
   
   👍 
   
   This is a good unit test of `ParquetObjectReader::spawn` -- 
   
   I like how the previous test verifies that the relevant functions in `ParquetObjectReader` call `spawn` rather than running directly on the same runtime
   



##########
parquet/src/arrow/async_reader/store.rs:
##########
@@ -180,10 +231,69 @@ mod tests {
                 let err = e.to_string();
                 assert!(
                     err.contains("not found: No such file or directory (os error 2)"),
-                    "{}",
-                    err
+                    "{err}",
                 );
             }
         }
     }
+
+    #[tokio::test]
+    // We need to mark this with the `target_has_atomic` because the spawned_tasks_count() fn is
+    // only available for that cfg
+    async fn test_runtime_is_used() {
+        let num_actions = Arc::new(AtomicUsize::new(0));
+
+        let (a1, a2) = (num_actions.clone(), num_actions.clone());
+        let rt = tokio::runtime::Builder::new_multi_thread()
+            .on_thread_park(move || {
+                a1.fetch_add(1, Ordering::Relaxed);
+            })
+            .on_thread_unpark(move || {
+                a2.fetch_add(1, Ordering::Relaxed);
+            })
+            .build()
+            .unwrap();
+
+        let (meta, store) = get_meta_store().await;
+
+        let initial_actions = num_actions.load(Ordering::Relaxed);
+
+        let reader = ParquetObjectReader::new(store, meta).with_runtime(rt.handle().clone());
+
+        let builder = ParquetRecordBatchStreamBuilder::new(reader).await.unwrap();
+        let batches: Vec<_> = builder.build().unwrap().try_collect().await.unwrap();
+
+        // Just copied these assert_eqs from the `test_timple` above
+        assert_eq!(batches.len(), 1);
+        assert_eq!(batches[0].num_rows(), 8);
+
+        assert!(num_actions.load(Ordering::Relaxed) - initial_actions > 0);
+
+        // Runtimes have to be dropped in blocking contexts, so we need to move this one to a new
+        // blocking thread to drop it.
+        tokio::runtime::Handle::current().spawn_blocking(move || drop(rt));
+    }
+
+    #[tokio::test]
+    async fn test_runtime_thread_id_different() {
+        let rt = tokio::runtime::Builder::new_multi_thread()
+            .worker_threads(1)
+            .build()
+            .unwrap();
+
+        let (meta, store) = get_meta_store().await;
+
+        let reader = ParquetObjectReader::new(store, meta).with_runtime(rt.handle().clone());
+
+        let current_id = std::thread::current().id();
+
+        let other_id = reader
+            .spawn(|_, _| async move { Ok::<_, Infallible>(std::thread::current().id()) }.boxed())
+            .await
+            .unwrap();
+
+        assert_ne!(current_id, other_id);
+
+        tokio::runtime::Handle::current().spawn_blocking(move || drop(rt));

Review Comment:
   Can you also add unit tests for each of the three APIs in ParquetObjectReader that spawn is used?
   * `get_bytes`
   * `get_byte_ranges`
   * `get_metadata`?



##########
parquet/src/arrow/async_reader/store.rs:
##########
@@ -99,27 +99,62 @@ impl ParquetObjectReader {
             ..self
         }
     }
+
+    /// Perform IO on the provided tokio runtime
+    ///
+    /// Tokio is a cooperative scheduler, and relies on tasks yielding in a timely manner
+    /// to service IO. Therefore, running IO and CPU-bound tasks, such as parquet decoding,
+    /// on the same tokio runtime can lead to degraded throughput, dropped connections and
+    /// other issues. For more information see [here].
+    ///
+    /// [here]: https://www.influxdata.com/blog/using-rustlangs-async-tokio-runtime-for-cpu-bound-tasks/
+    pub fn with_runtime(self, handle: Handle) -> Self {
+        Self {
+            runtime: Some(handle),
+            ..self
+        }
+    }
+
+    fn spawn<F, O, E>(&self, f: F) -> BoxFuture<'_, Result<O>>
+    where
+        F: for<'a> FnOnce(&'a Arc<dyn ObjectStore>, &'a Path) -> BoxFuture<'a, Result<O, E>>
+            + Send
+            + 'static,
+        O: Send + 'static,
+        E: Into<ParquetError> + Send + 'static,
+    {
+        match &self.runtime {
+            Some(handle) => {
+                let path = self.meta.location.clone();
+                let store = Arc::clone(&self.store);
+                handle
+                    .spawn(async move { f(&store, &path).await })
+                    .map_ok_or_else(
+                        |e| match e.try_into_panic() {
+                            Err(e) => Err(ParquetError::External(Box::new(e))),
+                            Ok(p) => std::panic::resume_unwind(p),
+                        },
+                        |res| res.map_err(|e| e.into()),
+                    )
+                    .boxed()
+            }
+            None => f(&self.store, &self.meta.location)
+                .map_err(|e| e.into())
+                .boxed(),
+        }
+    }
 }
 
 impl AsyncFileReader for ParquetObjectReader {
     fn get_bytes(&mut self, range: Range<usize>) -> BoxFuture<'_, Result<Bytes>> {
-        self.store
-            .get_range(&self.meta.location, range)
-            .map_err(|e| e.into())
-            .boxed()
+        self.spawn(|store, path| store.get_range(path, range))
     }
 
     fn get_byte_ranges(&mut self, ranges: Vec<Range<usize>>) -> BoxFuture<'_, Result<Vec<Bytes>>>
     where
         Self: Send,
     {
-        async move {
-            self.store
-                .get_ranges(&self.meta.location, &ranges)
-                .await
-                .map_err(|e| e.into())
-        }
-        .boxed()
+        self.spawn(|store, path| async move { store.get_ranges(path, &ranges).await }.boxed())

Review Comment:
   I believe we also need to wrap the future returned by `get_metadata` as well in `spawn`



##########
parquet/src/arrow/async_reader/store.rs:
##########
@@ -180,10 +231,69 @@ mod tests {
                 let err = e.to_string();
                 assert!(
                     err.contains("not found: No such file or directory (os error 2)"),
-                    "{}",
-                    err
+                    "{err}",
                 );
             }
         }
     }
+
+    #[tokio::test]
+    // We need to mark this with the `target_has_atomic` because the spawned_tasks_count() fn is

Review Comment:
   I don't understand this comment -- I don't see any other reference to `target_has_atomic` in this PR 🤔 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `ParquetObjectReader::with_runtime` [arrow-rs]

Posted by "itsjunetime (via GitHub)" <gi...@apache.org>.
itsjunetime commented on code in PR #6612:
URL: https://github.com/apache/arrow-rs/pull/6612#discussion_r1821210251


##########
parquet/src/arrow/async_reader/store.rs:
##########
@@ -180,10 +231,69 @@ mod tests {
                 let err = e.to_string();
                 assert!(
                     err.contains("not found: No such file or directory (os error 2)"),
-                    "{}",
-                    err
+                    "{err}",
                 );
             }
         }
     }
+
+    #[tokio::test]
+    // We need to mark this with the `target_has_atomic` because the spawned_tasks_count() fn is

Review Comment:
   This was left over from a previous commit, thanks for catching it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `ParquetObjectReader::with_runtime` [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6612:
URL: https://github.com/apache/arrow-rs/pull/6612#discussion_r1821275908


##########
parquet/src/arrow/async_reader/store.rs:
##########
@@ -99,27 +99,62 @@ impl ParquetObjectReader {
             ..self
         }
     }
+
+    /// Perform IO on the provided tokio runtime
+    ///
+    /// Tokio is a cooperative scheduler, and relies on tasks yielding in a timely manner
+    /// to service IO. Therefore, running IO and CPU-bound tasks, such as parquet decoding,
+    /// on the same tokio runtime can lead to degraded throughput, dropped connections and
+    /// other issues. For more information see [here].
+    ///
+    /// [here]: https://www.influxdata.com/blog/using-rustlangs-async-tokio-runtime-for-cpu-bound-tasks/
+    pub fn with_runtime(self, handle: Handle) -> Self {
+        Self {
+            runtime: Some(handle),
+            ..self
+        }
+    }
+
+    fn spawn<F, O, E>(&self, f: F) -> BoxFuture<'_, Result<O>>
+    where
+        F: for<'a> FnOnce(&'a Arc<dyn ObjectStore>, &'a Path) -> BoxFuture<'a, Result<O, E>>
+            + Send
+            + 'static,
+        O: Send + 'static,
+        E: Into<ParquetError> + Send + 'static,
+    {
+        match &self.runtime {
+            Some(handle) => {
+                let path = self.meta.location.clone();
+                let store = Arc::clone(&self.store);
+                handle
+                    .spawn(async move { f(&store, &path).await })
+                    .map_ok_or_else(
+                        |e| match e.try_into_panic() {
+                            Err(e) => Err(ParquetError::External(Box::new(e))),
+                            Ok(p) => std::panic::resume_unwind(p),
+                        },
+                        |res| res.map_err(|e| e.into()),
+                    )
+                    .boxed()
+            }
+            None => f(&self.store, &self.meta.location)
+                .map_err(|e| e.into())
+                .boxed(),
+        }
+    }
 }
 
 impl AsyncFileReader for ParquetObjectReader {
     fn get_bytes(&mut self, range: Range<usize>) -> BoxFuture<'_, Result<Bytes>> {
-        self.store
-            .get_range(&self.meta.location, range)
-            .map_err(|e| e.into())
-            .boxed()
+        self.spawn(|store, path| store.get_range(path, range))
     }
 
     fn get_byte_ranges(&mut self, ranges: Vec<Range<usize>>) -> BoxFuture<'_, Result<Vec<Bytes>>>
     where
         Self: Send,
     {
-        async move {
-            self.store
-                .get_ranges(&self.meta.location, &ranges)
-                .await
-                .map_err(|e| e.into())
-        }
-        .boxed()
+        self.spawn(|store, path| async move { store.get_ranges(path, &ranges).await }.boxed())

Review Comment:
   I see -- I think the thing I was missing is that `load_and_finish` takes `self` which is used to actually do the IO. Thank you for the clarification
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `ParquetObjectReader::with_runtime` [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #6612:
URL: https://github.com/apache/arrow-rs/pull/6612#discussion_r1821618619


##########
parquet/src/errors.rs:
##########
@@ -107,6 +107,13 @@ impl From<str::Utf8Error> for ParquetError {
     }
 }
 
+#[cfg(test)]
+impl From<std::convert::Infallible> for ParquetError {

Review Comment:
   I'm not sure about this, the whole point of infallible is that it can't be constructed and so doesn't need to be handled



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `ParquetObjectReader::with_runtime` [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6612:
URL: https://github.com/apache/arrow-rs/pull/6612#discussion_r1821512268


##########
parquet/src/errors.rs:
##########
@@ -107,6 +107,13 @@ impl From<str::Utf8Error> for ParquetError {
     }
 }
 
+#[cfg(test)]
+impl From<std::convert::Infallible> for ParquetError {

Review Comment:
   This is a nice improvement too. Thank you. Maybe it is worth adding publically as well



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `ParquetObjectReader::with_runtime` [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6612:
URL: https://github.com/apache/arrow-rs/pull/6612#discussion_r1822382038


##########
parquet/src/errors.rs:
##########
@@ -107,6 +107,13 @@ impl From<str::Utf8Error> for ParquetError {
     }
 }
 
+#[cfg(test)]
+impl From<std::convert::Infallible> for ParquetError {

Review Comment:
   Well, it can't be constructed, but it often does need to be "handled" (aka to transform a `Result<.., Infallible>` to `Result<.., Error>` type expected by an API)
   
   I don't feel strongly about this particular code. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `ParquetObjectReader::with_runtime` [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6612:
URL: https://github.com/apache/arrow-rs/pull/6612#discussion_r1822382038


##########
parquet/src/errors.rs:
##########
@@ -107,6 +107,13 @@ impl From<str::Utf8Error> for ParquetError {
     }
 }
 
+#[cfg(test)]
+impl From<std::convert::Infallible> for ParquetError {

Review Comment:
   Well, it can't be constructed, but it often does need to be handled (aka to transform a `Result<.., Infallible>` to `Result<.., Error>` type expected by an API)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `ParquetObjectReader::with_runtime` [arrow-rs]

Posted by "itsjunetime (via GitHub)" <gi...@apache.org>.
itsjunetime commented on code in PR #6612:
URL: https://github.com/apache/arrow-rs/pull/6612#discussion_r1821221462


##########
parquet/src/arrow/async_reader/store.rs:
##########
@@ -99,27 +99,62 @@ impl ParquetObjectReader {
             ..self
         }
     }
+
+    /// Perform IO on the provided tokio runtime
+    ///
+    /// Tokio is a cooperative scheduler, and relies on tasks yielding in a timely manner
+    /// to service IO. Therefore, running IO and CPU-bound tasks, such as parquet decoding,
+    /// on the same tokio runtime can lead to degraded throughput, dropped connections and
+    /// other issues. For more information see [here].
+    ///
+    /// [here]: https://www.influxdata.com/blog/using-rustlangs-async-tokio-runtime-for-cpu-bound-tasks/
+    pub fn with_runtime(self, handle: Handle) -> Self {
+        Self {
+            runtime: Some(handle),
+            ..self
+        }
+    }
+
+    fn spawn<F, O, E>(&self, f: F) -> BoxFuture<'_, Result<O>>
+    where
+        F: for<'a> FnOnce(&'a Arc<dyn ObjectStore>, &'a Path) -> BoxFuture<'a, Result<O, E>>
+            + Send
+            + 'static,
+        O: Send + 'static,
+        E: Into<ParquetError> + Send + 'static,
+    {
+        match &self.runtime {
+            Some(handle) => {
+                let path = self.meta.location.clone();
+                let store = Arc::clone(&self.store);
+                handle
+                    .spawn(async move { f(&store, &path).await })
+                    .map_ok_or_else(
+                        |e| match e.try_into_panic() {
+                            Err(e) => Err(ParquetError::External(Box::new(e))),
+                            Ok(p) => std::panic::resume_unwind(p),
+                        },
+                        |res| res.map_err(|e| e.into()),
+                    )
+                    .boxed()
+            }
+            None => f(&self.store, &self.meta.location)
+                .map_err(|e| e.into())
+                .boxed(),
+        }
+    }
 }
 
 impl AsyncFileReader for ParquetObjectReader {
     fn get_bytes(&mut self, range: Range<usize>) -> BoxFuture<'_, Result<Bytes>> {
-        self.store
-            .get_range(&self.meta.location, range)
-            .map_err(|e| e.into())
-            .boxed()
+        self.spawn(|store, path| store.get_range(path, range))
     }
 
     fn get_byte_ranges(&mut self, ranges: Vec<Range<usize>>) -> BoxFuture<'_, Result<Vec<Bytes>>>
     where
         Self: Send,
     {
-        async move {
-            self.store
-                .get_ranges(&self.meta.location, &ranges)
-                .await
-                .map_err(|e| e.into())
-        }
-        .boxed()
+        self.spawn(|store, path| async move { store.get_ranges(path, &ranges).await }.boxed())

Review Comment:
   I don't think this should be necessary, actually - if you look through the call chain of `get_metadata`, it ends up just doing IO through the `get_bytes` and `get_byte_ranges` methods on `Self`. So all the IO it does, down the line, is already spawned off to the other handle. I'll add a comment about this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `ParquetObjectReader::with_runtime` [arrow-rs]

Posted by "itsjunetime (via GitHub)" <gi...@apache.org>.
itsjunetime commented on code in PR #6612:
URL: https://github.com/apache/arrow-rs/pull/6612#discussion_r1821261048


##########
parquet/src/arrow/async_reader/store.rs:
##########
@@ -180,10 +231,69 @@ mod tests {
                 let err = e.to_string();
                 assert!(
                     err.contains("not found: No such file or directory (os error 2)"),
-                    "{}",
-                    err
+                    "{err}",
                 );
             }
         }
     }
+
+    #[tokio::test]
+    // We need to mark this with the `target_has_atomic` because the spawned_tasks_count() fn is
+    // only available for that cfg
+    async fn test_runtime_is_used() {

Review Comment:
   That's a good idea, I'll add that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org