[I] `ObjectStore::get_range` returns `Bytes` instead of `GetResult` [arrow-rs-object-store]

["Dandandan (via GitHub)" <gi...@apache.org>]

Dandandan opened a new issue, #195:
URL: https://github.com/apache/arrow-rs-object-store/issues/195

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   While using the object store (S3 indirectly via DataFusion/Ballista) and using `get_range` requests, we can not stream the results but are required to fetch entire column chunks instead of being able to stream the data directly.
   
   There are some limitations by not providing such a method in the `ObjectStore`
   * Reducing memory usage/scalability by fetching / incrementally
   * Processing more incrementally instead of waiting for the entire range to be fetched
   * Being able to push down limits to every part of the query (e.g. row filters) and aborting early.
   
   **Describe the solution you'd like**
   Change the `get_range` implementation to return `GetResult` or introduce a new `get_range_stream` implementation, returning a stream, allowing to process / stream incrementally.
   
   I checked this to be possible for S3, and I expect it to be for other implementations as well:
   ```rust
       async fn get_range_stream(&self, location: &Path, range: Range<usize>) -> Result<GetResult> {
           let stream = self
               .client
               .get_request(location, Some(range), false)
               .await?
               .bytes_stream()
               .map_err(|source| crate::Error::Generic {
                   store: "S3",
                   source: Box::new(source),
               })
               .boxed();
           Ok(GetResult::Stream(stream))
       }
   ```
   
   We can provide a default implementation based on `get_range` in the meantime for other object stores.
   
   **Describe alternatives you've considered**
   -->
   
   **Additional context**
   <!--
   Add any other context or screenshots about the feature request here.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org