You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "konjac (via GitHub)" <gi...@apache.org> on 2025/02/05 11:33:19 UTC

[PR] refactor: collect dataframe as stream in `__repr__` [datafusion-python]

konjac opened a new pull request, #1015:
URL: https://github.com/apache/datafusion-python/pull/1015

   # Which issue does this PR close?
   
   Closes #1014.
   
    # Rationale for this change
   The root cause for #1014 is `_repr_` appends a limit clause and causes EXPLAIN not the top plan.
   
   # What changes are included in this PR?
   This change refactors `_repr_` to collect the dataframe as a stream.
   
   # Are there any user-facing changes?
   No. Bugfix only.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor: collect dataframe as stream in `__repr__` [datafusion-python]

Posted by "konjac (via GitHub)" <gi...@apache.org>.
konjac commented on code in PR #1015:
URL: https://github.com/apache/datafusion-python/pull/1015#discussion_r1959666298


##########
src/dataframe.rs:
##########
@@ -90,8 +91,16 @@ impl PyDataFrame {
     }
 
     fn __repr__(&self, py: Python) -> PyDataFusionResult<String> {
-        let df = self.df.as_ref().clone().limit(0, Some(10))?;
-        let batches = wait_for_future(py, df.collect())?;
+        let df = self.df.as_ref().clone();
+
+        let stream = wait_for_future(py, df.execute_stream()).map_err(py_datafusion_err)?;
+
+        let batches: Vec<RecordBatch>  = wait_for_future(
+            py,
+            stream.take(10).collect::<Vec<_>>())
+            .into_iter()
+            .collect::<Result<Vec<_>,_>>()?;
+

Review Comment:
   @timsaucer My bad. I used a dataset so trivial that failed to find the bug. 
   
   I just pushed a new iteration, which iterates over batches until getting 10 rows. This should be identical to the previous implementation adding limit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor: collect dataframe as stream in `__repr__` [datafusion-python]

Posted by "konjac (via GitHub)" <gi...@apache.org>.
konjac commented on code in PR #1015:
URL: https://github.com/apache/datafusion-python/pull/1015#discussion_r1959678559


##########
src/dataframe.rs:
##########
@@ -90,8 +91,16 @@ impl PyDataFrame {
     }
 
     fn __repr__(&self, py: Python) -> PyDataFusionResult<String> {
-        let df = self.df.as_ref().clone().limit(0, Some(10))?;
-        let batches = wait_for_future(py, df.collect())?;
+        let df = self.df.as_ref().clone();
+
+        let stream = wait_for_future(py, df.execute_stream()).map_err(py_datafusion_err)?;
+
+        let batches: Vec<RecordBatch>  = wait_for_future(
+            py,
+            stream.take(10).collect::<Vec<_>>())
+            .into_iter()
+            .collect::<Result<Vec<_>,_>>()?;
+

Review Comment:
   @timsaucer I think enhancement could be a separate issue #1026



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor: collect dataframe as stream in `__repr__` [datafusion-python]

Posted by "konjac (via GitHub)" <gi...@apache.org>.
konjac commented on PR #1015:
URL: https://github.com/apache/datafusion-python/pull/1015#issuecomment-2676912860

   > This looks good. It took me a while to parse through the logic of the `get_batches`. I think it's worth adding some documentation within the file to explain why we are doing this so that future maintainers can understand the necessity of doing it this way.
   > 
   > Thank you for the PR!
   
   Thank you for the reviewing! A new iteration is pushed to add code comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor: collect dataframe as stream in `__repr__` [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on code in PR #1015:
URL: https://github.com/apache/datafusion-python/pull/1015#discussion_r1957173874


##########
src/dataframe.rs:
##########
@@ -90,8 +91,16 @@ impl PyDataFrame {
     }
 
     fn __repr__(&self, py: Python) -> PyDataFusionResult<String> {
-        let df = self.df.as_ref().clone().limit(0, Some(10))?;
-        let batches = wait_for_future(py, df.collect())?;
+        let df = self.df.as_ref().clone();
+
+        let stream = wait_for_future(py, df.execute_stream()).map_err(py_datafusion_err)?;
+
+        let batches: Vec<RecordBatch>  = wait_for_future(
+            py,
+            stream.take(10).collect::<Vec<_>>())
+            .into_iter()
+            .collect::<Result<Vec<_>,_>>()?;
+

Review Comment:
   I did a test and this changes how `__repr__` works from what we currently have. With this change it looks like it is returning the first 10 record batches instead of the first 10 rows, as I would expect. The idea of putting the `limit(0, Some(10))` into the logical plan was so that you can get a small sampling of the data.
   
   I think we need to change this to support the bug but also to make sure we don't change the output here.
   
   I suspect we have the same problem for `__repr_html__`



##########
src/dataframe.rs:
##########
@@ -90,8 +91,16 @@ impl PyDataFrame {
     }
 
     fn __repr__(&self, py: Python) -> PyDataFusionResult<String> {
-        let df = self.df.as_ref().clone().limit(0, Some(10))?;
-        let batches = wait_for_future(py, df.collect())?;
+        let df = self.df.as_ref().clone();
+
+        let stream = wait_for_future(py, df.execute_stream()).map_err(py_datafusion_err)?;
+
+        let batches: Vec<RecordBatch>  = wait_for_future(
+            py,
+            stream.take(10).collect::<Vec<_>>())
+            .into_iter()
+            .collect::<Result<Vec<_>,_>>()?;
+

Review Comment:
   As a side note, I wonder if we want to enhance `__repr__` to also check to see the total number of rows in the DataFrame. My guess is that we don't want to do that. But if we did we could add a line int the return that was something like `... {} additional rows`. A lighter weight would be to change the limit to 11, get the number returned, show the first 10 and if there was 11 returned to sad `... and additional rows` just so the end user knows that you're only seeing a portion of the DF.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor: collect dataframe as stream in `__repr__` [datafusion-python]

Posted by "konjac (via GitHub)" <gi...@apache.org>.
konjac commented on PR #1015:
URL: https://github.com/apache/datafusion-python/pull/1015#issuecomment-2644492439

   Hi @timsaucer , could you help to trigger CI and also review the proposed change? Thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor: collect dataframe as stream in `__repr__` [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer closed pull request #1015: refactor: collect dataframe as stream in `__repr__`
URL: https://github.com/apache/datafusion-python/pull/1015


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor: collect dataframe as stream in `__repr__` [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on PR #1015:
URL: https://github.com/apache/datafusion-python/pull/1015#issuecomment-2970821413

   Original problem was resolved by https://github.com/apache/datafusion-python/pull/1036 which I believe used part of this work. Thank you very much for the contribution!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor: collect dataframe as stream in `__repr__` [datafusion-python]

Posted by "konjac (via GitHub)" <gi...@apache.org>.
konjac commented on code in PR #1015:
URL: https://github.com/apache/datafusion-python/pull/1015#discussion_r1959666298


##########
src/dataframe.rs:
##########
@@ -90,8 +91,16 @@ impl PyDataFrame {
     }
 
     fn __repr__(&self, py: Python) -> PyDataFusionResult<String> {
-        let df = self.df.as_ref().clone().limit(0, Some(10))?;
-        let batches = wait_for_future(py, df.collect())?;
+        let df = self.df.as_ref().clone();
+
+        let stream = wait_for_future(py, df.execute_stream()).map_err(py_datafusion_err)?;
+
+        let batches: Vec<RecordBatch>  = wait_for_future(
+            py,
+            stream.take(10).collect::<Vec<_>>())
+            .into_iter()
+            .collect::<Result<Vec<_>,_>>()?;
+

Review Comment:
   @timsaucer My bad. I used a dataset so trivial that failed to find the bug. 
   
   I just pushed a new iteration, which iterates over batches until getting 10 rows. This should be identical to the previous implementation by adding limit caluse.



##########
src/dataframe.rs:
##########
@@ -90,8 +91,16 @@ impl PyDataFrame {
     }
 
     fn __repr__(&self, py: Python) -> PyDataFusionResult<String> {
-        let df = self.df.as_ref().clone().limit(0, Some(10))?;
-        let batches = wait_for_future(py, df.collect())?;
+        let df = self.df.as_ref().clone();
+
+        let stream = wait_for_future(py, df.execute_stream()).map_err(py_datafusion_err)?;
+
+        let batches: Vec<RecordBatch>  = wait_for_future(
+            py,
+            stream.take(10).collect::<Vec<_>>())
+            .into_iter()
+            .collect::<Result<Vec<_>,_>>()?;
+

Review Comment:
   @timsaucer My bad. I used a dataset so trivial that failed to find the bug. 
   
   I just pushed a new iteration, which iterates over batches until getting 10 rows. This should be identical to the previous implementation by adding limit clause.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org