You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "H0TB0X420 (via GitHub)" <gi...@apache.org> on 2025/09/16 19:23:36 UTC
[PR] Add to_batches() and interpolate() methods to DataFrame [datafusion-python]
H0TB0X420 opened a new pull request, #1241:
URL: https://github.com/apache/datafusion-python/pull/1241
- Add to_batches() as alias for collect() returning RecordBatch list
- Add interpolate() method with forward_fill support
- Add deprecation warning to collect() method
- Add comprehensive tests for both methods
- All existing tests pass (with expected deprecation warnings)
Addresses items from RFC #875
This is my first PR! Please let me know what I can improve.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add to_batches() and interpolate() methods to DataFrame [datafusion-python]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on code in PR #1241:
URL: https://github.com/apache/datafusion-python/pull/1241#discussion_r2355804076
##########
python/datafusion/dataframe.py:
##########
@@ -370,6 +375,39 @@ def schema(self) -> pa.Schema:
Describing schema of the DataFrame
"""
return self.df.schema()
+
+ def to_batches(self) -> list[pa.RecordBatch]:
+ """Convert DataFrame to list of RecordBatches."""
+ return self.collect() # delegate to existing method
Review Comment:
> Now if there is a way we can remove this dependency and it doesn't break existing workflows
I think the straightforward way to do that is to remove pyarrow as a required dependency and error if it's not installed. But a separate question is whether we should be adding _new methods_ that explicitly depend on pyarrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add to_batches() and interpolate() methods to DataFrame [datafusion-python]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on code in PR #1241:
URL: https://github.com/apache/datafusion-python/pull/1241#discussion_r2355723443
##########
python/datafusion/dataframe.py:
##########
@@ -370,6 +375,39 @@ def schema(self) -> pa.Schema:
Describing schema of the DataFrame
"""
return self.df.schema()
+
+ def to_batches(self) -> list[pa.RecordBatch]:
+ """Convert DataFrame to list of RecordBatches."""
+ return self.collect() # delegate to existing method
Review Comment:
Perhaps we should get @timsaucer 's thoughts in https://github.com/apache/datafusion-python/issues/1227: is having a required dependency on pyarrow a problem? What should we do about it? Do we want to depend on arro3-core instead? Or have functions that rely on pyarrow but error if pyarrow isn't installed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add to_batches() and interpolate() methods to DataFrame [datafusion-python]
Posted by "ion-elgreco (via GitHub)" <gi...@apache.org>.
ion-elgreco commented on code in PR #1241:
URL: https://github.com/apache/datafusion-python/pull/1241#discussion_r2353697365
##########
python/datafusion/dataframe.py:
##########
@@ -370,6 +375,39 @@ def schema(self) -> pa.Schema:
Describing schema of the DataFrame
"""
return self.df.schema()
+
+ def to_batches(self) -> list[pa.RecordBatch]:
+ """Convert DataFrame to list of RecordBatches."""
+ return self.collect() # delegate to existing method
Review Comment:
100% agree, this should return arro3 recordbatches
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add to_batches() and interpolate() methods to DataFrame [datafusion-python]
Posted by "H0TB0X420 (via GitHub)" <gi...@apache.org>.
H0TB0X420 commented on code in PR #1241:
URL: https://github.com/apache/datafusion-python/pull/1241#discussion_r2356344778
##########
python/datafusion/dataframe.py:
##########
@@ -370,6 +375,39 @@ def schema(self) -> pa.Schema:
Describing schema of the DataFrame
"""
return self.df.schema()
+
+ def to_batches(self) -> list[pa.RecordBatch]:
+ """Convert DataFrame to list of RecordBatches."""
+ return self.collect() # delegate to existing method
Review Comment:
I could do a try-except to throw an ImportError if it is not installed. But it might make more sense to drop `to_batches()` from this PR given the dependency discussion. I don't want to add to the PyArrow dependency concerns.
I'll remove it and focus on fixing the `interpolate()` implementation. I appreciate you taking the time to give feedback!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add to_batches() and interpolate() methods to DataFrame [datafusion-python]
Posted by "ion-elgreco (via GitHub)" <gi...@apache.org>.
ion-elgreco commented on code in PR #1241:
URL: https://github.com/apache/datafusion-python/pull/1241#discussion_r2354427053
##########
python/datafusion/dataframe.py:
##########
@@ -370,6 +375,39 @@ def schema(self) -> pa.Schema:
Describing schema of the DataFrame
"""
return self.df.schema()
+
+ def to_batches(self) -> list[pa.RecordBatch]:
+ """Convert DataFrame to list of RecordBatches."""
+ return self.collect() # delegate to existing method
Review Comment:
> I'm not saying it should even return arro3 RecordBatches... because that's still an external dependency that datafusion is requiring on users. Datafusion could return a minimal batch object that just holds the RecordBatch pointer and then the user transfers it to their library of choice
That's possible but I would argue it's more convenient if it's in some way already usable instead of just a pointer. Arro3 is that small that it's negligible in size overall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add to_batches() and interpolate() methods to DataFrame [datafusion-python]
Posted by "H0TB0X420 (via GitHub)" <gi...@apache.org>.
H0TB0X420 commented on code in PR #1241:
URL: https://github.com/apache/datafusion-python/pull/1241#discussion_r2356326534
##########
python/datafusion/dataframe.py:
##########
@@ -370,6 +375,39 @@ def schema(self) -> pa.Schema:
Describing schema of the DataFrame
"""
return self.df.schema()
+
+ def to_batches(self) -> list[pa.RecordBatch]:
+ """Convert DataFrame to list of RecordBatches."""
+ return self.collect() # delegate to existing method
+
+ def interpolate(self, method: str = "forward_fill", **kwargs) -> DataFrame:
+ """Interpolate missing values per column.
+
+ Args:
+ method: Interpolation method ('linear', 'forward_fill', 'backward_fill')
Review Comment:
Two quick questions:
1. Should we have separate methods for filling vs interpolation?
2. Do we wait for DataFusion core features before adding Python APIs, or is it okay to create stand-in Python implementations using existing primitives?
Thank you for your time!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add to_batches() and interpolate() methods to DataFrame [datafusion-python]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on code in PR #1241:
URL: https://github.com/apache/datafusion-python/pull/1241#discussion_r2354157056
##########
python/datafusion/dataframe.py:
##########
@@ -370,6 +375,39 @@ def schema(self) -> pa.Schema:
Describing schema of the DataFrame
"""
return self.df.schema()
+
+ def to_batches(self) -> list[pa.RecordBatch]:
+ """Convert DataFrame to list of RecordBatches."""
+ return self.collect() # delegate to existing method
Review Comment:
I'm not saying it should even return arro3 RecordBatches... because that's still an external dependency that datafusion is requiring on users. Datafusion could return a minimal batch object that just holds the RecordBatch pointer and then the user transfers it to their library of choice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add to_batches() and interpolate() methods to DataFrame [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on code in PR #1241:
URL: https://github.com/apache/datafusion-python/pull/1241#discussion_r2355771369
##########
python/datafusion/dataframe.py:
##########
@@ -370,6 +375,39 @@ def schema(self) -> pa.Schema:
Describing schema of the DataFrame
"""
return self.df.schema()
+
+ def to_batches(self) -> list[pa.RecordBatch]:
+ """Convert DataFrame to list of RecordBatches."""
+ return self.collect() # delegate to existing method
Review Comment:
I understand that `pyarrow` is a large dependency, but I also have a feeling from the community that nearly everyone who is using `datafusion-python` to do things like `to_batches` is using `pyarrow` in the next portions of their code. I'm sure we can always find exceptions.
Now if there is a way we can remove this dependency and it doesn't break existing workflows, that would be even better. I haven't made the time to sit down and play with it, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add to_batches() and interpolate() methods to DataFrame [datafusion-python]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on code in PR #1241:
URL: https://github.com/apache/datafusion-python/pull/1241#discussion_r2353461791
##########
python/datafusion/dataframe.py:
##########
@@ -370,6 +375,39 @@ def schema(self) -> pa.Schema:
Describing schema of the DataFrame
"""
return self.df.schema()
+
+ def to_batches(self) -> list[pa.RecordBatch]:
+ """Convert DataFrame to list of RecordBatches."""
+ return self.collect() # delegate to existing method
Review Comment:
My opinion (see https://github.com/apache/datafusion-python/issues/1227) is to limit the surface area where we explicitly depend on pyarrow. Especially in this case it's just an alias?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add to_batches() and interpolate() methods to DataFrame [datafusion-python]
Posted by "H0TB0X420 (via GitHub)" <gi...@apache.org>.
H0TB0X420 commented on PR #1241:
URL: https://github.com/apache/datafusion-python/pull/1241#issuecomment-3304031710
Thanks for the feedback!
I will revert those changes and deprecation warnings to collect and schema. I will focus on the interpolation and to_batches changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add to_batches() and interpolate() methods to DataFrame [datafusion-python]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on code in PR #1241:
URL: https://github.com/apache/datafusion-python/pull/1241#discussion_r2355723443
##########
python/datafusion/dataframe.py:
##########
@@ -370,6 +375,39 @@ def schema(self) -> pa.Schema:
Describing schema of the DataFrame
"""
return self.df.schema()
+
+ def to_batches(self) -> list[pa.RecordBatch]:
+ """Convert DataFrame to list of RecordBatches."""
+ return self.collect() # delegate to existing method
Review Comment:
Perhaps we should get @timsaucer 's thoughts in https://github.com/apache/datafusion-python/issues/1227: is having a required dependency on pyarrow a problem? What should we do about it? Do we want to depend on arro3-core instead? Or have functions that rely on pyarrow but error if pyarrow isn't installed?
@ion-elgreco feel free to write down your thoughts there too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add to_batches() and interpolate() methods to DataFrame [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on code in PR #1241:
URL: https://github.com/apache/datafusion-python/pull/1241#discussion_r2355817186
##########
python/datafusion/dataframe.py:
##########
@@ -370,6 +375,39 @@ def schema(self) -> pa.Schema:
Describing schema of the DataFrame
"""
return self.df.schema()
+
+ def to_batches(self) -> list[pa.RecordBatch]:
+ """Convert DataFrame to list of RecordBatches."""
+ return self.collect() # delegate to existing method
+
+ def interpolate(self, method: str = "forward_fill", **kwargs) -> DataFrame:
+ """Interpolate missing values per column.
+
+ Args:
+ method: Interpolation method ('linear', 'forward_fill', 'backward_fill')
Review Comment:
At the outset this doesn't look quite right to me.
- The method is called `interpolate` but the one interpolation method is the one not supported. The others are filling operations, not interpolations.
- This looks like it's going to create a very wrong filling - every field in the schema gets sorted and then filled? I would expect we would need an ordering column to determine the filling method.
- If we were to do this I think the most pleasant experience would have an enum for the possible values with the simple string conversions necessary.
##########
python/tests/test_dataframe.py:
##########
@@ -185,6 +185,47 @@ def get_header_style(self) -> str:
"padding: 10px; border: 1px solid #3367d6;"
)
+def test_to_batches(df):
+ """Test to_batches method returns list of RecordBatches."""
+ batches = df.to_batches()
+ assert isinstance(batches, list)
+ assert len(batches) > 0
+ assert all(isinstance(batch, pa.RecordBatch) for batch in batches)
+
+
+ collect_batches = df.collect()
+ assert len(batches) == len(collect_batches)
+ for i, batch in enumerate(batches):
+ assert batch.equals(collect_batches[i])
+
+
+def test_interpolate_forward_fill(ctx):
+ """Test interpolate method with forward_fill."""
+
+ batch = pa.RecordBatch.from_arrays(
+ [pa.array([1, None, 3, None]), pa.array([4.0, None, 6.0, None])],
+ names=["int_col", "float_col"],
+ )
+ df = ctx.create_dataframe([[batch]])
+
+ result = df.interpolate("forward_fill")
+
+ assert isinstance(result, DataFrame)
Review Comment:
We would probably want to collect the results and verify they fill as expected.
##########
python/datafusion/dataframe.py:
##########
@@ -592,6 +630,9 @@ def tail(self, n: int = 5) -> DataFrame:
"""
return DataFrame(self.df.limit(n, max(0, self.count() - n)))
+ @deprecated(
+ "collect() returning RecordBatch list is deprecated. Use to_batches() for RecordBatch list or collect() will return DataFrame in future versions"
+ )
Review Comment:
I disagree with deprecating `collect`. I think we should keep our main functions close to those in `datafusion`, but with the appropriate sugar to make them pleasant to use in Python. Also, changing `collect` to return a dataframe instead of a list of record batches would be a *major* breaking change to many people's workflows.
##########
python/datafusion/dataframe.py:
##########
@@ -360,6 +362,9 @@ def describe(self) -> DataFrame:
"""
return DataFrame(self.df.describe())
+ @deprecated(
+ "schema() is deprecated. Use :py:meth:`~DataFrame.get_schema` instead"
+ )
Review Comment:
I'm confused by this. I don't see a `get_schema` method in this file and I don't see why we'd want to deprecate this method anyways.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org