You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "p-vdp (via GitHub)" <gi...@apache.org> on 2025/10/02 22:20:25 UTC
[PR] Implement DataFrame methods `__len__`, `columns`, and `shape` [sedona-db]
p-vdp opened a new pull request, #176:
URL: https://github.com/apache/sedona-db/pull/176
Basic/standard data frame methods like in NumPy and Pandas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement DataFrame methods `__len__`, `columns`, and `shape` [sedona-db]
Posted by "p-vdp (via GitHub)" <gi...@apache.org>.
p-vdp commented on code in PR #176:
URL: https://github.com/apache/sedona-db/pull/176#discussion_r2400342813
##########
python/sedonadb/python/sedonadb/dataframe.py:
##########
@@ -151,6 +151,29 @@ def count(self) -> int:
"""
return self._impl.count()
+ def __len__(self) -> int:
+ """Compute the number of rows in the DataFrame"""
+ return self.count()
+
+ @property
+ def columns(self) -> list[str]:
+ """Return the column names in the DataFrame"""
+ columns = list()
+ field_index = 0
+ while True:
+ try:
+ columns.append(self._impl.schema().field(field_index).name)
+ field_index += 1
+ except IndexError:
+ break
+
Review Comment:
Yes this is an ugly loop but `self._impl.schema()` isn't iterable. This suggestion won't work without also modifying `PySedonaSchema.schema()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement DataFrame methods `__len__`, `columns`, and `shape` [sedona-db]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #176:
URL: https://github.com/apache/sedona-db/pull/176#issuecomment-3387655714
@p-vdp great work on this PR. Thank you very much! What is your plan on this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement DataFrame methods `__len__`, `columns`, and `shape` [sedona-db]
Posted by "p-vdp (via GitHub)" <gi...@apache.org>.
p-vdp commented on PR #176:
URL: https://github.com/apache/sedona-db/pull/176#issuecomment-3387758476
@jiayuasu from the discussion it sounds like __len__() and shape() aren't great ideas at this stage, and columns() is best implemented on the Rust side. Closing this PR and I might work on the columns() implementation in Rust if no one else gets to it first. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement DataFrame methods `__len__`, `columns`, and `shape` [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #176:
URL: https://github.com/apache/sedona-db/pull/176#discussion_r2400522763
##########
python/sedonadb/python/sedonadb/dataframe.py:
##########
@@ -151,6 +151,29 @@ def count(self) -> int:
"""
return self._impl.count()
+ def __len__(self) -> int:
+ """Compute the number of rows in the DataFrame"""
+ return self.count()
+
+ @property
+ def columns(self) -> list[str]:
+ """Return the column names in the DataFrame"""
+ columns = list()
+ field_index = 0
+ while True:
+ try:
+ columns.append(self._impl.schema().field(field_index).name)
+ field_index += 1
+ except IndexError:
+ break
+
+ return columns
Review Comment:
Agreed...I mostly just didn't expose enough from the schema object when I wrote it 😬
##########
python/sedonadb/python/sedonadb/dataframe.py:
##########
@@ -151,6 +151,29 @@ def count(self) -> int:
"""
return self._impl.count()
+ def __len__(self) -> int:
+ """Compute the number of rows in the DataFrame"""
+ return self.count()
+
+ @property
+ def columns(self) -> list[str]:
+ """Return the column names in the DataFrame"""
+ columns = list()
+ field_index = 0
+ while True:
+ try:
+ columns.append(self._impl.schema().field(field_index).name)
+ field_index += 1
+ except IndexError:
+ break
+
+ return columns
+
+ @property
+ def shape(self) -> tuple[int, int]:
+ """Return the shape of the DataFrame as a tuple of integers corresponding to (rows, columns)"""
+ return self.count(), len(self.columns)
Review Comment:
Neither Ibis nor DuckDB implement a `.shape` accessor. Given that this would also trigger execution, I don't think it's a good idea to include this 😬
##########
python/sedonadb/python/sedonadb/dataframe.py:
##########
@@ -151,6 +151,29 @@ def count(self) -> int:
"""
return self._impl.count()
+ def __len__(self) -> int:
+ """Compute the number of rows in the DataFrame"""
+ return self.count()
Review Comment:
I wonder if we should include this or not. Our "DataFrame" hasn't been materialized yet and might well take quite a long time to do so.
As a data point, Ibis implements `__len__()` but returns an error:
```
ExpressionError: Use .count() instead
```
Another data point...duckdb implements `__len__()` and executes the query like this implementation. I'm not sure what pyspark does here.
I would personally lean towards the Ibis approach (return an error forcing a user to explicitly count).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement DataFrame methods `__len__`, `columns`, and `shape` [sedona-db]
Posted by "petern48 (via GitHub)" <gi...@apache.org>.
petern48 commented on code in PR #176:
URL: https://github.com/apache/sedona-db/pull/176#discussion_r2400453179
##########
python/sedonadb/python/sedonadb/dataframe.py:
##########
@@ -151,6 +151,29 @@ def count(self) -> int:
"""
return self._impl.count()
+ def __len__(self) -> int:
+ """Compute the number of rows in the DataFrame"""
+ return self.count()
+
+ @property
+ def columns(self) -> list[str]:
+ """Return the column names in the DataFrame"""
+ columns = list()
+ field_index = 0
+ while True:
+ try:
+ columns.append(self._impl.schema().field(field_index).name)
+ field_index += 1
+ except IndexError:
+ break
+
+ return columns
Review Comment:
```suggestion
return self._impl.columns()
```
I think we should implement this in Rust, adding the below code [here](https://github.com/apache/sedona-db/blob/a15844b4ff9c7e9416b8f1c1c07ad81d908a89cd/python/sedonadb/src/dataframe.rs#L62). To test the updated Rust code locally, you'll want to run `maturin develop` from the `python/sedonadb/` directory. You might need to `pip install maturin`.
```rust
fn columns(&self) -> Vec<String> {
self.inner
.schema()
.fields()
.iter()
.map(|f| f.name().to_string())
.collect()
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement DataFrame methods `__len__`, `columns`, and `shape` [sedona-db]
Posted by "p-vdp (via GitHub)" <gi...@apache.org>.
p-vdp commented on code in PR #176:
URL: https://github.com/apache/sedona-db/pull/176#discussion_r2400342813
##########
python/sedonadb/python/sedonadb/dataframe.py:
##########
@@ -151,6 +151,29 @@ def count(self) -> int:
"""
return self._impl.count()
+ def __len__(self) -> int:
+ """Compute the number of rows in the DataFrame"""
+ return self.count()
+
+ @property
+ def columns(self) -> list[str]:
+ """Return the column names in the DataFrame"""
+ columns = list()
+ field_index = 0
+ while True:
+ try:
+ columns.append(self._impl.schema().field(field_index).name)
+ field_index += 1
+ except IndexError:
+ break
+
Review Comment:
Yes this is an ugly loop but `self._impl.schema()` isn't iterable. This suggestion won't work without also modifying the return from `PySedonaSchema.schema()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement DataFrame methods `__len__`, `columns`, and `shape` [sedona-db]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #176:
URL: https://github.com/apache/sedona-db/pull/176#discussion_r2400298297
##########
python/sedonadb/python/sedonadb/dataframe.py:
##########
@@ -151,6 +151,29 @@ def count(self) -> int:
"""
return self._impl.count()
+ def __len__(self) -> int:
+ """Compute the number of rows in the DataFrame"""
+ return self.count()
+
+ @property
+ def columns(self) -> list[str]:
+ """Return the column names in the DataFrame"""
+ columns = list()
+ field_index = 0
+ while True:
+ try:
+ columns.append(self._impl.schema().field(field_index).name)
+ field_index += 1
+ except IndexError:
+ break
+
+ return columns
+
+ @property
+ def shape(self) -> tuple[int, int]:
+ """Return the shape of the DataFrame as a tuple of integers corresponding to (rows, columns)"""
+ return self.count(), len(self.columns)
Review Comment:
The `shape` property calls both `count()` and `len(self.columns)`, which results in two separate operations. Since `columns` property already performs schema iteration, consider caching or optimizing this to avoid redundant work when both dimensions are needed.
```suggestion
columns = self.columns
return self.count(), len(columns)
```
##########
python/sedonadb/python/sedonadb/dataframe.py:
##########
@@ -151,6 +151,29 @@ def count(self) -> int:
"""
return self._impl.count()
+ def __len__(self) -> int:
+ """Compute the number of rows in the DataFrame"""
+ return self.count()
+
+ @property
+ def columns(self) -> list[str]:
+ """Return the column names in the DataFrame"""
+ columns = list()
+ field_index = 0
+ while True:
+ try:
+ columns.append(self._impl.schema().field(field_index).name)
+ field_index += 1
+ except IndexError:
+ break
+
Review Comment:
The manual loop with exception handling to iterate through schema fields is unnecessarily complex. Consider using a more Pythonic approach with list comprehension or iterating over the schema directly if the API supports it.
```suggestion
# Use a more Pythonic approach to extract column names
columns = [field.name for field in self._impl.schema()]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement DataFrame methods `__len__`, `columns`, and `shape` [sedona-db]
Posted by "p-vdp (via GitHub)" <gi...@apache.org>.
p-vdp closed pull request #176: Implement DataFrame methods `__len__`, `columns`, and `shape`
URL: https://github.com/apache/sedona-db/pull/176
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement DataFrame methods `__len__`, `columns`, and `shape` [sedona-db]
Posted by "p-vdp (via GitHub)" <gi...@apache.org>.
p-vdp commented on code in PR #176:
URL: https://github.com/apache/sedona-db/pull/176#discussion_r2400342813
##########
python/sedonadb/python/sedonadb/dataframe.py:
##########
@@ -151,6 +151,29 @@ def count(self) -> int:
"""
return self._impl.count()
+ def __len__(self) -> int:
+ """Compute the number of rows in the DataFrame"""
+ return self.count()
+
+ @property
+ def columns(self) -> list[str]:
+ """Return the column names in the DataFrame"""
+ columns = list()
+ field_index = 0
+ while True:
+ try:
+ columns.append(self._impl.schema().field(field_index).name)
+ field_index += 1
+ except IndexError:
+ break
+
Review Comment:
Yes this is an ugly loop but `self._impl.schema()` returns a string, not an iterable of `fields`. This suggestion won't work without also modifying `PySedonaSchema.schema()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement DataFrame methods `__len__`, `columns`, and `shape` [sedona-db]
Posted by "p-vdp (via GitHub)" <gi...@apache.org>.
p-vdp commented on code in PR #176:
URL: https://github.com/apache/sedona-db/pull/176#discussion_r2400342813
##########
python/sedonadb/python/sedonadb/dataframe.py:
##########
@@ -151,6 +151,29 @@ def count(self) -> int:
"""
return self._impl.count()
+ def __len__(self) -> int:
+ """Compute the number of rows in the DataFrame"""
+ return self.count()
+
+ @property
+ def columns(self) -> list[str]:
+ """Return the column names in the DataFrame"""
+ columns = list()
+ field_index = 0
+ while True:
+ try:
+ columns.append(self._impl.schema().field(field_index).name)
+ field_index += 1
+ except IndexError:
+ break
+
Review Comment:
Yes this is an ugly loop but `self._impl.schema()` returns a string, not an iterable object. This suggestion won't work without also modifying `PySedonaSchema.schema()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org