You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/02/08 03:10:56 UTC
[PR] Minor: Improve `DataFrame` docs, add examples [arrow-datafusion]
alamb opened a new pull request, #9159:
URL: https://github.com/apache/arrow-datafusion/pull/9159
## Which issue does this PR close?
Part of #7013
## Rationale for this change
I was watching @carols10cents try to find how to write a parquet file via a `DataFrame` in the API docs and they left something to be desired
## What changes are included in this PR?
1. Add examples to `DataFrame::` `write_json`, `write_csv`, `write_parquet`
2. Improve more of the DataFrame documentations
## Are these changes tested?
Covered by CI doc tests
## Are there any user-facing changes?
Better docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: Improve `DataFrame` docs, add examples [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9159:
URL: https://github.com/apache/arrow-datafusion/pull/9159#discussion_r1483774623
##########
datafusion/core/src/dataframe/mod.rs:
##########
@@ -1025,6 +1107,7 @@ impl DataFrame {
}
/// Write this DataFrame to the referenced table by name.
+ ///
/// This method uses on the same underlying implementation
Review Comment:
I missed this comment - will make a follow on PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: Improve `DataFrame` docs, add examples [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9159:
URL: https://github.com/apache/arrow-datafusion/pull/9159#issuecomment-1935199462
> Great work thanks @alamb
Thanks @comphead ❤️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: Improve `DataFrame` docs, add examples [arrow-datafusion]
Posted by "carols10cents (via GitHub)" <gi...@apache.org>.
carols10cents commented on code in PR #9159:
URL: https://github.com/apache/arrow-datafusion/pull/9159#discussion_r1483029958
##########
datafusion/core/src/dataframe/mod.rs:
##########
@@ -109,27 +109,54 @@ impl Default for DataFrameWriteOptions {
}
}
-/// DataFrame represents a logical set of rows with the same named columns.
-/// Similar to a [Pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) or
-/// [Spark DataFrame](https://spark.apache.org/docs/latest/sql-programming-guide.html)
+/// Represents a logical set of rows with the same named columns.
///
-/// DataFrames are typically created by the `read_csv` and `read_parquet` methods on the
-/// [SessionContext](../execution/context/struct.SessionContext.html) and can then be modified
-/// by calling the transformation methods, such as `filter`, `select`, `aggregate`, and `limit`
-/// to build up a query definition.
+/// Similar to a [Pandas DataFrame] or [Spark DataFrame], a DataFusion DataFrame
+/// represents a 2 dimensional table of rows and columns.
///
-/// The query can be executed by calling the `collect` method.
+/// The typical workflow using DataFrames looks like
///
+/// 1. Create a DataFrame via methods on [SessionContext], such as [`read_csv`]
+/// and [`read_parquet`].
+///
+/// 2. Build a desired calculation by calling methods such as [`filter`],
+/// [`select`], [`aggregate`], and [`limit`]
+///
+/// 3. Execute into [`RecordBatch`]es by calling [`collect`]
+///
+/// A `DataFrame` is a wrapper around a [`LogicalPlan`] and the [`SessionState`]
+/// required for execution.
+///
+/// DataFrames are "lazy" in the sense that most methods do not actually compute
+/// anything, they just build up a plan. Calling [`collect`] executes the plan
Review Comment:
Nit: double space
```suggestion
/// anything, they just build up a plan. Calling [`collect`] executes the plan
```
##########
datafusion/core/src/dataframe/mod.rs:
##########
@@ -739,9 +804,14 @@ impl DataFrame {
Ok(len)
}
- /// Convert the logical plan represented by this DataFrame into a physical plan and
- /// execute it, collecting all resulting batches into memory
- /// Executes this DataFrame and collects all results into a vector of RecordBatch.
+ /// Execute this `DataFrame` and buffer all resulting `RecordBatch`es into memory.
Review Comment:
This change in particular is _excellent_! The reason I'd want to call this method is precisely because I don't want to think about logical plans and physical plans and such 😉
##########
datafusion/core/src/dataframe/mod.rs:
##########
@@ -562,8 +617,12 @@ impl DataFrame {
))
}
- /// Sort the DataFrame by the specified sorting expressions. Any expression can be turned into
- /// a sort expression by calling its [sort](../logical_plan/enum.Expr.html#method.sort) method.
+ /// Sort the DataFrame by the specified sorting expressions.
+ ///
+ /// Note that Any expression can be turned into
Review Comment:
Nit: capitalization in the middle of a sentence
```suggestion
/// Note that any expression can be turned into
```
##########
datafusion/core/src/dataframe/mod.rs:
##########
@@ -961,6 +1042,7 @@ impl DataFrame {
/// Return a `FunctionRegistry` used to plan udf's calls
///
+ ///
Review Comment:
Is this extra newline what you intended? It seems like in most places you were adding an `# Example` heading, so perhaps you meant:
```suggestion
/// # Example
```
##########
datafusion/core/src/dataframe/mod.rs:
##########
@@ -1025,6 +1107,7 @@ impl DataFrame {
}
/// Write this DataFrame to the referenced table by name.
+ ///
/// This method uses on the same underlying implementation
Review Comment:
This line wasn't changed in this PR, but it caught my eye... it seems like there's an extra word?
```suggestion
/// This method uses the same underlying implementation
```
##########
datafusion/core/src/dataframe/mod.rs:
##########
@@ -706,9 +768,12 @@ impl DataFrame {
Ok(DataFrame::new(self.session_state, plan))
}
- /// Run a count aggregate on the DataFrame and execute the DataFrame to collect this
- /// count and return it as a usize, to find the total number of rows after executing
- /// the DataFrame.
+ /// Return the total number of rows in this `DataFrame`.
Review Comment:
Nit: double space
```suggestion
/// Return the total number of rows in this `DataFrame`.
```
##########
datafusion/core/src/dataframe/mod.rs:
##########
@@ -899,20 +979,21 @@ impl DataFrame {
(self.session_state, self.plan)
}
- /// Return the logical plan represented by this DataFrame without running the optimizers
+ /// Return the [`LogicalPlan`] represented by this DataFrame without running
+ /// any optimizers
///
- /// Note: This method should not be used outside testing, as it loses the snapshot
- /// of the [`SessionState`] attached to this [`DataFrame`] and consequently subsequent
- /// operations may take place against a different state
+ /// Note: This method should not be used outside testing, as it loses the
+ /// snapshot of the [`SessionState`] attached to this [`DataFrame`] and
+ /// consequently subsequent operations may take place against a different
+ /// state (e.g. a different value of `now()`)
Review Comment:
Love this example!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: Improve `DataFrame` docs, add examples [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9159:
URL: https://github.com/apache/arrow-datafusion/pull/9159#discussion_r1483780362
##########
datafusion/core/src/dataframe/mod.rs:
##########
@@ -1025,6 +1107,7 @@ impl DataFrame {
}
/// Write this DataFrame to the referenced table by name.
+ ///
/// This method uses on the same underlying implementation
Review Comment:
https://github.com/apache/arrow-datafusion/pull/9169
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: Improve `DataFrame` docs, add examples [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9159:
URL: https://github.com/apache/arrow-datafusion/pull/9159#discussion_r1482361878
##########
datafusion/core/src/dataframe/parquet.rs:
##########
@@ -26,7 +26,27 @@ use super::{
};
impl DataFrame {
- /// Write a `DataFrame` to a Parquet file.
+ /// Execute the `DataFrame` and write the results to Parquet file(s).
+ ///
+ /// # Example
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// use datafusion::dataframe::DataFrameWriteOptions;
+ /// let ctx = SessionContext::new();
+ /// // Sort the data by column "b" and write it to a new location
+ /// ctx.read_csv("tests/data/example.csv", CsvReadOptions::new()).await?
+ /// .sort(vec![col("b").sort(true, true)])? // sort by b asc, nulls first
+ /// .write_parquet(
+ /// "output.parquet",
+ /// DataFrameWriteOptions::new(),
+ /// None, // can also specify parquet writing options here
+ /// ).await?;
+ /// # Ok(())
+ /// # }
+ /// ```
pub async fn write_parquet(
Review Comment:
This is the example I wanted to write, but I started clicking through the API to get here and I saw too much along the way that needed to be fixed too, so I got a bit carried away and worked on quite a bit of the docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: Improve `DataFrame` docs, add examples [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9159:
URL: https://github.com/apache/arrow-datafusion/pull/9159#issuecomment-1934506870
Thank you for the review @carols10cents 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: Improve `DataFrame` docs, add examples [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9159:
URL: https://github.com/apache/arrow-datafusion/pull/9159#discussion_r1483274579
##########
datafusion/core/src/dataframe/mod.rs:
##########
@@ -961,6 +1042,7 @@ impl DataFrame {
/// Return a `FunctionRegistry` used to plan udf's calls
///
+ ///
Review Comment:
no that was not intentional -- thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: Improve `DataFrame` docs, add examples [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #9159:
URL: https://github.com/apache/arrow-datafusion/pull/9159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org