You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jiangzhx (via GitHub)" <gi...@apache.org> on 2023/02/09 13:46:16 UTC
[GitHub] [arrow-datafusion] jiangzhx opened a new pull request, #5226: add a describe method on DataFrame like Polars
jiangzhx opened a new pull request, #5226:
URL: https://github.com/apache/arrow-datafusion/pull/5226
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #4974 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jiangzhx commented on pull request #5226: add a describe method on DataFrame like Polars
Posted by "jiangzhx (via GitHub)" <gi...@apache.org>.
jiangzhx commented on PR #5226:
URL: https://github.com/apache/arrow-datafusion/pull/5226#issuecomment-1424219632
Try to complete this work, there are still many places that can be optimized.
<img width="796" alt="image" src="https://user-images.githubusercontent.com/494507/217830064-2ecf1de4-2604-445e-8f7e-99c7cb878c07.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ursabot commented on pull request #5226: add a describe method on DataFrame like Polars
Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #5226:
URL: https://github.com/apache/arrow-datafusion/pull/5226#issuecomment-1448560062
Benchmark runs are scheduled for baseline = ea3b965dd4861f966a5ec1c4a65f748f0e5dcc12 and contender = 96aa2a677f1f2b2bb8dd803841d71f2a08301961. 96aa2a677f1f2b2bb8dd803841d71f2a08301961 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/54a3bbb0b18c42cf9b515835ad76a2de...0fda86e06c4a4ea3a16493c183ff2667/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] [test-mac-arm](https://conbench.ursa.dev/compare/runs/8495a37e815140629f37249605a63d62...b1b8432c413d4875859e704726a34fb2/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/761fcef6eda541f9a7e52814bce249e2...cc8ed035ad9241c081612dd704584cfe/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/2195e83baaaf46eaa1b595e64e1aa9cd...7b5cb342b87d45399d09e6c53825c9cd/)
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #5226: add a describe method on DataFrame like Polars
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #5226:
URL: https://github.com/apache/arrow-datafusion/pull/5226#discussion_r1120419464
##########
datafusion/core/src/dataframe.rs:
##########
@@ -302,6 +305,155 @@ impl DataFrame {
))
}
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.describe().await.unwrap();
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn describe(self) -> Result<Self> {
+ //the functions now supported
+ let supported_describe_functions = vec!["count", "null_count", "max", "min"];
+
+ let fields_iter = self.schema().fields().iter();
+
+ //define describe column
+ let mut describe_schemas = fields_iter
+ .clone()
+ .map(|field| {
+ if field.data_type().is_numeric() {
+ Field::new(field.name(), DataType::Float64, true)
+ } else {
+ Field::new(field.name(), DataType::Utf8, true)
+ }
+ })
+ .collect::<Vec<_>>();
+ describe_schemas.insert(0, Field::new("describe", DataType::Utf8, false));
+
+ //collect recordBatch
+ let describe_record_batch = vec![
+ // count aggregation
+ self.clone()
+ .aggregate(
+ vec![],
+ fields_iter
+ .clone()
+ .map(|f| datafusion_expr::count(col(f.name())).alias(f.name()))
+ .collect::<Vec<_>>(),
+ )?
+ .collect()
+ .await?,
+ // null_count aggregation
+ self.clone()
+ .aggregate(
+ vec![],
+ fields_iter
+ .clone()
+ .map(|f| {
+ datafusion_expr::count(datafusion_expr::is_null(
+ col(f.name()),
+ ))
+ .alias(f.name())
+ })
+ .collect::<Vec<_>>(),
+ )?
+ .collect()
+ .await?,
+ // max aggregation
+ self.clone()
+ .aggregate(
+ vec![],
+ fields_iter
+ .clone()
+ .filter(|f| {
+ !matches!(f.data_type(), DataType::Binary | DataType::Boolean)
+ })
+ .map(|f| datafusion_expr::max(col(f.name())).alias(f.name()))
+ .collect::<Vec<_>>(),
+ )?
+ .collect()
+ .await?,
+ // min aggregation
+ self.clone()
+ .aggregate(
+ vec![],
+ fields_iter
+ .clone()
+ .filter(|f| {
+ !matches!(f.data_type(), DataType::Binary | DataType::Boolean)
+ })
+ .map(|f| datafusion_expr::min(col(f.name())).alias(f.name()))
+ .collect::<Vec<_>>(),
+ )?
+ .collect()
+ .await?,
+ ];
+
+ let mut array_ref_vec: Vec<ArrayRef> = vec![];
+ for field in fields_iter {
+ let mut array_datas = vec![];
+ for record_batch in describe_record_batch.iter() {
+ let column = record_batch.get(0).unwrap().column_by_name(field.name());
+ match column {
+ Some(c) => {
+ if field.data_type().is_numeric() {
+ array_datas.push(cast(c, &DataType::Float64)?);
+ } else {
+ array_datas.push(cast(c, &DataType::Utf8)?);
+ }
+ }
+ //if None mean the column cannot be min/max aggregation
+ None => {
+ array_datas.push(Arc::new(StringArray::from_slice(["null"])));
+ }
+ }
+ }
+
+ array_ref_vec.push(concat(
+ array_datas
+ .iter()
+ .map(|af| af.as_ref())
+ .collect::<Vec<_>>()
+ .as_slice(),
+ )?);
+ }
+
+ //insert first column with function names
+ array_ref_vec.insert(
+ 0,
+ Arc::new(StringArray::from_slice(
+ supported_describe_functions.clone(),
+ )),
+ );
+
+ let describe_record_batch =
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jiangzhx commented on a diff in pull request #5226: add a describe method on DataFrame like Polars
Posted by "jiangzhx (via GitHub)" <gi...@apache.org>.
jiangzhx commented on code in PR #5226:
URL: https://github.com/apache/arrow-datafusion/pull/5226#discussion_r1118239437
##########
datafusion/core/src/dataframe.rs:
##########
@@ -302,6 +306,177 @@ impl DataFrame {
))
}
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.describe().await?;
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn describe(self) -> Result<()> {
+ Ok(print_batches(
+ &self.clone().collect_describe().await.unwrap(),
+ )?)
+ }
+
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.collect_describe().await.unwrap();
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn collect_describe(self) -> Result<Vec<RecordBatch>> {
+ //the functions now supported
+ let supported_describe_functions = vec!["count", "null_count", "max", "min"]; //"count", "max", "min",
Review Comment:
forgot to delete,fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jiangzhx commented on a diff in pull request #5226: add a describe method on DataFrame like Polars
Posted by "jiangzhx (via GitHub)" <gi...@apache.org>.
jiangzhx commented on code in PR #5226:
URL: https://github.com/apache/arrow-datafusion/pull/5226#discussion_r1118246773
##########
datafusion/core/src/dataframe.rs:
##########
@@ -302,6 +306,177 @@ impl DataFrame {
))
}
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.describe().await?;
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn describe(self) -> Result<()> {
+ Ok(print_batches(
+ &self.clone().collect_describe().await.unwrap(),
+ )?)
+ }
+
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.collect_describe().await.unwrap();
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn collect_describe(self) -> Result<Vec<RecordBatch>> {
+ //the functions now supported
+ let supported_describe_functions = vec!["count", "null_count", "max", "min"]; //"count", "max", "min",
+
+ let fields_iter = self.schema().fields().iter();
+
+ //define describe column
+ let mut describe_schemas = fields_iter
+ .clone()
+ .map(|field| {
+ if field.data_type().is_numeric() {
+ Field::new(field.name(), DataType::Float64, true)
+ } else {
+ Field::new(field.name(), DataType::Utf8, true)
+ }
+ })
+ .collect::<Vec<_>>();
+ describe_schemas.insert(0, Field::new("describe", DataType::Utf8, false));
+
+ //collect recordBatch
+ let describe_record_batch = vec![
+ // count aggregation
+ self.clone()
+ .aggregate(
+ vec![],
+ fields_iter
+ .clone()
+ .map(|f| {
+ Expr::Alias(
+ Box::new(datafusion_expr::count(col(f.name()))),
+ f.name().to_string(),
+ )
+ })
+ .collect::<Vec<_>>(),
+ )?
+ .collect()
+ .await?,
+ // null_count aggregation
+ self.clone()
+ .aggregate(
+ vec![],
+ fields_iter
+ .clone()
+ .map(|f| {
+ Expr::Alias(
+ Box::new(datafusion_expr::count(
+ datafusion_expr::is_null(col(f.name())),
+ )),
+ f.name().to_string(),
+ )
+ })
+ .collect::<Vec<_>>(),
+ )?
+ .collect()
+ .await?,
+ // max aggregation
+ self.clone()
+ .aggregate(
+ vec![],
+ fields_iter
+ .clone()
+ .filter(|f| matches!(f.data_type().is_numeric(), true))
Review Comment:
boolean and binary not work with min/max.
filter out DataType::Binary , DataType::Boolean will be better.
!matches!(f.data_type(), DataType::Binary | DataType::Boolean)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jiangzhx commented on a diff in pull request #5226: add a describe method on DataFrame like Polars
Posted by "jiangzhx (via GitHub)" <gi...@apache.org>.
jiangzhx commented on code in PR #5226:
URL: https://github.com/apache/arrow-datafusion/pull/5226#discussion_r1118277844
##########
datafusion/core/src/dataframe.rs:
##########
@@ -302,6 +306,177 @@ impl DataFrame {
))
}
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.describe().await?;
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn describe(self) -> Result<()> {
+ Ok(print_batches(
+ &self.clone().collect_describe().await.unwrap(),
+ )?)
+ }
+
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.collect_describe().await.unwrap();
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn collect_describe(self) -> Result<Vec<RecordBatch>> {
+ //the functions now supported
+ let supported_describe_functions = vec!["count", "null_count", "max", "min"]; //"count", "max", "min",
+
+ let fields_iter = self.schema().fields().iter();
+
+ //define describe column
+ let mut describe_schemas = fields_iter
+ .clone()
+ .map(|field| {
+ if field.data_type().is_numeric() {
+ Field::new(field.name(), DataType::Float64, true)
+ } else {
+ Field::new(field.name(), DataType::Utf8, true)
+ }
+ })
+ .collect::<Vec<_>>();
+ describe_schemas.insert(0, Field::new("describe", DataType::Utf8, false));
+
+ //collect recordBatch
+ let describe_record_batch = vec![
+ // count aggregation
+ self.clone()
+ .aggregate(
+ vec![],
+ fields_iter
+ .clone()
+ .map(|f| {
+ Expr::Alias(
+ Box::new(datafusion_expr::count(col(f.name()))),
+ f.name().to_string(),
+ )
+ })
+ .collect::<Vec<_>>(),
+ )?
+ .collect()
+ .await?,
+ // null_count aggregation
+ self.clone()
+ .aggregate(
+ vec![],
+ fields_iter
+ .clone()
+ .map(|f| {
+ Expr::Alias(
+ Box::new(datafusion_expr::count(
+ datafusion_expr::is_null(col(f.name())),
+ )),
+ f.name().to_string(),
+ )
+ })
+ .collect::<Vec<_>>(),
+ )?
+ .collect()
+ .await?,
+ // max aggregation
+ self.clone()
+ .aggregate(
+ vec![],
+ fields_iter
+ .clone()
+ .filter(|f| matches!(f.data_type().is_numeric(), true))
Review Comment:
date_string_col, string_col ’s datatype also Binary.
called `Result::unwrap()` on an `Err` value: Internal("Min/Max accumulator not implemented for type Binary")
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #5226: add a describe method on DataFrame like Polars
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #5226:
URL: https://github.com/apache/arrow-datafusion/pull/5226#discussion_r1118075835
##########
datafusion/core/src/dataframe.rs:
##########
@@ -302,6 +306,177 @@ impl DataFrame {
))
}
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.describe().await?;
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn describe(self) -> Result<()> {
+ Ok(print_batches(
+ &self.clone().collect_describe().await.unwrap(),
+ )?)
+ }
+
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.collect_describe().await.unwrap();
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn collect_describe(self) -> Result<Vec<RecordBatch>> {
+ //the functions now supported
+ let supported_describe_functions = vec!["count", "null_count", "max", "min"]; //"count", "max", "min",
+
+ let fields_iter = self.schema().fields().iter();
+
+ //define describe column
+ let mut describe_schemas = fields_iter
+ .clone()
+ .map(|field| {
+ if field.data_type().is_numeric() {
+ Field::new(field.name(), DataType::Float64, true)
+ } else {
+ Field::new(field.name(), DataType::Utf8, true)
+ }
+ })
+ .collect::<Vec<_>>();
+ describe_schemas.insert(0, Field::new("describe", DataType::Utf8, false));
+
+ //collect recordBatch
+ let describe_record_batch = vec![
+ // count aggregation
+ self.clone()
+ .aggregate(
+ vec![],
+ fields_iter
+ .clone()
+ .map(|f| {
+ Expr::Alias(
+ Box::new(datafusion_expr::count(col(f.name()))),
+ f.name().to_string(),
+ )
Review Comment:
I think you can write this more concisely like this (untested)
```suggestion
datafusion_expr::count(col(f.name()).alias(f.name())
```
##########
datafusion/core/src/dataframe.rs:
##########
@@ -302,6 +306,177 @@ impl DataFrame {
))
}
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.describe().await?;
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn describe(self) -> Result<()> {
+ Ok(print_batches(
Review Comment:
I think a nicer interface would be
```rust
pub async fn describe(self) -> Result<Self> {
```
And then we don't need a separate `collect_describe` instead we could reuse the existing `collect()`
One way might be to return a new `DataFrame` from the created RecordBatch -- perhaps following the model of
https://docs.rs/datafusion/18.0.0/src/datafusion/execution/context.rs.html#719-730
##########
datafusion/core/src/dataframe.rs:
##########
@@ -302,6 +306,177 @@ impl DataFrame {
))
}
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.describe().await?;
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn describe(self) -> Result<()> {
+ Ok(print_batches(
+ &self.clone().collect_describe().await.unwrap(),
+ )?)
+ }
+
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.collect_describe().await.unwrap();
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn collect_describe(self) -> Result<Vec<RecordBatch>> {
+ //the functions now supported
+ let supported_describe_functions = vec!["count", "null_count", "max", "min"]; //"count", "max", "min",
+
+ let fields_iter = self.schema().fields().iter();
+
+ //define describe column
+ let mut describe_schemas = fields_iter
+ .clone()
+ .map(|field| {
+ if field.data_type().is_numeric() {
+ Field::new(field.name(), DataType::Float64, true)
+ } else {
+ Field::new(field.name(), DataType::Utf8, true)
Review Comment:
I would expect that the schema for `count` and `null_count` were always `Int64` and the schema for min/max were always `Utf8`
##########
datafusion/core/src/dataframe.rs:
##########
@@ -302,6 +306,177 @@ impl DataFrame {
))
}
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.describe().await?;
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn describe(self) -> Result<()> {
+ Ok(print_batches(
+ &self.clone().collect_describe().await.unwrap(),
+ )?)
+ }
+
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.collect_describe().await.unwrap();
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn collect_describe(self) -> Result<Vec<RecordBatch>> {
+ //the functions now supported
+ let supported_describe_functions = vec!["count", "null_count", "max", "min"]; //"count", "max", "min",
Review Comment:
Is the `// count` comment on purpose?
##########
datafusion/core/src/dataframe.rs:
##########
@@ -302,6 +306,177 @@ impl DataFrame {
))
}
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.describe().await?;
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn describe(self) -> Result<()> {
+ Ok(print_batches(
+ &self.clone().collect_describe().await.unwrap(),
+ )?)
+ }
+
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.collect_describe().await.unwrap();
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn collect_describe(self) -> Result<Vec<RecordBatch>> {
+ //the functions now supported
+ let supported_describe_functions = vec!["count", "null_count", "max", "min"]; //"count", "max", "min",
+
+ let fields_iter = self.schema().fields().iter();
+
+ //define describe column
+ let mut describe_schemas = fields_iter
+ .clone()
+ .map(|field| {
+ if field.data_type().is_numeric() {
+ Field::new(field.name(), DataType::Float64, true)
+ } else {
+ Field::new(field.name(), DataType::Utf8, true)
+ }
+ })
+ .collect::<Vec<_>>();
+ describe_schemas.insert(0, Field::new("describe", DataType::Utf8, false));
+
+ //collect recordBatch
+ let describe_record_batch = vec![
+ // count aggregation
+ self.clone()
+ .aggregate(
+ vec![],
+ fields_iter
+ .clone()
+ .map(|f| {
+ Expr::Alias(
+ Box::new(datafusion_expr::count(col(f.name()))),
+ f.name().to_string(),
+ )
+ })
+ .collect::<Vec<_>>(),
+ )?
+ .collect()
+ .await?,
+ // null_count aggregation
+ self.clone()
+ .aggregate(
+ vec![],
+ fields_iter
+ .clone()
+ .map(|f| {
+ Expr::Alias(
+ Box::new(datafusion_expr::count(
+ datafusion_expr::is_null(col(f.name())),
+ )),
+ f.name().to_string(),
+ )
+ })
+ .collect::<Vec<_>>(),
+ )?
+ .collect()
+ .await?,
+ // max aggregation
+ self.clone()
+ .aggregate(
+ vec![],
+ fields_iter
+ .clone()
+ .filter(|f| matches!(f.data_type().is_numeric(), true))
Review Comment:
I wonder why restrict the min/max aggregation to numeric fields?
In order to get the min/max values in all columns to work, you could call `cast` to cast them to the same datatype
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jiangzhx closed pull request #5226: add a describe method on DataFrame like Polars
Posted by "jiangzhx (via GitHub)" <gi...@apache.org>.
jiangzhx closed pull request #5226: add a describe method on DataFrame like Polars
URL: https://github.com/apache/arrow-datafusion/pull/5226
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jiangzhx commented on a diff in pull request #5226: add a describe method on DataFrame like Polars
Posted by "jiangzhx (via GitHub)" <gi...@apache.org>.
jiangzhx commented on code in PR #5226:
URL: https://github.com/apache/arrow-datafusion/pull/5226#discussion_r1118296489
##########
datafusion/core/src/dataframe.rs:
##########
@@ -302,6 +306,177 @@ impl DataFrame {
))
}
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.describe().await?;
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn describe(self) -> Result<()> {
+ Ok(print_batches(
+ &self.clone().collect_describe().await.unwrap(),
+ )?)
+ }
+
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.collect_describe().await.unwrap();
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn collect_describe(self) -> Result<Vec<RecordBatch>> {
+ //the functions now supported
+ let supported_describe_functions = vec!["count", "null_count", "max", "min"]; //"count", "max", "min",
+
+ let fields_iter = self.schema().fields().iter();
+
+ //define describe column
+ let mut describe_schemas = fields_iter
+ .clone()
+ .map(|field| {
+ if field.data_type().is_numeric() {
+ Field::new(field.name(), DataType::Float64, true)
+ } else {
+ Field::new(field.name(), DataType::Utf8, true)
Review Comment:
the describe method return schema like this.
<img width="1392" alt="image" src="https://user-images.githubusercontent.com/494507/221468185-82ede8f7-9efd-4678-8fe8-bf5af713256b.png">
the each column should have same datatype .
for example :
- `bool_col` on `count/null_count` return Int64 ; error on `min/max` , so make `bool_col` datatype `UTF8`;
- `float_col` on `count/null_count` return Int64 ; on `min/max` return float, so make `float_col` datatype `Float64`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jiangzhx commented on a diff in pull request #5226: add a describe method on DataFrame like Polars
Posted by "jiangzhx (via GitHub)" <gi...@apache.org>.
jiangzhx commented on code in PR #5226:
URL: https://github.com/apache/arrow-datafusion/pull/5226#discussion_r1118246773
##########
datafusion/core/src/dataframe.rs:
##########
@@ -302,6 +306,177 @@ impl DataFrame {
))
}
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.describe().await?;
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn describe(self) -> Result<()> {
+ Ok(print_batches(
+ &self.clone().collect_describe().await.unwrap(),
+ )?)
+ }
+
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.collect_describe().await.unwrap();
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn collect_describe(self) -> Result<Vec<RecordBatch>> {
+ //the functions now supported
+ let supported_describe_functions = vec!["count", "null_count", "max", "min"]; //"count", "max", "min",
+
+ let fields_iter = self.schema().fields().iter();
+
+ //define describe column
+ let mut describe_schemas = fields_iter
+ .clone()
+ .map(|field| {
+ if field.data_type().is_numeric() {
+ Field::new(field.name(), DataType::Float64, true)
+ } else {
+ Field::new(field.name(), DataType::Utf8, true)
+ }
+ })
+ .collect::<Vec<_>>();
+ describe_schemas.insert(0, Field::new("describe", DataType::Utf8, false));
+
+ //collect recordBatch
+ let describe_record_batch = vec![
+ // count aggregation
+ self.clone()
+ .aggregate(
+ vec![],
+ fields_iter
+ .clone()
+ .map(|f| {
+ Expr::Alias(
+ Box::new(datafusion_expr::count(col(f.name()))),
+ f.name().to_string(),
+ )
+ })
+ .collect::<Vec<_>>(),
+ )?
+ .collect()
+ .await?,
+ // null_count aggregation
+ self.clone()
+ .aggregate(
+ vec![],
+ fields_iter
+ .clone()
+ .map(|f| {
+ Expr::Alias(
+ Box::new(datafusion_expr::count(
+ datafusion_expr::is_null(col(f.name())),
+ )),
+ f.name().to_string(),
+ )
+ })
+ .collect::<Vec<_>>(),
+ )?
+ .collect()
+ .await?,
+ // max aggregation
+ self.clone()
+ .aggregate(
+ vec![],
+ fields_iter
+ .clone()
+ .filter(|f| matches!(f.data_type().is_numeric(), true))
Review Comment:
boolean and binary not work with min/max
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb merged pull request #5226: add a describe method on DataFrame like Polars
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #5226:
URL: https://github.com/apache/arrow-datafusion/pull/5226
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jiangzhx commented on a diff in pull request #5226: add a describe method on DataFrame like Polars
Posted by "jiangzhx (via GitHub)" <gi...@apache.org>.
jiangzhx commented on code in PR #5226:
URL: https://github.com/apache/arrow-datafusion/pull/5226#discussion_r1118296004
##########
datafusion/core/src/dataframe.rs:
##########
@@ -302,6 +306,177 @@ impl DataFrame {
))
}
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.describe().await?;
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn describe(self) -> Result<()> {
+ Ok(print_batches(
Review Comment:
thanks , it's great suggestion.
fixed
##########
datafusion/core/src/dataframe.rs:
##########
@@ -302,6 +306,177 @@ impl DataFrame {
))
}
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.describe().await?;
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn describe(self) -> Result<()> {
+ Ok(print_batches(
Review Comment:
thanks , it's a great suggestion.
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jiangzhx commented on pull request #5226: add a describe method on DataFrame like Polars
Posted by "jiangzhx (via GitHub)" <gi...@apache.org>.
jiangzhx commented on PR #5226:
URL: https://github.com/apache/arrow-datafusion/pull/5226#issuecomment-1429816797
make describe method return right datatype for record batch。
<img width="1864" alt="image" src="https://user-images.githubusercontent.com/494507/218763741-a272a244-8c30-480d-8ce5-8145805da949.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jiangzhx commented on a diff in pull request #5226: add a describe method on DataFrame like Polars
Posted by "jiangzhx (via GitHub)" <gi...@apache.org>.
jiangzhx commented on code in PR #5226:
URL: https://github.com/apache/arrow-datafusion/pull/5226#discussion_r1118239192
##########
datafusion/core/src/dataframe.rs:
##########
@@ -302,6 +306,177 @@ impl DataFrame {
))
}
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.describe().await?;
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn describe(self) -> Result<()> {
+ Ok(print_batches(
+ &self.clone().collect_describe().await.unwrap(),
+ )?)
+ }
+
+ /// Summary statistics for a DataFrame. Only summarizes numeric datatypes at the moment and
+ /// returns nulls for non numeric datatypes. Try in keep output similar to pandas
+ ///
+ /// ```
+ /// # use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use arrow::util::pretty;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_csv("tests/tpch-csv/customer.csv", CsvReadOptions::new()).await?;
+ /// df.collect_describe().await.unwrap();
+ ///
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn collect_describe(self) -> Result<Vec<RecordBatch>> {
+ //the functions now supported
+ let supported_describe_functions = vec!["count", "null_count", "max", "min"]; //"count", "max", "min",
+
+ let fields_iter = self.schema().fields().iter();
+
+ //define describe column
+ let mut describe_schemas = fields_iter
+ .clone()
+ .map(|field| {
+ if field.data_type().is_numeric() {
+ Field::new(field.name(), DataType::Float64, true)
+ } else {
+ Field::new(field.name(), DataType::Utf8, true)
+ }
+ })
+ .collect::<Vec<_>>();
+ describe_schemas.insert(0, Field::new("describe", DataType::Utf8, false));
+
+ //collect recordBatch
+ let describe_record_batch = vec![
+ // count aggregation
+ self.clone()
+ .aggregate(
+ vec![],
+ fields_iter
+ .clone()
+ .map(|f| {
+ Expr::Alias(
+ Box::new(datafusion_expr::count(col(f.name()))),
+ f.name().to_string(),
+ )
Review Comment:
thanks, This API is more convenient to use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org