You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Veeupup (via GitHub)" <gi...@apache.org> on 2023/12/04 14:01:04 UTC
[PR] Support parquet_metadata for datafusion-cli [arrow-datafusion]
Veeupup opened a new pull request, #8413:
URL: https://github.com/apache/arrow-datafusion/pull/8413
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #8367
make each `RowGroupMetadata` each row to display detailed information for all row groups, just like:
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support parquet_metadata for datafusion-cli [arrow-datafusion]
Posted by "Veeupup (via GitHub)" <gi...@apache.org>.
Veeupup commented on PR #8413:
URL: https://github.com/apache/arrow-datafusion/pull/8413#issuecomment-1842994662
@alamb @simonvandel Hi! I made `parquet_metadata` behave just like `duckdb`!
The above is from duckdb, and the below is from datafusion. You can click on the picture to see more clearly!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support parquet_metadata for datafusion-cli [arrow-datafusion]
Posted by "Veeupup (via GitHub)" <gi...@apache.org>.
Veeupup commented on code in PR #8413:
URL: https://github.com/apache/arrow-datafusion/pull/8413#discussion_r1415325108
##########
datafusion-cli/src/functions.rs:
##########
@@ -196,3 +209,110 @@ pub fn display_all_functions() -> Result<()> {
println!("{}", pretty_format_batches(&[batch]).unwrap());
Ok(())
}
+
+/// PARQUET_META table function
+struct ParquetMetadataTable {
+ schema: SchemaRef,
+ batch: RecordBatch,
+}
+
+#[async_trait]
+impl TableProvider for ParquetMetadataTable {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn schema(&self) -> arrow::datatypes::SchemaRef {
+ self.schema.clone()
+ }
+
+ fn table_type(&self) -> datafusion::logical_expr::TableType {
+ datafusion::logical_expr::TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &SessionState,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ _limit: Option<usize>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ Ok(Arc::new(MemoryExec::try_new(
+ &[vec![self.batch.clone()]],
+ TableProvider::schema(self),
+ projection.cloned(),
+ )?))
+ }
+}
+
+pub struct ParquetMetadataFunc {}
+
+impl TableFunctionImpl for ParquetMetadataFunc {
+ fn call(&self, exprs: &[Expr]) -> Result<Arc<dyn TableProvider>> {
+ let Some(Expr::Column(Column { name, .. })) = exprs.get(0) else {
+ return plan_err!("read_csv requires at least one string argument");
Review Comment:
aha, thank you!
I'll make it right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support parquet_metadata for datafusion-cli [arrow-datafusion]
Posted by "Veeupup (via GitHub)" <gi...@apache.org>.
Veeupup commented on code in PR #8413:
URL: https://github.com/apache/arrow-datafusion/pull/8413#discussion_r1415456066
##########
datafusion-cli/src/functions.rs:
##########
@@ -196,3 +209,110 @@ pub fn display_all_functions() -> Result<()> {
println!("{}", pretty_format_batches(&[batch]).unwrap());
Ok(())
}
+
+/// PARQUET_META table function
+struct ParquetMetadataTable {
+ schema: SchemaRef,
+ batch: RecordBatch,
+}
+
+#[async_trait]
+impl TableProvider for ParquetMetadataTable {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn schema(&self) -> arrow::datatypes::SchemaRef {
+ self.schema.clone()
+ }
+
+ fn table_type(&self) -> datafusion::logical_expr::TableType {
+ datafusion::logical_expr::TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &SessionState,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ _limit: Option<usize>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ Ok(Arc::new(MemoryExec::try_new(
+ &[vec![self.batch.clone()]],
+ TableProvider::schema(self),
+ projection.cloned(),
+ )?))
+ }
+}
+
+pub struct ParquetMetadataFunc {}
+
+impl TableFunctionImpl for ParquetMetadataFunc {
+ fn call(&self, exprs: &[Expr]) -> Result<Arc<dyn TableProvider>> {
+ let Some(Expr::Column(Column { name, .. })) = exprs.get(0) else {
+ return plan_err!("read_csv requires at least one string argument");
+ };
+
+ let file = File::open(name)?;
+ let reader = SerializedFileReader::new(file)?;
+ let metadata = reader.metadata();
+
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("version", DataType::Int32, false),
+ Field::new("num_rows", DataType::Int64, false),
+ Field::new("created_by", DataType::Utf8, false),
+ Field::new("columns_order", DataType::Utf8, false),
+ Field::new("num_row_groups", DataType::Int64, false),
+ Field::new("row_groups", DataType::Utf8, false),
Review Comment:
each row group will be one row (field name should be `row_group` rather than `row_groups`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support parquet_metadata for datafusion-cli [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #8413:
URL: https://github.com/apache/datafusion/pull/8413#issuecomment-2746219812
I'd like to add `column_name` to the output. Would that be okay with folks or are we trying to match DuckDB 1:1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support parquet_metadata for datafusion-cli [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8413:
URL: https://github.com/apache/arrow-datafusion/pull/8413#discussion_r1418071917
##########
datafusion-cli/src/functions.rs:
##########
@@ -196,3 +209,197 @@ pub fn display_all_functions() -> Result<()> {
println!("{}", pretty_format_batches(&[batch]).unwrap());
Ok(())
}
+
+/// PARQUET_META table function
+struct ParquetMetadataTable {
+ schema: SchemaRef,
+ batch: RecordBatch,
+}
+
+#[async_trait]
+impl TableProvider for ParquetMetadataTable {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn schema(&self) -> arrow::datatypes::SchemaRef {
+ self.schema.clone()
+ }
+
+ fn table_type(&self) -> datafusion::logical_expr::TableType {
+ datafusion::logical_expr::TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &SessionState,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ _limit: Option<usize>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ Ok(Arc::new(MemoryExec::try_new(
+ &[vec![self.batch.clone()]],
+ TableProvider::schema(self),
+ projection.cloned(),
+ )?))
+ }
+}
+
+pub struct ParquetMetadataFunc {}
+
+impl TableFunctionImpl for ParquetMetadataFunc {
+ fn call(&self, exprs: &[Expr]) -> Result<Arc<dyn TableProvider>> {
+ let Some(Expr::Column(Column { name, .. })) = exprs.get(0) else {
+ return plan_err!("parquet_metadata requires string argument as its input");
+ };
+
+ let file = File::open(name.clone())?;
+ let reader = SerializedFileReader::new(file)?;
+ let metadata = reader.metadata();
+
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("filename", DataType::Utf8, true),
+ Field::new("row_group_id", DataType::Int64, true),
+ Field::new("row_group_num_rows", DataType::Int64, true),
+ Field::new("row_group_num_columns", DataType::Int64, true),
+ Field::new("row_group_bytes", DataType::Int64, true),
+ Field::new("column_id", DataType::Int64, true),
+ Field::new("file_offset", DataType::Int64, true),
+ Field::new("num_values", DataType::Int64, true),
+ Field::new("path_in_schema", DataType::Utf8, true),
+ Field::new("type", DataType::Utf8, true),
+ Field::new("stats_min", DataType::Utf8, true),
+ Field::new("stats_max", DataType::Utf8, true),
+ Field::new("stats_null_count", DataType::Int64, true),
+ Field::new("stats_distinct_count", DataType::Int64, true),
+ Field::new("stats_min_value", DataType::Utf8, true),
+ Field::new("stats_max_value", DataType::Utf8, true),
+ Field::new("compression", DataType::Utf8, true),
+ Field::new("encodings", DataType::Utf8, true),
+ Field::new("index_page_offset", DataType::Int64, true),
+ Field::new("dictionary_page_offset", DataType::Int64, true),
+ Field::new("data_page_offset", DataType::Int64, true),
+ Field::new("total_compressed_size", DataType::Int64, true),
+ Field::new("total_uncompressed_size", DataType::Int64, true),
+ ]));
+
+ // construct recordbatch from metadata
+ let mut filename_arr = vec![];
+ let mut row_group_id_arr = vec![];
+ let mut row_group_num_rows_arr = vec![];
+ let mut row_group_num_columns_arr = vec![];
+ let mut row_group_bytes_arr = vec![];
+ let mut column_id_arr = vec![];
+ let mut file_offset_arr = vec![];
+ let mut num_values_arr = vec![];
+ let mut path_in_schema_arr = vec![];
+ let mut type_arr = vec![];
+ let mut stats_min_arr = vec![];
+ let mut stats_max_arr = vec![];
+ let mut stats_null_count_arr = vec![];
+ let mut stats_distinct_count_arr = vec![];
+ let mut stats_min_value_arr = vec![];
+ let mut stats_max_value_arr = vec![];
+ let mut compression_arr = vec![];
+ let mut encodings_arr = vec![];
+ let mut index_page_offset_arr = vec![];
+ let mut dictionary_page_offset_arr = vec![];
+ let mut data_page_offset_arr = vec![];
+ let mut total_compressed_size_arr = vec![];
+ let mut total_uncompressed_size_arr = vec![];
+ for (rg_idx, row_group) in metadata.row_groups().iter().enumerate() {
+ for (col_idx, column) in row_group.columns().iter().enumerate() {
+ filename_arr.push(name.clone());
+ row_group_id_arr.push(rg_idx as i64);
+ row_group_num_rows_arr.push(row_group.num_rows());
+ row_group_num_columns_arr.push(row_group.num_columns() as i64);
+ row_group_bytes_arr.push(row_group.total_byte_size());
+ column_id_arr.push(col_idx as i64);
+ file_offset_arr.push(column.file_offset());
+ num_values_arr.push(column.num_values());
+ path_in_schema_arr.push(column.column_path().to_string());
+ type_arr.push(column.column_type().to_string());
+ if let Some(s) = column.statistics() {
Review Comment:
For some reason, I think you also have to check `has_min_max_set` before calling min/max https://docs.rs/parquet/latest/parquet/file/statistics/enum.Statistics.html#method.has_min_max_set
(this is what is causing the panic I think)
##########
datafusion-cli/src/main.rs:
##########
@@ -185,6 +186,8 @@ pub async fn main() -> Result<()> {
ctx.state().catalog_list(),
ctx.state_weak_ref(),
)));
+ // register `parquet_metadata` table function to get metadata from parquet files
+ ctx.register_udtf("parquet_metadata", Arc::new(ParquetMetadataFunc {}));
Review Comment:
🥳
##########
datafusion-cli/src/functions.rs:
##########
@@ -196,3 +209,197 @@ pub fn display_all_functions() -> Result<()> {
println!("{}", pretty_format_batches(&[batch]).unwrap());
Ok(())
}
+
+/// PARQUET_META table function
+struct ParquetMetadataTable {
+ schema: SchemaRef,
+ batch: RecordBatch,
+}
+
+#[async_trait]
+impl TableProvider for ParquetMetadataTable {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn schema(&self) -> arrow::datatypes::SchemaRef {
+ self.schema.clone()
+ }
+
+ fn table_type(&self) -> datafusion::logical_expr::TableType {
+ datafusion::logical_expr::TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &SessionState,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ _limit: Option<usize>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ Ok(Arc::new(MemoryExec::try_new(
+ &[vec![self.batch.clone()]],
+ TableProvider::schema(self),
+ projection.cloned(),
+ )?))
+ }
+}
+
+pub struct ParquetMetadataFunc {}
+
+impl TableFunctionImpl for ParquetMetadataFunc {
+ fn call(&self, exprs: &[Expr]) -> Result<Arc<dyn TableProvider>> {
+ let Some(Expr::Column(Column { name, .. })) = exprs.get(0) else {
Review Comment:
If this also handled `ScalarValue::String` here it would let the `read_metadata('file')` (single quotes `'` work.
##########
datafusion-cli/src/main.rs:
##########
@@ -185,6 +186,8 @@ pub async fn main() -> Result<()> {
ctx.state().catalog_list(),
ctx.state_weak_ref(),
)));
+ // register `parquet_metadata` table function to get metadata from parquet files
+ ctx.register_udtf("parquet_metadata", Arc::new(ParquetMetadataFunc {}));
Review Comment:
🥳
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support parquet_metadata for datafusion-cli [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8413:
URL: https://github.com/apache/arrow-datafusion/pull/8413#issuecomment-1838727996
Thank you @Veeupup -- I will check this out later today!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support parquet_metadata for datafusion-cli [arrow-datafusion]
Posted by "simonvandel (via GitHub)" <gi...@apache.org>.
simonvandel commented on code in PR #8413:
URL: https://github.com/apache/arrow-datafusion/pull/8413#discussion_r1415139549
##########
datafusion-cli/src/functions.rs:
##########
@@ -196,3 +209,110 @@ pub fn display_all_functions() -> Result<()> {
println!("{}", pretty_format_batches(&[batch]).unwrap());
Ok(())
}
+
+/// PARQUET_META table function
+struct ParquetMetadataTable {
+ schema: SchemaRef,
+ batch: RecordBatch,
+}
+
+#[async_trait]
+impl TableProvider for ParquetMetadataTable {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn schema(&self) -> arrow::datatypes::SchemaRef {
+ self.schema.clone()
+ }
+
+ fn table_type(&self) -> datafusion::logical_expr::TableType {
+ datafusion::logical_expr::TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &SessionState,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ _limit: Option<usize>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ Ok(Arc::new(MemoryExec::try_new(
+ &[vec![self.batch.clone()]],
+ TableProvider::schema(self),
+ projection.cloned(),
+ )?))
+ }
+}
+
+pub struct ParquetMetadataFunc {}
+
+impl TableFunctionImpl for ParquetMetadataFunc {
+ fn call(&self, exprs: &[Expr]) -> Result<Arc<dyn TableProvider>> {
+ let Some(Expr::Column(Column { name, .. })) = exprs.get(0) else {
+ return plan_err!("read_csv requires at least one string argument");
Review Comment:
This should be "parquet_metadata", right?
##########
datafusion-cli/src/functions.rs:
##########
@@ -196,3 +209,110 @@ pub fn display_all_functions() -> Result<()> {
println!("{}", pretty_format_batches(&[batch]).unwrap());
Ok(())
}
+
+/// PARQUET_META table function
+struct ParquetMetadataTable {
+ schema: SchemaRef,
+ batch: RecordBatch,
+}
+
+#[async_trait]
+impl TableProvider for ParquetMetadataTable {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn schema(&self) -> arrow::datatypes::SchemaRef {
+ self.schema.clone()
+ }
+
+ fn table_type(&self) -> datafusion::logical_expr::TableType {
+ datafusion::logical_expr::TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &SessionState,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ _limit: Option<usize>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ Ok(Arc::new(MemoryExec::try_new(
+ &[vec![self.batch.clone()]],
+ TableProvider::schema(self),
+ projection.cloned(),
+ )?))
+ }
+}
+
+pub struct ParquetMetadataFunc {}
+
+impl TableFunctionImpl for ParquetMetadataFunc {
+ fn call(&self, exprs: &[Expr]) -> Result<Arc<dyn TableProvider>> {
+ let Some(Expr::Column(Column { name, .. })) = exprs.get(0) else {
+ return plan_err!("read_csv requires at least one string argument");
+ };
+
+ let file = File::open(name)?;
+ let reader = SerializedFileReader::new(file)?;
Review Comment:
Is SerializedFileReader buffering reads? Otherwise, would wrapping a https://doc.rust-lang.org/std/io/struct.BufReader.html be a good idea here?
##########
datafusion-cli/src/functions.rs:
##########
@@ -196,3 +209,110 @@ pub fn display_all_functions() -> Result<()> {
println!("{}", pretty_format_batches(&[batch]).unwrap());
Ok(())
}
+
+/// PARQUET_META table function
+struct ParquetMetadataTable {
+ schema: SchemaRef,
+ batch: RecordBatch,
+}
+
+#[async_trait]
+impl TableProvider for ParquetMetadataTable {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn schema(&self) -> arrow::datatypes::SchemaRef {
+ self.schema.clone()
+ }
+
+ fn table_type(&self) -> datafusion::logical_expr::TableType {
+ datafusion::logical_expr::TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &SessionState,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ _limit: Option<usize>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ Ok(Arc::new(MemoryExec::try_new(
+ &[vec![self.batch.clone()]],
+ TableProvider::schema(self),
+ projection.cloned(),
+ )?))
+ }
+}
+
+pub struct ParquetMetadataFunc {}
+
+impl TableFunctionImpl for ParquetMetadataFunc {
+ fn call(&self, exprs: &[Expr]) -> Result<Arc<dyn TableProvider>> {
+ let Some(Expr::Column(Column { name, .. })) = exprs.get(0) else {
+ return plan_err!("read_csv requires at least one string argument");
+ };
+
+ let file = File::open(name)?;
+ let reader = SerializedFileReader::new(file)?;
+ let metadata = reader.metadata();
+
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("version", DataType::Int32, false),
+ Field::new("num_rows", DataType::Int64, false),
+ Field::new("created_by", DataType::Utf8, false),
+ Field::new("columns_order", DataType::Utf8, false),
+ Field::new("num_row_groups", DataType::Int64, false),
+ Field::new("row_groups", DataType::Utf8, false),
Review Comment:
Perhaps this field being a list of structs would be more precise?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support parquet_metadata for datafusion-cli [arrow-datafusion]
Posted by "simonvandel (via GitHub)" <gi...@apache.org>.
simonvandel commented on code in PR #8413:
URL: https://github.com/apache/arrow-datafusion/pull/8413#discussion_r1415401513
##########
datafusion-cli/src/functions.rs:
##########
@@ -196,3 +209,110 @@ pub fn display_all_functions() -> Result<()> {
println!("{}", pretty_format_batches(&[batch]).unwrap());
Ok(())
}
+
+/// PARQUET_META table function
+struct ParquetMetadataTable {
+ schema: SchemaRef,
+ batch: RecordBatch,
+}
+
+#[async_trait]
+impl TableProvider for ParquetMetadataTable {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn schema(&self) -> arrow::datatypes::SchemaRef {
+ self.schema.clone()
+ }
+
+ fn table_type(&self) -> datafusion::logical_expr::TableType {
+ datafusion::logical_expr::TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &SessionState,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ _limit: Option<usize>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ Ok(Arc::new(MemoryExec::try_new(
+ &[vec![self.batch.clone()]],
+ TableProvider::schema(self),
+ projection.cloned(),
+ )?))
+ }
+}
+
+pub struct ParquetMetadataFunc {}
+
+impl TableFunctionImpl for ParquetMetadataFunc {
+ fn call(&self, exprs: &[Expr]) -> Result<Arc<dyn TableProvider>> {
+ let Some(Expr::Column(Column { name, .. })) = exprs.get(0) else {
+ return plan_err!("read_csv requires at least one string argument");
+ };
+
+ let file = File::open(name)?;
+ let reader = SerializedFileReader::new(file)?;
+ let metadata = reader.metadata();
+
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("version", DataType::Int32, false),
+ Field::new("num_rows", DataType::Int64, false),
+ Field::new("created_by", DataType::Utf8, false),
+ Field::new("columns_order", DataType::Utf8, false),
+ Field::new("num_row_groups", DataType::Int64, false),
+ Field::new("row_groups", DataType::Utf8, false),
Review Comment:
If there are many row_groups, wouldn't the single String also fill up the single cell?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support parquet_metadata for datafusion-cli [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #8413:
URL: https://github.com/apache/arrow-datafusion/pull/8413
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support parquet_metadata for datafusion-cli [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8413:
URL: https://github.com/apache/arrow-datafusion/pull/8413#issuecomment-1846193880
This feature was so cool I started using it, and I found one suggestion for improvement already: https://github.com/apache/arrow-datafusion/issues/8464
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support parquet_metadata for datafusion-cli [arrow-datafusion]
Posted by "Veeupup (via GitHub)" <gi...@apache.org>.
Veeupup commented on code in PR #8413:
URL: https://github.com/apache/arrow-datafusion/pull/8413#discussion_r1415338241
##########
datafusion-cli/src/functions.rs:
##########
@@ -196,3 +209,110 @@ pub fn display_all_functions() -> Result<()> {
println!("{}", pretty_format_batches(&[batch]).unwrap());
Ok(())
}
+
+/// PARQUET_META table function
+struct ParquetMetadataTable {
+ schema: SchemaRef,
+ batch: RecordBatch,
+}
+
+#[async_trait]
+impl TableProvider for ParquetMetadataTable {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn schema(&self) -> arrow::datatypes::SchemaRef {
+ self.schema.clone()
+ }
+
+ fn table_type(&self) -> datafusion::logical_expr::TableType {
+ datafusion::logical_expr::TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &SessionState,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ _limit: Option<usize>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ Ok(Arc::new(MemoryExec::try_new(
+ &[vec![self.batch.clone()]],
+ TableProvider::schema(self),
+ projection.cloned(),
+ )?))
+ }
+}
+
+pub struct ParquetMetadataFunc {}
+
+impl TableFunctionImpl for ParquetMetadataFunc {
+ fn call(&self, exprs: &[Expr]) -> Result<Arc<dyn TableProvider>> {
+ let Some(Expr::Column(Column { name, .. })) = exprs.get(0) else {
+ return plan_err!("read_csv requires at least one string argument");
+ };
+
+ let file = File::open(name)?;
+ let reader = SerializedFileReader::new(file)?;
+ let metadata = reader.metadata();
+
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("version", DataType::Int32, false),
+ Field::new("num_rows", DataType::Int64, false),
+ Field::new("created_by", DataType::Utf8, false),
+ Field::new("columns_order", DataType::Utf8, false),
+ Field::new("num_row_groups", DataType::Int64, false),
+ Field::new("row_groups", DataType::Utf8, false),
Review Comment:
Yeah, agree with you! a list of structs would be more precise! I implement the first version schema using `List(Struct)`.
But if the list contains lots of row_groups (100+ or more), then all 100+ rows groups will be displayed in one cell.
Then the display may look not so user-friendly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support parquet_metadata for datafusion-cli [arrow-datafusion]
Posted by "Veeupup (via GitHub)" <gi...@apache.org>.
Veeupup commented on PR #8413:
URL: https://github.com/apache/arrow-datafusion/pull/8413#issuecomment-1838702265
@alamb PTAL : )
how do you like the display format?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support parquet_metadata for datafusion-cli [arrow-datafusion]
Posted by "Veeupup (via GitHub)" <gi...@apache.org>.
Veeupup commented on code in PR #8413:
URL: https://github.com/apache/arrow-datafusion/pull/8413#discussion_r1418989159
##########
datafusion-cli/src/functions.rs:
##########
@@ -196,3 +209,197 @@ pub fn display_all_functions() -> Result<()> {
println!("{}", pretty_format_batches(&[batch]).unwrap());
Ok(())
}
+
+/// PARQUET_META table function
+struct ParquetMetadataTable {
+ schema: SchemaRef,
+ batch: RecordBatch,
+}
+
+#[async_trait]
+impl TableProvider for ParquetMetadataTable {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn schema(&self) -> arrow::datatypes::SchemaRef {
+ self.schema.clone()
+ }
+
+ fn table_type(&self) -> datafusion::logical_expr::TableType {
+ datafusion::logical_expr::TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &SessionState,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ _limit: Option<usize>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ Ok(Arc::new(MemoryExec::try_new(
+ &[vec![self.batch.clone()]],
+ TableProvider::schema(self),
+ projection.cloned(),
+ )?))
+ }
+}
+
+pub struct ParquetMetadataFunc {}
+
+impl TableFunctionImpl for ParquetMetadataFunc {
+ fn call(&self, exprs: &[Expr]) -> Result<Arc<dyn TableProvider>> {
+ let Some(Expr::Column(Column { name, .. })) = exprs.get(0) else {
+ return plan_err!("parquet_metadata requires string argument as its input");
+ };
+
+ let file = File::open(name.clone())?;
+ let reader = SerializedFileReader::new(file)?;
+ let metadata = reader.metadata();
+
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("filename", DataType::Utf8, true),
+ Field::new("row_group_id", DataType::Int64, true),
+ Field::new("row_group_num_rows", DataType::Int64, true),
+ Field::new("row_group_num_columns", DataType::Int64, true),
+ Field::new("row_group_bytes", DataType::Int64, true),
+ Field::new("column_id", DataType::Int64, true),
+ Field::new("file_offset", DataType::Int64, true),
+ Field::new("num_values", DataType::Int64, true),
+ Field::new("path_in_schema", DataType::Utf8, true),
+ Field::new("type", DataType::Utf8, true),
+ Field::new("stats_min", DataType::Utf8, true),
+ Field::new("stats_max", DataType::Utf8, true),
+ Field::new("stats_null_count", DataType::Int64, true),
+ Field::new("stats_distinct_count", DataType::Int64, true),
+ Field::new("stats_min_value", DataType::Utf8, true),
+ Field::new("stats_max_value", DataType::Utf8, true),
+ Field::new("compression", DataType::Utf8, true),
+ Field::new("encodings", DataType::Utf8, true),
+ Field::new("index_page_offset", DataType::Int64, true),
+ Field::new("dictionary_page_offset", DataType::Int64, true),
+ Field::new("data_page_offset", DataType::Int64, true),
+ Field::new("total_compressed_size", DataType::Int64, true),
+ Field::new("total_uncompressed_size", DataType::Int64, true),
+ ]));
+
+ // construct recordbatch from metadata
+ let mut filename_arr = vec![];
+ let mut row_group_id_arr = vec![];
+ let mut row_group_num_rows_arr = vec![];
+ let mut row_group_num_columns_arr = vec![];
+ let mut row_group_bytes_arr = vec![];
+ let mut column_id_arr = vec![];
+ let mut file_offset_arr = vec![];
+ let mut num_values_arr = vec![];
+ let mut path_in_schema_arr = vec![];
+ let mut type_arr = vec![];
+ let mut stats_min_arr = vec![];
+ let mut stats_max_arr = vec![];
+ let mut stats_null_count_arr = vec![];
+ let mut stats_distinct_count_arr = vec![];
+ let mut stats_min_value_arr = vec![];
+ let mut stats_max_value_arr = vec![];
+ let mut compression_arr = vec![];
+ let mut encodings_arr = vec![];
+ let mut index_page_offset_arr = vec![];
+ let mut dictionary_page_offset_arr = vec![];
+ let mut data_page_offset_arr = vec![];
+ let mut total_compressed_size_arr = vec![];
+ let mut total_uncompressed_size_arr = vec![];
+ for (rg_idx, row_group) in metadata.row_groups().iter().enumerate() {
+ for (col_idx, column) in row_group.columns().iter().enumerate() {
+ filename_arr.push(name.clone());
+ row_group_id_arr.push(rg_idx as i64);
+ row_group_num_rows_arr.push(row_group.num_rows());
+ row_group_num_columns_arr.push(row_group.num_columns() as i64);
+ row_group_bytes_arr.push(row_group.total_byte_size());
+ column_id_arr.push(col_idx as i64);
+ file_offset_arr.push(column.file_offset());
+ num_values_arr.push(column.num_values());
+ path_in_schema_arr.push(column.column_path().to_string());
+ type_arr.push(column.column_type().to_string());
+ if let Some(s) = column.statistics() {
Review Comment:
fixed! thank you~ I should be more careful next time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support parquet_metadata for datafusion-cli [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8413:
URL: https://github.com/apache/datafusion/pull/8413#issuecomment-2746383644
> I'd like to add `column_name` to the output. Would that be okay with folks or are we trying to match DuckDB 1:1?
I don't think we need to match DuckDB 1:1
BTW you have probably seen https://parquet-viewer.xiangpeng.systems/ which is also a cool way to visualize this data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support parquet_metadata for datafusion-cli [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8413:
URL: https://github.com/apache/arrow-datafusion/pull/8413#discussion_r1416268467
##########
datafusion-cli/src/functions.rs:
##########
@@ -196,3 +209,110 @@ pub fn display_all_functions() -> Result<()> {
println!("{}", pretty_format_batches(&[batch]).unwrap());
Ok(())
}
+
+/// PARQUET_META table function
+struct ParquetMetadataTable {
+ schema: SchemaRef,
+ batch: RecordBatch,
+}
+
+#[async_trait]
+impl TableProvider for ParquetMetadataTable {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn schema(&self) -> arrow::datatypes::SchemaRef {
+ self.schema.clone()
+ }
+
+ fn table_type(&self) -> datafusion::logical_expr::TableType {
+ datafusion::logical_expr::TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &SessionState,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ _limit: Option<usize>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ Ok(Arc::new(MemoryExec::try_new(
+ &[vec![self.batch.clone()]],
+ TableProvider::schema(self),
+ projection.cloned(),
+ )?))
+ }
+}
+
+pub struct ParquetMetadataFunc {}
+
+impl TableFunctionImpl for ParquetMetadataFunc {
+ fn call(&self, exprs: &[Expr]) -> Result<Arc<dyn TableProvider>> {
+ let Some(Expr::Column(Column { name, .. })) = exprs.get(0) else {
+ return plan_err!("read_csv requires at least one string argument");
+ };
+
+ let file = File::open(name)?;
+ let reader = SerializedFileReader::new(file)?;
+ let metadata = reader.metadata();
+
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("version", DataType::Int32, false),
+ Field::new("num_rows", DataType::Int64, false),
+ Field::new("created_by", DataType::Utf8, false),
+ Field::new("columns_order", DataType::Utf8, false),
+ Field::new("num_row_groups", DataType::Int64, false),
+ Field::new("row_groups", DataType::Utf8, false),
+ ]));
+
+ // construct recordbatch from metadata
+ let num_groups = metadata.num_row_groups();
+ let row_groups = metadata
+ .row_groups()
+ .iter()
+ .map(|rg| {
+ format!(
+ "num_columns: {}, num_rows: {}, total_byte_size: {}, sorting_columns: {:?}",
Review Comment:
every row group should have the same number of columns, for what it is worth
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support parquet_metadata for datafusion-cli [arrow-datafusion]
Posted by "Veeupup (via GitHub)" <gi...@apache.org>.
Veeupup commented on code in PR #8413:
URL: https://github.com/apache/arrow-datafusion/pull/8413#discussion_r1417291602
##########
datafusion-cli/src/functions.rs:
##########
@@ -196,3 +209,110 @@ pub fn display_all_functions() -> Result<()> {
println!("{}", pretty_format_batches(&[batch]).unwrap());
Ok(())
}
+
+/// PARQUET_META table function
+struct ParquetMetadataTable {
+ schema: SchemaRef,
+ batch: RecordBatch,
+}
+
+#[async_trait]
+impl TableProvider for ParquetMetadataTable {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn schema(&self) -> arrow::datatypes::SchemaRef {
+ self.schema.clone()
+ }
+
+ fn table_type(&self) -> datafusion::logical_expr::TableType {
+ datafusion::logical_expr::TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &SessionState,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ _limit: Option<usize>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ Ok(Arc::new(MemoryExec::try_new(
+ &[vec![self.batch.clone()]],
+ TableProvider::schema(self),
+ projection.cloned(),
+ )?))
+ }
+}
+
+pub struct ParquetMetadataFunc {}
+
+impl TableFunctionImpl for ParquetMetadataFunc {
+ fn call(&self, exprs: &[Expr]) -> Result<Arc<dyn TableProvider>> {
+ let Some(Expr::Column(Column { name, .. })) = exprs.get(0) else {
+ return plan_err!("read_csv requires at least one string argument");
+ };
+
+ let file = File::open(name)?;
+ let reader = SerializedFileReader::new(file)?;
+ let metadata = reader.metadata();
+
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("version", DataType::Int32, false),
+ Field::new("num_rows", DataType::Int64, false),
+ Field::new("created_by", DataType::Utf8, false),
+ Field::new("columns_order", DataType::Utf8, false),
+ Field::new("num_row_groups", DataType::Int64, false),
+ Field::new("row_groups", DataType::Utf8, false),
+ ]));
+
+ // construct recordbatch from metadata
+ let num_groups = metadata.num_row_groups();
+ let row_groups = metadata
+ .row_groups()
+ .iter()
+ .map(|rg| {
+ format!(
+ "num_columns: {}, num_rows: {}, total_byte_size: {}, sorting_columns: {:?}",
Review Comment:
it seems that each column in each row_group will be one row in duckdb, so there will be `Num(cols) * Num(row_group)` rows
I'll just mock the duckdb display format then ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support parquet_metadata for datafusion-cli [arrow-datafusion]
Posted by "Veeupup (via GitHub)" <gi...@apache.org>.
Veeupup commented on code in PR #8413:
URL: https://github.com/apache/arrow-datafusion/pull/8413#discussion_r1418987291
##########
datafusion-cli/src/functions.rs:
##########
@@ -196,3 +209,197 @@ pub fn display_all_functions() -> Result<()> {
println!("{}", pretty_format_batches(&[batch]).unwrap());
Ok(())
}
+
+/// PARQUET_META table function
+struct ParquetMetadataTable {
+ schema: SchemaRef,
+ batch: RecordBatch,
+}
+
+#[async_trait]
+impl TableProvider for ParquetMetadataTable {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn schema(&self) -> arrow::datatypes::SchemaRef {
+ self.schema.clone()
+ }
+
+ fn table_type(&self) -> datafusion::logical_expr::TableType {
+ datafusion::logical_expr::TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &SessionState,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ _limit: Option<usize>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ Ok(Arc::new(MemoryExec::try_new(
+ &[vec![self.batch.clone()]],
+ TableProvider::schema(self),
+ projection.cloned(),
+ )?))
+ }
+}
+
+pub struct ParquetMetadataFunc {}
+
+impl TableFunctionImpl for ParquetMetadataFunc {
+ fn call(&self, exprs: &[Expr]) -> Result<Arc<dyn TableProvider>> {
+ let Some(Expr::Column(Column { name, .. })) = exprs.get(0) else {
Review Comment:
thanks! I have fixed it! : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support parquet_metadata for datafusion-cli [arrow-datafusion]
Posted by "Veeupup (via GitHub)" <gi...@apache.org>.
Veeupup commented on code in PR #8413:
URL: https://github.com/apache/arrow-datafusion/pull/8413#discussion_r1415456066
##########
datafusion-cli/src/functions.rs:
##########
@@ -196,3 +209,110 @@ pub fn display_all_functions() -> Result<()> {
println!("{}", pretty_format_batches(&[batch]).unwrap());
Ok(())
}
+
+/// PARQUET_META table function
+struct ParquetMetadataTable {
+ schema: SchemaRef,
+ batch: RecordBatch,
+}
+
+#[async_trait]
+impl TableProvider for ParquetMetadataTable {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn schema(&self) -> arrow::datatypes::SchemaRef {
+ self.schema.clone()
+ }
+
+ fn table_type(&self) -> datafusion::logical_expr::TableType {
+ datafusion::logical_expr::TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ _state: &SessionState,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ _limit: Option<usize>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ Ok(Arc::new(MemoryExec::try_new(
+ &[vec![self.batch.clone()]],
+ TableProvider::schema(self),
+ projection.cloned(),
+ )?))
+ }
+}
+
+pub struct ParquetMetadataFunc {}
+
+impl TableFunctionImpl for ParquetMetadataFunc {
+ fn call(&self, exprs: &[Expr]) -> Result<Arc<dyn TableProvider>> {
+ let Some(Expr::Column(Column { name, .. })) = exprs.get(0) else {
+ return plan_err!("read_csv requires at least one string argument");
+ };
+
+ let file = File::open(name)?;
+ let reader = SerializedFileReader::new(file)?;
+ let metadata = reader.metadata();
+
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("version", DataType::Int32, false),
+ Field::new("num_rows", DataType::Int64, false),
+ Field::new("created_by", DataType::Utf8, false),
+ Field::new("columns_order", DataType::Utf8, false),
+ Field::new("num_row_groups", DataType::Int64, false),
+ Field::new("row_groups", DataType::Utf8, false),
Review Comment:
each row group will be one row
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support parquet_metadata for datafusion-cli [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #8413:
URL: https://github.com/apache/datafusion/pull/8413#issuecomment-2746528066
I did not know of it, that's great!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org