You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jonahgao (via GitHub)" <gi...@apache.org> on 2024/08/10 16:30:36 UTC

[PR] Parse Sqllogictest column types from physical schema [datafusion]

jonahgao opened a new pull request, #11929:
URL: https://github.com/apache/datafusion/pull/11929

   ## Which issue does this PR close?
   
   N/A
   
   ## Rationale for this change
   
   Currently, the column types in Sqllogictest are [parsed](https://github.com/apache/datafusion/blob/27304239ef79b50a443320791755bf74eed4a85d/datafusion/sqllogictest/src/engines/datafusion_engine/runner.rs#L73) from the logical plan's schema, which can lead to inconsistencies with the values of the query results. For example
   https://github.com/apache/datafusion/blob/27304239ef79b50a443320791755bf74eed4a85d/datafusion/sqllogictest/test_files/copy.slt#L56-L60
   
   We have 3 column types, but the data has only one column. 
   
   Physical schemas will be closer to the query results.
   
   
   ## What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   ## Are these changes tested?
   Yes
   ## Are there any user-facing changes?
   No
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Parse Sqllogictest column types from physical schema [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on code in PR #11929:
URL: https://github.com/apache/datafusion/pull/11929#discussion_r1713127966


##########
datafusion/sqllogictest/src/engines/datafusion_engine/runner.rs:
##########
@@ -69,9 +71,11 @@ impl sqllogictest::AsyncDB for DataFusion {
 
 async fn run_query(ctx: &SessionContext, sql: impl Into<String>) -> Result<DFOutput> {
     let df = ctx.sql(sql.into().as_str()).await?;
+    let task_ctx = Arc::new(df.task_ctx());
+    let plan = df.create_physical_plan().await?;
 
-    let types = normalize::convert_schema_to_types(df.schema().fields());
-    let results: Vec<RecordBatch> = df.collect().await?;
+    let types = normalize::convert_schema_to_types(plan.schema().fields());
+    let results: Vec<RecordBatch> = collect(plan, task_ctx).await?;

Review Comment:
   It would be better to keep them the same.  Changed to using `SendableRecordBatchStream`. Thanks @alamb 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Parse Sqllogictest column types from physical schema [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #11929:
URL: https://github.com/apache/datafusion/pull/11929#discussion_r1712995933


##########
datafusion/sqllogictest/src/engines/datafusion_engine/runner.rs:
##########
@@ -69,9 +71,11 @@ impl sqllogictest::AsyncDB for DataFusion {
 
 async fn run_query(ctx: &SessionContext, sql: impl Into<String>) -> Result<DFOutput> {
     let df = ctx.sql(sql.into().as_str()).await?;
+    let task_ctx = Arc::new(df.task_ctx());
+    let plan = df.create_physical_plan().await?;
 
-    let types = normalize::convert_schema_to_types(df.schema().fields());
-    let results: Vec<RecordBatch> = df.collect().await?;
+    let types = normalize::convert_schema_to_types(plan.schema().fields());
+    let results: Vec<RecordBatch> = collect(plan, task_ctx).await?;

Review Comment:
   Another thing you could do that might be even closer to the results, would be to call `execute` to get a `SendableRecordBatchStream` and retrieve the schema from that  (e.g `stream.schema()`) which should be exactly the same schema as RecordBatch produced
   
   I think using the physical plan's schema is fine too though and better than what is on main



##########
datafusion/sqllogictest/test_files/copy.slt:
##########
@@ -378,7 +378,7 @@ select * from validate_parquet_with_options;
 2 Bar
 
 # Copy from table to single file
-query IT
+query I

Review Comment:
   that certainly makes more send 👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Parse Sqllogictest column types from physical schema [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on code in PR #11929:
URL: https://github.com/apache/datafusion/pull/11929#discussion_r1712671711


##########
datafusion/sqllogictest/test_files/coalesce.slt:
##########
@@ -23,7 +23,7 @@ select coalesce(1, 2, 3);
 1
 
 # test with first null
-query ?T
+query IT
 select coalesce(null, 3, 2, 1), arrow_typeof(coalesce(null, 3, 2, 1));

Review Comment:
   Same as `arrow_typeof` now.
   ```sh
   > select arrow_typeof(coalesce(null, 3, 2, 1));
   +---------------------------------------------------------+
   | arrow_typeof(coalesce(NULL,Int64(3),Int64(2),Int64(1))) |
   +---------------------------------------------------------+
   | Int64                                                   |
   +---------------------------------------------------------+
   ```
   The datatype of the initial logical plan is incorrect until TypeCoercion is executed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Parse Sqllogictest column types from physical schema [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on PR #11929:
URL: https://github.com/apache/datafusion/pull/11929#issuecomment-2283419507

   Thanks for the review @alamb @jayzhan211 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Parse Sqllogictest column types from physical schema [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao merged PR #11929:
URL: https://github.com/apache/datafusion/pull/11929


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Parse Sqllogictest column types from physical schema [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #11929:
URL: https://github.com/apache/datafusion/pull/11929#discussion_r1713973317


##########
datafusion/sqllogictest/src/engines/datafusion_engine/runner.rs:
##########
@@ -69,9 +72,12 @@ impl sqllogictest::AsyncDB for DataFusion {
 
 async fn run_query(ctx: &SessionContext, sql: impl Into<String>) -> Result<DFOutput> {
     let df = ctx.sql(sql.into().as_str()).await?;
+    let task_ctx = Arc::new(df.task_ctx());
+    let plan = df.create_physical_plan().await?;
 
-    let types = normalize::convert_schema_to_types(df.schema().fields());
-    let results: Vec<RecordBatch> = df.collect().await?;
+    let stream = execute_stream(plan, task_ctx)?;
+    let types = normalize::convert_schema_to_types(stream.schema().fields());

Review Comment:
   👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Parse Sqllogictest column types from physical schema [datafusion]

Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on code in PR #11929:
URL: https://github.com/apache/datafusion/pull/11929#discussion_r1712672999


##########
datafusion/sqllogictest/test_files/aggregate.slt:
##########
@@ -5553,4 +5553,4 @@ drop table employee_csv;
 query I??III?T
 select count(null), min(null), max(null), bit_and(NULL), bit_or(NULL), bit_xor(NULL), nth_value(NULL, 1), string_agg(NULL, ',');
 ----
-0 NULL NULL NULL NULL NULL NULL NULL

Review Comment:
   Complete mode added a newline at the end of the file.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org