You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "rkrishn7 (via GitHub)" <gi...@apache.org> on 2025/02/07 08:33:58 UTC

[PR] feat: Implement UNION ALL BY NAME [datafusion]

rkrishn7 opened a new pull request, #14538:
URL: https://github.com/apache/datafusion/pull/14538

   ## Which issue does this PR close?
   
   - Closes #14508
   
   ## What changes are included in this PR?
   
   UNION BY NAME implementation and associated tests
   
   ## Are these changes tested?
   
   Yes
   
   ## Are there any user-facing changes?
   
   Yes, no breaking changes, but new functionality.
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement UNION ALL BY NAME [datafusion]

Posted by "rkrishn7 (via GitHub)" <gi...@apache.org>.
rkrishn7 commented on code in PR #14538:
URL: https://github.com/apache/datafusion/pull/14538#discussion_r1948419349


##########
datafusion/sqllogictest/test_files/union_by_name.slt:
##########
@@ -0,0 +1,264 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+
+#   http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+statement ok
+CREATE TABLE t1 (x INT, y INT);
+
+statement ok
+INSERT INTO t1 VALUES (3, 3), (3, 3), (1, 1);
+
+statement ok
+CREATE TABLE t2 (y INT, z INT);
+
+statement ok
+INSERT INTO t2 VALUES (2, 2), (4, 4);
+
+
+# Test binding
+query I
+SELECT t1.x FROM t1 UNION BY NAME SELECT x FROM t1 ORDER BY t1.x;
+----
+1
+3
+
+query I
+SELECT t1.x FROM t1 UNION ALL BY NAME SELECT x FROM t1 ORDER BY t1.x;
+----
+1
+1
+3
+3
+3
+3
+
+query I
+SELECT x FROM t1 UNION BY NAME SELECT x FROM t1 ORDER BY t1.x;
+----
+1
+3
+
+query I
+SELECT x FROM t1 UNION ALL BY NAME SELECT x FROM t1 ORDER BY t1.x;
+----
+1
+1
+3
+3
+3
+3
+
+query II
+(SELECT x FROM t1 UNION ALL SELECT x FROM t1) UNION BY NAME SELECT 5 ORDER BY x;
+----
+NULL 1
+NULL 3
+5 NULL
+
+# TODO: This should pass, but the sanity checker isn't allowing it.

Review Comment:
   A few of the `UNION ALL BY NAME` tests are failing during the sanity checker. Not completely sure why atm 🤔 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement UNION ALL BY NAME [datafusion]

Posted by "rkrishn7 (via GitHub)" <gi...@apache.org>.
rkrishn7 commented on PR #14538:
URL: https://github.com/apache/datafusion/pull/14538#issuecomment-2661623965

   > Thank you @rkrishn7, I really like this PR. I'm just curios about why the SanityChecker does not allow those queries. I guess it should be related with equivalence reset during UnionExec's (perhaps we should sustain some info under some conditions). However, it's not a problem coming along with this PR, and I think it shouldn't be a blocker.
   
   Thanks for the review @berkaysynnada!
   
   I was thinking I can open up an issue to fix the failing tests, and leave that as a follow-up once this is merged. Let me know if that sounds good to you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement UNION ALL BY NAME [datafusion]

Posted by "berkaysynnada (via GitHub)" <gi...@apache.org>.
berkaysynnada commented on PR #14538:
URL: https://github.com/apache/datafusion/pull/14538#issuecomment-2653402466

   I'll take a look ASAP, but any additional reviewers are welcomed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement UNION ALL BY NAME [datafusion]

Posted by "berkaysynnada (via GitHub)" <gi...@apache.org>.
berkaysynnada merged PR #14538:
URL: https://github.com/apache/datafusion/pull/14538


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement UNION ALL BY NAME [datafusion]

Posted by "rkrishn7 (via GitHub)" <gi...@apache.org>.
rkrishn7 commented on code in PR #14538:
URL: https://github.com/apache/datafusion/pull/14538#discussion_r1951574316


##########
datafusion/sqllogictest/test_files/union_by_name.slt:
##########
@@ -0,0 +1,264 @@
+# Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   Thank you for catching that @Omega359. Can I include the notice there in conjunction with this header?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement UNION ALL BY NAME [datafusion]

Posted by "berkaysynnada (via GitHub)" <gi...@apache.org>.
berkaysynnada commented on code in PR #14538:
URL: https://github.com/apache/datafusion/pull/14538#discussion_r1957407828


##########
datafusion/sqllogictest/test_files/union_by_name.slt:
##########
@@ -0,0 +1,264 @@
+# Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   If there is anything to be done about this, let's do it before the merge



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement UNION ALL BY NAME [datafusion]

Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #14538:
URL: https://github.com/apache/datafusion/pull/14538#discussion_r1957421115


##########
datafusion/sqllogictest/test_files/union_by_name.slt:
##########
@@ -0,0 +1,264 @@
+# Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   I think having both should be ok @rkrishn7 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement UNION ALL BY NAME [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #14538:
URL: https://github.com/apache/datafusion/pull/14538#issuecomment-2648118134

   FYI @berkaysynnada  and @ozankabak 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement UNION ALL BY NAME [datafusion]

Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #14538:
URL: https://github.com/apache/datafusion/pull/14538#discussion_r1949980327


##########
datafusion/sqllogictest/test_files/union_by_name.slt:
##########
@@ -0,0 +1,264 @@
+# Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   DuckDB to the best of my knowledge is [*NOT* Apache licensed.](https://github.com/duckdb/duckdb/blob/main/LICENSE). Any code or tests taken from them should be licensed appropriately.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement UNION ALL BY NAME [datafusion]

Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on PR #14538:
URL: https://github.com/apache/datafusion/pull/14538#issuecomment-2649370553

   I ran the sqlite sqllogictests against your branch and it passed so none of those files covered union (all) by name


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement UNION ALL BY NAME [datafusion]

Posted by "rkrishn7 (via GitHub)" <gi...@apache.org>.
rkrishn7 commented on code in PR #14538:
URL: https://github.com/apache/datafusion/pull/14538#discussion_r1948419729


##########
datafusion/sqllogictest/test_files/union_by_name.slt:
##########
@@ -0,0 +1,264 @@
+# Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   Most tests here were taken from DuckDB's SLT suite!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement UNION ALL BY NAME [datafusion]

Posted by "berkaysynnada (via GitHub)" <gi...@apache.org>.
berkaysynnada commented on PR #14538:
URL: https://github.com/apache/datafusion/pull/14538#issuecomment-2662269400

   > I was thinking I can open up an issue to fix the failing tests due to the sanity checker, and leave that as a follow-up once this is merged. Let me know if that sounds good to you.
   
   Sounds great, let's do that
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement UNION ALL BY NAME [datafusion]

Posted by "berkaysynnada (via GitHub)" <gi...@apache.org>.
berkaysynnada commented on PR #14538:
URL: https://github.com/apache/datafusion/pull/14538#issuecomment-2661604041

   > I ran the sqlite sqllogictests against your branch and it passed so none of those files covered union (all) by name
   
   sorry @Omega359, I couldn't quite get what you mean. Do we expect a failure?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement UNION ALL BY NAME [datafusion]

Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on PR #14538:
URL: https://github.com/apache/datafusion/pull/14538#issuecomment-2661630680

   > > I ran the sqlite sqllogictests against your branch and it passed so none of those files covered union (all) by name
   > 
   > sorry @Omega359, I couldn't quite get what you mean. Do we expect a failure?
   
   No, it just means that this union isn't covered at all with sqlite tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement UNION ALL BY NAME [datafusion]

Posted by "rkrishn7 (via GitHub)" <gi...@apache.org>.
rkrishn7 commented on code in PR #14538:
URL: https://github.com/apache/datafusion/pull/14538#discussion_r1957518449


##########
datafusion/sqllogictest/test_files/union_by_name.slt:
##########
@@ -0,0 +1,264 @@
+# Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   Done @Omega359



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org