You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "comphead (via GitHub)" <gi...@apache.org> on 2024/02/23 00:59:14 UTC

[I] LAG window function schema issue. Non null conflict. [arrow-datafusion]

comphead opened a new issue, #9320:
URL: https://github.com/apache/arrow-datafusion/issues/9320

   ### Describe the bug
   
   ```
   select lag(a) over () as x1
           from 
           (select 2 id, 'b' a union all select 1 id, null a union all select 3 id, null);
   
   Arrow error: Invalid argument error: Column 'a' is declared as non-nullable but contains null values
   ```
   
   ### To Reproduce
   
   Above
   
   ### Expected behavior
   
   Query should work
   
   ### Additional context
   
   We faced this issue not once, latest is https://github.com/apache/arrow-datafusion/issues/4828
   Found when working on https://github.com/apache/arrow-datafusion/pull/9221


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] LAG window function schema issue. Non null conflict. [arrow-datafusion]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #9320:
URL: https://github.com/apache/arrow-datafusion/issues/9320#issuecomment-1984963256

   Weird thing, it might be a bigger problem behind
   
   ```
   DataFusion CLI v36.0.0
   ❯ select lag(a) over () as x1
           from 
           (select 2 id, 'b' a union all select 1 id, null a union all select 3 id, null);
   Arrow error: Invalid argument error: Column 'a' is declared as non-nullable but contains null values
   ❯ select lag(a) over () as x1
           from 
           (select 2 id, 'b' a union all select 1 id, null a union all select 3 id, null);
   Arrow error: Invalid argument error: Column 'a' is declared as non-nullable but contains null values
   ❯ select lag(a) over () as x1
           from 
           (select 2 id, 'b' a union all select 1 id, null a union all select 3 id, null);
   +----+
   | x1 |
   +----+
   |    |
   | b  |
   |    |
   +----+
   3 rows in set. Query took 0.013 seconds.
   ```
   
   Will dive into it really quick


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] LAG window function schema issue. Non null conflict. [arrow-datafusion]

Posted by "mustafasrepo (via GitHub)" <gi...@apache.org>.
mustafasrepo commented on issue #9320:
URL: https://github.com/apache/arrow-datafusion/issues/9320#issuecomment-1960869179

   When I execute this query. I recognized, that sometimes it works, sometimes it fails. It seems like a strange bug. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] LAG window function schema issue. Non null conflict. [arrow-datafusion]

Posted by "mustafasrepo (via GitHub)" <gi...@apache.org>.
mustafasrepo commented on issue #9320:
URL: https://github.com/apache/arrow-datafusion/issues/9320#issuecomment-1985769243

   @comphead, I debugged this behaviour also. It seems that during concatenation in the `BoundedWindowAggExec`. We were effectively using the `schema` of the first batch in the partition (The schema of the first batch may not reflect the worst case scenario. As in this example, first batch may come from one of the input branches of the `union` with `is_nullable=false`). I have filed the [PR9508](https://github.com/apache/arrow-datafusion/pull/9508) to fix this problem. 
   
   If you have time, I would appreciate your review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] LAG window function schema issue. Non null conflict. [arrow-datafusion]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead closed issue #9320: LAG window function schema issue. Non null conflict.
URL: https://github.com/apache/arrow-datafusion/issues/9320


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org