[GitHub] [arrow-datafusion] kmitchener opened a new issue, #2869: window functions not supported in expressions

[GitBox <gi...@apache.org>]

kmitchener opened a new issue, #2869:
URL: https://github.com/apache/arrow-datafusion/issues/2869

   **Describe the bug**
   A clear and concise description of what the bug is.
   
   Was attempting to concisely calculate growth rates using window functions. Queries like this:
   
   ```sql
   select col1, idx, count(*), sum(amount), lag(sum(amount), 1) over (order by idx) as prev_amount, 
   sum(amount) - lag(sum(amount), 1) over (order by idx) as difference from (
   select * from (values ('a', 1, 100), ('a', 2, 150)) as t (col1, idx, amount)
   ) a 
   group by col1, idx;
   ```
   or 
   ```sql
   select 1 - lag(amount, 1) over (order by idx) from (values ('a', 1, 100), ('a', 2, 150)) as t (col1, idx, amount)
   ```
   Results in 
   ```
   NotImplemented("Physical plan does not support logical expression LAG(#SUM(a.amount), Int64(1)) ORDER BY [#a.idx ASC NULLS LAST]")
   ```
   
   **To Reproduce**
   Steps to reproduce the behavior:
   
   use a window function in an expression
   
   **Expected behavior**
   A clear and concise description of what you expected to happen.
   
   **Additional context**
   Add any other context about the problem here.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] dovijoel commented on issue #2869: window functions not supported in expressions

[GitBox <gi...@apache.org>]

dovijoel commented on issue #2869:
URL: https://github.com/apache/arrow-datafusion/issues/2869#issuecomment-1196815138

   I am still having this issue while using the master branch with using the API. Am I doing something incorrectly?
   ```rust
   // create expression for close price with a lead window function
   let close_expr = Expr::WindowFunction {
           fun: WindowFunction::BuiltInWindowFunction(BuiltInWindowFunction::Lead),
           args: vec![col("PX_LAST"), lit(10)],
           partition_by: vec![col("InstrumentID")],
           order_by: vec![Sort {
               expr: Box::new(col("dates")),
               asc: ascending,
               nulls_first: false
           }],
           window_frame: None
       }
   // calculate the return
   let return_expr = (close_expr / col("PX_LAST")) - 1.0;
   let returnsDf = df.select(vec![
           col("dates"),
           col("InstrumentID"),
           col("PX_LAST").alias("open_price"),
           close_expr.alias("close_price"),
           return_expr.alias("return")
       ])?;
   
   dailyReturnsDf.show_limit(100).await?;
   ```
   
   This results in: 
   `Error: NotImplemented("Physical plan does not support logical expression LEAD(#raw_data.PX_LAST, Int64(10)) PARTITION BY [#raw_data.InstrumentID] ORDER BY [#raw_data.dates ASC NULLS LAST]")`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] kmitchener commented on issue #2869: window functions not supported in expressions

[GitBox <gi...@apache.org>]

kmitchener commented on issue #2869:
URL: https://github.com/apache/arrow-datafusion/issues/2869#issuecomment-1188151525

   Thank you @AssHero! This is awesome!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] alamb closed issue #2869: window functions not supported in expressions

[GitBox <gi...@apache.org>]

alamb closed issue #2869: window functions not supported in expressions
URL: https://github.com/apache/arrow-datafusion/issues/2869


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org