You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "timsaucer (via GitHub)" <gi...@apache.org> on 2025/10/15 13:50:34 UTC
[I] Expand use of sql parsing string expressions in DataFrame [datafusion-python]
timsaucer opened a new issue, #1278:
URL: https://github.com/apache/datafusion-python/issues/1278
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
This is a follow on to https://github.com/apache/datafusion-python/issues/1273
There are a number of places where it would be convenient to pass SQL strings as expressions. For example, it would be nice to do
```
df.select(
"a",
"a - b",
col("c"),
)
```
This should intuitively know that we are getting column `a`, followed by `col("a") - col("b")` followed by column 'c'.
**Describe the solution you'd like**
Using the sql parsing on the DataFrame make the following functions handle SQL strings. We must be *very* careful that we do not break things like cases where users have a column name that is not SQL parseable.
**Describe alternatives you've considered**
Status quo
**Additional context**
DataFrame functions to update:
- [ ] select
- [ ] remove `select_exprs`
- [ ] with_column
- [ ] with_columns
- [ ] aggregate
- [ ] repartition_by_hash
We do **not** want to apply this treatment to joins because there is no easy way to know *which* DataFrame to perform the SQL parsing against.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expand use of sql parsing string expressions in DataFrame [datafusion-python]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #1278:
URL: https://github.com/apache/datafusion-python/issues/1278#issuecomment-3407425098
I have no strong opinion Tim, I'm split between keeping it as it is and change it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expand use of sql parsing string expressions in DataFrame [datafusion-python]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #1278:
URL: https://github.com/apache/datafusion-python/issues/1278#issuecomment-3423430457
if we do not have special handle for column names I would agree with you, one select may make more sense. If we have to check for column name before we `parse_sql` I believe it would be simpler from user perspective to leave them as they are. Anyway leats get consensus
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expand use of sql parsing string expressions in DataFrame [datafusion-python]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #1278:
URL: https://github.com/apache/datafusion-python/issues/1278#issuecomment-3429365793
`aggregate` may have the same issue like `select` as it expects column name as a string, I'll leave those two to you @timsaucer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expand use of sql parsing string expressions in DataFrame [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #1278:
URL: https://github.com/apache/datafusion-python/issues/1278#issuecomment-3423415082
I can see how it might be preferable to some to have it split out, but I also feel like just having one `select` is very ergonomic. Asking for another opinion from some users I know @ntjohnson1 @MrPowers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expand use of sql parsing string expressions in DataFrame [datafusion-python]
Posted by "ntjohnson1 (via GitHub)" <gi...@apache.org>.
ntjohnson1 commented on issue #1278:
URL: https://github.com/apache/datafusion-python/issues/1278#issuecomment-3423477322
I definitely think a single `select` would be nice ergonomically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expand use of sql parsing string expressions in DataFrame [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #1278:
URL: https://github.com/apache/datafusion-python/issues/1278#issuecomment-3406551401
cc @milenkovicm @K-dash
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expand use of sql parsing string expressions in DataFrame [datafusion-python]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #1278:
URL: https://github.com/apache/datafusion-python/issues/1278#issuecomment-3406712056
Have you changed your mind about select Tim? I'm stuck next few days but may help if needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expand use of sql parsing string expressions in DataFrame [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #1278:
URL: https://github.com/apache/datafusion-python/issues/1278#issuecomment-3406555584
I think when we have a string we want to first check to see if it is a column name. If so, create a column expression. If not, try sql parsing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expand use of sql parsing string expressions in DataFrame [datafusion-python]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #1278:
URL: https://github.com/apache/datafusion-python/issues/1278#issuecomment-3423235372
[looking at spark](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.select.html#pyspark.sql.DataFrame.select) it looks like we're aligned with `select` and `selectExpr` maybe we keep both as they are ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expand use of sql parsing string expressions in DataFrame [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #1278:
URL: https://github.com/apache/datafusion-python/issues/1278#issuecomment-3406750424
I do think we want `select` but it just can't be the snippet I shared before. Instead I think it needs to check if the string matches a column name first. If so, make a column expression. Otherwise try parsing. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expand use of sql parsing string expressions in DataFrame [datafusion-python]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #1278:
URL: https://github.com/apache/datafusion-python/issues/1278#issuecomment-3412620111
i hope to tick off few of them off the list this weekend
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expand use of sql parsing string expressions in DataFrame [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer closed issue #1278: Expand use of sql parsing string expressions in DataFrame
URL: https://github.com/apache/datafusion-python/issues/1278
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org