You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "timsaucer (via GitHub)" <gi...@apache.org> on 2026/03/31 16:20:21 UTC
[PR] Add missing scalar functions [datafusion-python]
timsaucer opened a new pull request, #1470:
URL: https://github.com/apache/datafusion-python/pull/1470
# Which issue does this PR close?
Closes #1453
# Rationale for this change
These functions exist upstream but were not exposed to Python.
# What changes are included in this PR?
Expose functions to Python
Add unit testss
# Are there any user-facing changes?
New addition only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add missing scalar functions [datafusion-python]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1470:
URL: https://github.com/apache/datafusion-python/pull/1470#discussion_r3034074031
##########
python/tests/test_functions.py:
##########
@@ -1435,3 +1435,73 @@ def test_coalesce(df):
assert result.column(0) == pa.array(
["Hello", "fallback", "!"], type=pa.string_view()
)
+
+
+def test_get_field(df):
+ df = df.with_column(
+ "s",
+ f.named_struct(
+ [
+ ("x", column("a")),
+ ("y", column("b")),
+ ]
+ ),
+ )
+ result = df.select(
+ f.get_field(column("s"), string_literal("x")).alias("x_val"),
+ f.get_field(column("s"), string_literal("y")).alias("y_val"),
+ ).collect()[0]
+
+ assert result.column(0) == pa.array(["Hello", "World", "!"], type=pa.string_view())
+ assert result.column(1) == pa.array([4, 5, 6])
+
+
+def test_arrow_metadata(df):
+ result = df.select(
+ f.arrow_metadata(column("a")).alias("meta"),
+ ).collect()[0]
+ # The metadata column should be returned as a map type (possibly empty)
+ assert result.column(0).type == pa.map_(pa.utf8(), pa.utf8())
Review Comment:
`test_arrow_metadata` currently only asserts the returned Arrow type is `map<utf8, utf8>` and doesn’t verify contents or the 2-argument form (lookup by key). Consider extending this test to first create a column with known field metadata (e.g., via `alias(..., metadata={...})`) and then assert (1) the map contains the expected key/value and (2) `arrow_metadata(expr, key)` returns the expected value.
```suggestion
metadata_key = "source"
metadata_value = "unit-test"
df = df.select(
column("a").alias("a_with_meta", metadata={metadata_key: metadata_value})
)
result = df.select(
f.arrow_metadata(column("a_with_meta")).alias("meta"),
f.arrow_metadata(column("a_with_meta"), string_literal(metadata_key)).alias(
"meta_value"
),
).collect()[0]
# The metadata column should be returned as a map type and contain the
# expected key/value for each row.
assert result.column(0).type == pa.map_(pa.utf8(), pa.utf8())
assert result.column(0).to_pylist() == [[(metadata_key, metadata_value)]] * len(
result.column(0)
)
assert result.column(1).to_pylist() == [metadata_value] * len(result.column(1))
```
##########
python/datafusion/functions.py:
##########
@@ -90,6 +90,7 @@
"array_to_string",
"array_union",
"arrow_cast",
+ "arrow_metadata",
"arrow_typeof",
"ascii",
"asin",
Review Comment:
The PR description/linked issue (#1453) includes exposing `arrow_try_cast`, but this PR only adds `arrow_metadata` (and others). Please either implement/export `arrow_try_cast` (Rust binding + Python wrapper + tests) or update the PR scope/issue closure accordingly so we don’t close the issue with functionality still missing.
##########
crates/core/src/functions.rs:
##########
@@ -631,8 +631,29 @@ expr_fn_vec!(named_struct);
expr_fn!(from_unixtime, unixtime);
expr_fn!(arrow_typeof, arg_1);
expr_fn!(arrow_cast, arg_1 datatype);
+expr_fn_vec!(arrow_metadata);
+expr_fn!(union_tag, arg1);
expr_fn!(random);
Review Comment:
`arrow_try_cast` is part of the missing-function list for issue #1453, but there is no Rust binding/export added here (only `arrow_metadata`, `union_tag`, `get_field`, `union_extract`, `version`). Please add the binding for `arrow_try_cast` (and expose it in the module) or adjust the PR/issue closure to match what’s actually implemented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add missing scalar functions [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on code in PR #1470:
URL: https://github.com/apache/datafusion-python/pull/1470#discussion_r3039356529
##########
python/datafusion/functions.py:
##########
@@ -2628,22 +2634,155 @@ def arrow_typeof(arg: Expr) -> Expr:
return Expr(f.arrow_typeof(arg.expr))
-def arrow_cast(expr: Expr, data_type: Expr) -> Expr:
+def arrow_cast(expr: Expr, data_type: Expr | str) -> Expr:
Review Comment:
This should take a pyarrow data type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add missing scalar functions [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on PR #1470:
URL: https://github.com/apache/datafusion-python/pull/1470#issuecomment-4192266773
Thanks @ntjohnson1 !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add missing scalar functions [datafusion-python]
Posted by "ntjohnson1 (via GitHub)" <gi...@apache.org>.
ntjohnson1 commented on code in PR #1470:
URL: https://github.com/apache/datafusion-python/pull/1470#discussion_r3039132251
##########
python/datafusion/functions.py:
##########
@@ -2596,22 +2602,111 @@ def arrow_typeof(arg: Expr) -> Expr:
return Expr(f.arrow_typeof(arg.expr))
-def arrow_cast(expr: Expr, data_type: Expr) -> Expr:
+def arrow_cast(expr: Expr, data_type: Expr | str) -> Expr:
"""Casts an expression to a specified data type.
Examples:
>>> ctx = dfn.SessionContext()
>>> df = ctx.from_pydict({"a": [1]})
- >>> data_type = dfn.string_literal("Float64")
>>> result = df.select(
- ... dfn.functions.arrow_cast(dfn.col("a"), data_type).alias("c")
+ ... dfn.functions.arrow_cast(dfn.col("a"), "Float64").alias("c")
... )
>>> result.collect_column("c")[0].as_py()
1.0
"""
+ if isinstance(data_type, str):
+ data_type = Expr.string_literal(data_type)
return Expr(f.arrow_cast(expr.expr, data_type.expr))
+def arrow_metadata(expr: Expr, key: Expr | str | None = None) -> Expr:
+ """Returns the metadata of the input expression.
+
+ If called with one argument, returns a Map of all metadata key-value pairs.
+ If called with two arguments, returns the value for the specified metadata key.
+
+ Args:
+ expr: An expression whose metadata to retrieve.
+ key: Optional metadata key to look up. Can be a string or an Expr.
+
+ Returns:
+ A Map of metadata or a specific metadata value.
+ """
+ if key is None:
+ return Expr(f.arrow_metadata(expr.expr))
+ if isinstance(key, str):
+ key = Expr.string_literal(key)
+ return Expr(f.arrow_metadata(expr.expr, key.expr))
+
+
+def get_field(expr: Expr, name: Expr | str) -> Expr:
+ """Extracts a field from a struct or map by name.
+
+ Args:
+ expr: A struct or map expression.
+ name: The field name to extract.
+
+ Returns:
Review Comment:
Echo example/returns note above
##########
python/datafusion/functions.py:
##########
@@ -2596,22 +2602,111 @@ def arrow_typeof(arg: Expr) -> Expr:
return Expr(f.arrow_typeof(arg.expr))
-def arrow_cast(expr: Expr, data_type: Expr) -> Expr:
+def arrow_cast(expr: Expr, data_type: Expr | str) -> Expr:
"""Casts an expression to a specified data type.
Examples:
>>> ctx = dfn.SessionContext()
>>> df = ctx.from_pydict({"a": [1]})
- >>> data_type = dfn.string_literal("Float64")
>>> result = df.select(
- ... dfn.functions.arrow_cast(dfn.col("a"), data_type).alias("c")
+ ... dfn.functions.arrow_cast(dfn.col("a"), "Float64").alias("c")
... )
>>> result.collect_column("c")[0].as_py()
1.0
"""
+ if isinstance(data_type, str):
+ data_type = Expr.string_literal(data_type)
return Expr(f.arrow_cast(expr.expr, data_type.expr))
+def arrow_metadata(expr: Expr, key: Expr | str | None = None) -> Expr:
+ """Returns the metadata of the input expression.
+
+ If called with one argument, returns a Map of all metadata key-value pairs.
+ If called with two arguments, returns the value for the specified metadata key.
+
+ Args:
+ expr: An expression whose metadata to retrieve.
+ key: Optional metadata key to look up. Can be a string or an Expr.
+
+ Returns:
Review Comment:
No example and I think nothing else in the file uses the Returns category. I'm not sure how consistent that is across the code base.
##########
python/datafusion/functions.py:
##########
@@ -2596,22 +2602,111 @@ def arrow_typeof(arg: Expr) -> Expr:
return Expr(f.arrow_typeof(arg.expr))
-def arrow_cast(expr: Expr, data_type: Expr) -> Expr:
+def arrow_cast(expr: Expr, data_type: Expr | str) -> Expr:
"""Casts an expression to a specified data type.
Examples:
>>> ctx = dfn.SessionContext()
>>> df = ctx.from_pydict({"a": [1]})
- >>> data_type = dfn.string_literal("Float64")
>>> result = df.select(
- ... dfn.functions.arrow_cast(dfn.col("a"), data_type).alias("c")
+ ... dfn.functions.arrow_cast(dfn.col("a"), "Float64").alias("c")
... )
>>> result.collect_column("c")[0].as_py()
1.0
"""
+ if isinstance(data_type, str):
+ data_type = Expr.string_literal(data_type)
return Expr(f.arrow_cast(expr.expr, data_type.expr))
+def arrow_metadata(expr: Expr, key: Expr | str | None = None) -> Expr:
+ """Returns the metadata of the input expression.
+
+ If called with one argument, returns a Map of all metadata key-value pairs.
+ If called with two arguments, returns the value for the specified metadata key.
+
+ Args:
+ expr: An expression whose metadata to retrieve.
+ key: Optional metadata key to look up. Can be a string or an Expr.
+
+ Returns:
+ A Map of metadata or a specific metadata value.
+ """
+ if key is None:
+ return Expr(f.arrow_metadata(expr.expr))
+ if isinstance(key, str):
+ key = Expr.string_literal(key)
+ return Expr(f.arrow_metadata(expr.expr, key.expr))
+
+
+def get_field(expr: Expr, name: Expr | str) -> Expr:
+ """Extracts a field from a struct or map by name.
+
+ Args:
+ expr: A struct or map expression.
+ name: The field name to extract.
+
+ Returns:
+ The value of the named field.
+ """
+ if isinstance(name, str):
+ name = Expr.string_literal(name)
+ return Expr(f.get_field(expr.expr, name.expr))
+
+
+def union_extract(union_expr: Expr, field_name: Expr | str) -> Expr:
+ """Extracts a value from a union type by field name.
+
+ Returns the value of the named field if it is the currently selected
+ variant, otherwise returns NULL.
+
+ Args:
+ union_expr: A union-typed expression.
+ field_name: The name of the field to extract.
+
+ Returns:
+ The extracted value or NULL.
+ """
+ if isinstance(field_name, str):
+ field_name = Expr.string_literal(field_name)
+ return Expr(f.union_extract(union_expr.expr, field_name.expr))
+
+
+def union_tag(union_expr: Expr) -> Expr:
+ """Returns the tag (active field name) of a union type.
+
+ Args:
+ union_expr: A union-typed expression.
+
+ Returns:
+ The name of the currently selected field in the union.
+ """
+ return Expr(f.union_tag(union_expr.expr))
+
+
+def version() -> Expr:
+ """Returns the DataFusion version string.
+
+ Returns:
+ A string describing the DataFusion version.
+ """
+ return Expr(f.version())
+
+
+def row(*args: Expr) -> Expr:
+ """Returns a struct with the given arguments.
+
+ This is an alias for :py:func:`struct`.
Review Comment:
Doesn't use the See Also block
##########
python/datafusion/functions.py:
##########
@@ -2596,22 +2602,111 @@ def arrow_typeof(arg: Expr) -> Expr:
return Expr(f.arrow_typeof(arg.expr))
-def arrow_cast(expr: Expr, data_type: Expr) -> Expr:
+def arrow_cast(expr: Expr, data_type: Expr | str) -> Expr:
"""Casts an expression to a specified data type.
Examples:
>>> ctx = dfn.SessionContext()
>>> df = ctx.from_pydict({"a": [1]})
- >>> data_type = dfn.string_literal("Float64")
>>> result = df.select(
- ... dfn.functions.arrow_cast(dfn.col("a"), data_type).alias("c")
+ ... dfn.functions.arrow_cast(dfn.col("a"), "Float64").alias("c")
... )
>>> result.collect_column("c")[0].as_py()
1.0
"""
+ if isinstance(data_type, str):
Review Comment:
Nice! I don't know if anyone has run into it yet but I wonder if a helper around strings might be nice. Hopefully most common is people just passing python strings, but I could image someone passing a numpy string or pyarrow string extracted from some other operation. Definitely follow on work/issue
##########
python/datafusion/functions.py:
##########
@@ -2596,22 +2602,111 @@ def arrow_typeof(arg: Expr) -> Expr:
return Expr(f.arrow_typeof(arg.expr))
-def arrow_cast(expr: Expr, data_type: Expr) -> Expr:
+def arrow_cast(expr: Expr, data_type: Expr | str) -> Expr:
"""Casts an expression to a specified data type.
Examples:
>>> ctx = dfn.SessionContext()
>>> df = ctx.from_pydict({"a": [1]})
- >>> data_type = dfn.string_literal("Float64")
>>> result = df.select(
- ... dfn.functions.arrow_cast(dfn.col("a"), data_type).alias("c")
+ ... dfn.functions.arrow_cast(dfn.col("a"), "Float64").alias("c")
... )
>>> result.collect_column("c")[0].as_py()
1.0
"""
+ if isinstance(data_type, str):
+ data_type = Expr.string_literal(data_type)
return Expr(f.arrow_cast(expr.expr, data_type.expr))
+def arrow_metadata(expr: Expr, key: Expr | str | None = None) -> Expr:
+ """Returns the metadata of the input expression.
+
+ If called with one argument, returns a Map of all metadata key-value pairs.
+ If called with two arguments, returns the value for the specified metadata key.
+
+ Args:
+ expr: An expression whose metadata to retrieve.
+ key: Optional metadata key to look up. Can be a string or an Expr.
+
+ Returns:
+ A Map of metadata or a specific metadata value.
+ """
+ if key is None:
+ return Expr(f.arrow_metadata(expr.expr))
+ if isinstance(key, str):
+ key = Expr.string_literal(key)
+ return Expr(f.arrow_metadata(expr.expr, key.expr))
+
+
+def get_field(expr: Expr, name: Expr | str) -> Expr:
+ """Extracts a field from a struct or map by name.
+
+ Args:
+ expr: A struct or map expression.
+ name: The field name to extract.
+
+ Returns:
+ The value of the named field.
+ """
+ if isinstance(name, str):
+ name = Expr.string_literal(name)
+ return Expr(f.get_field(expr.expr, name.expr))
+
+
+def union_extract(union_expr: Expr, field_name: Expr | str) -> Expr:
+ """Extracts a value from a union type by field name.
+
+ Returns the value of the named field if it is the currently selected
+ variant, otherwise returns NULL.
+
+ Args:
+ union_expr: A union-typed expression.
+ field_name: The name of the field to extract.
+
+ Returns:
+ The extracted value or NULL.
+ """
+ if isinstance(field_name, str):
+ field_name = Expr.string_literal(field_name)
+ return Expr(f.union_extract(union_expr.expr, field_name.expr))
+
+
+def union_tag(union_expr: Expr) -> Expr:
+ """Returns the tag (active field name) of a union type.
+
+ Args:
+ union_expr: A union-typed expression.
+
+ Returns:
+ The name of the currently selected field in the union.
+ """
+ return Expr(f.union_tag(union_expr.expr))
+
+
+def version() -> Expr:
+ """Returns the DataFusion version string.
+
+ Returns:
+ A string describing the DataFusion version.
+ """
+ return Expr(f.version())
+
+
+def row(*args: Expr) -> Expr:
+ """Returns a struct with the given arguments.
+
+ This is an alias for :py:func:`struct`.
+
+ Args:
+ args: The expressions to include in the struct.
+
+ Returns:
Review Comment:
example/returns
##########
python/datafusion/functions.py:
##########
@@ -2596,22 +2602,111 @@ def arrow_typeof(arg: Expr) -> Expr:
return Expr(f.arrow_typeof(arg.expr))
-def arrow_cast(expr: Expr, data_type: Expr) -> Expr:
+def arrow_cast(expr: Expr, data_type: Expr | str) -> Expr:
"""Casts an expression to a specified data type.
Examples:
>>> ctx = dfn.SessionContext()
>>> df = ctx.from_pydict({"a": [1]})
- >>> data_type = dfn.string_literal("Float64")
>>> result = df.select(
- ... dfn.functions.arrow_cast(dfn.col("a"), data_type).alias("c")
+ ... dfn.functions.arrow_cast(dfn.col("a"), "Float64").alias("c")
... )
>>> result.collect_column("c")[0].as_py()
1.0
"""
+ if isinstance(data_type, str):
+ data_type = Expr.string_literal(data_type)
return Expr(f.arrow_cast(expr.expr, data_type.expr))
+def arrow_metadata(expr: Expr, key: Expr | str | None = None) -> Expr:
+ """Returns the metadata of the input expression.
+
+ If called with one argument, returns a Map of all metadata key-value pairs.
+ If called with two arguments, returns the value for the specified metadata key.
+
+ Args:
+ expr: An expression whose metadata to retrieve.
+ key: Optional metadata key to look up. Can be a string or an Expr.
+
+ Returns:
+ A Map of metadata or a specific metadata value.
+ """
+ if key is None:
+ return Expr(f.arrow_metadata(expr.expr))
+ if isinstance(key, str):
+ key = Expr.string_literal(key)
+ return Expr(f.arrow_metadata(expr.expr, key.expr))
+
+
+def get_field(expr: Expr, name: Expr | str) -> Expr:
+ """Extracts a field from a struct or map by name.
+
+ Args:
+ expr: A struct or map expression.
+ name: The field name to extract.
+
+ Returns:
+ The value of the named field.
+ """
+ if isinstance(name, str):
+ name = Expr.string_literal(name)
+ return Expr(f.get_field(expr.expr, name.expr))
+
+
+def union_extract(union_expr: Expr, field_name: Expr | str) -> Expr:
+ """Extracts a value from a union type by field name.
+
+ Returns the value of the named field if it is the currently selected
+ variant, otherwise returns NULL.
+
+ Args:
+ union_expr: A union-typed expression.
+ field_name: The name of the field to extract.
+
+ Returns:
Review Comment:
example/returns
##########
python/datafusion/functions.py:
##########
@@ -2596,22 +2602,111 @@ def arrow_typeof(arg: Expr) -> Expr:
return Expr(f.arrow_typeof(arg.expr))
-def arrow_cast(expr: Expr, data_type: Expr) -> Expr:
+def arrow_cast(expr: Expr, data_type: Expr | str) -> Expr:
"""Casts an expression to a specified data type.
Examples:
>>> ctx = dfn.SessionContext()
>>> df = ctx.from_pydict({"a": [1]})
- >>> data_type = dfn.string_literal("Float64")
>>> result = df.select(
- ... dfn.functions.arrow_cast(dfn.col("a"), data_type).alias("c")
+ ... dfn.functions.arrow_cast(dfn.col("a"), "Float64").alias("c")
... )
>>> result.collect_column("c")[0].as_py()
1.0
"""
+ if isinstance(data_type, str):
+ data_type = Expr.string_literal(data_type)
return Expr(f.arrow_cast(expr.expr, data_type.expr))
+def arrow_metadata(expr: Expr, key: Expr | str | None = None) -> Expr:
+ """Returns the metadata of the input expression.
+
+ If called with one argument, returns a Map of all metadata key-value pairs.
+ If called with two arguments, returns the value for the specified metadata key.
+
+ Args:
+ expr: An expression whose metadata to retrieve.
+ key: Optional metadata key to look up. Can be a string or an Expr.
+
+ Returns:
+ A Map of metadata or a specific metadata value.
+ """
+ if key is None:
+ return Expr(f.arrow_metadata(expr.expr))
+ if isinstance(key, str):
+ key = Expr.string_literal(key)
+ return Expr(f.arrow_metadata(expr.expr, key.expr))
+
+
+def get_field(expr: Expr, name: Expr | str) -> Expr:
+ """Extracts a field from a struct or map by name.
+
+ Args:
+ expr: A struct or map expression.
+ name: The field name to extract.
+
+ Returns:
+ The value of the named field.
+ """
+ if isinstance(name, str):
+ name = Expr.string_literal(name)
+ return Expr(f.get_field(expr.expr, name.expr))
+
+
+def union_extract(union_expr: Expr, field_name: Expr | str) -> Expr:
+ """Extracts a value from a union type by field name.
+
+ Returns the value of the named field if it is the currently selected
+ variant, otherwise returns NULL.
+
+ Args:
+ union_expr: A union-typed expression.
+ field_name: The name of the field to extract.
+
+ Returns:
+ The extracted value or NULL.
+ """
+ if isinstance(field_name, str):
+ field_name = Expr.string_literal(field_name)
+ return Expr(f.union_extract(union_expr.expr, field_name.expr))
+
+
+def union_tag(union_expr: Expr) -> Expr:
+ """Returns the tag (active field name) of a union type.
+
+ Args:
+ union_expr: A union-typed expression.
+
+ Returns:
+ The name of the currently selected field in the union.
+ """
+ return Expr(f.union_tag(union_expr.expr))
+
+
+def version() -> Expr:
+ """Returns the DataFusion version string.
+
+ Returns:
Review Comment:
example/returns
In this case the returns is definitely redundant with the definition
##########
python/datafusion/functions.py:
##########
@@ -2596,22 +2602,111 @@ def arrow_typeof(arg: Expr) -> Expr:
return Expr(f.arrow_typeof(arg.expr))
-def arrow_cast(expr: Expr, data_type: Expr) -> Expr:
+def arrow_cast(expr: Expr, data_type: Expr | str) -> Expr:
"""Casts an expression to a specified data type.
Examples:
>>> ctx = dfn.SessionContext()
>>> df = ctx.from_pydict({"a": [1]})
- >>> data_type = dfn.string_literal("Float64")
>>> result = df.select(
- ... dfn.functions.arrow_cast(dfn.col("a"), data_type).alias("c")
+ ... dfn.functions.arrow_cast(dfn.col("a"), "Float64").alias("c")
... )
>>> result.collect_column("c")[0].as_py()
1.0
"""
+ if isinstance(data_type, str):
+ data_type = Expr.string_literal(data_type)
return Expr(f.arrow_cast(expr.expr, data_type.expr))
+def arrow_metadata(expr: Expr, key: Expr | str | None = None) -> Expr:
+ """Returns the metadata of the input expression.
+
+ If called with one argument, returns a Map of all metadata key-value pairs.
+ If called with two arguments, returns the value for the specified metadata key.
+
+ Args:
+ expr: An expression whose metadata to retrieve.
+ key: Optional metadata key to look up. Can be a string or an Expr.
+
+ Returns:
+ A Map of metadata or a specific metadata value.
+ """
+ if key is None:
+ return Expr(f.arrow_metadata(expr.expr))
+ if isinstance(key, str):
+ key = Expr.string_literal(key)
+ return Expr(f.arrow_metadata(expr.expr, key.expr))
+
+
+def get_field(expr: Expr, name: Expr | str) -> Expr:
+ """Extracts a field from a struct or map by name.
+
+ Args:
+ expr: A struct or map expression.
+ name: The field name to extract.
+
+ Returns:
+ The value of the named field.
+ """
+ if isinstance(name, str):
+ name = Expr.string_literal(name)
+ return Expr(f.get_field(expr.expr, name.expr))
+
+
+def union_extract(union_expr: Expr, field_name: Expr | str) -> Expr:
+ """Extracts a value from a union type by field name.
+
+ Returns the value of the named field if it is the currently selected
+ variant, otherwise returns NULL.
+
+ Args:
+ union_expr: A union-typed expression.
+ field_name: The name of the field to extract.
+
+ Returns:
+ The extracted value or NULL.
+ """
+ if isinstance(field_name, str):
+ field_name = Expr.string_literal(field_name)
+ return Expr(f.union_extract(union_expr.expr, field_name.expr))
+
+
+def union_tag(union_expr: Expr) -> Expr:
+ """Returns the tag (active field name) of a union type.
+
+ Args:
+ union_expr: A union-typed expression.
+
+ Returns:
Review Comment:
example/returns
##########
python/tests/test_functions.py:
##########
@@ -20,7 +20,7 @@
import numpy as np
import pyarrow as pa
import pytest
-from datafusion import SessionContext, column, literal, string_literal
Review Comment:
Love that this is no longer needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add missing scalar functions [datafusion-python]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1470:
URL: https://github.com/apache/datafusion-python/pull/1470#discussion_r3034209158
##########
python/datafusion/functions.py:
##########
@@ -2596,22 +2602,111 @@ def arrow_typeof(arg: Expr) -> Expr:
return Expr(f.arrow_typeof(arg.expr))
-def arrow_cast(expr: Expr, data_type: Expr) -> Expr:
+def arrow_cast(expr: Expr, data_type: Expr | str) -> Expr:
"""Casts an expression to a specified data type.
Examples:
>>> ctx = dfn.SessionContext()
Review Comment:
This PR declares `Closes #1453`, but the issue also lists `arrow_try_cast` as a missing scalar function. I verified there is no `arrow_try_cast` wrapper anywhere in the repo (no Python wrapper in `python/datafusion/functions.py` and no Rust binding in `crates/core/src/functions.rs`). Either add `arrow_try_cast` (and a corresponding unit test) or adjust the PR description/linked issue closure so we’re not closing the issue prematurely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add missing scalar functions [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer merged PR #1470:
URL: https://github.com/apache/datafusion-python/pull/1470
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org