You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "goldmedal (via GitHub)" <gi...@apache.org> on 2024/07/13 15:59:04 UTC
[PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
goldmedal opened a new pull request, #11452:
URL: https://github.com/apache/datafusion/pull/11452
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Parietally solve #11434
## Rationale for this change
The benchmark result:
```
Gnuplot not found, using plotters backend
make_map_1000 time: [234.07 ยตs 237.23 ยตs 240.86 ยตs]
change: [-91.706% -91.396% -91.136%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
3 (3.00%) high mild
2 (2.00%) high severe
```
It's much faster than the previous implementation #11361. Although the benchmark doesn't invoke the function, it contains the bottleneck of the original scalar function, aggregating the keys and values.
Thanks to @jayzhan211 for the nice suggestion.
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
Remove the scalar function `make_map`, and then plan it in `ExprPlanner`.
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Are these changes tested?
yes
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
no
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1681920330
##########
datafusion/functions-array/src/make_array.rs:
##########
@@ -131,6 +138,77 @@ impl ScalarUDFImpl for MakeArray {
}
}
+#[derive(Debug)]
+pub struct MakeArrayStrict {
Review Comment:
I still think we should find a way to avoid `make_array_strict` ๐ค
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1682783513
##########
datafusion/functions-array/src/make_array.rs:
##########
@@ -131,6 +138,77 @@ impl ScalarUDFImpl for MakeArray {
}
}
+#[derive(Debug)]
+pub struct MakeArrayStrict {
Review Comment:
I have some concerns for it. If we make `MapFunc` to accept one array, it would be used like
```
SELECT map([1,2,3,'a','b','c'])
```
After planning, the input array would be `['1','2','3','a','b','c']` because of the type coercion for array elements. I think the behavior is wrong. If we change the signature of `MapFunc`, we might need to have another implementation to solve it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1681967883
##########
datafusion/functions-array/src/make_array.rs:
##########
@@ -131,6 +138,77 @@ impl ScalarUDFImpl for MakeArray {
}
}
+#[derive(Debug)]
+pub struct MakeArrayStrict {
Review Comment:
Thanks! I will check it tonight.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1679372260
##########
datafusion/functions-array/src/make_array.rs:
##########
@@ -131,6 +138,77 @@ impl ScalarUDFImpl for MakeArray {
}
}
+#[derive(Debug)]
+pub struct MakeArrayStrict {
Review Comment:
I think the high level idea is that
`SELECT MAKE_MAP('POST', 41, 'PAST', 33,'PATCH', 30)`
We arrange args to ['POST', 'PAST', 'PATCH'], [41, 33, 30], and call
`MAP(['POST', 'PAST', 'PATCH'], [41, 33, 30])`
I just noticed that we can't directly pass these two array to `MapFunc` ๐
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#issuecomment-2238757741
Thanks @goldmedal . I will file an issue about the `map` API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1676853439
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -131,17 +131,23 @@ SELECT MAKE_MAP([1,2], ['a', 'b'], [3,4], ['b']);
----
{[1, 2]: [a, b], [3, 4]: [b]}
-query error
+query ?
SELECT MAKE_MAP('POST', 41, 'HEAD', 'ab', 'PATCH', 30);
+----
+{POST: 41, HEAD: ab, PATCH: 30}
Review Comment:
I expected the query would fail because similar behavior isn't allowed in other databases (e.g. DuckDB). However, seems `make_array` will coercion the value to find a suitable type for them. In this case, all of them will be converted to `utf8`.
```
> select make_array(1,'a',3);
+-----------------------------------------+
| make_array(Int64(1),Utf8("a"),Int64(3)) |
+-----------------------------------------+
| [1, a, 3] |
+-----------------------------------------+
1 row(s) fetched.
Elapsed 0.004 seconds.
> select arrow_typeof(make_array(1,'a',3));
+-----------------------------------------------------------------------------------------------------------------+
| arrow_typeof(make_array(Int64(1),Utf8("a"),Int64(3))) |
+-----------------------------------------------------------------------------------------------------------------+
| List(Field { name: "item", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) |
+-----------------------------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.002 seconds.
```
I think if DataFusion allows this type of coercion for `make_array`, we can allow it for `make_map` too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1679399236
##########
datafusion/functions-array/src/make_array.rs:
##########
@@ -131,6 +138,77 @@ impl ScalarUDFImpl for MakeArray {
}
}
+#[derive(Debug)]
+pub struct MakeArrayStrict {
Review Comment:
For this PR, we just call `make_array_inner` and instead of `make_array_strict`, others we could deal with them in another PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1682861404
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -131,17 +131,23 @@ SELECT MAKE_MAP([1,2], ['a', 'b'], [3,4], ['b']);
----
{[1, 2]: [a, b], [3, 4]: [b]}
-query error
+query ?
SELECT MAKE_MAP('POST', 41, 'HEAD', 'ab', 'PATCH', 30);
+----
+{POST: 41, HEAD: ab, PATCH: 30}
Review Comment:
Alright, so the behaviour is actually depend on `array` itself.
I think we can use `make_array` in this case.
But, if we want to introduce nice dataframe API `map(keys: Vec<Expr>, values: Vec<Expr>)`, I think we still need to pass `Vec<Expr>` instead of the result of `make_array`. However, we can introduce that in another PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1682952179
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -131,17 +131,23 @@ SELECT MAKE_MAP([1,2], ['a', 'b'], [3,4], ['b']);
----
{[1, 2]: [a, b], [3, 4]: [b]}
-query error
+query ?
SELECT MAKE_MAP('POST', 41, 'HEAD', 'ab', 'PATCH', 30);
+----
+{POST: 41, HEAD: ab, PATCH: 30}
Review Comment:
> Alright, so the behaviour is actually depend on `array` itself. I think we can use `make_array` in this case.
Ok, I'll roll back to `make_array` first.
> current API expects `map(vec![make_array(vec![lit("a"), lit("b")]), make_array(vec![lit("1"), lit("2")])])` a slightly better API is `map(vec![lit("a"), lit("b")], vec![lit(1), lit(2)])`
I'm not very familiar with the data frame implementation. Curiously, does the API for data frames also use the UDF `map`? I think the UDF is a logical layer function, but we don't have a corresponding logical expression for `vec!` other than `make_array`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1681932892
##########
datafusion/functions-array/src/make_array.rs:
##########
@@ -131,6 +138,77 @@ impl ScalarUDFImpl for MakeArray {
}
}
+#[derive(Debug)]
+pub struct MakeArrayStrict {
Review Comment:
We can change the MapFunc first, let it takes arguments with Vec<Expr>. The first half is `keys`, the other is `values`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1677342379
##########
datafusion/functions-array/src/planner.rs:
##########
@@ -97,6 +98,27 @@ impl ExprPlanner for ArrayFunctionPlanner {
) -> Result<PlannerResult<Vec<Expr>>> {
Ok(PlannerResult::Planned(make_array(exprs)))
}
+
+ fn plan_make_map(&self, args: Vec<Expr>) -> Result<PlannerResult<Vec<Expr>>> {
+ if args.len() % 2 != 0 {
+ return exec_err!("make_map requires an even number of arguments");
+ }
+
+ let (keys, values): (Vec<_>, Vec<_>) = args
+ .chunks_exact(2)
+ .map(|chunk| (chunk[0].clone(), chunk[1].clone()))
+ .unzip();
+
+ let keys = make_array(keys);
+ let values = make_array(values);
Review Comment:
I guess we can reuse `make_array_inner` if we move `make_array` to `functions` crate.
The alternative is to keep the code here in `functions-array`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1677923547
##########
datafusion/functions-array/src/planner.rs:
##########
@@ -97,6 +98,27 @@ impl ExprPlanner for ArrayFunctionPlanner {
) -> Result<PlannerResult<Vec<Expr>>> {
Ok(PlannerResult::Planned(make_array(exprs)))
}
+
+ fn plan_make_map(&self, args: Vec<Expr>) -> Result<PlannerResult<Vec<Expr>>> {
+ if args.len() % 2 != 0 {
+ return exec_err!("make_map requires an even number of arguments");
+ }
+
+ let (keys, values): (Vec<_>, Vec<_>) = args
+ .chunks_exact(2)
+ .map(|chunk| (chunk[0].clone(), chunk[1].clone()))
+ .unzip();
+
+ let keys = make_array(keys);
+ let values = make_array(values);
Review Comment:
Yes, I think moving `make_array` to functions is a good idea. It would be beneficial for many scenarios.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1677998536
##########
datafusion/functions-array/src/planner.rs:
##########
@@ -97,6 +98,27 @@ impl ExprPlanner for ArrayFunctionPlanner {
) -> Result<PlannerResult<Vec<Expr>>> {
Ok(PlannerResult::Planned(make_array(exprs)))
}
+
+ fn plan_make_map(&self, args: Vec<Expr>) -> Result<PlannerResult<Vec<Expr>>> {
+ if args.len() % 2 != 0 {
+ return exec_err!("make_map requires an even number of arguments");
+ }
+
+ let (keys, values): (Vec<_>, Vec<_>) = args
+ .chunks_exact(2)
+ .map(|chunk| (chunk[0].clone(), chunk[1].clone()))
+ .unzip();
+
+ let keys = make_array(keys);
+ let values = make_array(values);
Review Comment:
Hmm, okay. After some research, I believe it's not easy to move `make_array` to `functions`. It's tied to methods in `utils.rs` and `macro.rs`. Moving all the required methods to `functions` could make the codebase chaotic. For now, I prefer to keep them in `functions-arrays`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1677237773
##########
datafusion/functions-array/src/planner.rs:
##########
@@ -97,6 +98,27 @@ impl ExprPlanner for ArrayFunctionPlanner {
) -> Result<PlannerResult<Vec<Expr>>> {
Ok(PlannerResult::Planned(make_array(exprs)))
}
+
+ fn plan_make_map(&self, args: Vec<Expr>) -> Result<PlannerResult<Vec<Expr>>> {
+ if args.len() % 2 != 0 {
+ return exec_err!("make_map requires an even number of arguments");
+ }
+
+ let (keys, values): (Vec<_>, Vec<_>) = args
+ .chunks_exact(2)
+ .map(|chunk| (chunk[0].clone(), chunk[1].clone()))
+ .unzip();
+
+ let keys = make_array(keys);
+ let values = make_array(values);
Review Comment:
Maybe we could move `make-array` to functions core-feature?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1677315950
##########
datafusion/functions-array/src/planner.rs:
##########
@@ -97,6 +98,27 @@ impl ExprPlanner for ArrayFunctionPlanner {
) -> Result<PlannerResult<Vec<Expr>>> {
Ok(PlannerResult::Planned(make_array(exprs)))
}
+
+ fn plan_make_map(&self, args: Vec<Expr>) -> Result<PlannerResult<Vec<Expr>>> {
+ if args.len() % 2 != 0 {
+ return exec_err!("make_map requires an even number of arguments");
+ }
+
+ let (keys, values): (Vec<_>, Vec<_>) = args
+ .chunks_exact(2)
+ .map(|chunk| (chunk[0].clone(), chunk[1].clone()))
+ .unzip();
+
+ let keys = make_array(keys);
+ let values = make_array(values);
Review Comment:
> Do we have any downside of adding functions-array as depedency to functions
Then you need to import unnecessary array function crate if you only care about `funcitons`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1677998536
##########
datafusion/functions-array/src/planner.rs:
##########
@@ -97,6 +98,27 @@ impl ExprPlanner for ArrayFunctionPlanner {
) -> Result<PlannerResult<Vec<Expr>>> {
Ok(PlannerResult::Planned(make_array(exprs)))
}
+
+ fn plan_make_map(&self, args: Vec<Expr>) -> Result<PlannerResult<Vec<Expr>>> {
+ if args.len() % 2 != 0 {
+ return exec_err!("make_map requires an even number of arguments");
+ }
+
+ let (keys, values): (Vec<_>, Vec<_>) = args
+ .chunks_exact(2)
+ .map(|chunk| (chunk[0].clone(), chunk[1].clone()))
+ .unzip();
+
+ let keys = make_array(keys);
+ let values = make_array(values);
Review Comment:
Hmm, okay. After some research, I believe it's not easy to move `make_array` to `functions`. It's tied to methods in `utils.rs` and `macro.rs`. Moving all the required methods to `functions` could make the codebase chaotic. For now, I prefer to keep them in `functions-arrays` first. We can do it in another PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1682861404
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -131,17 +131,23 @@ SELECT MAKE_MAP([1,2], ['a', 'b'], [3,4], ['b']);
----
{[1, 2]: [a, b], [3, 4]: [b]}
-query error
+query ?
SELECT MAKE_MAP('POST', 41, 'HEAD', 'ab', 'PATCH', 30);
+----
+{POST: 41, HEAD: ab, PATCH: 30}
Review Comment:
Alright, so the behaviour is actually depend on `array` itself.
I think we can use `make_array` in this case.
But, if we want to introduce nice dataframe API `map(keys: Vec<Expr>, values: Vec<Expr>)`, I think we still need to pass `Vec<Expr>` instead of the result of `make_array`. However, we can introduce that in another PR.
current API expects `map(vec![make_array(vec![lit("a"), lit("b")]), make_array(vec![lit("1"), lit("2")])])`
a slightly better API is `map(vec![lit("a"), lit("b")], vec![lit(1), lit(2)])`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1677920841
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -131,17 +131,23 @@ SELECT MAKE_MAP([1,2], ['a', 'b'], [3,4], ['b']);
----
{[1, 2]: [a, b], [3, 4]: [b]}
-query error
+query ?
SELECT MAKE_MAP('POST', 41, 'HEAD', 'ab', 'PATCH', 30);
+----
+{POST: 41, HEAD: ab, PATCH: 30}
Review Comment:
Agreed. That's why I planned to implement another scalar function for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "dharanad (via GitHub)" <gi...@apache.org>.
dharanad commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1677181297
##########
datafusion/functions/src/core/mod.rs:
##########
@@ -81,10 +80,6 @@ pub mod expr_fn {
coalesce,
"Returns `coalesce(args...)`, which evaluates to the value of the first expr which is not NULL",
args,
- ),(
- make_map,
- "Returns a map created from the given keys and values pairs. This function isn't efficient for large maps. Use the `map` function instead.",
- args,
Review Comment:
Agreed, We can use to document this function in https://datafusion.apache.org/user-guide/sql/scalar_functions.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1681932892
##########
datafusion/functions-array/src/make_array.rs:
##########
@@ -131,6 +138,77 @@ impl ScalarUDFImpl for MakeArray {
}
}
+#[derive(Debug)]
+pub struct MakeArrayStrict {
Review Comment:
We can change the MapFunc first, let it takes arguments with `Vec<Expr>`. The first half is `keys`, the other is `values`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#issuecomment-2238780395
Thanks @jayzhan211 and @dharanad for reviewing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1684115054
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -131,17 +131,23 @@ SELECT MAKE_MAP([1,2], ['a', 'b'], [3,4], ['b']);
----
{[1, 2]: [a, b], [3, 4]: [b]}
-query error
+query ?
SELECT MAKE_MAP('POST', 41, 'HEAD', 'ab', 'PATCH', 30);
+----
+{POST: 41, HEAD: ab, PATCH: 30}
Review Comment:
dataframe API is used for building `Expr`.
`map(vec![make_array(vec![lit("a"), lit("b")]), make_array(vec![lit("1"), lit("2")])])` is actually like `Expr::ScalarFunction(map_udf(), args: ...)`.
The idea is something like
```rust
fn map(keys: Vec<Expr>, values: Vec<Expr>) {
let args: Vec<Expr> = concat (keys, values)
Expr::ScalarFunction(map_udf(), args)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1681023548
##########
datafusion/functions-array/src/make_array.rs:
##########
@@ -131,6 +138,77 @@ impl ScalarUDFImpl for MakeArray {
}
}
+#[derive(Debug)]
+pub struct MakeArrayStrict {
Review Comment:
I think we can't call `make_array_inner` in `ExprPlanner`. In my opinion, it should be used in the execution phase, but `ExprPlanner` is used in the planning phase. Our purpose is to arrange the input expressions into two array logical expressions (which is more efficient than arranging `ColumnarValue`).
However, I noticed that we don't have another way to create an array logical expression aside from `make_array`. We plan any array literal to `make_array`.
```
> explain select [c1,2] from t1;
+---------------+-----------------------------------------------------------------------------------------+
| plan_type | plan |
+---------------+-----------------------------------------------------------------------------------------+
| logical_plan | Projection: make_array(CAST(t1.c1 AS Int64), Int64(2)) |
| | TableScan: t1 projection=[c1] |
| physical_plan | ProjectionExec: expr=[make_array(CAST(c1@0 AS Int64), 2) as make_array(t1.c1,Int64(2))] |
| | MemoryExec: partitions=1, partition_sizes=[1] |
| | |
+---------------+-----------------------------------------------------------------------------------------+
2 row(s) fetched.
Elapsed 0.008 seconds.
```
Is there any other way to aggregate the expressions into an array or list in the planning phase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 merged PR #11452:
URL: https://github.com/apache/datafusion/pull/11452
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1682778258
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -131,17 +131,23 @@ SELECT MAKE_MAP([1,2], ['a', 'b'], [3,4], ['b']);
----
{[1, 2]: [a, b], [3, 4]: [b]}
-query error
+query ?
SELECT MAKE_MAP('POST', 41, 'HEAD', 'ab', 'PATCH', 30);
+----
+{POST: 41, HEAD: ab, PATCH: 30}
Review Comment:
> I think we need another `make_array` to does not apply coercion. I prefer to align the behaviour to other system unless there is a good reason not to.
I did more tests for DuckDB behavior and found something interesting. I found they also try to coercion types when building arrays or maps.
I arranged some notes for the behaviors:
## How DuckDB build a map
It seems that they also transform to two lists and call map function. Just like my first design, using `make_array`.
```sql
D select map {1:102, 2:20};
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ main.map(main.list_value(1, 2), main.list_value(102, 20)) โ
โ map(integer, integer) โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ {1=102, 2=20} โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
```
## How DuckDB and DataFusion coercion array type
### DuckDB
- Array constructed from `INT32 and numeric string`: DuckDB will make it be `INTEGER[]`.
```sql
D select array[1,2,'3'];
โโโโโโโโโโโโโโโโโโโโโโ
โ (ARRAY[1, 2, '3']) โ
โ int32[] โ
โโโโโโโโโโโโโโโโโโโโโโค
โ [1, 2, 3] โ
โโโโโโโโโโโโโโโโโโโโโโ
D select typeof(array[1,2,'3']);
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ typeof((ARRAY[1, 2, '3'])) โ
โ varchar โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ INTEGER[] โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
```
- Array constructed from `INT32 and non-numeric string`: DuckDB can't construct the array.
```sql
D select array[1,2,'a'];
Conversion Error: Could not convert the string 'a' to INT32
LINE 1: select array[1,2,'a'];
```
### DataFusion
- Array constructed from `INT32 and numeric string`: DataFusion will make it be `Uf8 array`.
```sql
> select [1,2,'1'];
+-----------------------------------------+
| make_array(Int64(1),Int64(2),Utf8("1")) |
+-----------------------------------------+
| [1, 2, 1] |
+-----------------------------------------+
1 row(s) fetched.
Elapsed 0.001 seconds.
> select arrow_typeof([1,2,'1']);
+-----------------------------------------------------------------------------------------------------------------+
| arrow_typeof(make_array(Int64(1),Int64(2),Utf8("1"))) |
+-----------------------------------------------------------------------------------------------------------------+
| List(Field { name: "item", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) |
+-----------------------------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.001 seconds.
```
- Array constructed from `INT32 and non-numeric string`: DataFusion will make it be `Uf8 array`.
```sql
> select [1,2,'a'];
+-----------------------------------------+
| make_array(Int64(1),Int64(2),Utf8("a")) |
+-----------------------------------------+
| [1, 2, a] |
+-----------------------------------------+
1 row(s) fetched.
Elapsed 0.001 seconds.
> select arrow_typeof([1,2,'a']);
+-----------------------------------------------------------------------------------------------------------------+
| arrow_typeof(make_array(Int64(1),Int64(2),Utf8("a"))) |
+-----------------------------------------------------------------------------------------------------------------+
| List(Field { name: "item", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) |
+-----------------------------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.001 seconds.
```
The behavior of type coercion between INT32 and String is really different.
## How DuckDB coercion map type
- INT32 value and numeric string value: We can find the value `'20'` has been converted to `20`.
```sql
D select map {1:10, 2:'20'};
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ main.map(main.list_value(1, 2), main.list_value(10, '20')) โ
โ map(integer, integer) โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ {1=10, 2=20} โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
```
- INT32 value and non-numeric string value. (It's what I tried in the first time. That's why I thought it shouldn't be allowed)
```sql
D select map {1:10, 2:'abc'};
Conversion Error: Could not convert string 'abc' to INT32
LINE 1: select map {1:10, 2:'abc'};
^
```
## Conclusion
Referring to these behaviors, I think we can just back to using `make_array` to implement this. Because the behavior of type coercion is different, Our `make_map` can allow `map {1:10, 2:'a'}` but DuckDB can't do it. It makes sense for me.
@jayzhan211 WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1676854128
##########
datafusion/functions/src/core/mod.rs:
##########
@@ -81,10 +80,6 @@ pub mod expr_fn {
coalesce,
"Returns `coalesce(args...)`, which evaluates to the value of the first expr which is not NULL",
args,
- ),(
- make_map,
- "Returns a map created from the given keys and values pairs. This function isn't efficient for large maps. Use the `map` function instead.",
- args,
Review Comment:
I'm not sure where can put this doc. Maybe we can do it when https://github.com/apache/datafusion/issues/11435
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1676853774
##########
datafusion/functions-array/src/planner.rs:
##########
@@ -97,6 +98,27 @@ impl ExprPlanner for ArrayFunctionPlanner {
) -> Result<PlannerResult<Vec<Expr>>> {
Ok(PlannerResult::Planned(make_array(exprs)))
}
+
+ fn plan_make_map(&self, args: Vec<Expr>) -> Result<PlannerResult<Vec<Expr>>> {
+ if args.len() % 2 != 0 {
+ return exec_err!("make_map requires an even number of arguments");
+ }
+
+ let (keys, values): (Vec<_>, Vec<_>) = args
+ .chunks_exact(2)
+ .map(|chunk| (chunk[0].clone(), chunk[1].clone()))
+ .unzip();
+
+ let keys = make_array(keys);
+ let values = make_array(values);
Review Comment:
I want to invoke the `make_array` to do the aggregation. That's why I put the implementation in `functions-array`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1676985308
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -131,17 +131,23 @@ SELECT MAKE_MAP([1,2], ['a', 'b'], [3,4], ['b']);
----
{[1, 2]: [a, b], [3, 4]: [b]}
-query error
+query ?
SELECT MAKE_MAP('POST', 41, 'HEAD', 'ab', 'PATCH', 30);
+----
+{POST: 41, HEAD: ab, PATCH: 30}
Review Comment:
I think we need another `make_array` to does not apply coercion. I prefer to align the behaviour to other system unless there is a good reason not to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1677147399
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -131,17 +131,23 @@ SELECT MAKE_MAP([1,2], ['a', 'b'], [3,4], ['b']);
----
{[1, 2]: [a, b], [3, 4]: [b]}
-query error
+query ?
SELECT MAKE_MAP('POST', 41, 'HEAD', 'ab', 'PATCH', 30);
+----
+{POST: 41, HEAD: ab, PATCH: 30}
Review Comment:
I see. Maybe I can create a scalar function `make_array_strict` that won't implement the `coerce_types` method for `ScalarUDFImpl`, but other implementations are the same as `make_array`.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1677923547
##########
datafusion/functions-array/src/planner.rs:
##########
@@ -97,6 +98,27 @@ impl ExprPlanner for ArrayFunctionPlanner {
) -> Result<PlannerResult<Vec<Expr>>> {
Ok(PlannerResult::Planned(make_array(exprs)))
}
+
+ fn plan_make_map(&self, args: Vec<Expr>) -> Result<PlannerResult<Vec<Expr>>> {
+ if args.len() % 2 != 0 {
+ return exec_err!("make_map requires an even number of arguments");
+ }
+
+ let (keys, values): (Vec<_>, Vec<_>) = args
+ .chunks_exact(2)
+ .map(|chunk| (chunk[0].clone(), chunk[1].clone()))
+ .unzip();
+
+ let keys = make_array(keys);
+ let values = make_array(values);
Review Comment:
Yes, I think moving `make_array_inner` to functions is a good idea. It would be beneficial for many scenarios.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "dharanad (via GitHub)" <gi...@apache.org>.
dharanad commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1677181830
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -131,17 +131,23 @@ SELECT MAKE_MAP([1,2], ['a', 'b'], [3,4], ['b']);
----
{[1, 2]: [a, b], [3, 4]: [b]}
-query error
+query ?
SELECT MAKE_MAP('POST', 41, 'HEAD', 'ab', 'PATCH', 30);
+----
+{POST: 41, HEAD: ab, PATCH: 30}
Review Comment:
> I think we need another `make_array` to does not apply coercion. I prefer to align the behaviour to other system unless there is a good reason not to.
Instead can we pass a boolean arg `should_coercion` with default value as false, to control such behaviour
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1677340968
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -131,17 +131,23 @@ SELECT MAKE_MAP([1,2], ['a', 'b'], [3,4], ['b']);
----
{[1, 2]: [a, b], [3, 4]: [b]}
-query error
+query ?
SELECT MAKE_MAP('POST', 41, 'HEAD', 'ab', 'PATCH', 30);
+----
+{POST: 41, HEAD: ab, PATCH: 30}
Review Comment:
> > I think we need another `make_array` to does not apply coercion. I prefer to align the behaviour to other system unless there is a good reason not to.
>
> Instead can we pass a boolean arg `should_coercion` with default value as false, to control such behaviour
The coercion logic is not simply work like if-else statement. The `make_array_inner` doesn't care about coercion, the coercion is in `type_coercion` pass in analzyer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "dharanad (via GitHub)" <gi...@apache.org>.
dharanad commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1677181029
##########
datafusion/functions-array/src/planner.rs:
##########
@@ -97,6 +98,27 @@ impl ExprPlanner for ArrayFunctionPlanner {
) -> Result<PlannerResult<Vec<Expr>>> {
Ok(PlannerResult::Planned(make_array(exprs)))
}
+
+ fn plan_make_map(&self, args: Vec<Expr>) -> Result<PlannerResult<Vec<Expr>>> {
+ if args.len() % 2 != 0 {
+ return exec_err!("make_map requires an even number of arguments");
+ }
+
+ let (keys, values): (Vec<_>, Vec<_>) = args
+ .chunks_exact(2)
+ .map(|chunk| (chunk[0].clone(), chunk[1].clone()))
+ .unzip();
+
+ let keys = make_array(keys);
+ let values = make_array(values);
Review Comment:
Ideally i think this should be implemented in `functions` inside `core`.
Do we have any downside of adding `functions-array` as depedency to `functions` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1682861404
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -131,17 +131,23 @@ SELECT MAKE_MAP([1,2], ['a', 'b'], [3,4], ['b']);
----
{[1, 2]: [a, b], [3, 4]: [b]}
-query error
+query ?
SELECT MAKE_MAP('POST', 41, 'HEAD', 'ab', 'PATCH', 30);
+----
+{POST: 41, HEAD: ab, PATCH: 30}
Review Comment:
Alright, so the behaviour is actually depend on `array` itself.
I think we can use `make_array` for this.
But, if we want to introduce nice dataframe API `map(keys: Vec<Expr>, values: Vec<Expr>)`, I think we still need to pass `Vec<Expr>` instead of the result of `make_array`. However, we can introduce that in another PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1679386760
##########
datafusion/functions-array/src/make_array.rs:
##########
@@ -131,6 +138,77 @@ impl ScalarUDFImpl for MakeArray {
}
}
+#[derive(Debug)]
+pub struct MakeArrayStrict {
Review Comment:
I think we could figure how to build with dataframe API, map(keys, values)
Current function is like
```rust
pub fn map($($arg: datafusion_expr::Expr),*) -> datafusion_expr::Expr {
super::$FUNC().call(vec![$($arg),*])
}
```
Expected
```rust
pub fn map(keys: Vec<Expr>, values: Vec<Expr>) -> Expr
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1681036797
##########
datafusion/functions-array/src/make_array.rs:
##########
@@ -131,6 +138,77 @@ impl ScalarUDFImpl for MakeArray {
}
}
+#[derive(Debug)]
+pub struct MakeArrayStrict {
Review Comment:
> I think we could figure how to build with dataframe API, map(keys, values)
>
Sounds great. I think we can file another issue for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1679354355
##########
datafusion/functions-array/src/make_array.rs:
##########
@@ -131,6 +138,77 @@ impl ScalarUDFImpl for MakeArray {
}
}
+#[derive(Debug)]
+pub struct MakeArrayStrict {
Review Comment:
This function are public function that could be used in datafusion-cli or other project. We are just converting keys to array, we just need internal private function for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1676984126
##########
datafusion/functions-array/src/planner.rs:
##########
@@ -97,6 +98,27 @@ impl ExprPlanner for ArrayFunctionPlanner {
) -> Result<PlannerResult<Vec<Expr>>> {
Ok(PlannerResult::Planned(make_array(exprs)))
}
+
+ fn plan_make_map(&self, args: Vec<Expr>) -> Result<PlannerResult<Vec<Expr>>> {
+ if args.len() % 2 != 0 {
+ return exec_err!("make_map requires an even number of arguments");
+ }
+
+ let (keys, values): (Vec<_>, Vec<_>) = args
Review Comment:
It is possible to avoid clone
```suggestion
let (keys, values): (Vec<_>, Vec<_>) = args.into_iter().enumerate().partition(|(i, _)| i % 2 == 0);
let keys = make_array(keys.into_iter().map(|(_, expr)| expr).collect());
let values = make_array(values.into_iter().map(|(_, expr)| expr).collect());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1679399236
##########
datafusion/functions-array/src/make_array.rs:
##########
@@ -131,6 +138,77 @@ impl ScalarUDFImpl for MakeArray {
}
}
+#[derive(Debug)]
+pub struct MakeArrayStrict {
Review Comment:
For this PR, we just call `make_array_inner` and instead of `make_array_strict`, we could deal with others in another PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1679348352
##########
datafusion/functions-array/src/make_array.rs:
##########
@@ -131,6 +138,77 @@ impl ScalarUDFImpl for MakeArray {
}
}
+#[derive(Debug)]
+pub struct MakeArrayStrict {
Review Comment:
Can we just add the function that convert keys and values to list of expr instead of introducing another udf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Move `MAKE_MAP` to ExprPlanner [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11452:
URL: https://github.com/apache/datafusion/pull/11452#discussion_r1681963551
##########
datafusion/functions-array/src/make_array.rs:
##########
@@ -131,6 +138,77 @@ impl ScalarUDFImpl for MakeArray {
}
}
+#[derive(Debug)]
+pub struct MakeArrayStrict {
Review Comment:
I play around to make sure the suggestion makes sense https://github.com/apache/datafusion/pull/11526
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org