You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "izveigor (via GitHub)" <gi...@apache.org> on 2023/07/15 19:31:21 UTC
[GitHub] [arrow-datafusion] izveigor opened a new issue, #6980: General ticket for the concept of the practical implementation of the array
izveigor opened a new issue, #6980:
URL: https://github.com/apache/arrow-datafusion/issues/6980
### Is your feature request related to a problem or challenge?
It was decided that further development of array functions will be carried out according to the following concept: https://github.com/apache/arrow-datafusion/discussions/6855.
According to the concept, any changes may be subject to rational criticism, so some aspects may change.
It should be noted that this list is incomplete and will be updated in the future.
The ticket contains many tickets. Each ticket is divided into respective categories:
# List of features
A complete list of features, the implementation of which is desirable from the point of view of this concept 🎯:
## Managing array elements:
### Via index:
- [ ] `array_element` https://github.com/apache/arrow-datafusion/issues/6974
- [ ] `array_slice` https://github.com/apache/arrow-datafusion/issues/6975
## Set language:
### Basic operations on sets:
- [ ] `array_intersect` https://github.com/apache/arrow-datafusion/issues/6978
- [ ] `array_except` https://github.com/apache/arrow-datafusion/issues/6979
### Checking if elements belong:
- [ ] `array_has_all` https://github.com/apache/arrow-datafusion/issues/6973
- [ ] `array_has_any` https://github.com/apache/arrow-datafusion/issues/6976
- [ ] `array_has` https://github.com/apache/arrow-datafusion/issues/6977
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] izveigor commented on issue #6980: [Epic] General ticket for the concept of the practical implementation of `ARRAY`
Posted by "izveigor (via GitHub)" <gi...@apache.org>.
izveigor commented on issue #6980:
URL: https://github.com/apache/arrow-datafusion/issues/6980#issuecomment-1667382174
> I would like to try on `array_aggregate`. https://duckdb.org/docs/sql/functions/nested#list-aggregates There are many kinds of rewrite functions, but most of them are not supported yet. I think we can start with the easy one or an existing function, like `sum`, `min`, or `max`.
It sounds great! Thank you, @jayzhan211! I created the tickets about this topic: https://github.com/apache/arrow-datafusion/issues/7213 and https://github.com/apache/arrow-datafusion/issues/7214.
> @izveigor , I dont fully understand about the difference between `list_xxx` and `array_xxx` in duckdb. As you said, list is for mutable, and array is for immutable. But like `append` a mutable function, why do they have `array_append`? `contains` a immutable function, but they have `list_contains`. For aggregate rewrite functions like `list_min`, `list_sum`, they don't alias with array, no `array_min` or `array_sum`, but we can do the same things with `array_aggregate()`.
>
> How do we differentiate `list`, and `array` in datafusion?
I don't fully understand DuckDB principles myself. In my opinion every function should have `list_` and `array_` prefix due to the habit of using a single word. About mutability, this is purely my assumption of separation of functions, we may not adhere to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jayzhan211 commented on issue #6980: [Epic] General ticket for the concept of the practical implementation of `ARRAY`
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #6980:
URL: https://github.com/apache/arrow-datafusion/issues/6980#issuecomment-1667976685
> > I would like to try on `array_aggregate`. https://duckdb.org/docs/sql/functions/nested#list-aggregates There are many kinds of rewrite functions, but most of them are not supported yet. I think we can start with the easy one or an existing function, like `sum`, `min`, or `max`.
>
> It sounds great! Thank you, @jayzhan211! I created the tickets about this topic: #7213 and #7214.
>
> > @izveigor , I dont fully understand about the difference between `list_xxx` and `array_xxx` in duckdb. As you said, list is for mutable, and array is for immutable. But like `append` a mutable function, why do they have `array_append`? `contains` a immutable function, but they have `list_contains`. For aggregate rewrite functions like `list_min`, `list_sum`, they don't alias with array, no `array_min` or `array_sum`, but we can do the same things with `array_aggregate()`.
> > How do we differentiate `list`, and `array` in datafusion?
>
> I don't fully understand DuckDB principles myself. In my opinion every function should have `list_` and `array_` prefix due to the habit of using a single word. About mutability, this is purely my assumption of separation of functions, we may not adhere to it.
I asked someone that is familiar with duckdb, he says the reason that they have list_xxx and array_xxx is due to compatibility, postgres for array_xxx, and others for list_xxx. In this case, since we heavily adopted from duckdb, clickhouse and others, we should also use similar naming from them, but I don't think mutability is the thing that differentiates list_xxx and array_xxx, it might be misleading if we said that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] izveigor commented on issue #6980: General ticket for the concept of the practical implementation of the array
Posted by "izveigor (via GitHub)" <gi...@apache.org>.
izveigor commented on issue #6980:
URL: https://github.com/apache/arrow-datafusion/issues/6980#issuecomment-1636870291
cc @alamb @jayzhan211
I also send to people who have recently been interested in the topic of the array: @parkma99 @maxburke @BubbaJoe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jayzhan211 commented on issue #6980: [Epic] General ticket for the concept of the practical implementation of `ARRAY`
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #6980:
URL: https://github.com/apache/arrow-datafusion/issues/6980#issuecomment-1667040581
@izveigor , I dont fully understand about the difference between `list_xxx` and `array_xxx` in duckdb. As you said, list is for mutable, and array is for immutable. But like `append` a mutable function, why do they have `array_append`? `contains` a immutable function, but they have `list_contains`. For aggregate rewrite functions like `list_min`, `list_sum`, they don't alias with array, no `array_min` or `array_sum`, but we can do the same things with `array_aggregate()`.
How do we differentiate `list`, and `array` in datafusion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jayzhan211 commented on issue #6980: General ticket for the concept of the practical implementation of the array
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #6980:
URL: https://github.com/apache/arrow-datafusion/issues/6980#issuecomment-1637018276
Should we accept nested arrays for `array_has_all` and array_has_any`, unlike `array_has` the rhs could be `array` not `element`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #6980: [Epic] General ticket for the concept of the practical implementation of `ARRAY`
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6980:
URL: https://github.com/apache/arrow-datafusion/issues/6980#issuecomment-1668001398
> I asked someone that is familiar with duckdb, he says the reason that they have list_xxx and array_xxx is due to compatibility, postgres for array_xxx, and others for list_xxx.
This would be great to add to the documentation if you had a chance -- thank you for the research @jayzhan211
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] izveigor commented on issue #6980: [Epic] General ticket for the concept of the practical implementation of `ARRAY`
Posted by "izveigor (via GitHub)" <gi...@apache.org>.
izveigor commented on issue #6980:
URL: https://github.com/apache/arrow-datafusion/issues/6980#issuecomment-1712492734
Hello, @edmondop!
Yes, you can pick all tickets if there are no assignments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] izveigor commented on issue #6980: General ticket for the concept of the practical implementation of the array
Posted by "izveigor (via GitHub)" <gi...@apache.org>.
izveigor commented on issue #6980:
URL: https://github.com/apache/arrow-datafusion/issues/6980#issuecomment-1637045457
> @izveigor, since there are array_intersect and array_except, did you find any function similar to array_union, the function that returns a list of all elements that exist in `one of the l1 and l2` without duplicates? list_intersect([1,2,3], [3,4,5]) -> [1,2,3,4,5]. It seems to me a helpful set operation, but not sure why Duckdb does not have this one.
I think we can use Apache Spark SQL and Azure DataBricks function `array_union` (See: https://sparkbyexamples.com/spark/spark-sql-array-functions/ and https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/functions/array_union). Yes, I agree that the full set of set operations is a very powerful tool.
> It makes sense to allow nested array as the lhs value of three of the array_has function. We just need to check the element one by one.
Absolutely agree! I think this is the main mistake with PostgreSQL's array function set. Each element, including the array/list itself, must be processed by any function.
> The difference of array_has and array_has_all is the rhs value. The former is element (non-array), and the latter is array (one-dimension). Should we consider them as two different functions as Duckdb does?
I prefer to use ClickHouse/DuckDB version of these functions, because there are a lot of possibilities for making full use of the set language (specifically we should check whether an element belongs to a set and check whether a set is a subset of a certain set).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] izveigor commented on issue #6980: [Epic] General ticket for the concept of the practical implementation of `ARRAY`
Posted by "izveigor (via GitHub)" <gi...@apache.org>.
izveigor commented on issue #6980:
URL: https://github.com/apache/arrow-datafusion/issues/6980#issuecomment-1668021161
@jayzhan211, thank you for the research. I agree with you and we should create the subpage of the documentation that represents our syntax (we diccussed it in https://github.com/apache/arrow-datafusion/discussions/6855#discussioncomment-6536597). I will deal with it soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] izveigor commented on issue #6980: [Epic] General ticket for the concept of the practical implementation of `ARRAY`
Posted by "izveigor (via GitHub)" <gi...@apache.org>.
izveigor commented on issue #6980:
URL: https://github.com/apache/arrow-datafusion/issues/6980#issuecomment-1722270150
Hello, @alamb and @jayzhan211.
I would like to inform you that, unfortunately, due to life circumstances, I will no longer be able to take part in the project, or at least in the near future.
I hope that my ideas helped with the development of the project. It was a pleasure working with you and this experience really improved my programming skills.
Finally, I’ll leave my thoughts on the last problem, I hope someone can implement it: https://github.com/apache/arrow-datafusion/issues/7580.
Good luck!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] edmondop commented on issue #6980: [Epic] General ticket for the concept of the practical implementation of `ARRAY`
Posted by "edmondop (via GitHub)" <gi...@apache.org>.
edmondop commented on issue #6980:
URL: https://github.com/apache/arrow-datafusion/issues/6980#issuecomment-1710757743
Can I pick something @jiangzhx that nobody else is looking to?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jayzhan211 commented on issue #6980: General ticket for the concept of the practical implementation of the array
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #6980:
URL: https://github.com/apache/arrow-datafusion/issues/6980#issuecomment-1637221722
I will work on three of `array_has`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] General ticket for the concept of the practical implementation of `ARRAY` [arrow-datafusion]
Posted by "edmondop (via GitHub)" <gi...@apache.org>.
edmondop commented on issue #6980:
URL: https://github.com/apache/arrow-datafusion/issues/6980#issuecomment-1808601673
I propose to merge https://github.com/apache/arrow-datafusion/issues/6978 and then refactor both that pr and this https://github.com/apache/arrow-datafusion/issues/6981 to make easy to implement #6979 .
All these functions traverse arrays and use some accumulator, so we can create an accumulator trait and three instances (union, intersection, except). @alamb what do yout hink?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] General ticket for the concept of the practical implementation of `ARRAY` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6980:
URL: https://github.com/apache/datafusion/issues/6980#issuecomment-2214863083
This looks pretty good to me now, I think we have done all the major items so closing this epic as good. We can handle small cleanup items as normal tickets. Please let me know if you disagree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] General ticket for the concept of the practical implementation of `ARRAY` [arrow-datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #6980:
URL: https://github.com/apache/arrow-datafusion/issues/6980#issuecomment-1848395251
TODO:
- #7194 `array_resize`
- #7142 / #7143 Null / Empty Handling
- #7213 / #7214 `array aggregate`
- #6555 `unnest`
- Struct / Fixed Size List / Large List ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jayzhan211 commented on issue #6980: [Epic] General ticket for the concept of the practical implementation of `ARRAY`
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #6980:
URL: https://github.com/apache/arrow-datafusion/issues/6980#issuecomment-1667028423
I would like to try on `array_aggregate`. https://duckdb.org/docs/sql/functions/nested#list-aggregates
There are many kinds of rewrite functions, but most of them are not supported yet. I think we can start with the easy one or an existing function, like `sum`, `min`, or `max`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jayzhan211 commented on issue #6980: General ticket for the concept of the practical implementation of the array
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #6980:
URL: https://github.com/apache/arrow-datafusion/issues/6980#issuecomment-1637009025
Since there is array_intersect and array_except, did you find any function similar to array_union, the function that returns a list of all elements that exist in `one of the l1 and l2` without duplicates? list_intersect([1,2,3], [3,4,5]) -> [1,2,3,4,5]. It seems to me a helpful set operation, but not sure why Duckdb does not have this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] General ticket for the concept of the practical implementation of `ARRAY` [arrow-datafusion]
Posted by "Weijun-H (via GitHub)" <gi...@apache.org>.
Weijun-H commented on issue #6980:
URL: https://github.com/apache/arrow-datafusion/issues/6980#issuecomment-1913170640
#9021 support list_reverse in array
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] General ticket for the concept of the practical implementation of `ARRAY` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #6980: [Epic] General ticket for the concept of the practical implementation of `ARRAY`
URL: https://github.com/apache/datafusion/issues/6980
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #6980: [Epic] General ticket for the concept of the practical implementation of `ARRAY`
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6980:
URL: https://github.com/apache/arrow-datafusion/issues/6980#issuecomment-1722474095
@izveigor -- thank you for letting us know and for all your work so far in this project. I think it has driven DataFusion significantly and we have all benefited from your work and inspiration.
Good luck with your next adventure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org