You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/07/07 08:28:13 UTC
[GitHub] [doris] cambyzju opened a new pull request, #10673: [feature-wip](array-type) explode support more sub types
cambyzju opened a new pull request, #10673:
URL: https://github.com/apache/doris/pull/10673
# Proposed changes
Issue Number: close #10052
## Problem Summary:
refractor explode and explode_outer:
1. support more sub types;
2. support nullable elements of array;
Usage example:
```
mysql> set enable_vectorized_engine = true
mysql> set enable_array_type = true
mysql> select e1 from (select 1 k1) as t lateral view explode([1,2,3]) tmp1 as e1;
+------+
| e1 |
+------+
| 1 |
| 2 |
| 3 |
+------+
mysql> select e1 from (select 1 k1) as t lateral view explode_outer(null) tmp1 as e1;
+------+
| e1 |
+------+
| NULL |
+------+
mysql> select e1 from (select 1 k1) as t lateral view explode([]) tmp1 as e1;
Empty set (0.010 sec)
mysql> select e1 from (select 1 k1) as t lateral view explode([null,1,null]) tmp1 as e1;
+------+
| e1 |
+------+
| NULL |
| 1 |
| NULL |
+------+
mysql> select e1 from (select 1 k1) as t lateral view explode_outer([null,1,null]) tmp1 as e1;
+------+
| e1 |
+------+
| NULL |
| 1 |
| NULL |
+------+
```
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] cambyzju commented on a diff in pull request #10673: [feature-wip](array-type) explode support more sub types
Posted by GitBox <gi...@apache.org>.
cambyzju commented on code in PR #10673:
URL: https://github.com/apache/doris/pull/10673#discussion_r919689917
##########
be/src/vec/exprs/table_function/vexplode.cpp:
##########
@@ -56,26 +48,22 @@ Status VExplodeTableFunction::process_row(size_t row_idx) {
DCHECK(row_idx < _array_column->size());
_is_current_empty = false;
_eos = false;
+ _cur_offset = 0;
+ _array_off = (*_detail.offsets_ptr)[row_idx - 1];
+ _cur_size = (*_detail.offsets_ptr)[row_idx] - _array_off;
Review Comment:
table function framework also need `_cur_size`, so we set it here.
##########
be/src/vec/exprs/table_function/vexplode.h:
##########
@@ -39,9 +40,9 @@ class VExplodeTableFunction : public TableFunction {
virtual Status get_value_length(int64_t* length) override;
private:
- const UInt8* _array_null_map;
- const ColumnArray* _array_column;
- size_t _pos;
+ ColumnPtr _array_column;
+ ColumnArrayExecutionData _detail;
+ size_t _array_off; // start offset of array[row_idx]
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #10673: [feature-wip](array-type) explode support more sub types
Posted by GitBox <gi...@apache.org>.
HappenLee commented on code in PR #10673:
URL: https://github.com/apache/doris/pull/10673#discussion_r919624433
##########
be/src/vec/exprs/table_function/vexplode.cpp:
##########
@@ -56,26 +48,22 @@ Status VExplodeTableFunction::process_row(size_t row_idx) {
DCHECK(row_idx < _array_column->size());
_is_current_empty = false;
_eos = false;
+ _cur_offset = 0;
+ _array_off = (*_detail.offsets_ptr)[row_idx - 1];
+ _cur_size = (*_detail.offsets_ptr)[row_idx] - _array_off;
Review Comment:
maybe the `_cur_size` should use use `Member variables`
##########
be/src/vec/exprs/table_function/vexplode.h:
##########
@@ -39,9 +40,9 @@ class VExplodeTableFunction : public TableFunction {
virtual Status get_value_length(int64_t* length) override;
private:
- const UInt8* _array_null_map;
- const ColumnArray* _array_column;
- size_t _pos;
+ ColumnPtr _array_column;
+ ColumnArrayExecutionData _detail;
+ size_t _array_off; // start offset of array[row_idx]
Review Comment:
`_array_offset`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xy720 commented on a diff in pull request #10673: [feature-wip](array-type) explode support more sub types
Posted by GitBox <gi...@apache.org>.
xy720 commented on code in PR #10673:
URL: https://github.com/apache/doris/pull/10673#discussion_r921011660
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/Type.java:
##########
@@ -156,6 +157,23 @@ public abstract class Type {
supportedTypes.add(TIME);
supportedTypes.add(TIMEV2);
supportedTypes.add(STRING);
+
+ arraySubTypes = Lists.newArrayList();
+ arraySubTypes.add(BOOLEAN);
+ arraySubTypes.add(TINYINT);
+ arraySubTypes.add(SMALLINT);
+ arraySubTypes.add(INT);
+ arraySubTypes.add(BIGINT);
+ arraySubTypes.add(LARGEINT);
+ arraySubTypes.add(FLOAT);
+ arraySubTypes.add(DOUBLE);
+ arraySubTypes.add(VARCHAR);
+ arraySubTypes.add(CHAR);
+ arraySubTypes.add(DATE);
+ arraySubTypes.add(DATETIME);
+ arraySubTypes.add(DECIMALV2);
+ arraySubTypes.add(TIME);
Review Comment:
TIME?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] cambyzju commented on pull request #10673: [feature-wip](array-type) explode support more sub types
Posted by GitBox <gi...@apache.org>.
cambyzju commented on PR #10673:
URL: https://github.com/apache/doris/pull/10673#issuecomment-1178768701
rebase to trigger P0 regression again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10673: [feature-wip](array-type) explode support more sub types
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #10673:
URL: https://github.com/apache/doris/pull/10673#issuecomment-1185385705
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xy720 merged pull request #10673: [feature-wip](array-type) explode support more sub types
Posted by GitBox <gi...@apache.org>.
xy720 merged PR #10673:
URL: https://github.com/apache/doris/pull/10673
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] cambyzju commented on pull request #10673: [feature-wip](array-type) explode support more sub types
Posted by GitBox <gi...@apache.org>.
cambyzju commented on PR #10673:
URL: https://github.com/apache/doris/pull/10673#issuecomment-1183222616
rebase to trigger regression test again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xy720 commented on a diff in pull request #10673: [feature-wip](array-type) explode support more sub types
Posted by GitBox <gi...@apache.org>.
xy720 commented on code in PR #10673:
URL: https://github.com/apache/doris/pull/10673#discussion_r920987948
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java:
##########
@@ -2711,11 +2711,10 @@ private void initTableFunction() {
"_ZN5doris19DummyTableFunctions22explode_numbersEPN9doris_udf15FunctionContextERKNS1_9IntValE");
initTableFunctionListWithCombinator(EXPLODE);
- addTableFunctionWithCombinator(EXPLODE, Type.INT, Function.NullableMode.ALWAYS_NULLABLE,
- Lists.newArrayList(new ArrayType(Type.INT)), false,
- "_ZN5doris19DummyTableFunctions7explodeEPN9doris_udf15FunctionContextERKNS1_13CollectionValE");
- addTableFunctionWithCombinator(EXPLODE, Type.VARCHAR, Function.NullableMode.ALWAYS_NULLABLE,
- Lists.newArrayList(new ArrayType(Type.VARCHAR)), false,
- "_ZN5doris19DummyTableFunctions7explodeEPN9doris_udf15FunctionContextERKNS1_13CollectionValE");
Review Comment:
What is the dummy table function?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] cambyzju commented on a diff in pull request #10673: [feature-wip](array-type) explode support more sub types
Posted by GitBox <gi...@apache.org>.
cambyzju commented on code in PR #10673:
URL: https://github.com/apache/doris/pull/10673#discussion_r921098964
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/Type.java:
##########
@@ -156,6 +157,23 @@ public abstract class Type {
supportedTypes.add(TIME);
supportedTypes.add(TIMEV2);
supportedTypes.add(STRING);
+
+ arraySubTypes = Lists.newArrayList();
+ arraySubTypes.add(BOOLEAN);
+ arraySubTypes.add(TINYINT);
+ arraySubTypes.add(SMALLINT);
+ arraySubTypes.add(INT);
+ arraySubTypes.add(BIGINT);
+ arraySubTypes.add(LARGEINT);
+ arraySubTypes.add(FLOAT);
+ arraySubTypes.add(DOUBLE);
+ arraySubTypes.add(VARCHAR);
+ arraySubTypes.add(CHAR);
+ arraySubTypes.add(DATE);
+ arraySubTypes.add(DATETIME);
+ arraySubTypes.add(DECIMALV2);
+ arraySubTypes.add(TIME);
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10673: [feature-wip](array-type) explode support more sub types
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #10673:
URL: https://github.com/apache/doris/pull/10673#issuecomment-1185385740
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] cambyzju commented on a diff in pull request #10673: [feature-wip](array-type) explode support more sub types
Posted by GitBox <gi...@apache.org>.
cambyzju commented on code in PR #10673:
URL: https://github.com/apache/doris/pull/10673#discussion_r921099726
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java:
##########
@@ -2711,11 +2711,10 @@ private void initTableFunction() {
"_ZN5doris19DummyTableFunctions22explode_numbersEPN9doris_udf15FunctionContextERKNS1_9IntValE");
initTableFunctionListWithCombinator(EXPLODE);
- addTableFunctionWithCombinator(EXPLODE, Type.INT, Function.NullableMode.ALWAYS_NULLABLE,
- Lists.newArrayList(new ArrayType(Type.INT)), false,
- "_ZN5doris19DummyTableFunctions7explodeEPN9doris_udf15FunctionContextERKNS1_13CollectionValE");
- addTableFunctionWithCombinator(EXPLODE, Type.VARCHAR, Function.NullableMode.ALWAYS_NULLABLE,
- Lists.newArrayList(new ArrayType(Type.VARCHAR)), false,
- "_ZN5doris19DummyTableFunctions7explodeEPN9doris_udf15FunctionContextERKNS1_13CollectionValE");
Review Comment:
after non-vec codes removed, dummy functions also can removed later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yangzhg commented on pull request #10673: [feature-wip](array-type) explode support more sub types
Posted by GitBox <gi...@apache.org>.
yangzhg commented on PR #10673:
URL: https://github.com/apache/doris/pull/10673#issuecomment-1178426253
maybe need rebase to fix p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org