You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "ektravel (via GitHub)" <gi...@apache.org> on 2023/04/25 20:52:21 UTC
[GitHub] [druid] ektravel opened a new pull request, #14161: docs: remove the note about segments
ektravel opened a new pull request, #14161:
URL: https://github.com/apache/druid/pull/14161
### Description
This PR updates the paragraph about array types and adds a link to [Schema auto-discovery for dimensions](https://github.com/apache/druid/pull/14065/files#diff-f3fb439425e47c9c5e0a587b01c2690c8eb138d37e523db1d6507d7177ff8156R244).
This PR has:
- [x] been self-reviewed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vtlim merged pull request #14161: docs: remove the note about segments
Posted by "vtlim (via GitHub)" <gi...@apache.org>.
vtlim merged PR #14161:
URL: https://github.com/apache/druid/pull/14161
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] 317brian commented on a diff in pull request #14161: docs: remove the note about segments
Posted by "317brian (via GitHub)" <gi...@apache.org>.
317brian commented on code in PR #14161:
URL: https://github.com/apache/druid/pull/14161#discussion_r1198137486
##########
docs/querying/sql-data-types.md:
##########
@@ -74,14 +74,13 @@ The following table describes how Druid maps SQL types onto native types when ru
## Multi-value strings
-Druid's native type system allows strings to potentially have multiple values. These
-[multi-value string dimensions](multi-value-dimensions.md) are reported in SQL as VARCHAR typed, and can be
-syntactically used like any other VARCHAR. Regular string functions that refer to multi-value string dimensions are
-applied to all values for each row individually. Multi-value string dimensions can also be treated as arrays via special
-[multi-value string functions](sql-multivalue-string-functions.md), which can perform powerful array-aware operations, but retain
-their VARCHAR typing and behavior.
+Druid's native type system allows strings to have multiple values. These [multi-value string dimensions](multi-value-dimensions.md) are reported in SQL as type VARCHAR and can be
+syntactically used like any other VARCHAR. Regular string functions that refer to multi-value string dimensions are applied to all values for each row individually.
-Grouping by a multi-value expression observes the native Druid multi-value aggregation behavior, which is similar to the SQL UNNEST operator. Refer to the documentation on [multi-value string dimensions](multi-value-dimensions.md) for additional details.
+You can treat multi-value string dimensions as arrays using special
+[multi-value string functions](sql-multivalue-string-functions.md), which perform powerful array-aware operations, but retain their VARCHAR type and behavior.
+
+When grouping on a multi-value dimension, Druid uses all values from matching rows to generate one group per value, similar to using the UNNEST operator on an ARRAY type. See [Grouping](multi-value-dimensions.md#grouping) for more information.
Review Comment:
The similarity isn't that there's one group per value. UNNEST doesn't do any grouping. It's that when you do a group by for N MVDs, that can return more than N results because each MVD has multiple values. This is like when you unnest N arrays, you can have more than N results because each array is comprised of multiple values
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] ektravel commented on a diff in pull request #14161: docs: remove the note about segments
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #14161:
URL: https://github.com/apache/druid/pull/14161#discussion_r1197852699
##########
docs/querying/sql-data-types.md:
##########
@@ -67,53 +66,51 @@ The following table describes how Druid maps SQL types onto native types when ru
|SMALLINT|LONG|`0`||
|INTEGER|LONG|`0`||
|BIGINT|LONG|`0`|Druid LONG columns (except `__time`) are reported as BIGINT|
-|TIMESTAMP|LONG|`0`, meaning 1970-01-01 00:00:00 UTC|Druid's `__time` column is reported as TIMESTAMP. Casts between string and timestamp types assume standard SQL formatting, e.g. `2000-01-02 03:04:05`, _not_ ISO8601 formatting. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
-|DATE|LONG|`0`, meaning 1970-01-01|Casting TIMESTAMP to DATE rounds down the timestamp to the nearest day. Casts between string and date types assume standard SQL formatting, e.g. `2000-01-02`. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
-|ARRAY|ARRAY|`NULL`|Druid native array types work as SQL arrays, and multi-value strings can be converted to arrays. See the [`ARRAY` details](#arrays).|
+|TIMESTAMP|LONG|`0`, meaning 1970-01-01 00:00:00 UTC|Druid's `__time` column is reported as TIMESTAMP. Casts between string and timestamp types assume standard SQL formatting—for example, `2000-01-02 03:04:05`—not ISO8601 formatting. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
+|DATE|LONG|`0`, meaning 1970-01-01|Casting TIMESTAMP to DATE rounds down the timestamp to the nearest day. Casts between string and date types assume standard SQL formatting—for example, `2000-01-02`. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
+|ARRAY|ARRAY|`NULL`|Druid native array types work as SQL arrays, and multi-value strings can be converted to arrays. See [Arrays](#arrays) for more information.|
|OTHER|COMPLEX|none|May represent various Druid column types such as hyperUnique, approxHistogram, etc.|
<sup>*</sup> Default value applies if `druid.generic.useDefaultValueForNull = true` (the default mode). Otherwise, the default value is `NULL` for all types.
## Multi-value strings
Druid's native type system allows strings to potentially have multiple values. These
-[multi-value string dimensions](multi-value-dimensions.md) are reported in SQL as `VARCHAR` typed, and can be
-syntactically used like any other `VARCHAR`. Regular string functions that refer to multi-value string dimensions are
+[multi-value string dimensions](multi-value-dimensions.md) are reported in SQL as VARCHAR typed, and can be
+syntactically used like any other VARCHAR. Regular string functions that refer to multi-value string dimensions are
applied to all values for each row individually. Multi-value string dimensions can also be treated as arrays via special
[multi-value string functions](sql-multivalue-string-functions.md), which can perform powerful array-aware operations, but retain
-their `VARCHAR` typing and behavior.
+their VARCHAR typing and behavior.
Grouping by a multi-value expression observes the native Druid multi-value aggregation behavior, which is similar to
-an implicit SQL `UNNEST`. Refer to the documentation on [multi-value string dimensions](multi-value-dimensions.md)
+the implicit SQL UNNEST operator. Refer to the documentation on [multi-value string dimensions](multi-value-dimensions.md)
Review Comment:
Removed the word "implicit".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] 317brian commented on a diff in pull request #14161: docs: remove the note about segments
Posted by "317brian (via GitHub)" <gi...@apache.org>.
317brian commented on code in PR #14161:
URL: https://github.com/apache/druid/pull/14161#discussion_r1198137486
##########
docs/querying/sql-data-types.md:
##########
@@ -74,14 +74,13 @@ The following table describes how Druid maps SQL types onto native types when ru
## Multi-value strings
-Druid's native type system allows strings to potentially have multiple values. These
-[multi-value string dimensions](multi-value-dimensions.md) are reported in SQL as VARCHAR typed, and can be
-syntactically used like any other VARCHAR. Regular string functions that refer to multi-value string dimensions are
-applied to all values for each row individually. Multi-value string dimensions can also be treated as arrays via special
-[multi-value string functions](sql-multivalue-string-functions.md), which can perform powerful array-aware operations, but retain
-their VARCHAR typing and behavior.
+Druid's native type system allows strings to have multiple values. These [multi-value string dimensions](multi-value-dimensions.md) are reported in SQL as type VARCHAR and can be
+syntactically used like any other VARCHAR. Regular string functions that refer to multi-value string dimensions are applied to all values for each row individually.
-Grouping by a multi-value expression observes the native Druid multi-value aggregation behavior, which is similar to the SQL UNNEST operator. Refer to the documentation on [multi-value string dimensions](multi-value-dimensions.md) for additional details.
+You can treat multi-value string dimensions as arrays using special
+[multi-value string functions](sql-multivalue-string-functions.md), which perform powerful array-aware operations, but retain their VARCHAR type and behavior.
+
+When grouping on a multi-value dimension, Druid uses all values from matching rows to generate one group per value, similar to using the UNNEST operator on an ARRAY type. See [Grouping](multi-value-dimensions.md#grouping) for more information.
Review Comment:
The similarity isn't that there's one group per value. UNNEST doesn't do any grouping. It's that when you do a group by for N MVDs, that can return more than N results because each MVD has multiple values. This is like when you unnest N arrays, you can have more than N results because each array is comprised of multiple values.
Or at least that's how I read the link. I might be wrong. maybe we should check with Somu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] ektravel commented on a diff in pull request #14161: docs: remove the note about segments
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #14161:
URL: https://github.com/apache/druid/pull/14161#discussion_r1178300697
##########
docs/querying/sql-data-types.md:
##########
@@ -96,14 +96,15 @@ for additional details.
> nearly all possible native functionality.
## Arrays
-Druid supports `ARRAY` types constructed at query time, though it currently lacks the ability to store them in
-segments. `ARRAY` types behave as standard SQL arrays, where results are grouped by matching entire arrays. This is in
-contrast to the implicit `UNNEST` that occurs when grouping on multi-value dimensions directly or when used with the
-multi-value functions. You can convert multi-value dimensions to standard SQL arrays either by explicitly by converting
-them with `MV_TO_ARRAY` or implicitly when used within the [array functions](./sql-array-functions.md). Arrays may
-also be constructed from multiple columns using the array functions.
+
+Druid supports `ARRAY` types constructed at query time. `ARRAY` types behave as standard SQL arrays, where results are grouped by matching entire arrays. This is in contrast to the implicit `UNNEST` that occurs when grouping on multi-value dimensions directly or when used with the multi-value functions.
+
+You can convert multi-value dimensions to standard SQL arrays explicitly by converting them with `MV_TO_ARRAY` or implicitly when used within the [array functions](./sql-array-functions.md). You can also use the array functions to construct arrays from multiple columns.
+
+You can use [schema auto-discovery for dimensions](../ingestion/schema-design.md#schema-auto-discovery-for-dimensions) to store arrays.
Review Comment:
Update the text. Please take another look.
##########
docs/querying/sql-data-types.md:
##########
@@ -96,14 +96,15 @@ for additional details.
> nearly all possible native functionality.
## Arrays
-Druid supports `ARRAY` types constructed at query time, though it currently lacks the ability to store them in
-segments. `ARRAY` types behave as standard SQL arrays, where results are grouped by matching entire arrays. This is in
-contrast to the implicit `UNNEST` that occurs when grouping on multi-value dimensions directly or when used with the
-multi-value functions. You can convert multi-value dimensions to standard SQL arrays either by explicitly by converting
-them with `MV_TO_ARRAY` or implicitly when used within the [array functions](./sql-array-functions.md). Arrays may
-also be constructed from multiple columns using the array functions.
+
+Druid supports `ARRAY` types constructed at query time. `ARRAY` types behave as standard SQL arrays, where results are grouped by matching entire arrays. This is in contrast to the implicit `UNNEST` that occurs when grouping on multi-value dimensions directly or when used with the multi-value functions.
+
+You can convert multi-value dimensions to standard SQL arrays explicitly by converting them with `MV_TO_ARRAY` or implicitly when used within the [array functions](./sql-array-functions.md). You can also use the array functions to construct arrays from multiple columns.
+
+You can use [schema auto-discovery for dimensions](../ingestion/schema-design.md#schema-auto-discovery-for-dimensions) to store arrays.
Review Comment:
Updated the text. Please take another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] 317brian commented on a diff in pull request #14161: docs: remove the note about segments
Posted by "317brian (via GitHub)" <gi...@apache.org>.
317brian commented on code in PR #14161:
URL: https://github.com/apache/druid/pull/14161#discussion_r1212088242
##########
docs/querying/sql-data-types.md:
##########
@@ -39,13 +38,12 @@ Druid natively supports the following basic column types:
* COMPLEX: non-standard data types, such as nested JSON, hyperUnique and approxHistogram, and DataSketches
* ARRAY: arrays composed of any of these types
-Timestamps (including the `__time` column) are treated by Druid as longs, with the value being the number of
+Druid treats timestamps (including the `__time` column) as LONG, with the value being the number of
milliseconds since 1970-01-01 00:00:00 UTC, not counting leap seconds. Therefore, timestamps in Druid do not carry any
-timezone information, but only carry information about the exact moment in time they represent. See the
-[Time functions](sql-scalar.md#date-and-time-functions) section for more information about timestamp handling.
+timezone information, but only carry information about the exact moment in time they represent. See
Review Comment:
```suggestion
timezone information. They only carry information about the exact moment in time they represent. See
```
##########
docs/querying/sql-data-types.md:
##########
@@ -67,53 +65,48 @@ The following table describes how Druid maps SQL types onto native types when ru
|SMALLINT|LONG|`0`||
|INTEGER|LONG|`0`||
|BIGINT|LONG|`0`|Druid LONG columns (except `__time`) are reported as BIGINT|
-|TIMESTAMP|LONG|`0`, meaning 1970-01-01 00:00:00 UTC|Druid's `__time` column is reported as TIMESTAMP. Casts between string and timestamp types assume standard SQL formatting, e.g. `2000-01-02 03:04:05`, _not_ ISO8601 formatting. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
-|DATE|LONG|`0`, meaning 1970-01-01|Casting TIMESTAMP to DATE rounds down the timestamp to the nearest day. Casts between string and date types assume standard SQL formatting, e.g. `2000-01-02`. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
-|ARRAY|ARRAY|`NULL`|Druid native array types work as SQL arrays, and multi-value strings can be converted to arrays. See the [`ARRAY` details](#arrays).|
+|TIMESTAMP|LONG|`0`, meaning 1970-01-01 00:00:00 UTC|Druid's `__time` column is reported as TIMESTAMP. Casts between string and timestamp types assume standard SQL formatting—for example, `2000-01-02 03:04:05`—not ISO 8601 formatting. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
Review Comment:
technically em dashes are meant to emphasize things vs either appositive commas or (). One of the latter 2 woudl be more appropriate for examples
##########
docs/querying/sql-data-types.md:
##########
@@ -39,13 +38,12 @@ Druid natively supports the following basic column types:
* COMPLEX: non-standard data types, such as nested JSON, hyperUnique and approxHistogram, and DataSketches
* ARRAY: arrays composed of any of these types
-Timestamps (including the `__time` column) are treated by Druid as longs, with the value being the number of
+Druid treats timestamps (including the `__time` column) as LONG, with the value being the number of
milliseconds since 1970-01-01 00:00:00 UTC, not counting leap seconds. Therefore, timestamps in Druid do not carry any
-timezone information, but only carry information about the exact moment in time they represent. See the
-[Time functions](sql-scalar.md#date-and-time-functions) section for more information about timestamp handling.
+timezone information, but only carry information about the exact moment in time they represent. See
+[Time functions](sql-scalar.md#date-and-time-functions) for more information about timestamp handling.
-Casts between two SQL types with the same Druid runtime type (see below table) have no effect, other than exceptions
-noted in the table. Casts between two SQL types that have different Druid runtime types generate a runtime cast in
+Casts between two SQL types with the same Druid runtime type have no effect, other than the exceptions noted in the following table. Casts between two SQL types that have different Druid runtime types generate a runtime cast in
Review Comment:
No comma between `effect` and `other`. The sequencing of this section is a little odd. The table comes 2 sentences after 2 other cases.
Maybe
Casts w/ same runtime
TABLE
Casts w/ different runtime
Can't cast
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vtlim commented on a diff in pull request #14161: docs: remove the note about segments
Posted by "vtlim (via GitHub)" <gi...@apache.org>.
vtlim commented on code in PR #14161:
URL: https://github.com/apache/druid/pull/14161#discussion_r1197048998
##########
docs/querying/sql-data-types.md:
##########
@@ -67,53 +66,51 @@ The following table describes how Druid maps SQL types onto native types when ru
|SMALLINT|LONG|`0`||
|INTEGER|LONG|`0`||
|BIGINT|LONG|`0`|Druid LONG columns (except `__time`) are reported as BIGINT|
-|TIMESTAMP|LONG|`0`, meaning 1970-01-01 00:00:00 UTC|Druid's `__time` column is reported as TIMESTAMP. Casts between string and timestamp types assume standard SQL formatting, e.g. `2000-01-02 03:04:05`, _not_ ISO8601 formatting. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
-|DATE|LONG|`0`, meaning 1970-01-01|Casting TIMESTAMP to DATE rounds down the timestamp to the nearest day. Casts between string and date types assume standard SQL formatting, e.g. `2000-01-02`. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
-|ARRAY|ARRAY|`NULL`|Druid native array types work as SQL arrays, and multi-value strings can be converted to arrays. See the [`ARRAY` details](#arrays).|
+|TIMESTAMP|LONG|`0`, meaning 1970-01-01 00:00:00 UTC|Druid's `__time` column is reported as TIMESTAMP. Casts between string and timestamp types assume standard SQL formatting—for example, `2000-01-02 03:04:05`—not ISO8601 formatting. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
+|DATE|LONG|`0`, meaning 1970-01-01|Casting TIMESTAMP to DATE rounds down the timestamp to the nearest day. Casts between string and date types assume standard SQL formatting—for example, `2000-01-02`. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
+|ARRAY|ARRAY|`NULL`|Druid native array types work as SQL arrays, and multi-value strings can be converted to arrays. See [Arrays](#arrays) for more information.|
|OTHER|COMPLEX|none|May represent various Druid column types such as hyperUnique, approxHistogram, etc.|
<sup>*</sup> Default value applies if `druid.generic.useDefaultValueForNull = true` (the default mode). Otherwise, the default value is `NULL` for all types.
## Multi-value strings
Druid's native type system allows strings to potentially have multiple values. These
-[multi-value string dimensions](multi-value-dimensions.md) are reported in SQL as `VARCHAR` typed, and can be
-syntactically used like any other `VARCHAR`. Regular string functions that refer to multi-value string dimensions are
+[multi-value string dimensions](multi-value-dimensions.md) are reported in SQL as VARCHAR typed, and can be
+syntactically used like any other VARCHAR. Regular string functions that refer to multi-value string dimensions are
applied to all values for each row individually. Multi-value string dimensions can also be treated as arrays via special
[multi-value string functions](sql-multivalue-string-functions.md), which can perform powerful array-aware operations, but retain
-their `VARCHAR` typing and behavior.
+their VARCHAR typing and behavior.
Grouping by a multi-value expression observes the native Druid multi-value aggregation behavior, which is similar to
-an implicit SQL `UNNEST`. Refer to the documentation on [multi-value string dimensions](multi-value-dimensions.md)
+the implicit SQL UNNEST operator. Refer to the documentation on [multi-value string dimensions](multi-value-dimensions.md)
for additional details.
-> Because multi-value dimensions are treated by the SQL planner as `VARCHAR`, there are some inconsistencies between how
-> they are handled in Druid SQL and in native queries. For example, expressions involving multi-value dimensions may be
-> incorrectly optimized by the Druid SQL planner: `multi_val_dim = 'a' AND multi_val_dim = 'b'` is optimized to
-> `false`, even though it is possible for a single row to have both "a" and "b" as values for `multi_val_dim`. The
-> SQL behavior of multi-value dimensions may change in a future release to more closely align with their behavior
-> in native queries, but the [multi-value string functions](./sql-multivalue-string-functions.md) should be able to provide
-> nearly all possible native functionality.
+> Because the SQL planner treats multi-value dimensions as VARCHAR, there are some inconsistencies between how they are handled in Druid SQL and in native queries. For instance, expressions involving multi-value dimensions may be incorrectly optimized by the Druid SQL planner. For example, `multi_val_dim = 'a' AND multi_val_dim = 'b'` is optimized to
+`false`, even though it is possible for a single row to have both `'a'` and `'b'` as values for `multi_val_dim`.
+>
+> The SQL behavior of multi-value dimensions may change in a future release to more closely align with their behavior in native queries, but the [multi-value string functions](./sql-multivalue-string-functions.md) should be able to provide nearly all possible native functionality.
## Arrays
-Druid supports `ARRAY` types constructed at query time, though it currently lacks the ability to store them in
Review Comment:
Oh never mind, from the PR description this was intentional
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] techdocsmith commented on a diff in pull request #14161: docs: remove the note about segments
Posted by "techdocsmith (via GitHub)" <gi...@apache.org>.
techdocsmith commented on code in PR #14161:
URL: https://github.com/apache/druid/pull/14161#discussion_r1177177728
##########
docs/querying/sql-data-types.md:
##########
@@ -96,14 +96,15 @@ for additional details.
> nearly all possible native functionality.
## Arrays
-Druid supports `ARRAY` types constructed at query time, though it currently lacks the ability to store them in
-segments. `ARRAY` types behave as standard SQL arrays, where results are grouped by matching entire arrays. This is in
-contrast to the implicit `UNNEST` that occurs when grouping on multi-value dimensions directly or when used with the
-multi-value functions. You can convert multi-value dimensions to standard SQL arrays either by explicitly by converting
-them with `MV_TO_ARRAY` or implicitly when used within the [array functions](./sql-array-functions.md). Arrays may
-also be constructed from multiple columns using the array functions.
+
+Druid supports `ARRAY` types constructed at query time. `ARRAY` types behave as standard SQL arrays, where results are grouped by matching entire arrays. This is in contrast to the implicit `UNNEST` that occurs when grouping on multi-value dimensions directly or when used with the multi-value functions.
Review Comment:
Check if this should be code font. In the table line 72 it is not code font.
##########
docs/querying/sql-data-types.md:
##########
@@ -96,14 +96,15 @@ for additional details.
> nearly all possible native functionality.
## Arrays
-Druid supports `ARRAY` types constructed at query time, though it currently lacks the ability to store them in
-segments. `ARRAY` types behave as standard SQL arrays, where results are grouped by matching entire arrays. This is in
-contrast to the implicit `UNNEST` that occurs when grouping on multi-value dimensions directly or when used with the
-multi-value functions. You can convert multi-value dimensions to standard SQL arrays either by explicitly by converting
-them with `MV_TO_ARRAY` or implicitly when used within the [array functions](./sql-array-functions.md). Arrays may
-also be constructed from multiple columns using the array functions.
+
+Druid supports `ARRAY` types constructed at query time. `ARRAY` types behave as standard SQL arrays, where results are grouped by matching entire arrays. This is in contrast to the implicit `UNNEST` that occurs when grouping on multi-value dimensions directly or when used with the multi-value functions.
+
+You can convert multi-value dimensions to standard SQL arrays explicitly by converting them with `MV_TO_ARRAY` or implicitly when used within the [array functions](./sql-array-functions.md). You can also use the array functions to construct arrays from multiple columns.
+
+You can use [schema auto-discovery for dimensions](../ingestion/schema-design.md#schema-auto-discovery-for-dimensions) to store arrays.
Review Comment:
This reads confusing to me. "schema auto-discovery...to store". I feel like auto discovery would detect an array that would later be stored as an array type? Not sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] 317brian commented on a diff in pull request #14161: docs: remove the note about segments
Posted by "317brian (via GitHub)" <gi...@apache.org>.
317brian commented on code in PR #14161:
URL: https://github.com/apache/druid/pull/14161#discussion_r1198032562
##########
docs/querying/sql-data-types.md:
##########
@@ -67,53 +66,51 @@ The following table describes how Druid maps SQL types onto native types when ru
|SMALLINT|LONG|`0`||
|INTEGER|LONG|`0`||
|BIGINT|LONG|`0`|Druid LONG columns (except `__time`) are reported as BIGINT|
-|TIMESTAMP|LONG|`0`, meaning 1970-01-01 00:00:00 UTC|Druid's `__time` column is reported as TIMESTAMP. Casts between string and timestamp types assume standard SQL formatting, e.g. `2000-01-02 03:04:05`, _not_ ISO8601 formatting. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
-|DATE|LONG|`0`, meaning 1970-01-01|Casting TIMESTAMP to DATE rounds down the timestamp to the nearest day. Casts between string and date types assume standard SQL formatting, e.g. `2000-01-02`. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
-|ARRAY|ARRAY|`NULL`|Druid native array types work as SQL arrays, and multi-value strings can be converted to arrays. See the [`ARRAY` details](#arrays).|
+|TIMESTAMP|LONG|`0`, meaning 1970-01-01 00:00:00 UTC|Druid's `__time` column is reported as TIMESTAMP. Casts between string and timestamp types assume standard SQL formatting—for example, `2000-01-02 03:04:05`—not ISO8601 formatting. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
+|DATE|LONG|`0`, meaning 1970-01-01|Casting TIMESTAMP to DATE rounds down the timestamp to the nearest day. Casts between string and date types assume standard SQL formatting—for example, `2000-01-02`. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
+|ARRAY|ARRAY|`NULL`|Druid native array types work as SQL arrays, and multi-value strings can be converted to arrays. See [Arrays](#arrays) for more information.|
|OTHER|COMPLEX|none|May represent various Druid column types such as hyperUnique, approxHistogram, etc.|
<sup>*</sup> Default value applies if `druid.generic.useDefaultValueForNull = true` (the default mode). Otherwise, the default value is `NULL` for all types.
## Multi-value strings
Druid's native type system allows strings to potentially have multiple values. These
-[multi-value string dimensions](multi-value-dimensions.md) are reported in SQL as `VARCHAR` typed, and can be
-syntactically used like any other `VARCHAR`. Regular string functions that refer to multi-value string dimensions are
+[multi-value string dimensions](multi-value-dimensions.md) are reported in SQL as VARCHAR typed, and can be
+syntactically used like any other VARCHAR. Regular string functions that refer to multi-value string dimensions are
applied to all values for each row individually. Multi-value string dimensions can also be treated as arrays via special
[multi-value string functions](sql-multivalue-string-functions.md), which can perform powerful array-aware operations, but retain
-their `VARCHAR` typing and behavior.
+their VARCHAR typing and behavior.
Grouping by a multi-value expression observes the native Druid multi-value aggregation behavior, which is similar to
-an implicit SQL `UNNEST`. Refer to the documentation on [multi-value string dimensions](multi-value-dimensions.md)
+the implicit SQL UNNEST operator. Refer to the documentation on [multi-value string dimensions](multi-value-dimensions.md)
Review Comment:
MVD groupby behaves like UNNEST in the sense that it can return more rows in your result than there are in the original table. I think we'd want to include some of the detail that exists on the MVD page. Specifically, this section: https://druid.apache.org/docs/latest/querying/multi-value-dimensions.html#grouping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] ektravel commented on pull request #14161: docs: remove the note about segments
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on PR #14161:
URL: https://github.com/apache/druid/pull/14161#issuecomment-1542564201
The broken link is a known issue and will be resolved by PR https://github.com/apache/druid/pull/14065
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vtlim commented on a diff in pull request #14161: docs: remove the note about segments
Posted by "vtlim (via GitHub)" <gi...@apache.org>.
vtlim commented on code in PR #14161:
URL: https://github.com/apache/druid/pull/14161#discussion_r1197044178
##########
docs/querying/sql-data-types.md:
##########
@@ -27,7 +27,7 @@ sidebar_label: "SQL data types"
> This document describes the SQL language.
-Columns in Druid are associated with a specific data type. This topic describes supported data types in [Druid SQL](./sql.md).
+Druid associates columns with a specific data type. This topic describes supported data types in [Druid SQL](./sql.md).
Review Comment:
We should be consistent in singular/plural (I realize this wasn't your change). Maybe something like "Druid associates each column with..."
##########
docs/querying/sql-data-types.md:
##########
@@ -67,53 +66,51 @@ The following table describes how Druid maps SQL types onto native types when ru
|SMALLINT|LONG|`0`||
|INTEGER|LONG|`0`||
|BIGINT|LONG|`0`|Druid LONG columns (except `__time`) are reported as BIGINT|
-|TIMESTAMP|LONG|`0`, meaning 1970-01-01 00:00:00 UTC|Druid's `__time` column is reported as TIMESTAMP. Casts between string and timestamp types assume standard SQL formatting, e.g. `2000-01-02 03:04:05`, _not_ ISO8601 formatting. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
-|DATE|LONG|`0`, meaning 1970-01-01|Casting TIMESTAMP to DATE rounds down the timestamp to the nearest day. Casts between string and date types assume standard SQL formatting, e.g. `2000-01-02`. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
-|ARRAY|ARRAY|`NULL`|Druid native array types work as SQL arrays, and multi-value strings can be converted to arrays. See the [`ARRAY` details](#arrays).|
+|TIMESTAMP|LONG|`0`, meaning 1970-01-01 00:00:00 UTC|Druid's `__time` column is reported as TIMESTAMP. Casts between string and timestamp types assume standard SQL formatting—for example, `2000-01-02 03:04:05`—not ISO8601 formatting. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
+|DATE|LONG|`0`, meaning 1970-01-01|Casting TIMESTAMP to DATE rounds down the timestamp to the nearest day. Casts between string and date types assume standard SQL formatting—for example, `2000-01-02`. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
+|ARRAY|ARRAY|`NULL`|Druid native array types work as SQL arrays, and multi-value strings can be converted to arrays. See [Arrays](#arrays) for more information.|
|OTHER|COMPLEX|none|May represent various Druid column types such as hyperUnique, approxHistogram, etc.|
<sup>*</sup> Default value applies if `druid.generic.useDefaultValueForNull = true` (the default mode). Otherwise, the default value is `NULL` for all types.
## Multi-value strings
Druid's native type system allows strings to potentially have multiple values. These
-[multi-value string dimensions](multi-value-dimensions.md) are reported in SQL as `VARCHAR` typed, and can be
-syntactically used like any other `VARCHAR`. Regular string functions that refer to multi-value string dimensions are
+[multi-value string dimensions](multi-value-dimensions.md) are reported in SQL as VARCHAR typed, and can be
+syntactically used like any other VARCHAR. Regular string functions that refer to multi-value string dimensions are
applied to all values for each row individually. Multi-value string dimensions can also be treated as arrays via special
[multi-value string functions](sql-multivalue-string-functions.md), which can perform powerful array-aware operations, but retain
-their `VARCHAR` typing and behavior.
+their VARCHAR typing and behavior.
Grouping by a multi-value expression observes the native Druid multi-value aggregation behavior, which is similar to
-an implicit SQL `UNNEST`. Refer to the documentation on [multi-value string dimensions](multi-value-dimensions.md)
+the implicit SQL UNNEST operator. Refer to the documentation on [multi-value string dimensions](multi-value-dimensions.md)
Review Comment:
I think this changes the meaning slightly? UNNEST isn't implicit in itself, if I understand correctly
##########
docs/querying/sql-data-types.md:
##########
@@ -67,53 +66,51 @@ The following table describes how Druid maps SQL types onto native types when ru
|SMALLINT|LONG|`0`||
|INTEGER|LONG|`0`||
|BIGINT|LONG|`0`|Druid LONG columns (except `__time`) are reported as BIGINT|
-|TIMESTAMP|LONG|`0`, meaning 1970-01-01 00:00:00 UTC|Druid's `__time` column is reported as TIMESTAMP. Casts between string and timestamp types assume standard SQL formatting, e.g. `2000-01-02 03:04:05`, _not_ ISO8601 formatting. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
-|DATE|LONG|`0`, meaning 1970-01-01|Casting TIMESTAMP to DATE rounds down the timestamp to the nearest day. Casts between string and date types assume standard SQL formatting, e.g. `2000-01-02`. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
-|ARRAY|ARRAY|`NULL`|Druid native array types work as SQL arrays, and multi-value strings can be converted to arrays. See the [`ARRAY` details](#arrays).|
+|TIMESTAMP|LONG|`0`, meaning 1970-01-01 00:00:00 UTC|Druid's `__time` column is reported as TIMESTAMP. Casts between string and timestamp types assume standard SQL formatting—for example, `2000-01-02 03:04:05`—not ISO8601 formatting. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
Review Comment:
```suggestion
|TIMESTAMP|LONG|`0`, meaning 1970-01-01 00:00:00 UTC|Druid's `__time` column is reported as TIMESTAMP. Casts between string and timestamp types assume standard SQL formatting—for example, `2000-01-02 03:04:05`—not ISO 8601 formatting. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
```
##########
docs/querying/sql-data-types.md:
##########
@@ -67,53 +66,51 @@ The following table describes how Druid maps SQL types onto native types when ru
|SMALLINT|LONG|`0`||
|INTEGER|LONG|`0`||
|BIGINT|LONG|`0`|Druid LONG columns (except `__time`) are reported as BIGINT|
-|TIMESTAMP|LONG|`0`, meaning 1970-01-01 00:00:00 UTC|Druid's `__time` column is reported as TIMESTAMP. Casts between string and timestamp types assume standard SQL formatting, e.g. `2000-01-02 03:04:05`, _not_ ISO8601 formatting. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
-|DATE|LONG|`0`, meaning 1970-01-01|Casting TIMESTAMP to DATE rounds down the timestamp to the nearest day. Casts between string and date types assume standard SQL formatting, e.g. `2000-01-02`. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
-|ARRAY|ARRAY|`NULL`|Druid native array types work as SQL arrays, and multi-value strings can be converted to arrays. See the [`ARRAY` details](#arrays).|
+|TIMESTAMP|LONG|`0`, meaning 1970-01-01 00:00:00 UTC|Druid's `__time` column is reported as TIMESTAMP. Casts between string and timestamp types assume standard SQL formatting—for example, `2000-01-02 03:04:05`—not ISO8601 formatting. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
+|DATE|LONG|`0`, meaning 1970-01-01|Casting TIMESTAMP to DATE rounds down the timestamp to the nearest day. Casts between string and date types assume standard SQL formatting—for example, `2000-01-02`. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
+|ARRAY|ARRAY|`NULL`|Druid native array types work as SQL arrays, and multi-value strings can be converted to arrays. See [Arrays](#arrays) for more information.|
|OTHER|COMPLEX|none|May represent various Druid column types such as hyperUnique, approxHistogram, etc.|
<sup>*</sup> Default value applies if `druid.generic.useDefaultValueForNull = true` (the default mode). Otherwise, the default value is `NULL` for all types.
## Multi-value strings
Druid's native type system allows strings to potentially have multiple values. These
-[multi-value string dimensions](multi-value-dimensions.md) are reported in SQL as `VARCHAR` typed, and can be
-syntactically used like any other `VARCHAR`. Regular string functions that refer to multi-value string dimensions are
+[multi-value string dimensions](multi-value-dimensions.md) are reported in SQL as VARCHAR typed, and can be
+syntactically used like any other VARCHAR. Regular string functions that refer to multi-value string dimensions are
applied to all values for each row individually. Multi-value string dimensions can also be treated as arrays via special
[multi-value string functions](sql-multivalue-string-functions.md), which can perform powerful array-aware operations, but retain
-their `VARCHAR` typing and behavior.
+their VARCHAR typing and behavior.
Grouping by a multi-value expression observes the native Druid multi-value aggregation behavior, which is similar to
-an implicit SQL `UNNEST`. Refer to the documentation on [multi-value string dimensions](multi-value-dimensions.md)
+the implicit SQL UNNEST operator. Refer to the documentation on [multi-value string dimensions](multi-value-dimensions.md)
for additional details.
-> Because multi-value dimensions are treated by the SQL planner as `VARCHAR`, there are some inconsistencies between how
-> they are handled in Druid SQL and in native queries. For example, expressions involving multi-value dimensions may be
-> incorrectly optimized by the Druid SQL planner: `multi_val_dim = 'a' AND multi_val_dim = 'b'` is optimized to
-> `false`, even though it is possible for a single row to have both "a" and "b" as values for `multi_val_dim`. The
-> SQL behavior of multi-value dimensions may change in a future release to more closely align with their behavior
-> in native queries, but the [multi-value string functions](./sql-multivalue-string-functions.md) should be able to provide
-> nearly all possible native functionality.
+> Because the SQL planner treats multi-value dimensions as VARCHAR, there are some inconsistencies between how they are handled in Druid SQL and in native queries. For instance, expressions involving multi-value dimensions may be incorrectly optimized by the Druid SQL planner. For example, `multi_val_dim = 'a' AND multi_val_dim = 'b'` is optimized to
+`false`, even though it is possible for a single row to have both `'a'` and `'b'` as values for `multi_val_dim`.
+>
+> The SQL behavior of multi-value dimensions may change in a future release to more closely align with their behavior in native queries, but the [multi-value string functions](./sql-multivalue-string-functions.md) should be able to provide nearly all possible native functionality.
## Arrays
-Druid supports `ARRAY` types constructed at query time, though it currently lacks the ability to store them in
Review Comment:
Should this have been removed?
>though it currently lacks the ability to store them in segments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vtlim commented on a diff in pull request #14161: docs: remove the note about segments
Posted by "vtlim (via GitHub)" <gi...@apache.org>.
vtlim commented on code in PR #14161:
URL: https://github.com/apache/druid/pull/14161#discussion_r1198011960
##########
docs/querying/sql-data-types.md:
##########
@@ -67,53 +66,51 @@ The following table describes how Druid maps SQL types onto native types when ru
|SMALLINT|LONG|`0`||
|INTEGER|LONG|`0`||
|BIGINT|LONG|`0`|Druid LONG columns (except `__time`) are reported as BIGINT|
-|TIMESTAMP|LONG|`0`, meaning 1970-01-01 00:00:00 UTC|Druid's `__time` column is reported as TIMESTAMP. Casts between string and timestamp types assume standard SQL formatting, e.g. `2000-01-02 03:04:05`, _not_ ISO8601 formatting. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
-|DATE|LONG|`0`, meaning 1970-01-01|Casting TIMESTAMP to DATE rounds down the timestamp to the nearest day. Casts between string and date types assume standard SQL formatting, e.g. `2000-01-02`. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
-|ARRAY|ARRAY|`NULL`|Druid native array types work as SQL arrays, and multi-value strings can be converted to arrays. See the [`ARRAY` details](#arrays).|
+|TIMESTAMP|LONG|`0`, meaning 1970-01-01 00:00:00 UTC|Druid's `__time` column is reported as TIMESTAMP. Casts between string and timestamp types assume standard SQL formatting—for example, `2000-01-02 03:04:05`—not ISO8601 formatting. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
+|DATE|LONG|`0`, meaning 1970-01-01|Casting TIMESTAMP to DATE rounds down the timestamp to the nearest day. Casts between string and date types assume standard SQL formatting—for example, `2000-01-02`. For handling other formats, use one of the [time functions](sql-scalar.md#date-and-time-functions).|
+|ARRAY|ARRAY|`NULL`|Druid native array types work as SQL arrays, and multi-value strings can be converted to arrays. See [Arrays](#arrays) for more information.|
|OTHER|COMPLEX|none|May represent various Druid column types such as hyperUnique, approxHistogram, etc.|
<sup>*</sup> Default value applies if `druid.generic.useDefaultValueForNull = true` (the default mode). Otherwise, the default value is `NULL` for all types.
## Multi-value strings
Druid's native type system allows strings to potentially have multiple values. These
-[multi-value string dimensions](multi-value-dimensions.md) are reported in SQL as `VARCHAR` typed, and can be
-syntactically used like any other `VARCHAR`. Regular string functions that refer to multi-value string dimensions are
+[multi-value string dimensions](multi-value-dimensions.md) are reported in SQL as VARCHAR typed, and can be
+syntactically used like any other VARCHAR. Regular string functions that refer to multi-value string dimensions are
applied to all values for each row individually. Multi-value string dimensions can also be treated as arrays via special
[multi-value string functions](sql-multivalue-string-functions.md), which can perform powerful array-aware operations, but retain
-their `VARCHAR` typing and behavior.
+their VARCHAR typing and behavior.
Grouping by a multi-value expression observes the native Druid multi-value aggregation behavior, which is similar to
-an implicit SQL `UNNEST`. Refer to the documentation on [multi-value string dimensions](multi-value-dimensions.md)
+the implicit SQL UNNEST operator. Refer to the documentation on [multi-value string dimensions](multi-value-dimensions.md)
Review Comment:
This is how I interpret the various sentences:
* Original: Grouping behaves similarly to an UNNEST operation that happens implicitly
* Change 1: Grouping behaves similarly to UNNEST, which is an implicit operator
* Change 2: Grouping behaves similarly to the UNNEST operator
Not sure if this is splitting hairs but it seems the meaning is slightly different for each.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org