You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/08/23 11:09:51 UTC
[GitHub] [druid] writer-jill opened a new pull request, #12946: Nested columns documentation
writer-jill opened a new pull request, #12946:
URL: https://github.com/apache/druid/pull/12946
Documentation for the nested columns feature.
This PR has:
- [x] been self-reviewed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r953570890
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,546 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ ~
+ ~ Nested columns is an experimental feature that is subject to
+ ~ change or removal at any time.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
+
+Druid SQL JSON functions let you extract, transform, and create `COMPLEX<json>` values. Additionally, you can use certain `JSONPath` operators to extract values from nested data structures.
+
+Druid currently supports ingesting JSON-format nested columns with this feature. If you want to ingest nested data in another format, consider using the [flattenSpec object](../ingestion/data-formats.md#flattenspec).
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ "time",
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ "time",
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](../misc/math-expr.md#nested-columns-functions) for information on the functions in the examples below.
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r954789335
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,519 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
+
+Druid SQL JSON functions let you extract, transform, and create `COMPLEX<json>` values. Additionally, you can use certain `JSONPath` operators to extract values from nested data structures.
+
+Druid currently supports ingesting JSON-format nested columns with this feature. If you want to ingest nested data in another format, consider using the [flattenSpec object](../ingestion/data-formats.md#flattenspec).
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ "time",
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ "time",
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- JSON_QUERY
+- JSON_OBJECT
+- PARSE_JSON
+- TO_JSON_STRING
+
+These functions are primarily intended for use with the Multi-Stage Query Architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
+
+#### Example query: Return results in a JSON object
+
+You can use the `JSON_QUERY` function to extract a partial structure from any JSON input and return results in a JSON object. Unlike `JSON_VALUE` it can extract objects and arrays.
+
+The following example query illustrates the differences in output between `JSON_VALUE` and `JSON_QUERY`.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ JSON_VALUE(shipTo, '$.address'),
+ JSON_QUERY(shipTo, '$.address'),
+ JSON_VALUE(shipTo, '$.phoneNumbers'),
+ JSON_QUERY(shipTo, '$.phoneNumbers')
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["STRING","COMPLEX","STRING","COMPLEX"],["VARCHAR","OTHER","VARCHAR","OTHER"],["","{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"}","","[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]"],["","{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"}","","[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]"],["","{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"}","","[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]"],["","{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCod
e\":\"98912-1195\"}","","[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]"],["","{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"}","","[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]"]]
Review Comment:
```suggestion
[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["VARCHAR","OTHER","VARCHAR","OTHER"],["","{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"}","","[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]"],["","{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"}","","[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]"],["","{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"}","","[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]"],["","{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW
\",\"postalCode\":\"98912-1195\"}","","[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]"],["","{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"}","","[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]"]]
```
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,519 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
+
+Druid SQL JSON functions let you extract, transform, and create `COMPLEX<json>` values. Additionally, you can use certain `JSONPath` operators to extract values from nested data structures.
+
+Druid currently supports ingesting JSON-format nested columns with this feature. If you want to ingest nested data in another format, consider using the [flattenSpec object](../ingestion/data-formats.md#flattenspec).
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ "time",
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ "time",
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- JSON_QUERY
+- JSON_OBJECT
+- PARSE_JSON
+- TO_JSON_STRING
+
+These functions are primarily intended for use with the Multi-Stage Query Architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
+
+#### Example query: Return results in a JSON object
+
+You can use the `JSON_QUERY` function to extract a partial structure from any JSON input and return results in a JSON object. Unlike `JSON_VALUE` it can extract objects and arrays.
+
+The following example query illustrates the differences in output between `JSON_VALUE` and `JSON_QUERY`.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ JSON_VALUE(shipTo, '$.address'),
+ JSON_QUERY(shipTo, '$.address'),
+ JSON_VALUE(shipTo, '$.phoneNumbers'),
+ JSON_QUERY(shipTo, '$.phoneNumbers')
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["STRING","COMPLEX","STRING","COMPLEX"],["VARCHAR","OTHER","VARCHAR","OTHER"],["","{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"}","","[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]"],["","{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"}","","[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]"],["","{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"}","","[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]"],["","{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCod
e\":\"98912-1195\"}","","[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]"],["","{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"}","","[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]"]]
+```
+
+#### Example query: Combine multiple JSON inputs into a single JSON object value
+
+ The following query illustrates how to use `JSON_OBJECT` to combine nested data elements into a new object.
+
+ > Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ JSON_OBJECT(KEY 'shipTo' VALUE JSON_QUERY(shipTo, '$'), KEY 'details' VALUE JSON_QUERY(details, '$')) as combinedJson
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["combinedJson"],["COMPLEX"],["OTHER"],["{\"details\":{\"color\":\"sky blue\",\"price\":542.0},\"shipTo\":{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}}"],["{\"details\":{\"color\":\"ivory\",\"price\":955.0},\"shipTo\":{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}}"],["{\"details\":{\"price\":8.0},\"shipTo\":{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"st
ate\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}}"],["{\"details\":{\"color\":\"olive\",\"price\":90.0},\"shipTo\":{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}}"],["{\"details\":{\"color\":\"plum\",\"price\":40.0},\"shipTo\":{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}}"]]
+```
+
+### Using other transform functions
+
+Druid provides the following additional transform functions:
+
+- `PARSE_JSON`: Deserializes a string value into a JSON object.
+- `TO_JSON_STRING`: Performs the operation of `TO_JSON` and then serializes the value into a string.
+
+#### Example query: Parse and deserialize data
+
+ The following query illustrates how to use the transform functions to parse and deserialize data.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ PARSE_JSON('{"x":"y"}'),
+ TO_JSON_STRING('{"x":"y"}'),
+ TO_JSON_STRING(PARSE_JSON('{"x":"y"}'))
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$2","EXPR$3"],["COMPLEX","STRING","STRING"],["OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+### Using helper operators
+
+The `JSON_KEYS` and `JSON_PATHS` functions are helper operators that you can use to examine JSON object schema. Use them to plan your queries, for example to work out which paths to use in `JSON_VALUE`.
+
+#### Example query: Examine JSON object schema
+
+ The following query illustrates how to use the helper operators to examine a nested data object.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.address')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_PATHS(shipTo))
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["COMPLEX","COMPLEX","STRING","STRING"],["OTHER","OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
Review Comment:
```suggestion
[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["COMPLEX<json>","COMPLEX<json>","STRING","STRING"],["OTHER","OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
```
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,519 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
+
+Druid SQL JSON functions let you extract, transform, and create `COMPLEX<json>` values. Additionally, you can use certain `JSONPath` operators to extract values from nested data structures.
+
+Druid currently supports ingesting JSON-format nested columns with this feature. If you want to ingest nested data in another format, consider using the [flattenSpec object](../ingestion/data-formats.md#flattenspec).
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ "time",
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ "time",
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- JSON_QUERY
+- JSON_OBJECT
+- PARSE_JSON
+- TO_JSON_STRING
+
+These functions are primarily intended for use with the Multi-Stage Query Architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
+
+#### Example query: Return results in a JSON object
+
+You can use the `JSON_QUERY` function to extract a partial structure from any JSON input and return results in a JSON object. Unlike `JSON_VALUE` it can extract objects and arrays.
+
+The following example query illustrates the differences in output between `JSON_VALUE` and `JSON_QUERY`.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ JSON_VALUE(shipTo, '$.address'),
+ JSON_QUERY(shipTo, '$.address'),
+ JSON_VALUE(shipTo, '$.phoneNumbers'),
+ JSON_QUERY(shipTo, '$.phoneNumbers')
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["STRING","COMPLEX","STRING","COMPLEX"],["VARCHAR","OTHER","VARCHAR","OTHER"],["","{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"}","","[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]"],["","{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"}","","[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]"],["","{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"}","","[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]"],["","{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCod
e\":\"98912-1195\"}","","[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]"],["","{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"}","","[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]"]]
+```
+
+#### Example query: Combine multiple JSON inputs into a single JSON object value
+
+ The following query illustrates how to use `JSON_OBJECT` to combine nested data elements into a new object.
+
+ > Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ JSON_OBJECT(KEY 'shipTo' VALUE JSON_QUERY(shipTo, '$'), KEY 'details' VALUE JSON_QUERY(details, '$')) as combinedJson
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["combinedJson"],["COMPLEX"],["OTHER"],["{\"details\":{\"color\":\"sky blue\",\"price\":542.0},\"shipTo\":{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}}"],["{\"details\":{\"color\":\"ivory\",\"price\":955.0},\"shipTo\":{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}}"],["{\"details\":{\"price\":8.0},\"shipTo\":{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"st
ate\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}}"],["{\"details\":{\"color\":\"olive\",\"price\":90.0},\"shipTo\":{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}}"],["{\"details\":{\"color\":\"plum\",\"price\":40.0},\"shipTo\":{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}}"]]
+```
+
+### Using other transform functions
+
+Druid provides the following additional transform functions:
+
+- `PARSE_JSON`: Deserializes a string value into a JSON object.
+- `TO_JSON_STRING`: Performs the operation of `TO_JSON` and then serializes the value into a string.
+
+#### Example query: Parse and deserialize data
+
+ The following query illustrates how to use the transform functions to parse and deserialize data.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ PARSE_JSON('{"x":"y"}'),
+ TO_JSON_STRING('{"x":"y"}'),
+ TO_JSON_STRING(PARSE_JSON('{"x":"y"}'))
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$2","EXPR$3"],["COMPLEX","STRING","STRING"],["OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
Review Comment:
```suggestion
[["EXPR$0","EXPR$2","EXPR$3"],["COMPLEX<json>","STRING","STRING"],["OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
```
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,519 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
+
+Druid SQL JSON functions let you extract, transform, and create `COMPLEX<json>` values. Additionally, you can use certain `JSONPath` operators to extract values from nested data structures.
+
+Druid currently supports ingesting JSON-format nested columns with this feature. If you want to ingest nested data in another format, consider using the [flattenSpec object](../ingestion/data-formats.md#flattenspec).
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ "time",
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ "time",
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- JSON_QUERY
+- JSON_OBJECT
+- PARSE_JSON
+- TO_JSON_STRING
+
+These functions are primarily intended for use with the Multi-Stage Query Architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
+
+#### Example query: Return results in a JSON object
+
+You can use the `JSON_QUERY` function to extract a partial structure from any JSON input and return results in a JSON object. Unlike `JSON_VALUE` it can extract objects and arrays.
+
+The following example query illustrates the differences in output between `JSON_VALUE` and `JSON_QUERY`.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ JSON_VALUE(shipTo, '$.address'),
+ JSON_QUERY(shipTo, '$.address'),
+ JSON_VALUE(shipTo, '$.phoneNumbers'),
+ JSON_QUERY(shipTo, '$.phoneNumbers')
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["STRING","COMPLEX","STRING","COMPLEX"],["VARCHAR","OTHER","VARCHAR","OTHER"],["","{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"}","","[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]"],["","{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"}","","[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]"],["","{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"}","","[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]"],["","{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCod
e\":\"98912-1195\"}","","[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]"],["","{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"}","","[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]"]]
+```
+
+#### Example query: Combine multiple JSON inputs into a single JSON object value
+
+ The following query illustrates how to use `JSON_OBJECT` to combine nested data elements into a new object.
+
+ > Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ JSON_OBJECT(KEY 'shipTo' VALUE JSON_QUERY(shipTo, '$'), KEY 'details' VALUE JSON_QUERY(details, '$')) as combinedJson
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["combinedJson"],["COMPLEX"],["OTHER"],["{\"details\":{\"color\":\"sky blue\",\"price\":542.0},\"shipTo\":{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}}"],["{\"details\":{\"color\":\"ivory\",\"price\":955.0},\"shipTo\":{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}}"],["{\"details\":{\"price\":8.0},\"shipTo\":{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"st
ate\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}}"],["{\"details\":{\"color\":\"olive\",\"price\":90.0},\"shipTo\":{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}}"],["{\"details\":{\"color\":\"plum\",\"price\":40.0},\"shipTo\":{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}}"]]
Review Comment:
```suggestion
[["combinedJson"],["COMPLEX<json>"],["OTHER"],["{\"details\":{\"color\":\"sky blue\",\"price\":542.0},\"shipTo\":{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}}"],["{\"details\":{\"color\":\"ivory\",\"price\":955.0},\"shipTo\":{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}}"],["{\"details\":{\"price\":8.0},\"shipTo\":{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserboroug
h\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}}"],["{\"details\":{\"color\":\"olive\",\"price\":90.0},\"shipTo\":{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}}"],["{\"details\":{\"color\":\"plum\",\"price\":40.0},\"shipTo\":{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}}"]]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
clintropolis commented on PR #12946:
URL: https://github.com/apache/druid/pull/12946#issuecomment-1232224043
looks like legitimate ci failures:
```
Site built successfully. Generated files in 'build' folder.
Could not find './nested-columns.md' linked from './build/ApacheDruid/docs/misc/math-expr.html'
Could not find '../multi-stage-query/index.md' linked from './build/ApacheDruid/docs/querying/nested-columns.html'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
writer-jill commented on PR #12946:
URL: https://github.com/apache/druid/pull/12946#issuecomment-1228292774
Ready for review by @techdocsmith
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r956494483
##########
website/sidebars.json:
##########
@@ -77,6 +77,7 @@
"querying/sql-scalar",
"querying/sql-aggregations",
"querying/sql-multivalue-string-functions",
+ "querying/nested-columns",
Review Comment:
i forgot to add sql-json-functions here in https://github.com/apache/druid/pull/12922, could you add it?
Also, i think maybe nested-columns should live in "concepts" section instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r953566654
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,546 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ ~
+ ~ Nested columns is an experimental feature that is subject to
+ ~ change or removal at any time.
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vtlim commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
vtlim commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r957876936
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
Review Comment:
```suggestion
Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format and specialized internal columns and indexes for nested literal values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
```
Suggest removing comma since these aren't two independent clauses, and suggest removing `'` from literal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r958651792
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
Review Comment:
Done
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r953576869
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,546 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ ~
+ ~ Nested columns is an experimental feature that is subject to
+ ~ change or removal at any time.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
+
+Druid SQL JSON functions let you extract, transform, and create `COMPLEX<json>` values. Additionally, you can use certain `JSONPath` operators to extract values from nested data structures.
+
+Druid currently supports ingesting JSON-format nested columns with this feature. If you want to ingest nested data in another format, consider using the [flattenSpec object](../ingestion/data-formats.md#flattenspec).
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ "time",
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ "time",
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](../misc/math-expr.md#nested-columns-functions) for information on the functions in the examples below.
+
+### JSONPath syntax
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures.
Review Comment:
I think we should nominate a single location for the JSONPath syntax reference and link to it from the other relevant pages. Minimising duplication is our preferred approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r953567671
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,546 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ ~
+ ~ Nested columns is an experimental feature that is subject to
+ ~ change or removal at any time.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
Review Comment:
This is our standard wording for new features - the same text is on the MSQ pages. @techdocsmith any comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r958651184
##########
website/sidebars.json:
##########
@@ -77,6 +77,7 @@
"querying/sql-scalar",
"querying/sql-aggregations",
"querying/sql-multivalue-string-functions",
+ "querying/nested-columns",
Review Comment:
Done
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r953576869
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,546 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ ~
+ ~ Nested columns is an experimental feature that is subject to
+ ~ change or removal at any time.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
+
+Druid SQL JSON functions let you extract, transform, and create `COMPLEX<json>` values. Additionally, you can use certain `JSONPath` operators to extract values from nested data structures.
+
+Druid currently supports ingesting JSON-format nested columns with this feature. If you want to ingest nested data in another format, consider using the [flattenSpec object](../ingestion/data-formats.md#flattenspec).
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ "time",
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ "time",
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](../misc/math-expr.md#nested-columns-functions) for information on the functions in the examples below.
+
+### JSONPath syntax
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures.
Review Comment:
I think we should nominate a single location for the JSONPath syntax reference and link to it from the other relevant pages. Minimising duplication is our preferred approach. Suggest we keep it with the JSON functions reference, I remove it from this page and link to it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r958652361
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
+
+
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- JSON_QUERY
+- JSON_OBJECT
+- PARSE_JSON
+- TO_JSON_STRING
+
+These functions are primarily intended for use with the Multi-Stage Query Architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
+
+#### Example query: Return results in a JSON object
+
+You can use the `JSON_QUERY` function to extract a partial structure from any JSON input and return results in a JSON object. Unlike `JSON_VALUE` it can extract objects and arrays.
+
+The following example query illustrates the differences in output between `JSON_VALUE` and `JSON_QUERY`.
+
+
+
+```sql
+SELECT
+ JSON_VALUE(shipTo, '$.address'),
+ JSON_QUERY(shipTo, '$.address'),
+ JSON_VALUE(shipTo, '$.phoneNumbers'),
+ JSON_QUERY(shipTo, '$.phoneNumbers')
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["VARCHAR","OTHER","VARCHAR","OTHER"],["","{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"}","","[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]"],["","{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"}","","[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]"],["","{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"}","","[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]"],["","{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\"
,\"postalCode\":\"98912-1195\"}","","[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]"],["","{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"}","","[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]"]]
+```
+
+#### Example query: Combine multiple JSON inputs into a single JSON object value
+
+ The following query illustrates how to use `JSON_OBJECT` to combine nested data elements into a new object.
+
+
+
+```sql
+SELECT
+ JSON_OBJECT(KEY 'shipTo' VALUE JSON_QUERY(shipTo, '$'), KEY 'details' VALUE JSON_QUERY(details, '$')) as combinedJson
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["combinedJson"],["COMPLEX<json>"],["OTHER"],["{\"details\":{\"color\":\"sky blue\",\"price\":542.0},\"shipTo\":{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}}"],["{\"details\":{\"color\":\"ivory\",\"price\":955.0},\"shipTo\":{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}}"],["{\"details\":{\"price\":8.0},\"shipTo\":{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\
",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}}"],["{\"details\":{\"color\":\"olive\",\"price\":90.0},\"shipTo\":{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}}"],["{\"details\":{\"color\":\"plum\",\"price\":40.0},\"shipTo\":{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}}"]]
+```
+
+### Using other transform functions
+
+Druid provides the following additional transform functions:
+
+- `PARSE_JSON`: Deserializes a string value into a JSON object.
+- `TO_JSON_STRING`: Performs the operation of `TO_JSON` and then serializes the value into a string.
+
+#### Example query: Parse and deserialize data
+
+ The following query illustrates how to use the transform functions to parse and deserialize data.
+
+
+
+```sql
+SELECT
+ PARSE_JSON('{"x":"y"}'),
+ TO_JSON_STRING('{"x":"y"}'),
+ TO_JSON_STRING(PARSE_JSON('{"x":"y"}'))
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$2","EXPR$3"],["COMPLEX<json>","STRING","STRING"],["OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+### Using helper operators
+
+The `JSON_KEYS` and `JSON_PATHS` functions are helper operators that you can use to examine JSON object schema. Use them to plan your queries, for example to work out which paths to use in `JSON_VALUE`.
+
+#### Example query: Examine JSON object schema
+
+ The following query illustrates how to use the helper operators to examine a nested data object.
+
+
+
+```sql
+SELECT
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.address')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_PATHS(shipTo))
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["COMPLEX<json>","COMPLEX<json>","STRING","STRING"],["OTHER","OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+## Known issues
+
+Before you start using the nested columns feature, consider the following known issues:
+
+- Directly using `COMPLEX<json>` typed columns and expressions is not well integrated into the Druid query engine. It can result in errors or undefined behavior when grouping and filtering, and when you use using `COMPLEX<json>` objects as inputs to aggregators. As a workaround, consider using `TO_JSON_STRING` to coerce the values to strings before you perform these operations.
Review Comment:
Done
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
+
+
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- JSON_QUERY
+- JSON_OBJECT
+- PARSE_JSON
+- TO_JSON_STRING
+
+These functions are primarily intended for use with the Multi-Stage Query Architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
+
+#### Example query: Return results in a JSON object
+
+You can use the `JSON_QUERY` function to extract a partial structure from any JSON input and return results in a JSON object. Unlike `JSON_VALUE` it can extract objects and arrays.
+
+The following example query illustrates the differences in output between `JSON_VALUE` and `JSON_QUERY`.
+
+
+
+```sql
+SELECT
+ JSON_VALUE(shipTo, '$.address'),
+ JSON_QUERY(shipTo, '$.address'),
+ JSON_VALUE(shipTo, '$.phoneNumbers'),
+ JSON_QUERY(shipTo, '$.phoneNumbers')
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["VARCHAR","OTHER","VARCHAR","OTHER"],["","{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"}","","[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]"],["","{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"}","","[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]"],["","{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"}","","[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]"],["","{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\"
,\"postalCode\":\"98912-1195\"}","","[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]"],["","{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"}","","[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]"]]
+```
+
+#### Example query: Combine multiple JSON inputs into a single JSON object value
+
+ The following query illustrates how to use `JSON_OBJECT` to combine nested data elements into a new object.
+
+
+
+```sql
+SELECT
+ JSON_OBJECT(KEY 'shipTo' VALUE JSON_QUERY(shipTo, '$'), KEY 'details' VALUE JSON_QUERY(details, '$')) as combinedJson
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["combinedJson"],["COMPLEX<json>"],["OTHER"],["{\"details\":{\"color\":\"sky blue\",\"price\":542.0},\"shipTo\":{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}}"],["{\"details\":{\"color\":\"ivory\",\"price\":955.0},\"shipTo\":{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}}"],["{\"details\":{\"price\":8.0},\"shipTo\":{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\
",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}}"],["{\"details\":{\"color\":\"olive\",\"price\":90.0},\"shipTo\":{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}}"],["{\"details\":{\"color\":\"plum\",\"price\":40.0},\"shipTo\":{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}}"]]
+```
+
+### Using other transform functions
+
+Druid provides the following additional transform functions:
+
+- `PARSE_JSON`: Deserializes a string value into a JSON object.
+- `TO_JSON_STRING`: Performs the operation of `TO_JSON` and then serializes the value into a string.
+
+#### Example query: Parse and deserialize data
+
+ The following query illustrates how to use the transform functions to parse and deserialize data.
+
+
+
+```sql
+SELECT
+ PARSE_JSON('{"x":"y"}'),
+ TO_JSON_STRING('{"x":"y"}'),
+ TO_JSON_STRING(PARSE_JSON('{"x":"y"}'))
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$2","EXPR$3"],["COMPLEX<json>","STRING","STRING"],["OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+### Using helper operators
+
+The `JSON_KEYS` and `JSON_PATHS` functions are helper operators that you can use to examine JSON object schema. Use them to plan your queries, for example to work out which paths to use in `JSON_VALUE`.
+
+#### Example query: Examine JSON object schema
+
+ The following query illustrates how to use the helper operators to examine a nested data object.
+
+
+
+```sql
+SELECT
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.address')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_PATHS(shipTo))
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["COMPLEX<json>","COMPLEX<json>","STRING","STRING"],["OTHER","OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+## Known issues
+
+Before you start using the nested columns feature, consider the following known issues:
+
+- Directly using `COMPLEX<json>` typed columns and expressions is not well integrated into the Druid query engine. It can result in errors or undefined behavior when grouping and filtering, and when you use using `COMPLEX<json>` objects as inputs to aggregators. As a workaround, consider using `TO_JSON_STRING` to coerce the values to strings before you perform these operations.
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r958652629
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
+
+
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- JSON_QUERY
+- JSON_OBJECT
+- PARSE_JSON
+- TO_JSON_STRING
+
+These functions are primarily intended for use with the Multi-Stage Query Architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
+
+#### Example query: Return results in a JSON object
+
+You can use the `JSON_QUERY` function to extract a partial structure from any JSON input and return results in a JSON object. Unlike `JSON_VALUE` it can extract objects and arrays.
+
+The following example query illustrates the differences in output between `JSON_VALUE` and `JSON_QUERY`.
+
+
+
+```sql
+SELECT
+ JSON_VALUE(shipTo, '$.address'),
+ JSON_QUERY(shipTo, '$.address'),
+ JSON_VALUE(shipTo, '$.phoneNumbers'),
+ JSON_QUERY(shipTo, '$.phoneNumbers')
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["VARCHAR","OTHER","VARCHAR","OTHER"],["","{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"}","","[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]"],["","{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"}","","[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]"],["","{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"}","","[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]"],["","{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\"
,\"postalCode\":\"98912-1195\"}","","[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]"],["","{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"}","","[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]"]]
+```
+
+#### Example query: Combine multiple JSON inputs into a single JSON object value
+
+ The following query illustrates how to use `JSON_OBJECT` to combine nested data elements into a new object.
+
+
+
+```sql
+SELECT
+ JSON_OBJECT(KEY 'shipTo' VALUE JSON_QUERY(shipTo, '$'), KEY 'details' VALUE JSON_QUERY(details, '$')) as combinedJson
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["combinedJson"],["COMPLEX<json>"],["OTHER"],["{\"details\":{\"color\":\"sky blue\",\"price\":542.0},\"shipTo\":{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}}"],["{\"details\":{\"color\":\"ivory\",\"price\":955.0},\"shipTo\":{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}}"],["{\"details\":{\"price\":8.0},\"shipTo\":{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\
",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}}"],["{\"details\":{\"color\":\"olive\",\"price\":90.0},\"shipTo\":{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}}"],["{\"details\":{\"color\":\"plum\",\"price\":40.0},\"shipTo\":{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}}"]]
+```
+
+### Using other transform functions
+
+Druid provides the following additional transform functions:
+
+- `PARSE_JSON`: Deserializes a string value into a JSON object.
+- `TO_JSON_STRING`: Performs the operation of `TO_JSON` and then serializes the value into a string.
+
+#### Example query: Parse and deserialize data
+
+ The following query illustrates how to use the transform functions to parse and deserialize data.
+
+
+
+```sql
+SELECT
+ PARSE_JSON('{"x":"y"}'),
+ TO_JSON_STRING('{"x":"y"}'),
+ TO_JSON_STRING(PARSE_JSON('{"x":"y"}'))
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$2","EXPR$3"],["COMPLEX<json>","STRING","STRING"],["OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+### Using helper operators
+
+The `JSON_KEYS` and `JSON_PATHS` functions are helper operators that you can use to examine JSON object schema. Use them to plan your queries, for example to work out which paths to use in `JSON_VALUE`.
+
+#### Example query: Examine JSON object schema
+
+ The following query illustrates how to use the helper operators to examine a nested data object.
+
+
+
+```sql
+SELECT
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.address')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_PATHS(shipTo))
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["COMPLEX<json>","COMPLEX<json>","STRING","STRING"],["OTHER","OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+## Known issues
+
+Before you start using the nested columns feature, consider the following known issues:
+
+- Directly using `COMPLEX<json>` typed columns and expressions is not well integrated into the Druid query engine. It can result in errors or undefined behavior when grouping and filtering, and when you use using `COMPLEX<json>` objects as inputs to aggregators. As a workaround, consider using `TO_JSON_STRING` to coerce the values to strings before you perform these operations.
+- Directly using array-typed outputs from `JSON_KEYS` and `JSON_PATHS` is moderately supported by the Druid query engine. You can group on these outputs, and there are a number of array expressions that can operate on these values, such as `ARRAY_CONCAT_AGG`. However, some operations are not well defined for use outside array-specific functions, such as filtering using `=` or `IS NULL`.
+- Input validation for JSON SQL operators is currently incomplete, which sometimes results in undefined behavior or unhelpful error messages.
+- Ingesting JSON columns with a very complex nested structure is potentially an expensive operation, and may require you to tune ingestion tasks and/or cluster parameters to account for increased memory usage or overall task run time. When you tune your ingestion configuration, treat each nested literal field inside a JSON object as a flattened top-level Druid column.
Review Comment:
Done
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
+
+
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- JSON_QUERY
+- JSON_OBJECT
+- PARSE_JSON
+- TO_JSON_STRING
+
+These functions are primarily intended for use with the Multi-Stage Query Architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
+
+#### Example query: Return results in a JSON object
+
+You can use the `JSON_QUERY` function to extract a partial structure from any JSON input and return results in a JSON object. Unlike `JSON_VALUE` it can extract objects and arrays.
+
+The following example query illustrates the differences in output between `JSON_VALUE` and `JSON_QUERY`.
+
+
+
+```sql
+SELECT
+ JSON_VALUE(shipTo, '$.address'),
+ JSON_QUERY(shipTo, '$.address'),
+ JSON_VALUE(shipTo, '$.phoneNumbers'),
+ JSON_QUERY(shipTo, '$.phoneNumbers')
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["VARCHAR","OTHER","VARCHAR","OTHER"],["","{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"}","","[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]"],["","{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"}","","[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]"],["","{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"}","","[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]"],["","{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\"
,\"postalCode\":\"98912-1195\"}","","[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]"],["","{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"}","","[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]"]]
+```
+
+#### Example query: Combine multiple JSON inputs into a single JSON object value
+
+ The following query illustrates how to use `JSON_OBJECT` to combine nested data elements into a new object.
+
+
+
+```sql
+SELECT
+ JSON_OBJECT(KEY 'shipTo' VALUE JSON_QUERY(shipTo, '$'), KEY 'details' VALUE JSON_QUERY(details, '$')) as combinedJson
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["combinedJson"],["COMPLEX<json>"],["OTHER"],["{\"details\":{\"color\":\"sky blue\",\"price\":542.0},\"shipTo\":{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}}"],["{\"details\":{\"color\":\"ivory\",\"price\":955.0},\"shipTo\":{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}}"],["{\"details\":{\"price\":8.0},\"shipTo\":{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\
",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}}"],["{\"details\":{\"color\":\"olive\",\"price\":90.0},\"shipTo\":{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}}"],["{\"details\":{\"color\":\"plum\",\"price\":40.0},\"shipTo\":{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}}"]]
+```
+
+### Using other transform functions
+
+Druid provides the following additional transform functions:
+
+- `PARSE_JSON`: Deserializes a string value into a JSON object.
+- `TO_JSON_STRING`: Performs the operation of `TO_JSON` and then serializes the value into a string.
+
+#### Example query: Parse and deserialize data
+
+ The following query illustrates how to use the transform functions to parse and deserialize data.
+
+
+
+```sql
+SELECT
+ PARSE_JSON('{"x":"y"}'),
+ TO_JSON_STRING('{"x":"y"}'),
+ TO_JSON_STRING(PARSE_JSON('{"x":"y"}'))
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$2","EXPR$3"],["COMPLEX<json>","STRING","STRING"],["OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+### Using helper operators
+
+The `JSON_KEYS` and `JSON_PATHS` functions are helper operators that you can use to examine JSON object schema. Use them to plan your queries, for example to work out which paths to use in `JSON_VALUE`.
+
+#### Example query: Examine JSON object schema
+
+ The following query illustrates how to use the helper operators to examine a nested data object.
+
+
+
+```sql
+SELECT
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.address')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_PATHS(shipTo))
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["COMPLEX<json>","COMPLEX<json>","STRING","STRING"],["OTHER","OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+## Known issues
+
+Before you start using the nested columns feature, consider the following known issues:
+
+- Directly using `COMPLEX<json>` typed columns and expressions is not well integrated into the Druid query engine. It can result in errors or undefined behavior when grouping and filtering, and when you use using `COMPLEX<json>` objects as inputs to aggregators. As a workaround, consider using `TO_JSON_STRING` to coerce the values to strings before you perform these operations.
+- Directly using array-typed outputs from `JSON_KEYS` and `JSON_PATHS` is moderately supported by the Druid query engine. You can group on these outputs, and there are a number of array expressions that can operate on these values, such as `ARRAY_CONCAT_AGG`. However, some operations are not well defined for use outside array-specific functions, such as filtering using `=` or `IS NULL`.
+- Input validation for JSON SQL operators is currently incomplete, which sometimes results in undefined behavior or unhelpful error messages.
+- Ingesting JSON columns with a very complex nested structure is potentially an expensive operation, and may require you to tune ingestion tasks and/or cluster parameters to account for increased memory usage or overall task run time. When you tune your ingestion configuration, treat each nested literal field inside a JSON object as a flattened top-level Druid column.
+
+## Further reading
+
+For more information, see the following pages:
+
+- [Nested columns functions reference](./sql-json-functions.md) for details of the functions used in the examples on this page.
+- [Ingestion spec reference](../ingestion/ingestion-spec.md) for information on native ingestion and [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+- [Multi-stage query architecture overview](../multi-stage-query/index.md) for information on how to set up and use this feature.
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vtlim merged pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
vtlim merged PR #12946:
URL: https://github.com/apache/druid/pull/12946
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r953180422
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,546 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ ~
+ ~ Nested columns is an experimental feature that is subject to
+ ~ change or removal at any time.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
+
+Druid SQL JSON functions let you extract, transform, and create `COMPLEX<json>` values. Additionally, you can use certain `JSONPath` operators to extract values from nested data structures.
+
+Druid currently supports ingesting JSON-format nested columns with this feature. If you want to ingest nested data in another format, consider using the [flattenSpec object](../ingestion/data-formats.md#flattenspec).
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ "time",
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ "time",
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](../misc/math-expr.md#nested-columns-functions) for information on the functions in the examples below.
+
+### JSONPath syntax
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures.
Review Comment:
this section also lives here https://github.com/apache/druid/blob/master/docs/querying/sql-json-functions.md#jsonpath-syntax (with slightly adjusted wording based on review) and then have another version that duplicates just the table and links to the sql page for the examples here https://github.com/apache/druid/blob/master/docs/misc/math-expr.md#jsonpath-syntax
I'm not sure what is the most appropriate way to handle this, at some point in the future we will likely update the supported subset (to add wildcards, etc) and so maybe should also be considering how we want to make sure we keep everything in sync.
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,546 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ ~
+ ~ Nested columns is an experimental feature that is subject to
+ ~ change or removal at any time.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
Review Comment:
I think we should consider customizing the wording here a bit to elaborate on what experimental means for nested columns. A very wordy version is included https://github.com/apache/druid/pull/12753#issuecomment-1182535667 which I summed up as
>So, long story short, due to the unknowns I think the answer for right now is that operators should experiment with COMPLEX<json> columns to see if they work well for their use case, and use them in production if so, otherwise provide feedback so that we can continue to make improvements and expand the use cases this is good for?
I mostly don't want to scare people away from using the feature; since it is a column type, using it means that the data is like that in your segments forever, so there is a bit of a commitment to start using it. But because of this, we included versioning and the like within the segment format, and so will basically always support backwards compatibility of being able to read segments written using this initial release of the feature with all future releases.
Not sure what the best (concise) way to frame it is, i'll try to think a bit about it
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,546 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ ~
+ ~ Nested columns is an experimental feature that is subject to
+ ~ change or removal at any time.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
+
+Druid SQL JSON functions let you extract, transform, and create `COMPLEX<json>` values. Additionally, you can use certain `JSONPath` operators to extract values from nested data structures.
+
+Druid currently supports ingesting JSON-format nested columns with this feature. If you want to ingest nested data in another format, consider using the [flattenSpec object](../ingestion/data-formats.md#flattenspec).
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ "time",
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ "time",
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](../misc/math-expr.md#nested-columns-functions) for information on the functions in the examples below.
Review Comment:
same comment about linking to https://github.com/apache/druid/blob/master/docs/querying/sql-json-functions.md instead of 'native' expressions
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,546 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ ~
+ ~ Nested columns is an experimental feature that is subject to
+ ~ change or removal at any time.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
+
+Druid SQL JSON functions let you extract, transform, and create `COMPLEX<json>` values. Additionally, you can use certain `JSONPath` operators to extract values from nested data structures.
+
+Druid currently supports ingesting JSON-format nested columns with this feature. If you want to ingest nested data in another format, consider using the [flattenSpec object](../ingestion/data-formats.md#flattenspec).
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ "time",
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ "time",
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](../misc/math-expr.md#nested-columns-functions) for information on the functions in the examples below.
+
+### JSONPath syntax
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures.
+
+|Operator|Description|
+| --- | --- |
+|`$`| Root element. All JSONPath expressions start with this operator. |
+|`.<name>`| Child element in dot notation. |
+|`['<name>']`| Child element in bracket notation. |
+|`[<number>]`| Array index. |
+
+Consider the following example input JSON:
+
+```json
+{"x":1, "y":[1, 2, 3]}
+```
+
+- To return the JSON object:<br>
+`$` -> `{"x":1, "y":[1, 2, 3]}`
+- To return the value of a key "x":<br>
+`$.x` -> `1`
+- For a key that contains an array, to return the entire array:<br>
+`$['y']` -> `[1, 2, 3]`
+- For a key that contains an array, to return an item in the array:<br>
+`$.y[1]` -> `2`
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- JSON_QUERY
+- JSON_OBJECT
+- PARSE_JSON
+- TO_JSON_STRING
+
+These functions are primarily intended for use with the Multi-Stage Query Architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
+
+#### Example query: Return results in a JSON object
+
+You can use the `JSON_QUERY` function to extract a partial structure from any JSON input and return results in a JSON object. Unlike `JSON_VALUE` it can extract objects and arrays.
+
+The following example query illustrates the differences in output between `JSON_VALUE` and `JSON_QUERY`.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ JSON_VALUE(shipTo, '$.address'),
+ JSON_QUERY(shipTo, '$.address'),
+ JSON_VALUE(shipTo, '$.phoneNumbers'),
+ JSON_QUERY(shipTo, '$.phoneNumbers')
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["STRING","COMPLEX","STRING","COMPLEX"],["VARCHAR","OTHER","VARCHAR","OTHER"],["","{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"}","","[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]"],["","{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"}","","[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]"],["","{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"}","","[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]"],["","{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCod
e\":\"98912-1195\"}","","[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]"],["","{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"}","","[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]"]]
+```
+
+#### Example query: Combine multiple JSON inputs into a single JSON object value
+
+ The following query illustrates how to use `JSON_OBJECT` to combine nested data elements into a new object.
+
+ > Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ JSON_OBJECT(KEY 'shipTo' VALUE JSON_QUERY(shipTo, '$'), KEY 'details' VALUE JSON_QUERY(details, '$')) as combinedJson
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["combinedJson"],["COMPLEX"],["OTHER"],["{\"details\":{\"color\":\"sky blue\",\"price\":542.0},\"shipTo\":{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}}"],["{\"details\":{\"color\":\"ivory\",\"price\":955.0},\"shipTo\":{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}}"],["{\"details\":{\"price\":8.0},\"shipTo\":{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"st
ate\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}}"],["{\"details\":{\"color\":\"olive\",\"price\":90.0},\"shipTo\":{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}}"],["{\"details\":{\"color\":\"plum\",\"price\":40.0},\"shipTo\":{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}}"]]
+```
+
+### Using other transform functions
+
+Druid provides the following additional transform functions:
+
+- `PARSE_JSON`: Deserializes a string value into a JSON object.
+- `TO_JSON_STRING`: Performs the operation of `TO_JSON` and then serializes the value into a string.
+
+#### Example query: Parse and deserialize data
+
+ The following query illustrates how to use the transform functions to parse and deserialize data.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ PARSE_JSON('{"x":"y"}'),
+ TO_JSON_STRING('{"x":"y"}'),
+ TO_JSON_STRING(PARSE_JSON('{"x":"y"}'))
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$2","EXPR$3"],["COMPLEX","STRING","STRING"],["OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+### Using helper operators
+
+The `JSON_KEYS` and `JSON_PATHS` functions are helper operators that you can use to examine JSON object schema. Use them to plan your queries, for example to work out which paths to use in `JSON_VALUE`.
+
+#### Example query: Examine JSON object schema
+
+ The following query illustrates how to use the helper operators to examine a nested data object.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.address')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_PATHS(shipTo))
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["COMPLEX","COMPLEX","STRING","STRING"],["OTHER","OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+## Known issues
+
+Before you start using the nested columns feature, consider the following known issues:
+
+- Directly using `COMPLEX<json>` typed columns and expressions is not well integrated into the Druid query engine. It can result in errors or undefined behavior when grouping and filtering, and when you use using `COMPLEX<json>` objects as inputs to aggregators. As a workaround, consider using `TO_JSON_STRING` to coerce the values to strings before you perform these operations.
+- Directly using array-typed outputs from `JSON_KEYS` and `JSON_PATHS` is moderately supported by the Druid query engine. You can group on these outputs, and there are a number of array expressions that can operate on these values, such as `ARRAY_CONCAT_AGG`. However, some operations are not well defined for use outside array-specific functions, such as filtering using `=` or `IS NULL`.
+- Input validation for JSON SQL operators is currently incomplete, which sometimes results in undefined behavior or unhelpful error messages.
+- Ingesting JSON columns with a very complex nested structure is potentially an expensive operation, and may require you to tune ingestion tasks and/or cluster parameters to account for increased memory usage or overall task run time. When you tune your ingestion configuration, treat each nested literal field inside a JSON object as a flattened top-level Druid column.
+
+## Learn more
+
+For more information, see the following pages:
+
+- [Nested columns functions reference](../misc/math-expr.md#nested-columns-functions) for details of the functions used in the examples on this page.
Review Comment:
this should link to sql functions page probably too
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,546 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ ~
+ ~ Nested columns is an experimental feature that is subject to
+ ~ change or removal at any time.
Review Comment:
any reason this is in the license part?
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,546 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ ~
+ ~ Nested columns is an experimental feature that is subject to
+ ~ change or removal at any time.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
Review Comment:
I think we should update
* https://github.com/apache/druid/blob/master/docs/querying/sql-json-functions.md
* https://github.com/apache/druid/blob/master/docs/querying/sql-data-types.md#nested-columns
* https://github.com/apache/druid/blob/master/docs/querying/virtual-columns.md#nested-field-virtual-column
* https://github.com/apache/druid/blob/master/docs/misc/math-expr.md#json-functions
to link to this page
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,546 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ ~
+ ~ Nested columns is an experimental feature that is subject to
+ ~ change or removal at any time.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
+
+Druid SQL JSON functions let you extract, transform, and create `COMPLEX<json>` values. Additionally, you can use certain `JSONPath` operators to extract values from nested data structures.
+
+Druid currently supports ingesting JSON-format nested columns with this feature. If you want to ingest nested data in another format, consider using the [flattenSpec object](../ingestion/data-formats.md#flattenspec).
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ "time",
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data in your ingestion query.
Review Comment:
this should link to https://github.com/apache/druid/blob/master/docs/querying/sql-json-functions.md instead of math-expr.md (which is native json query syntax that SQL gets translated into).
The native ingest specs should continue to link to math-expr.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r953572694
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,546 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ ~
+ ~ Nested columns is an experimental feature that is subject to
+ ~ change or removal at any time.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
+
+Druid SQL JSON functions let you extract, transform, and create `COMPLEX<json>` values. Additionally, you can use certain `JSONPath` operators to extract values from nested data structures.
+
+Druid currently supports ingesting JSON-format nested columns with this feature. If you want to ingest nested data in another format, consider using the [flattenSpec object](../ingestion/data-formats.md#flattenspec).
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ "time",
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ "time",
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](../misc/math-expr.md#nested-columns-functions) for information on the functions in the examples below.
+
+### JSONPath syntax
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures.
+
+|Operator|Description|
+| --- | --- |
+|`$`| Root element. All JSONPath expressions start with this operator. |
+|`.<name>`| Child element in dot notation. |
+|`['<name>']`| Child element in bracket notation. |
+|`[<number>]`| Array index. |
+
+Consider the following example input JSON:
+
+```json
+{"x":1, "y":[1, 2, 3]}
+```
+
+- To return the JSON object:<br>
+`$` -> `{"x":1, "y":[1, 2, 3]}`
+- To return the value of a key "x":<br>
+`$.x` -> `1`
+- For a key that contains an array, to return the entire array:<br>
+`$['y']` -> `[1, 2, 3]`
+- For a key that contains an array, to return an item in the array:<br>
+`$.y[1]` -> `2`
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- JSON_QUERY
+- JSON_OBJECT
+- PARSE_JSON
+- TO_JSON_STRING
+
+These functions are primarily intended for use with the Multi-Stage Query Architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
+
+#### Example query: Return results in a JSON object
+
+You can use the `JSON_QUERY` function to extract a partial structure from any JSON input and return results in a JSON object. Unlike `JSON_VALUE` it can extract objects and arrays.
+
+The following example query illustrates the differences in output between `JSON_VALUE` and `JSON_QUERY`.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ JSON_VALUE(shipTo, '$.address'),
+ JSON_QUERY(shipTo, '$.address'),
+ JSON_VALUE(shipTo, '$.phoneNumbers'),
+ JSON_QUERY(shipTo, '$.phoneNumbers')
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["STRING","COMPLEX","STRING","COMPLEX"],["VARCHAR","OTHER","VARCHAR","OTHER"],["","{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"}","","[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]"],["","{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"}","","[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]"],["","{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"}","","[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]"],["","{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCod
e\":\"98912-1195\"}","","[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]"],["","{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"}","","[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]"]]
+```
+
+#### Example query: Combine multiple JSON inputs into a single JSON object value
+
+ The following query illustrates how to use `JSON_OBJECT` to combine nested data elements into a new object.
+
+ > Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ JSON_OBJECT(KEY 'shipTo' VALUE JSON_QUERY(shipTo, '$'), KEY 'details' VALUE JSON_QUERY(details, '$')) as combinedJson
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["combinedJson"],["COMPLEX"],["OTHER"],["{\"details\":{\"color\":\"sky blue\",\"price\":542.0},\"shipTo\":{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}}"],["{\"details\":{\"color\":\"ivory\",\"price\":955.0},\"shipTo\":{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}}"],["{\"details\":{\"price\":8.0},\"shipTo\":{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"st
ate\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}}"],["{\"details\":{\"color\":\"olive\",\"price\":90.0},\"shipTo\":{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}}"],["{\"details\":{\"color\":\"plum\",\"price\":40.0},\"shipTo\":{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}}"]]
+```
+
+### Using other transform functions
+
+Druid provides the following additional transform functions:
+
+- `PARSE_JSON`: Deserializes a string value into a JSON object.
+- `TO_JSON_STRING`: Performs the operation of `TO_JSON` and then serializes the value into a string.
+
+#### Example query: Parse and deserialize data
+
+ The following query illustrates how to use the transform functions to parse and deserialize data.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ PARSE_JSON('{"x":"y"}'),
+ TO_JSON_STRING('{"x":"y"}'),
+ TO_JSON_STRING(PARSE_JSON('{"x":"y"}'))
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$2","EXPR$3"],["COMPLEX","STRING","STRING"],["OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+### Using helper operators
+
+The `JSON_KEYS` and `JSON_PATHS` functions are helper operators that you can use to examine JSON object schema. Use them to plan your queries, for example to work out which paths to use in `JSON_VALUE`.
+
+#### Example query: Examine JSON object schema
+
+ The following query illustrates how to use the helper operators to examine a nested data object.
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+SELECT
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.address')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_PATHS(shipTo))
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["COMPLEX","COMPLEX","STRING","STRING"],["OTHER","OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+## Known issues
+
+Before you start using the nested columns feature, consider the following known issues:
+
+- Directly using `COMPLEX<json>` typed columns and expressions is not well integrated into the Druid query engine. It can result in errors or undefined behavior when grouping and filtering, and when you use using `COMPLEX<json>` objects as inputs to aggregators. As a workaround, consider using `TO_JSON_STRING` to coerce the values to strings before you perform these operations.
+- Directly using array-typed outputs from `JSON_KEYS` and `JSON_PATHS` is moderately supported by the Druid query engine. You can group on these outputs, and there are a number of array expressions that can operate on these values, such as `ARRAY_CONCAT_AGG`. However, some operations are not well defined for use outside array-specific functions, such as filtering using `=` or `IS NULL`.
+- Input validation for JSON SQL operators is currently incomplete, which sometimes results in undefined behavior or unhelpful error messages.
+- Ingesting JSON columns with a very complex nested structure is potentially an expensive operation, and may require you to tune ingestion tasks and/or cluster parameters to account for increased memory usage or overall task run time. When you tune your ingestion configuration, treat each nested literal field inside a JSON object as a flattened top-level Druid column.
+
+## Learn more
+
+For more information, see the following pages:
+
+- [Nested columns functions reference](../misc/math-expr.md#nested-columns-functions) for details of the functions used in the examples on this page.
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r958966618
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,595 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format and specialized internal columns and indexes for nested literal values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#json-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion as an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to "flatten" nested data for ingestion.
+
+## Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed, a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [`flattenSpec`](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, specify the `json` type for the dimension in the `dimensions` list in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, define the output name and expression in the `transforms` list in the `transformSpec` object of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the value for `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [previous batch example](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName`, and `address` from `shipTo` and creates a composite JSON object containing `product`, `details`, and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Ingest deserialized JSON as COMPLEX\<json>
Review Comment:
i think 'JSON string' or 'stringified' might be the more appropriate way to describe this data?
I also wonder if it is worth calling out that this is required for 'text' input formats such as TSV
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,595 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format and specialized internal columns and indexes for nested literal values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#json-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion as an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to "flatten" nested data for ingestion.
+
+## Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed, a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [`flattenSpec`](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, specify the `json` type for the dimension in the `dimensions` list in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, define the output name and expression in the `transforms` list in the `transformSpec` object of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
Review Comment:
I don't think this necessarily needs to be done right now, but this file is quite small, I think we could just put it in `quickstart/tutorial/`, though it would require adjustment to the examples to use file instead of http input
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] techdocsmith commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
techdocsmith commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r957950343
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
+
+
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- JSON_QUERY
+- JSON_OBJECT
+- PARSE_JSON
+- TO_JSON_STRING
Review Comment:
@vtlim , @writer-jill I took this guidance, but we should sync on style
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vtlim commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
vtlim commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r958756426
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,595 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#json-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion as an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to "flatten" nested data for ingestion.
+
+## Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed, a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [`flattenSpec`](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, specify the `json` type for the dimension in the `dimensions` list in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, define the output name and expression in the `transforms` list in the `transformSpec` object of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the value for `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [previous batch example](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName`, and `address` from `shipTo` and creates a composite JSON object containing `product`, `details`, and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Ingest deserialized JSON as COMPLEX\<json>
+
+If your source data uses a string representation of your JSON column, you can still ingest the data as `COMPLEX<JSON>` as follows:
+- During native batch ingestion, call the `parse_json` function in a `transform` object in the `transformSpec`.
+- During SQL-based ingestion, use the PARSE_JSON keyword within your SELECT statement to transform the string values to JSON.
+- If you are concerned that your data may not contain valid JSON, you can use `try_parse_json` for native batch or `TRY_PARSE_JSON` for SQL-based ingestion. For cases where the column does not contain valid JSON, Druid inserts a null value.
+
+For example, consider the following deserialized row of the sample data set:
+
+```
+{"time": "2022-06-13T10:10:35Z", "product": "Bike", "department":"Sports", "shipTo":"{\"firstName\": \"Henry\",\"lastName\": \"Wuckert\",\"address\": {\"street\": \"5643 Jan Walk\",\"city\": \"Lake Bridget\",\"state\": \"HI\",\"country\":\"ME\",\"postalCode\": \"70204-2939\"},\"phoneNumbers\": [{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\" },{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}", "details":"{\"color\":\"ivory\", \"price\":955.00}"}
+```
+
+The following examples demonstrate how to ingest the `shipTo` and `details` columns both as string type and as `COMPLEX<json>` in the `shipTo_parsed` and `details_parsed` columns.
+
+<!--DOCUSAURUS_CODE_TABS-->
+<!--SQL-->
+```
+REPLACE INTO deserialized_example OVERWRITE ALL
+WITH source AS (SELECT * FROM TABLE(
+ EXTERN(
+ '{"type":"inline","data":"{\"time\": \"2022-06-13T10:10:35Z\", \"product\": \"Bike\", \"department\":\"Sports\", \"shipTo\":\"{\\\"firstName\\\": \\\"Henry\\\",\\\"lastName\\\": \\\"Wuckert\\\",\\\"address\\\": {\\\"street\\\": \\\"5643 Jan Walk\\\",\\\"city\\\": \\\"Lake Bridget\\\",\\\"state\\\": \\\"HI\\\",\\\"country\\\":\\\"ME\\\",\\\"postalCode\\\": \\\"70204-2939\\\"},\\\"phoneNumbers\\\": [{\\\"type\\\":\\\"primary\\\",\\\"number\\\":\\\"593.475.0449 x86733\\\" },{\\\"type\\\":\\\"secondary\\\",\\\"number\\\":\\\"638-372-1210\\\"}]}\", \"details\":\"{\\\"color\\\":\\\"ivory\\\", \\\"price\\\":955.00}\"}\n"}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"string"},{"name":"details","type":"string"},{"name":"shipTo_parsed","type":"json"},{"name":"details_parsed","type":"json"}]'
+ )
+))
+SELECT
+ TIME_PARSE("time") AS __time,
+ "product",
+ "department",
+ "shipTo",
+ "details",
+ PARSE_JSON("shipTo") as "shipTo_parsed",
+ PARSE_JSON("details") as "details_parsed"
+FROM source
+PARTITIONED BY DAY
+```
+<!--Native batch-->
+```{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "inline",
+ "data": "{\"time\": \"2022-06-13T10:10:35Z\", \"product\": \"Bike\", \"department\":\"Sports\", \"shipTo\":\"{\\\"firstName\\\": \\\"Henry\\\",\\\"lastName\\\": \\\"Wuckert\\\",\\\"address\\\": {\\\"street\\\": \\\"5643 Jan Walk\\\",\\\"city\\\": \\\"Lake Bridget\\\",\\\"state\\\": \\\"HI\\\",\\\"country\\\":\\\"ME\\\",\\\"postalCode\\\": \\\"70204-2939\\\"},\\\"phoneNumbers\\\": [{\\\"type\\\":\\\"primary\\\",\\\"number\\\":\\\"593.475.0449 x86733\\\" },{\\\"type\\\":\\\"secondary\\\",\\\"number\\\":\\\"638-372-1210\\\"}]}\", \"details\":\"{\\\"color\\\":\\\"ivory\\\", \\\"price\\\":955.00}\"}\n"
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ },
+ "dataSchema": {
+ "dataSource": "deserialized_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "iso"
+ },
+ "transformSpec": {
+ "transforms": [
+ {
+ "type": "expression",
+ "name": "shipTo_parsed",
+ "expression": "parse_json(shipTo)"
+ },
+ {
+ "type": "expression",
+ "name": "details_parsed",
+ "expression": "parse_json(details)"
+ }
+ ]
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ "shipTo",
+ "details",
+ "shipTo_parsed",
+ "details_parsed"
+ ]
+ },
+ "granularitySpec": {
+ "queryGranularity": "none",
+ "rollup": false,
+ "segmentGranularity": "day"
+```
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide native Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid DOUBLE, FLOAT, or LONG columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY, and ORDER BY operators with `JSON_VALUE`.
+
+
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- `JSON_QUERY`
+- `JSON_OBJECT`
+- `PARSE_JSON`
+- `TO_JSON_STRING`
+
+These functions are primarily intended for use with the multi-Stage Query Architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
Review Comment:
```suggestion
These functions are primarily intended for use with the multi-stage query architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] techdocsmith commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
techdocsmith commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r959001989
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,595 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format and specialized internal columns and indexes for nested literal values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#json-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion as an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to "flatten" nested data for ingestion.
+
+## Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed, a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [`flattenSpec`](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, specify the `json` type for the dimension in the `dimensions` list in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, define the output name and expression in the `transforms` list in the `transformSpec` object of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the value for `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [previous batch example](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName`, and `address` from `shipTo` and creates a composite JSON object containing `product`, `details`, and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Ingest deserialized JSON as COMPLEX\<json>
Review Comment:
```suggestion
## Ingest a JSON string as COMPLEX\<json>
```
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,595 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format and specialized internal columns and indexes for nested literal values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#json-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion as an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to "flatten" nested data for ingestion.
+
+## Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed, a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [`flattenSpec`](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, specify the `json` type for the dimension in the `dimensions` list in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, define the output name and expression in the `transforms` list in the `transformSpec` object of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the value for `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [previous batch example](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName`, and `address` from `shipTo` and creates a composite JSON object containing `product`, `details`, and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Ingest deserialized JSON as COMPLEX\<json>
+
+If your source data uses a string representation of your JSON column, you can still ingest the data as `COMPLEX<JSON>` as follows:
+- During native batch ingestion, call the `parse_json` function in a `transform` object in the `transformSpec`.
+- During SQL-based ingestion, use the PARSE_JSON keyword within your SELECT statement to transform the string values to JSON.
+- If you are concerned that your data may not contain valid JSON, you can use `try_parse_json` for native batch or `TRY_PARSE_JSON` for SQL-based ingestion. For cases where the column does not contain valid JSON, Druid inserts a null value.
+
Review Comment:
```suggestion
If you are using a text input format like `tsv`, you need to use this method to ingest data into a `COMPLEX<json>` column.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r953570130
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,546 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ ~
+ ~ Nested columns is an experimental feature that is subject to
+ ~ change or removal at any time.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
+
+Druid SQL JSON functions let you extract, transform, and create `COMPLEX<json>` values. Additionally, you can use certain `JSONPath` operators to extract values from nested data structures.
+
+Druid currently supports ingesting JSON-format nested columns with this feature. If you want to ingest nested data in another format, consider using the [flattenSpec object](../ingestion/data-formats.md#flattenspec).
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ "time",
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data in your ingestion query.
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r953613558
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,546 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ ~
+ ~ Nested columns is an experimental feature that is subject to
+ ~ change or removal at any time.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
Review Comment:
Added links to each of these pages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r953618556
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,546 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ ~
+ ~ Nested columns is an experimental feature that is subject to
+ ~ change or removal at any time.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
+
+Druid SQL JSON functions let you extract, transform, and create `COMPLEX<json>` values. Additionally, you can use certain `JSONPath` operators to extract values from nested data structures.
+
+Druid currently supports ingesting JSON-format nested columns with this feature. If you want to ingest nested data in another format, consider using the [flattenSpec object](../ingestion/data-formats.md#flattenspec).
Review Comment:
I think this part is a bit confusing about all of the functionality.. i think it focuses too much on using the transforms as an alternative to flattenSpec, and not enough that this is its own column type which allows storing nested data directly in druid segments and the 'flattening' extraction can instead be done at query time without performance penalty.
here is the set of stuff I think is worth mentioning in the lead in
```suggestion
Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. In addition to storing a copy of the structured data in JSON format, `COMPLEX<json>` columns also store specialized internal columns and indexes for nested 'literal' values (`STRING`, `LONG`, and `DOUBLE` types). An optimized [virtual-column](./virtual-columns.md#nested-field-virtual-column) allows for fast reading and filtering of these values at speeds which are consistent with standard Druid `LONG`, `DOUBLE`, and `STRING` columns.
The [Druid SQL JSON functions](./sql-json-functions.md) let you extract, transform, and create `COMPLEX<json>` values in SQL queries, taking advantage of the specialized virtual columns whenever possible. The [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions), can be used in ['native' queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
When used in MSQ INSERT and REPLACE statements, or native ingestion `transformSpec`, the JSON functions can serve as an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec).
```
feel free to remix however needed. if we cover this stuff here, we can dump the description https://github.com/apache/druid/pull/12946/files#diff-1509d7fece1f79b5fc731c62d0fd4f114d48e1c606b3d23a7779e408df4590deR330 and keep it focused on the example (maybe only mentioning that it is one of the functions that uses the special nested field virtual column)
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,546 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ ~
+ ~ Nested columns is an experimental feature that is subject to
+ ~ change or removal at any time.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
+
+Druid SQL JSON functions let you extract, transform, and create `COMPLEX<json>` values. Additionally, you can use certain `JSONPath` operators to extract values from nested data structures.
+
+Druid currently supports ingesting JSON-format nested columns with this feature. If you want to ingest nested data in another format, consider using the [flattenSpec object](../ingestion/data-formats.md#flattenspec).
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ "time",
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ "time",
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](../misc/math-expr.md#nested-columns-functions) for information on the functions in the examples below.
+
+### JSONPath syntax
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures.
Review Comment:
my vote is for sql-json-functions page
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r954768872
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,519 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
+
+Druid SQL JSON functions let you extract, transform, and create `COMPLEX<json>` values. Additionally, you can use certain `JSONPath` operators to extract values from nested data structures.
+
+Druid currently supports ingesting JSON-format nested columns with this feature. If you want to ingest nested data in another format, consider using the [flattenSpec object](../ingestion/data-formats.md#flattenspec).
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ "time",
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ "time",
Review Comment:
```suggestion
TIME_PARSE("time") as __time,
```
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,519 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
+
+Druid SQL JSON functions let you extract, transform, and create `COMPLEX<json>` values. Additionally, you can use certain `JSONPath` operators to extract values from nested data structures.
+
+Druid currently supports ingesting JSON-format nested columns with this feature. If you want to ingest nested data in another format, consider using the [flattenSpec object](../ingestion/data-formats.md#flattenspec).
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ "time",
Review Comment:
```suggestion
TIME_PARSE("time") as __time,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r954929494
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,546 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ ~
+ ~ Nested columns is an experimental feature that is subject to
+ ~ change or removal at any time.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
+
+Druid SQL JSON functions let you extract, transform, and create `COMPLEX<json>` values. Additionally, you can use certain `JSONPath` operators to extract values from nested data structures.
+
+Druid currently supports ingesting JSON-format nested columns with this feature. If you want to ingest nested data in another format, consider using the [flattenSpec object](../ingestion/data-formats.md#flattenspec).
Review Comment:
@clintropolis what's the description to (possibly) dump? This link doesn't show me anything specific.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] techdocsmith commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
techdocsmith commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r956361138
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,546 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ ~
+ ~ Nested columns is an experimental feature that is subject to
+ ~ change or removal at any time.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
Review Comment:
```suggestion
> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
```
@clintropolis WDYT?
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in query-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
Review Comment:
```suggestion
You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
```
not for this PR, but eventually SQL-based ingestion should link out
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in query-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
Review Comment:
we should indicate that `shipTo` and `details` are the nested JSON columns. (This happens later, but i think it would help to mention it here)
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in query-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
+
+
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- JSON_QUERY
+- JSON_OBJECT
+- PARSE_JSON
+- TO_JSON_STRING
+
+These functions are primarily intended for use with the Multi-Stage Query Architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
+
+#### Example query: Return results in a JSON object
+
+You can use the `JSON_QUERY` function to extract a partial structure from any JSON input and return results in a JSON object. Unlike `JSON_VALUE` it can extract objects and arrays.
+
+The following example query illustrates the differences in output between `JSON_VALUE` and `JSON_QUERY`.
+
+
+
+```sql
+SELECT
+ JSON_VALUE(shipTo, '$.address'),
+ JSON_QUERY(shipTo, '$.address'),
+ JSON_VALUE(shipTo, '$.phoneNumbers'),
+ JSON_QUERY(shipTo, '$.phoneNumbers')
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["VARCHAR","OTHER","VARCHAR","OTHER"],["","{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"}","","[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]"],["","{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"}","","[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]"],["","{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"}","","[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]"],["","{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\"
,\"postalCode\":\"98912-1195\"}","","[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]"],["","{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"}","","[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]"]]
+```
+
+#### Example query: Combine multiple JSON inputs into a single JSON object value
+
+ The following query illustrates how to use `JSON_OBJECT` to combine nested data elements into a new object.
+
+
+
+```sql
+SELECT
+ JSON_OBJECT(KEY 'shipTo' VALUE JSON_QUERY(shipTo, '$'), KEY 'details' VALUE JSON_QUERY(details, '$')) as combinedJson
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["combinedJson"],["COMPLEX<json>"],["OTHER"],["{\"details\":{\"color\":\"sky blue\",\"price\":542.0},\"shipTo\":{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}}"],["{\"details\":{\"color\":\"ivory\",\"price\":955.0},\"shipTo\":{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}}"],["{\"details\":{\"price\":8.0},\"shipTo\":{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\
",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}}"],["{\"details\":{\"color\":\"olive\",\"price\":90.0},\"shipTo\":{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}}"],["{\"details\":{\"color\":\"plum\",\"price\":40.0},\"shipTo\":{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}}"]]
+```
+
+### Using other transform functions
+
+Druid provides the following additional transform functions:
+
+- `PARSE_JSON`: Deserializes a string value into a JSON object.
+- `TO_JSON_STRING`: Performs the operation of `TO_JSON` and then serializes the value into a string.
+
+#### Example query: Parse and deserialize data
+
+ The following query illustrates how to use the transform functions to parse and deserialize data.
+
+
+
+```sql
+SELECT
+ PARSE_JSON('{"x":"y"}'),
+ TO_JSON_STRING('{"x":"y"}'),
+ TO_JSON_STRING(PARSE_JSON('{"x":"y"}'))
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$2","EXPR$3"],["COMPLEX<json>","STRING","STRING"],["OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+### Using helper operators
+
+The `JSON_KEYS` and `JSON_PATHS` functions are helper operators that you can use to examine JSON object schema. Use them to plan your queries, for example to work out which paths to use in `JSON_VALUE`.
+
+#### Example query: Examine JSON object schema
+
+ The following query illustrates how to use the helper operators to examine a nested data object.
+
+
+
+```sql
+SELECT
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.address')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_PATHS(shipTo))
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["COMPLEX<json>","COMPLEX<json>","STRING","STRING"],["OTHER","OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+## Known issues
+
+Before you start using the nested columns feature, consider the following known issues:
+
+- Directly using `COMPLEX<json>` typed columns and expressions is not well integrated into the Druid query engine. It can result in errors or undefined behavior when grouping and filtering, and when you use using `COMPLEX<json>` objects as inputs to aggregators. As a workaround, consider using `TO_JSON_STRING` to coerce the values to strings before you perform these operations.
+- Directly using array-typed outputs from `JSON_KEYS` and `JSON_PATHS` is moderately supported by the Druid query engine. You can group on these outputs, and there are a number of array expressions that can operate on these values, such as `ARRAY_CONCAT_AGG`. However, some operations are not well defined for use outside array-specific functions, such as filtering using `=` or `IS NULL`.
+- Input validation for JSON SQL operators is currently incomplete, which sometimes results in undefined behavior or unhelpful error messages.
+- Ingesting JSON columns with a very complex nested structure is potentially an expensive operation, and may require you to tune ingestion tasks and/or cluster parameters to account for increased memory usage or overall task run time. When you tune your ingestion configuration, treat each nested literal field inside a JSON object as a flattened top-level Druid column.
+
+## Learn more
Review Comment:
```suggestion
## Further reading
```
Apache Druid style
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r958652029
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
Review Comment:
Done
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
+
+
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- JSON_QUERY
+- JSON_OBJECT
+- PARSE_JSON
+- TO_JSON_STRING
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vtlim commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
vtlim commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r957877339
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
Review Comment:
```suggestion
Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#json-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
```
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
Review Comment:
```suggestion
You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to "flatten" nested data for ingestion.
```
Prefer straight double quotation marks when used for emphasis ([style guide](https://developers.google.com/style/quotation-marks?hl=en))
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
Review Comment:
```suggestion
When pretty-printed, a sample row in `nested_example_data` looks like this:
```
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
Review Comment:
> or in a `transformSpec` in native ingestion
Is this redundant with the previous sentence? "in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec)."
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
Review Comment:
Consider formatting query results as a Markdown table instead of a one line code block for readability
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
Review Comment:
```suggestion
To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the value for `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
```
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
Review Comment:
```suggestion
Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid DOUBLE, FLOAT, or LONG columns.
```
Remain consistent with style in the introduction
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+
Review Comment:
```suggestion

```
Looks like the screenshot has been added 😄
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
Review Comment:
```suggestion
The `JSON_VALUE` function is specially optimized to provide native Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
```
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
+
+
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- JSON_QUERY
+- JSON_OBJECT
+- PARSE_JSON
+- TO_JSON_STRING
+
+These functions are primarily intended for use with the Multi-Stage Query Architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
Review Comment:
```suggestion
These functions are primarily intended for use with the multi-stage query architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
```
Latest guidance is to not capitalize
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
+
+
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- JSON_QUERY
+- JSON_OBJECT
+- PARSE_JSON
+- TO_JSON_STRING
+
+These functions are primarily intended for use with the Multi-Stage Query Architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
+
+#### Example query: Return results in a JSON object
+
+You can use the `JSON_QUERY` function to extract a partial structure from any JSON input and return results in a JSON object. Unlike `JSON_VALUE` it can extract objects and arrays.
+
+The following example query illustrates the differences in output between `JSON_VALUE` and `JSON_QUERY`.
+
+
+
+```sql
+SELECT
+ JSON_VALUE(shipTo, '$.address'),
+ JSON_QUERY(shipTo, '$.address'),
+ JSON_VALUE(shipTo, '$.phoneNumbers'),
+ JSON_QUERY(shipTo, '$.phoneNumbers')
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["VARCHAR","OTHER","VARCHAR","OTHER"],["","{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"}","","[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]"],["","{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"}","","[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]"],["","{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"}","","[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]"],["","{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\"
,\"postalCode\":\"98912-1195\"}","","[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]"],["","{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"}","","[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]"]]
+```
+
+#### Example query: Combine multiple JSON inputs into a single JSON object value
+
+ The following query illustrates how to use `JSON_OBJECT` to combine nested data elements into a new object.
+
+
+
+```sql
+SELECT
+ JSON_OBJECT(KEY 'shipTo' VALUE JSON_QUERY(shipTo, '$'), KEY 'details' VALUE JSON_QUERY(details, '$')) as combinedJson
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["combinedJson"],["COMPLEX<json>"],["OTHER"],["{\"details\":{\"color\":\"sky blue\",\"price\":542.0},\"shipTo\":{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}}"],["{\"details\":{\"color\":\"ivory\",\"price\":955.0},\"shipTo\":{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}}"],["{\"details\":{\"price\":8.0},\"shipTo\":{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\
",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}}"],["{\"details\":{\"color\":\"olive\",\"price\":90.0},\"shipTo\":{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}}"],["{\"details\":{\"color\":\"plum\",\"price\":40.0},\"shipTo\":{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}}"]]
+```
+
+### Using other transform functions
+
+Druid provides the following additional transform functions:
+
+- `PARSE_JSON`: Deserializes a string value into a JSON object.
+- `TO_JSON_STRING`: Performs the operation of `TO_JSON` and then serializes the value into a string.
+
+#### Example query: Parse and deserialize data
+
+ The following query illustrates how to use the transform functions to parse and deserialize data.
+
+
+
+```sql
+SELECT
+ PARSE_JSON('{"x":"y"}'),
+ TO_JSON_STRING('{"x":"y"}'),
+ TO_JSON_STRING(PARSE_JSON('{"x":"y"}'))
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$2","EXPR$3"],["COMPLEX<json>","STRING","STRING"],["OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+### Using helper operators
+
+The `JSON_KEYS` and `JSON_PATHS` functions are helper operators that you can use to examine JSON object schema. Use them to plan your queries, for example to work out which paths to use in `JSON_VALUE`.
+
+#### Example query: Examine JSON object schema
+
+ The following query illustrates how to use the helper operators to examine a nested data object.
+
+
+
+```sql
+SELECT
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.address')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_PATHS(shipTo))
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["COMPLEX<json>","COMPLEX<json>","STRING","STRING"],["OTHER","OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+## Known issues
+
+Before you start using the nested columns feature, consider the following known issues:
+
+- Directly using `COMPLEX<json>` typed columns and expressions is not well integrated into the Druid query engine. It can result in errors or undefined behavior when grouping and filtering, and when you use using `COMPLEX<json>` objects as inputs to aggregators. As a workaround, consider using `TO_JSON_STRING` to coerce the values to strings before you perform these operations.
Review Comment:
```suggestion
- Directly using `COMPLEX<json>` typed columns and expressions is not well integrated into the Druid query engine. It can result in errors or undefined behavior when grouping and filtering and when you use `COMPLEX<json>` objects as inputs to aggregators. As a workaround, consider using `TO_JSON_STRING` to coerce the values to strings before you perform these operations.
```
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
Review Comment:
```suggestion
For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [`flattenSpec`](../ingestion/data-formats.md#flattenspec) input format.
```
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
Review Comment:
```suggestion
To configure a dimension as a nested data type, specify the `json` type for the dimension in the `dimensions` list in the `dimensionsSpec` property of your ingestion spec.
```
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
Review Comment:
```suggestion
For example, the following ingestion query is the SQL-based version of the [preceding batch example](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName`, and `address` from `shipTo` and creates a composite JSON object containing `product`, `details`, and `department`.
```
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
Review Comment:
```suggestion
To do this, define the output name and expression in the `transforms` list in the `transformSpec` property of your ingestion spec.
```
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
Review Comment:
```suggestion
The following example query illustrates how to use SUM, WHERE, GROUP BY, and ORDER BY operators with `JSON_VALUE`.
```
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
+
+
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- JSON_QUERY
+- JSON_OBJECT
+- PARSE_JSON
+- TO_JSON_STRING
+
+These functions are primarily intended for use with the Multi-Stage Query Architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
+
+#### Example query: Return results in a JSON object
+
+You can use the `JSON_QUERY` function to extract a partial structure from any JSON input and return results in a JSON object. Unlike `JSON_VALUE` it can extract objects and arrays.
+
+The following example query illustrates the differences in output between `JSON_VALUE` and `JSON_QUERY`.
Review Comment:
Add a line explaining the difference in output? Could be something like: The two output columns for `JSON_VALUE` contain null values only because `JSON_VALUE` only returns literal types.
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
+
+
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- JSON_QUERY
+- JSON_OBJECT
+- PARSE_JSON
+- TO_JSON_STRING
+
+These functions are primarily intended for use with the Multi-Stage Query Architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
+
+#### Example query: Return results in a JSON object
+
+You can use the `JSON_QUERY` function to extract a partial structure from any JSON input and return results in a JSON object. Unlike `JSON_VALUE` it can extract objects and arrays.
+
+The following example query illustrates the differences in output between `JSON_VALUE` and `JSON_QUERY`.
+
+
+
+```sql
+SELECT
+ JSON_VALUE(shipTo, '$.address'),
+ JSON_QUERY(shipTo, '$.address'),
+ JSON_VALUE(shipTo, '$.phoneNumbers'),
+ JSON_QUERY(shipTo, '$.phoneNumbers')
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["VARCHAR","OTHER","VARCHAR","OTHER"],["","{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"}","","[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]"],["","{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"}","","[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]"],["","{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"}","","[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]"],["","{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\"
,\"postalCode\":\"98912-1195\"}","","[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]"],["","{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"}","","[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]"]]
+```
+
+#### Example query: Combine multiple JSON inputs into a single JSON object value
+
+ The following query illustrates how to use `JSON_OBJECT` to combine nested data elements into a new object.
+
+
+
+```sql
+SELECT
+ JSON_OBJECT(KEY 'shipTo' VALUE JSON_QUERY(shipTo, '$'), KEY 'details' VALUE JSON_QUERY(details, '$')) as combinedJson
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["combinedJson"],["COMPLEX<json>"],["OTHER"],["{\"details\":{\"color\":\"sky blue\",\"price\":542.0},\"shipTo\":{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}}"],["{\"details\":{\"color\":\"ivory\",\"price\":955.0},\"shipTo\":{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}}"],["{\"details\":{\"price\":8.0},\"shipTo\":{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\
",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}}"],["{\"details\":{\"color\":\"olive\",\"price\":90.0},\"shipTo\":{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}}"],["{\"details\":{\"color\":\"plum\",\"price\":40.0},\"shipTo\":{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}}"]]
+```
+
+### Using other transform functions
+
+Druid provides the following additional transform functions:
+
+- `PARSE_JSON`: Deserializes a string value into a JSON object.
+- `TO_JSON_STRING`: Performs the operation of `TO_JSON` and then serializes the value into a string.
+
+#### Example query: Parse and deserialize data
+
+ The following query illustrates how to use the transform functions to parse and deserialize data.
+
+
+
+```sql
+SELECT
+ PARSE_JSON('{"x":"y"}'),
+ TO_JSON_STRING('{"x":"y"}'),
+ TO_JSON_STRING(PARSE_JSON('{"x":"y"}'))
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$2","EXPR$3"],["COMPLEX<json>","STRING","STRING"],["OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+### Using helper operators
+
+The `JSON_KEYS` and `JSON_PATHS` functions are helper operators that you can use to examine JSON object schema. Use them to plan your queries, for example to work out which paths to use in `JSON_VALUE`.
+
+#### Example query: Examine JSON object schema
+
+ The following query illustrates how to use the helper operators to examine a nested data object.
+
+
+
+```sql
+SELECT
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.address')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_PATHS(shipTo))
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["COMPLEX<json>","COMPLEX<json>","STRING","STRING"],["OTHER","OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+## Known issues
+
+Before you start using the nested columns feature, consider the following known issues:
+
+- Directly using `COMPLEX<json>` typed columns and expressions is not well integrated into the Druid query engine. It can result in errors or undefined behavior when grouping and filtering, and when you use using `COMPLEX<json>` objects as inputs to aggregators. As a workaround, consider using `TO_JSON_STRING` to coerce the values to strings before you perform these operations.
Review Comment:
> `COMPLEX<json>` typed columns
should probably have a hyphen before typed like "array-typed" in the line below.
Or to avoid weird formatting you could say something like
> `COMPLEX<json>` columns
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
+
+
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- JSON_QUERY
+- JSON_OBJECT
+- PARSE_JSON
+- TO_JSON_STRING
+
+These functions are primarily intended for use with the Multi-Stage Query Architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
+
+#### Example query: Return results in a JSON object
+
+You can use the `JSON_QUERY` function to extract a partial structure from any JSON input and return results in a JSON object. Unlike `JSON_VALUE` it can extract objects and arrays.
+
+The following example query illustrates the differences in output between `JSON_VALUE` and `JSON_QUERY`.
+
+
+
+```sql
+SELECT
+ JSON_VALUE(shipTo, '$.address'),
+ JSON_QUERY(shipTo, '$.address'),
+ JSON_VALUE(shipTo, '$.phoneNumbers'),
+ JSON_QUERY(shipTo, '$.phoneNumbers')
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["VARCHAR","OTHER","VARCHAR","OTHER"],["","{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"}","","[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]"],["","{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"}","","[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]"],["","{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"}","","[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]"],["","{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\"
,\"postalCode\":\"98912-1195\"}","","[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]"],["","{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"}","","[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]"]]
+```
+
+#### Example query: Combine multiple JSON inputs into a single JSON object value
+
+ The following query illustrates how to use `JSON_OBJECT` to combine nested data elements into a new object.
+
+
+
+```sql
+SELECT
+ JSON_OBJECT(KEY 'shipTo' VALUE JSON_QUERY(shipTo, '$'), KEY 'details' VALUE JSON_QUERY(details, '$')) as combinedJson
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["combinedJson"],["COMPLEX<json>"],["OTHER"],["{\"details\":{\"color\":\"sky blue\",\"price\":542.0},\"shipTo\":{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}}"],["{\"details\":{\"color\":\"ivory\",\"price\":955.0},\"shipTo\":{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}}"],["{\"details\":{\"price\":8.0},\"shipTo\":{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\
",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}}"],["{\"details\":{\"color\":\"olive\",\"price\":90.0},\"shipTo\":{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}}"],["{\"details\":{\"color\":\"plum\",\"price\":40.0},\"shipTo\":{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}}"]]
+```
+
+### Using other transform functions
+
+Druid provides the following additional transform functions:
+
+- `PARSE_JSON`: Deserializes a string value into a JSON object.
+- `TO_JSON_STRING`: Performs the operation of `TO_JSON` and then serializes the value into a string.
+
+#### Example query: Parse and deserialize data
+
+ The following query illustrates how to use the transform functions to parse and deserialize data.
+
+
+
+```sql
+SELECT
+ PARSE_JSON('{"x":"y"}'),
+ TO_JSON_STRING('{"x":"y"}'),
+ TO_JSON_STRING(PARSE_JSON('{"x":"y"}'))
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$2","EXPR$3"],["COMPLEX<json>","STRING","STRING"],["OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+### Using helper operators
+
+The `JSON_KEYS` and `JSON_PATHS` functions are helper operators that you can use to examine JSON object schema. Use them to plan your queries, for example to work out which paths to use in `JSON_VALUE`.
+
+#### Example query: Examine JSON object schema
+
+ The following query illustrates how to use the helper operators to examine a nested data object.
+
+
+
+```sql
+SELECT
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.address')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_PATHS(shipTo))
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["COMPLEX<json>","COMPLEX<json>","STRING","STRING"],["OTHER","OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+## Known issues
+
+Before you start using the nested columns feature, consider the following known issues:
+
+- Directly using `COMPLEX<json>` typed columns and expressions is not well integrated into the Druid query engine. It can result in errors or undefined behavior when grouping and filtering, and when you use using `COMPLEX<json>` objects as inputs to aggregators. As a workaround, consider using `TO_JSON_STRING` to coerce the values to strings before you perform these operations.
+- Directly using array-typed outputs from `JSON_KEYS` and `JSON_PATHS` is moderately supported by the Druid query engine. You can group on these outputs, and there are a number of array expressions that can operate on these values, such as `ARRAY_CONCAT_AGG`. However, some operations are not well defined for use outside array-specific functions, such as filtering using `=` or `IS NULL`.
+- Input validation for JSON SQL operators is currently incomplete, which sometimes results in undefined behavior or unhelpful error messages.
+- Ingesting JSON columns with a very complex nested structure is potentially an expensive operation, and may require you to tune ingestion tasks and/or cluster parameters to account for increased memory usage or overall task run time. When you tune your ingestion configuration, treat each nested literal field inside a JSON object as a flattened top-level Druid column.
+
+## Further reading
+
+For more information, see the following pages:
+
+- [Nested columns functions reference](./sql-json-functions.md) for details of the functions used in the examples on this page.
+- [Ingestion spec reference](../ingestion/ingestion-spec.md) for information on native ingestion and [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+- [Multi-stage query architecture overview](../multi-stage-query/index.md) for information on how to set up and use this feature.
Review Comment:
Suggest reordering to have `transformSpec` and `flattenSpec` details next to each other
```suggestion
- [Multi-stage query architecture overview](../multi-stage-query/index.md) for information on how to set up and use this feature.
- [Ingestion spec reference](../ingestion/ingestion-spec.md) for information on native ingestion and [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
```
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
+
+
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- JSON_QUERY
+- JSON_OBJECT
+- PARSE_JSON
+- TO_JSON_STRING
Review Comment:
```suggestion
- `JSON_QUERY`
- `JSON_OBJECT`
- `PARSE_JSON`
- `TO_JSON_STRING`
```
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide 'native' Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid double, float, or long columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+> Note to self: add screenshot
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY and ORDER BY operators with `JSON_VALUE`.
+
+
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- JSON_QUERY
+- JSON_OBJECT
+- PARSE_JSON
+- TO_JSON_STRING
+
+These functions are primarily intended for use with the Multi-Stage Query Architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
+
+#### Example query: Return results in a JSON object
+
+You can use the `JSON_QUERY` function to extract a partial structure from any JSON input and return results in a JSON object. Unlike `JSON_VALUE` it can extract objects and arrays.
+
+The following example query illustrates the differences in output between `JSON_VALUE` and `JSON_QUERY`.
+
+
+
+```sql
+SELECT
+ JSON_VALUE(shipTo, '$.address'),
+ JSON_QUERY(shipTo, '$.address'),
+ JSON_VALUE(shipTo, '$.phoneNumbers'),
+ JSON_QUERY(shipTo, '$.phoneNumbers')
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["VARCHAR","OTHER","VARCHAR","OTHER"],["","{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"}","","[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]"],["","{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"}","","[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]"],["","{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"}","","[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]"],["","{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\"
,\"postalCode\":\"98912-1195\"}","","[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]"],["","{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"}","","[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]"]]
+```
+
+#### Example query: Combine multiple JSON inputs into a single JSON object value
+
+ The following query illustrates how to use `JSON_OBJECT` to combine nested data elements into a new object.
+
+
+
+```sql
+SELECT
+ JSON_OBJECT(KEY 'shipTo' VALUE JSON_QUERY(shipTo, '$'), KEY 'details' VALUE JSON_QUERY(details, '$')) as combinedJson
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["combinedJson"],["COMPLEX<json>"],["OTHER"],["{\"details\":{\"color\":\"sky blue\",\"price\":542.0},\"shipTo\":{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}}"],["{\"details\":{\"color\":\"ivory\",\"price\":955.0},\"shipTo\":{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}}"],["{\"details\":{\"price\":8.0},\"shipTo\":{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\
",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}}"],["{\"details\":{\"color\":\"olive\",\"price\":90.0},\"shipTo\":{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}}"],["{\"details\":{\"color\":\"plum\",\"price\":40.0},\"shipTo\":{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Loischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}}"]]
+```
+
+### Using other transform functions
+
+Druid provides the following additional transform functions:
+
+- `PARSE_JSON`: Deserializes a string value into a JSON object.
+- `TO_JSON_STRING`: Performs the operation of `TO_JSON` and then serializes the value into a string.
+
+#### Example query: Parse and deserialize data
+
+ The following query illustrates how to use the transform functions to parse and deserialize data.
+
+
+
+```sql
+SELECT
+ PARSE_JSON('{"x":"y"}'),
+ TO_JSON_STRING('{"x":"y"}'),
+ TO_JSON_STRING(PARSE_JSON('{"x":"y"}'))
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$2","EXPR$3"],["COMPLEX<json>","STRING","STRING"],["OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+### Using helper operators
+
+The `JSON_KEYS` and `JSON_PATHS` functions are helper operators that you can use to examine JSON object schema. Use them to plan your queries, for example to work out which paths to use in `JSON_VALUE`.
+
+#### Example query: Examine JSON object schema
+
+ The following query illustrates how to use the helper operators to examine a nested data object.
+
+
+
+```sql
+SELECT
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_KEYS(shipTo, '$.address')),
+ ARRAY_CONCAT_AGG(DISTINCT JSON_PATHS(shipTo))
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["EXPR$0","EXPR$1","EXPR$2","EXPR$3"],["COMPLEX<json>","COMPLEX<json>","STRING","STRING"],["OTHER","OTHER","VARCHAR","VARCHAR"],["{\"x\":\"y\"}","\"{\\\"x\\\":\\\"y\\\"}\"","\"{\\\"x\\\":\\\"y\\\"}\"","{\"x\":\"y\"}"]]
+```
+
+## Known issues
+
+Before you start using the nested columns feature, consider the following known issues:
+
+- Directly using `COMPLEX<json>` typed columns and expressions is not well integrated into the Druid query engine. It can result in errors or undefined behavior when grouping and filtering, and when you use using `COMPLEX<json>` objects as inputs to aggregators. As a workaround, consider using `TO_JSON_STRING` to coerce the values to strings before you perform these operations.
+- Directly using array-typed outputs from `JSON_KEYS` and `JSON_PATHS` is moderately supported by the Druid query engine. You can group on these outputs, and there are a number of array expressions that can operate on these values, such as `ARRAY_CONCAT_AGG`. However, some operations are not well defined for use outside array-specific functions, such as filtering using `=` or `IS NULL`.
+- Input validation for JSON SQL operators is currently incomplete, which sometimes results in undefined behavior or unhelpful error messages.
+- Ingesting JSON columns with a very complex nested structure is potentially an expensive operation, and may require you to tune ingestion tasks and/or cluster parameters to account for increased memory usage or overall task run time. When you tune your ingestion configuration, treat each nested literal field inside a JSON object as a flattened top-level Druid column.
Review Comment:
```suggestion
- Ingesting JSON columns with a very complex nested structure is potentially an expensive operation and may require you to tune ingestion tasks and/or cluster parameters to account for increased memory usage or overall task run time. When you tune your ingestion configuration, treat each nested literal field inside a JSON object as a flattened top-level Druid column.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] paul-rogers commented on pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on PR #12946:
URL: https://github.com/apache/druid/pull/12946#issuecomment-1226213400
Because of a recent Travis change that introduced a "silent" merge conflict, you will need to rebase this PR on the latest master to pick up the newest Travis file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r953688947
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,546 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ ~
+ ~ Nested columns is an experimental feature that is subject to
+ ~ change or removal at any time.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. As an experimental feature, functionality documented on this page is subject to change or removal in future releases. Review the release notes and this page to stay up to date with changes.
+
+You can ingest and store nested JSON in an Apache Druid column as a `COMPLEX<json>` data type. Druid indexes and optimizes the nested data. This means you can use JSON functions to extract ‘literal’ values at ingestion time using the `transformSpec` or in the SELECT clause when using multi-stage query architecture.
+
+Druid SQL JSON functions let you extract, transform, and create `COMPLEX<json>` values. Additionally, you can use certain `JSONPath` operators to extract values from nested data structures.
+
+Druid currently supports ingesting JSON-format nested columns with this feature. If you want to ingest nested data in another format, consider using the [flattenSpec object](../ingestion/data-formats.md#flattenspec).
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+> Note to self: add screenshot
+
+<!----->
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ "time",
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ "time",
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](../misc/math-expr.md#nested-columns-functions) for information on the functions in the examples below.
+
+### JSONPath syntax
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures.
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] writer-jill commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
writer-jill commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r958651509
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
Review Comment:
Done
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] techdocsmith commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
techdocsmith commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r957947700
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,503 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format, and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#nested-columns-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion. This is an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to 'flatten' nested data for ingestion.
+
+### Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [flattenSpec](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, include a `dimensions` object in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, include a `transforms` object in the `transformSpec` property of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the column `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [batch example above](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
Review Comment:
Torn here. I think that the reader may want to see the output the way it is delivered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vtlim commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
vtlim commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r958762291
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,595 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
Review Comment:
```suggestion
Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format and specialized internal columns and indexes for nested literal values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vtlim commented on a diff in pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
vtlim commented on code in PR #12946:
URL: https://github.com/apache/druid/pull/12946#discussion_r958757128
##########
docs/querying/nested-columns.md:
##########
@@ -0,0 +1,595 @@
+---
+id: nested-columns
+title: "Nested columns"
+sidebar_label: Nested columns
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+> Nested columns is an experimental feature available starting in Apache Druid 24.0. Like most experimental features, functionality documented on this page is subject to change in future releases. However, the COMPLEX column type includes versioning to provide backward compatible support in future releases. We strongly encourage you you experiment with nested columns in your development environment to evaluate that they meet your use case. If so, you can use them in production scenarios. Review the release notes and this page to stay up to date with changes.
+
+Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format and specialized internal columns and indexes for nested 'literal' values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.
+
+Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](../misc/math-expr.md#json-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec).
+
+You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion as an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to "flatten" nested data for ingestion.
+
+## Example nested data
+
+The examples in this topic use the data in [nested_example_data.json](https://static.imply.io/data/nested_example_data.json). The file contains a simple fascimile of an order tracking and shipping table.
+
+When pretty-printed, a sample row in `nested_example_data` looks like this:
+
+```json
+{
+ "time":"2022-6-14T10:32:08Z",
+ "product":"Keyboard",
+ "department":"Computers",
+ "shipTo":{
+ "firstName": "Sandra",
+ "lastName": "Beatty",
+ "address": {
+ "street": "293 Grant Well",
+ "city": "Loischester",
+ "state": "FL",
+ "country": "TV",
+ "postalCode": "88845-0066"
+ },
+ "phoneNumbers": [
+ {"type":"primary","number":"1-788-771-7028 x8627" },
+ {"type":"secondary","number":"1-460-496-4884 x887"}
+ ]
+ },
+ "details"{"color":"plum","price":"40.00"}
+}
+```
+
+## Native batch ingestion
+
+For native batch ingestion, you can use the [JSON nested columns functions](./sql-json-functions.md) to extract nested data as an alternative to using the [`flattenSpec`](../ingestion/data-formats.md#flattenspec) input format.
+
+To configure a dimension as a nested data type, specify the `json` type for the dimension in the `dimensions` list in the `dimensionsSpec` property of your ingestion spec.
+
+For example, the following ingestion spec instructs Druid to ingest `shipTo` and `details` as JSON-type nested dimensions:
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ {
+ "type": "json",
+ "name": "shipTo"
+ },
+ {
+ "type": "json",
+ "name": "details"
+ }
+ ]
+ },
+ "transformSpec": {}
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+### Transform data during batch ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data and reference the transformed data in your ingestion spec.
+
+To do this, define the output name and expression in the `transforms` list in the `transformSpec` object of your ingestion spec.
+
+For example, the following ingestion spec extracts `firstName`, `lastName` and `address` from `shipTo` and creates a composite JSON object containing `product`, `details` and `department`.
+
+```json
+{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "http",
+ "uris": [
+ "https://static.imply.io/data/nested_example_data.json"
+ ]
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "dataSchema": {
+ "granularitySpec": {
+ "segmentGranularity": "day",
+ "queryGranularity": "none",
+ "rollup": false
+ },
+ "dataSource": "nested_data_transform_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "auto"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "firstName",
+ "lastName",
+ {
+ "type": "json",
+ "name": "address"
+ },
+ {
+ "type": "json",
+ "name": "productDetails"
+ }
+ ]
+ },
+ "transformSpec": {
+ "transforms":[
+ { "type":"expression", "name":"firstName", "expression":"json_value(shipTo, '$.firstName')"},
+ { "type":"expression", "name":"lastName", "expression":"json_value(shipTo, '$.lastName')"},
+ { "type":"expression", "name":"address", "expression":"json_query(shipTo, '$.address')"},
+ { "type":"expression", "name":"productDetails", "expression":"json_object('product', product, 'details', details, 'department', department)"}
+ ]
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ }
+ }
+}
+```
+
+## SQL-based ingestion
+
+To ingest nested data using multi-stage query architecture, specify `COMPLEX<json>` as the value for `type` when you define the row signature—`shipTo` and `details` in the following example ingestion spec:
+
+
+
+```sql
+REPLACE INTO msq_nested_data_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ product,
+ department,
+ shipTo,
+ details
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+### Transform data during SQL-based ingestion
+
+You can use the [JSON nested columns functions](./sql-json-functions.md) to transform JSON data in your ingestion query.
+
+For example, the following ingestion query is the SQL-based version of the [previous batch example](#transform-data-during-batch-ingestion)—it extracts `firstName`, `lastName`, and `address` from `shipTo` and creates a composite JSON object containing `product`, `details`, and `department`.
+
+
+
+```sql
+REPLACE INTO msq_nested_data_transform_example OVERWRITE ALL
+SELECT
+ TIME_PARSE("time") as __time,
+ JSON_VALUE(shipTo, '$.firstName') as firstName,
+ JSON_VALUE(shipTo, '$.lastName') as lastName,
+ JSON_QUERY(shipTo, '$.address') as address,
+ JSON_OBJECT('product':product,'details':details, 'department':department) as productDetails
+FROM (
+ SELECT * FROM
+ TABLE(
+ EXTERN(
+ '{"type":"http","uris":["https://static.imply.io/data/nested_example_data.json"]}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"COMPLEX<json>"},{"name":"details","type":"COMPLEX<json>"}]'
+ )
+ )
+)
+PARTITIONED BY ALL
+```
+
+## Ingest deserialized JSON as COMPLEX\<json>
+
+If your source data uses a string representation of your JSON column, you can still ingest the data as `COMPLEX<JSON>` as follows:
+- During native batch ingestion, call the `parse_json` function in a `transform` object in the `transformSpec`.
+- During SQL-based ingestion, use the PARSE_JSON keyword within your SELECT statement to transform the string values to JSON.
+- If you are concerned that your data may not contain valid JSON, you can use `try_parse_json` for native batch or `TRY_PARSE_JSON` for SQL-based ingestion. For cases where the column does not contain valid JSON, Druid inserts a null value.
+
+For example, consider the following deserialized row of the sample data set:
+
+```
+{"time": "2022-06-13T10:10:35Z", "product": "Bike", "department":"Sports", "shipTo":"{\"firstName\": \"Henry\",\"lastName\": \"Wuckert\",\"address\": {\"street\": \"5643 Jan Walk\",\"city\": \"Lake Bridget\",\"state\": \"HI\",\"country\":\"ME\",\"postalCode\": \"70204-2939\"},\"phoneNumbers\": [{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\" },{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}", "details":"{\"color\":\"ivory\", \"price\":955.00}"}
+```
+
+The following examples demonstrate how to ingest the `shipTo` and `details` columns both as string type and as `COMPLEX<json>` in the `shipTo_parsed` and `details_parsed` columns.
+
+<!--DOCUSAURUS_CODE_TABS-->
+<!--SQL-->
+```
+REPLACE INTO deserialized_example OVERWRITE ALL
+WITH source AS (SELECT * FROM TABLE(
+ EXTERN(
+ '{"type":"inline","data":"{\"time\": \"2022-06-13T10:10:35Z\", \"product\": \"Bike\", \"department\":\"Sports\", \"shipTo\":\"{\\\"firstName\\\": \\\"Henry\\\",\\\"lastName\\\": \\\"Wuckert\\\",\\\"address\\\": {\\\"street\\\": \\\"5643 Jan Walk\\\",\\\"city\\\": \\\"Lake Bridget\\\",\\\"state\\\": \\\"HI\\\",\\\"country\\\":\\\"ME\\\",\\\"postalCode\\\": \\\"70204-2939\\\"},\\\"phoneNumbers\\\": [{\\\"type\\\":\\\"primary\\\",\\\"number\\\":\\\"593.475.0449 x86733\\\" },{\\\"type\\\":\\\"secondary\\\",\\\"number\\\":\\\"638-372-1210\\\"}]}\", \"details\":\"{\\\"color\\\":\\\"ivory\\\", \\\"price\\\":955.00}\"}\n"}',
+ '{"type":"json"}',
+ '[{"name":"time","type":"string"},{"name":"product","type":"string"},{"name":"department","type":"string"},{"name":"shipTo","type":"string"},{"name":"details","type":"string"},{"name":"shipTo_parsed","type":"json"},{"name":"details_parsed","type":"json"}]'
+ )
+))
+SELECT
+ TIME_PARSE("time") AS __time,
+ "product",
+ "department",
+ "shipTo",
+ "details",
+ PARSE_JSON("shipTo") as "shipTo_parsed",
+ PARSE_JSON("details") as "details_parsed"
+FROM source
+PARTITIONED BY DAY
+```
+<!--Native batch-->
+```{
+ "type": "index_parallel",
+ "spec": {
+ "ioConfig": {
+ "type": "index_parallel",
+ "inputSource": {
+ "type": "inline",
+ "data": "{\"time\": \"2022-06-13T10:10:35Z\", \"product\": \"Bike\", \"department\":\"Sports\", \"shipTo\":\"{\\\"firstName\\\": \\\"Henry\\\",\\\"lastName\\\": \\\"Wuckert\\\",\\\"address\\\": {\\\"street\\\": \\\"5643 Jan Walk\\\",\\\"city\\\": \\\"Lake Bridget\\\",\\\"state\\\": \\\"HI\\\",\\\"country\\\":\\\"ME\\\",\\\"postalCode\\\": \\\"70204-2939\\\"},\\\"phoneNumbers\\\": [{\\\"type\\\":\\\"primary\\\",\\\"number\\\":\\\"593.475.0449 x86733\\\" },{\\\"type\\\":\\\"secondary\\\",\\\"number\\\":\\\"638-372-1210\\\"}]}\", \"details\":\"{\\\"color\\\":\\\"ivory\\\", \\\"price\\\":955.00}\"}\n"
+ },
+ "inputFormat": {
+ "type": "json"
+ }
+ },
+ "tuningConfig": {
+ "type": "index_parallel",
+ "partitionsSpec": {
+ "type": "dynamic"
+ }
+ },
+ "dataSchema": {
+ "dataSource": "deserialized_example",
+ "timestampSpec": {
+ "column": "time",
+ "format": "iso"
+ },
+ "transformSpec": {
+ "transforms": [
+ {
+ "type": "expression",
+ "name": "shipTo_parsed",
+ "expression": "parse_json(shipTo)"
+ },
+ {
+ "type": "expression",
+ "name": "details_parsed",
+ "expression": "parse_json(details)"
+ }
+ ]
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "product",
+ "department",
+ "shipTo",
+ "details",
+ "shipTo_parsed",
+ "details_parsed"
+ ]
+ },
+ "granularitySpec": {
+ "queryGranularity": "none",
+ "rollup": false,
+ "segmentGranularity": "day"
+```
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+## Querying nested columns
+
+Once ingested, Druid stores the JSON-typed columns as native JSON objects and presents them as `COMPLEX<json>`.
+
+See the [Nested columns functions reference](./sql-json-functions.md) for information on the functions in the examples below.
+
+Druid supports a small, simplified subset of the [JSONPath syntax](https://github.com/json-path/JsonPath/blob/master/README.md) operators, primarily limited to extracting individual values from nested data structures. See the [SQL JSON functions](./sql-json-functions.md#jsonpath-syntax) page for details.
+
+### Displaying data types
+
+The following example illustrates how you can display the data types for your columns. Note that `details` and `shipTo` display as `COMPLEX<json>`.
+
+#### Example query: Display data types
+
+
+
+```sql
+SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+FROM INFORMATION_SCHEMA.COLUMNS
+WHERE TABLE_NAME = 'nested_data_example'
+```
+
+Example query results:
+
+```json
+[["TABLE_NAME","COLUMN_NAME","DATA_TYPE"],["STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR"],["nested_data_example","__time","TIMESTAMP"],["nested_data_example","department","VARCHAR"],["nested_data_example","details","COMPLEX<json>"],["nested_data_example","product","VARCHAR"],["nested_data_example","shipTo","COMPLEX<json>"]]
+```
+
+### Retrieving JSON data
+
+You can retrieve JSON data directly from a table. Druid returns the results as a JSON object, so you can't use grouping, aggregation, or filtering operators.
+
+#### Example query: Retrieve JSON data
+
+The following example query extracts all data from `nested_data_example`:
+
+
+
+```sql
+SELECT * FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["__time","department","details","product","shipTo"],["LONG","STRING","COMPLEX<json>","STRING","COMPLEX<json>"],["TIMESTAMP","VARCHAR","OTHER","VARCHAR","OTHER"],["2022-06-13T07:52:29.000Z","Sports","{\"color\":\"sky blue\",\"price\":542.0}","Bike","{\"firstName\":\"Russ\",\"lastName\":\"Cole\",\"address\":{\"street\":\"77173 Rusty Station\",\"city\":\"South Yeseniabury\",\"state\":\"WA\",\"country\":\"BL\",\"postalCode\":\"01893\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"891-374-6188 x74568\"},{\"type\":\"secondary\",\"number\":\"1-248-998-4426 x33037\"}]}"],["2022-06-13T10:10:35.000Z","Sports","{\"color\":\"ivory\",\"price\":955.0}","Bike","{\"firstName\":\"Henry\",\"lastName\":\"Wuckert\",\"address\":{\"street\":\"5643 Jan Walk\",\"city\":\"Lake Bridget\",\"state\":\"HI\",\"country\":\"ME\",\"postalCode\":\"70204-2939\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"593.475.0449 x86733\"},{\"type\":\"secondary\",\"number\":\"638-372-1210\"}]}"],["2022-06-13T1
3:57:38.000Z","Grocery","{\"price\":8.0}","Sausages","{\"firstName\":\"Forrest\",\"lastName\":\"Brekke\",\"address\":{\"street\":\"41548 Collier Divide\",\"city\":\"Wintheiserborough\",\"state\":\"WA\",\"country\":\"AD\",\"postalCode\":\"27577-6784\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(904) 890-0696 x581\"},{\"type\":\"secondary\",\"number\":\"676.895.6759\"}]}"],["2022-06-13T21:37:06.000Z","Computers","{\"color\":\"olive\",\"price\":90.0}","Mouse","{\"firstName\":\"Rickey\",\"lastName\":\"Rempel\",\"address\":{\"street\":\"6232 Green Glens\",\"city\":\"New Fermin\",\"state\":\"HI\",\"country\":\"CW\",\"postalCode\":\"98912-1195\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"(689) 766-4272 x60778\"},{\"type\":\"secondary\",\"number\":\"375.662.4737 x24707\"}]}"],["2022-06-14T10:32:08.000Z","Computers","{\"color\":\"plum\",\"price\":40.0}","Keyboard","{\"firstName\":\"Sandra\",\"lastName\":\"Beatty\",\"address\":{\"street\":\"293 Grant Well\",\"city\":\"Lo
ischester\",\"state\":\"FL\",\"country\":\"TV\",\"postalCode\":\"88845-0066\"},\"phoneNumbers\":[{\"type\":\"primary\",\"number\":\"1-788-771-7028 x8627\"},{\"type\":\"secondary\",\"number\":\"1-460-496-4884 x887\"}]}"]]
+```
+
+### Extracting nested data elements
+
+The `JSON_VALUE` function is specially optimized to provide native Druid level performance when processing nested literal values, as if they were flattened, traditional, Druid column types. It does this by reading from the specialized nested columns and indexes that are built and stored in JSON objects when Druid creates segments.
+
+Some operations using `JSON_VALUE` run faster than those using native Druid columns. For example, filtering numeric types uses the indexes built for nested numeric columns, which are not available for Druid DOUBLE, FLOAT, or LONG columns.
+
+`JSON_VALUE` only returns literal types. Any paths that reference JSON objects or array types return null.
+
+> To achieve the best possible performance, use the `JSON_VALUE` function whenever you query JSON objects.
+
+#### Example query: Extract nested data elements
+
+The following example query illustrates how to use `JSON_VALUE` to extract specified elements from a `COMPLEX<json>` object. Note that the returned values default to type VARCHAR.
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(shipTo, '$.phoneNumbers[0].number') as primaryPhone,
+ JSON_VALUE(details, '$.price') as price
+FROM nested_data_example
+```
+
+Example query results:
+
+```json
+[["product","department","country","primaryPhone","price"],["STRING","STRING","STRING","STRING","STRING"],["VARCHAR","VARCHAR","VARCHAR","VARCHAR","VARCHAR"],["Bike","Sports","BL","891-374-6188 x74568","542.0"],["Bike","Sports","ME","593.475.0449 x86733","955.0"],["Sausages","Grocery","AD","(904) 890-0696 x581","8.0"],["Mouse","Computers","CW","(689) 766-4272 x60778","90.0"],["Keyboard","Computers","TV","1-788-771-7028 x8627","40.0"]]
+```
+
+### Extracting nested data elements as a suggested type
+
+You can use the `RETURNING` keyword to provide type hints to the `JSON_VALUE` function. This way the SQL planner produces the correct native Druid query, leading to expected results. This keyword allows you to specify a SQL type for the `path` value.
+
+#### Example query: Extract nested data elements as suggested types
+
+The following example query illustrates how to use `JSON_VALUE` and the `RETURNING` keyword to extract an element of nested data and return it as specified types.
+
+
+
+```sql
+SELECT
+ product,
+ department,
+ JSON_VALUE(shipTo, '$.address.country') as country,
+ JSON_VALUE(details, '$.price' RETURNING BIGINT) as price_int,
+ JSON_VALUE(details, '$.price' RETURNING DECIMAL) as price_decimal,
+ JSON_VALUE(details, '$.price' RETURNING VARCHAR) as price_varchar
+FROM nested_data_example
+```
+
+Query results:
+
+```json
+[["product","department","country","price_int","price_decimal","price_varchar"],["STRING","STRING","STRING","LONG","DOUBLE","STRING"],["VARCHAR","VARCHAR","VARCHAR","BIGINT","DECIMAL","VARCHAR"],["Bike","Sports","BL",542,542.0,"542.0"],["Bike","Sports","ME",955,955.0,"955.0"],["Sausages","Grocery","AD",8,8.0,"8.0"],["Mouse","Computers","CW",90,90.0,"90.0"],["Keyboard","Computers","TV",40,40.0,"40.0"]]
+```
+
+### Grouping, aggregating, and filtering
+
+You can use `JSON_VALUE` expressions in any context where you can use traditional Druid columns, such as grouping, aggregation, and filtering.
+
+#### Example query: Grouping and filtering
+
+The following example query illustrates how to use SUM, WHERE, GROUP BY, and ORDER BY operators with `JSON_VALUE`.
+
+
+
+```sql
+SELECT
+ product,
+ JSON_VALUE(shipTo, '$.address.country'),
+ SUM(JSON_VALUE(details, '$.price' RETURNING BIGINT))
+FROM nested_data_example
+WHERE JSON_VALUE(shipTo, '$.address.country') in ('BL', 'CW')
+GROUP BY 1,2
+ORDER BY 3 DESC
+```
+
+Example query results:
+
+```json
+[["product","EXPR$1","EXPR$2"],["STRING","STRING","LONG"],["VARCHAR","VARCHAR","BIGINT"],["Bike","BL",542],["Mouse","CW",90]]
+```
+
+### Transforming JSON object data
+
+In addition to `JSON_VALUE`, Druid offers a number of operators that focus on transforming JSON object data:
+
+- `JSON_QUERY`
+- `JSON_OBJECT`
+- `PARSE_JSON`
+- `TO_JSON_STRING`
+
+These functions are primarily intended for use with the multi-Stage Query Architecture to transform data during insert operations, but they also work in traditional Druid SQL queries. Because most of these functions output JSON objects, they have the same limitations when used in traditional Druid queries as interacting with the JSON objects directly.
+
+#### Example query: Return results in a JSON object
+
+You can use the `JSON_QUERY` function to extract a partial structure from any JSON input and return results in a JSON object. Unlike `JSON_VALUE` it can extract objects and arrays.
+
+The following example query illustrates the differences in output between `JSON_VALUE` and `JSON_QUERY`. The two output columns for JSON_VALUE contain null values only because JSON_VALUE only returns literal types.
Review Comment:
```suggestion
The following example query illustrates the differences in output between `JSON_VALUE` and `JSON_QUERY`. The two output columns for `JSON_VALUE` contain null values only because `JSON_VALUE` only returns literal types.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] techdocsmith commented on pull request #12946: Nested columns documentation
Posted by GitBox <gi...@apache.org>.
techdocsmith commented on PR #12946:
URL: https://github.com/apache/druid/pull/12946#issuecomment-1232290650
@clintropolis , I fixed the one broken link. The other one to MSQ docs is dependent on https://github.com/apache/druid/pull/12983/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org