You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "edgar2020 (via GitHub)" <gi...@apache.org> on 2024/08/05 23:43:00 UTC
[PR] [Docs] updating transformation during ingestion tutorial (druid)
edgar2020 opened a new pull request, #16845:
URL: https://github.com/apache/druid/pull/16845
### Description
Updating `tutorial-transform-spec.md` to to use MSQ inline ingestion. Replaces file segments associated with the defunct instructions. Updates language to use active voice.
This PR has:
- [X] been self-reviewed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "edgar2020 (via GitHub)" <gi...@apache.org>.
edgar2020 commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1716055771
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
+* `triple-number`: multiplies the `number` column by 3. This stores the results in a new column called `triple-number`. Note that it ingestions both the original and the transformed column.
+
+Additionally, a filter with the following three OR clauses are applied:
+* `animal` values that match "super-mongoose"
+* `triple-number` values that match 300
+* `location` values that match 3
+
+## Query the transformed data
+
+In the web console, open a new tab in the **Query** view. Run the following query to view the ingested data:
+
+```sql
+SELECT * FROM "transform_tutorial"
+```
+
+Returns the following:
+
+| `__time` | `animal` | `location` | `number` | `triple-number` |
+| -- | -- | -- | -- | -- |
+| `2018-01-01T05:01:35.000Z` | `super-mongoose` | `2` | `200` | `600` |
+| `2018-01-01T06:01:35.000Z` | `super-snake` | `3` | `300` | `900` |
+| `2018-01-01T07:01:35.000Z` | `super-octopus` | `1` | `100` | `300` |
+
+Note that the filter applies after the transformations. The only row that isn't ingested from the input data is the "lion" row because it doesn't meet any of the three filter condition listed.
Review Comment:
Added suggestion to the paragraph right before the filter examples
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1717156135
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,102 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
Review Comment:
Users don't need to know how to load a file to complete this tutorial. Use the same text as in [PR 16762](https://github.com/apache/druid/pull/16762/files#diff-36e70975ef02912047f7336cfd6e1a6988b53699f75008289a8e3f62e270c6d9R35):
> You should be familiar with data querying in Druid. If you haven't already, go through the [Query data](../tutorials/tutorial-query.md) tutorial first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "edgar2020 (via GitHub)" <gi...@apache.org>.
edgar2020 commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1716051788
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
Review Comment:
I can see the potential for confusion, replaced it with "Note that it only ingests the transformed data."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "edgar2020 (via GitHub)" <gi...@apache.org>.
edgar2020 commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1717375719
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,102 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
Review Comment:
Swapped to text from PR 16762
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "edgar2020 (via GitHub)" <gi...@apache.org>.
edgar2020 commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1719144131
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+Before proceeding, download Druid as described in [Quickstart (local)](index.md) and have it running on your local machine. You don't need to load any data into the Druid cluster.
+
+You should be familiar with data querying in Druid. If you haven't already, go through the [Query data](../tutorials/tutorial-query.md) tutorial first.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+
+```json
+{"timestamp":"2018-01-01T07:01:35Z", "animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z", "animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z", "animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z", "animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" = 100)
+PARTITIONED BY DAY
+```
+
+In the `SELECT` clause, you specify the following transformations:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. Note that it only ingests the transformed data.
+* `triple-number`: multiplies the `number` column by three and stores the results in a column named `triple-number`. Note that the query ingests both the original and the transformed data.
+
+Additionally, the `WHERE` clause applies the following three OR operators so that the query only ingests the rows where at least one of the following conditions is `true`:
+
+* `TEXTCAT('super-', "animal")` matches "super-mongoose"
+* `location` matches 3
+* `number` matches 100
+
+## Query the transformed data
+
+In the web console, open a new tab in the **Query** view. Run the following query to view the ingested data:
+
+```sql
+SELECT * FROM "transform_tutorial"
+```
+
+Returns the following:
+
+| `__time` | `animal` | `location` | `number` | `triple-number` |
+| -- | -- | -- | -- | -- |
+| `2018-01-01T05:01:35.000Z` | `super-mongoose` | `2` | `200` | `600` |
+| `2018-01-01T06:01:35.000Z` | `super-snake` | `3` | `300` | `900` |
+| `2018-01-01T07:01:35.000Z` | `super-octopus` | `1` | `100` | `300` |
+
+Once a row is accepted by the filter, the ingestion job applies the transformations. In this example, the filter selects the first three rows because each row meets at least one of the necessary OR conditions. Note that for the three rows selected, the ingestion job ingests the transformed `animal` column, the `location` column, and both the original `number` and the transformed `triple-number` column. The "lion" row is not accepted by the filter, so it is not ingested or transformed.
Review Comment:
I moved the paragraph to the end of "Transform data during ingestion" and added one sentance to the end of the "Query the transformed data" section to replicate the original concluding paragraph
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1718875694
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+Before proceeding, download Druid as described in [Quickstart (local)](index.md) and have it running on your local machine. You don't need to load any data into the Druid cluster.
Review Comment:
```suggestion
Before proceeding, download Apache Druid as described in [Quickstart (local)](index.md) and have it running on your local machine. You don't need to load any data into the Druid cluster.
```
Added "Apache" since it's the first time we mention Druid in this topic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1717153998
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,102 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
Review Comment:
For consistency, use the same text as in [PR 16762](https://github.com/apache/druid/pull/16762/files#diff-36e70975ef02912047f7336cfd6e1a6988b53699f75008289a8e3f62e270c6d9R33):
> Before proceeding, download Druid as described in [Quickstart (local)](index.md) and have it running on your local machine. You don't need to load any data into the Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1717206034
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,102 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+
+```json
+{"timestamp":"2018-01-01T07:01:35Z", "animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z", "animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z", "animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z", "animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+In the `SELECT` clause, you specify the following transformations:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. Note that it only ingests the transformed data.
+* `triple-number`: multiplies the `number` column by three and stores the results in a column named `triple-number`. Note that the query ingests both the original and the transformed data.
+
+Additionally, the `WHERE` clause applies the following three OR clauses so that the query only ingests the rows where at least one of the following conditions are `true`:
+
+* `TEXTCAT('super-', "animal")` value matches "super-mongoose"
+* `location` value matches 3
+* `'number' * 3` value matches 300
+
Review Comment:
The original tutorial concluded this section with the following sentence:
> This filter selects the first 3 rows, and it will exclude the final "lion" row in the input data. Note that the filter is applied after the transformation.
It would be helpful to include something similar here and explain why the "lion" row was excluded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1717206034
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,102 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+
+```json
+{"timestamp":"2018-01-01T07:01:35Z", "animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z", "animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z", "animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z", "animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+In the `SELECT` clause, you specify the following transformations:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. Note that it only ingests the transformed data.
+* `triple-number`: multiplies the `number` column by three and stores the results in a column named `triple-number`. Note that the query ingests both the original and the transformed data.
+
+Additionally, the `WHERE` clause applies the following three OR clauses so that the query only ingests the rows where at least one of the following conditions are `true`:
+
+* `TEXTCAT('super-', "animal")` value matches "super-mongoose"
+* `location` value matches 3
+* `'number' * 3` value matches 300
+
Review Comment:
The original tutorial concluded this section with the following sentence:
> This filter selects the first 3 rows, and it will exclude the final "lion" row in the input data. Note that the filter is applied after the transformation.
It would be helpful to include something similar here and explain why the `lion` row was excluded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "edgar2020 (via GitHub)" <gi...@apache.org>.
edgar2020 commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1716057805
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
+* `triple-number`: multiplies the `number` column by 3. This stores the results in a new column called `triple-number`. Note that it ingestions both the original and the transformed column.
+
+Additionally, a filter with the following three OR clauses are applied:
+* `animal` values that match "super-mongoose"
+* `triple-number` values that match 300
+* `location` values that match 3
+
+## Query the transformed data
+
+In the web console, open a new tab in the **Query** view. Run the following query to view the ingested data:
+
+```sql
+SELECT * FROM "transform_tutorial"
+```
+
+Returns the following:
+
+| `__time` | `animal` | `location` | `number` | `triple-number` |
+| -- | -- | -- | -- | -- |
+| `2018-01-01T05:01:35.000Z` | `super-mongoose` | `2` | `200` | `600` |
+| `2018-01-01T06:01:35.000Z` | `super-snake` | `3` | `300` | `900` |
+| `2018-01-01T07:01:35.000Z` | `super-octopus` | `1` | `100` | `300` |
+
+Note that the filter applies after the transformations. The only row that isn't ingested from the input data is the "lion" row because it doesn't meet any of the three filter condition listed.
+
+## Learn more
+
+See the following topics for more information:
+
+* [All functions](../querying/sql-functions.md) for a list of functions that can be used to transform data.
+* [Transform spec reference](../ingestion/ingestion-spec.md/#transformspec) to learn more about transforms in JSON-based batch ingestion.
+* [Query filters](../querying/filters.md) to learn about the types of filters available.
Review Comment:
I removed it for now, since it could be confusing since that is not the primary purpose of the filters page
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "edgar2020 (via GitHub)" <gi...@apache.org>.
edgar2020 commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1716055097
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
+* `triple-number`: multiplies the `number` column by 3. This stores the results in a new column called `triple-number`. Note that it ingestions both the original and the transformed column.
+
+Additionally, a filter with the following three OR clauses are applied:
+* `animal` values that match "super-mongoose"
Review Comment:
I changed it to match the expression, being compared:
ex: TEXTCAT('super-', "animal")` value matches "super-mongoose"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "techdocsmith (via GitHub)" <gi...@apache.org>.
techdocsmith merged PR #16845:
URL: https://github.com/apache/druid/pull/16845
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "vtlim (via GitHub)" <gi...@apache.org>.
vtlim commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1712439530
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
+* `triple-number`: multiplies the `number` column by 3. This stores the results in a new column called `triple-number`. Note that it ingestions both the original and the transformed column.
Review Comment:
```suggestion
* `triple-number`: multiplies the `number` column by three and stores the results in a column named `triple-number`. Note that the query ingests both the original and the transformed data.
```
All of the columns are new columns, so you don't need to specify that.
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
+* `triple-number`: multiplies the `number` column by 3. This stores the results in a new column called `triple-number`. Note that it ingestions both the original and the transformed column.
+
+Additionally, a filter with the following three OR clauses are applied:
+* `animal` values that match "super-mongoose"
Review Comment:
It's ambiguous to call this `animal` values since it has the transform applied
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
Review Comment:
No override is happening here -- you just don't ingest `animal`. I think I understand what you were getting at, but override makes it sound like an ingest and replace operation.
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
+* `triple-number`: multiplies the `number` column by 3. This stores the results in a new column called `triple-number`. Note that it ingestions both the original and the transformed column.
+
+Additionally, a filter with the following three OR clauses are applied:
+* `animal` values that match "super-mongoose"
+* `triple-number` values that match 300
+* `location` values that match 3
+
+## Query the transformed data
+
+In the web console, open a new tab in the **Query** view. Run the following query to view the ingested data:
+
+```sql
+SELECT * FROM "transform_tutorial"
+```
+
+Returns the following:
+
+| `__time` | `animal` | `location` | `number` | `triple-number` |
+| -- | -- | -- | -- | -- |
+| `2018-01-01T05:01:35.000Z` | `super-mongoose` | `2` | `200` | `600` |
+| `2018-01-01T06:01:35.000Z` | `super-snake` | `3` | `300` | `900` |
+| `2018-01-01T07:01:35.000Z` | `super-octopus` | `1` | `100` | `300` |
+
+Note that the filter applies after the transformations. The only row that isn't ingested from the input data is the "lion" row because it doesn't meet any of the three filter condition listed.
+
+## Learn more
+
+See the following topics for more information:
+
+* [All functions](../querying/sql-functions.md) for a list of functions that can be used to transform data.
+* [Transform spec reference](../ingestion/ingestion-spec.md/#transformspec) to learn more about transforms in JSON-based batch ingestion.
+* [Query filters](../querying/filters.md) to learn about the types of filters available.
+* [WHERE clause](../querying/sql.md#where) to learn how to create filters in SQL.
Review Comment:
```suggestion
* [WHERE clause](../querying/sql.md#where) to learn how to specify filters in Druid SQL.
```
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
Review Comment:
```suggestion
{"timestamp":"2018-01-01T07:01:35Z", "animal":"octopus", "location":1, "number":100}
{"timestamp":"2018-01-01T05:01:35Z", "animal":"mongoose", "location":2,"number":200}
{"timestamp":"2018-01-01T06:01:35Z", "animal":"snake", "location":3, "number":300}
{"timestamp":"2018-01-01T01:01:35Z", "animal":"lion", "location":4, "number":300}
```
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
Review Comment:
```suggestion
In the SELECT clause, you specify the following transformations:
```
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
+* `triple-number`: multiplies the `number` column by 3. This stores the results in a new column called `triple-number`. Note that it ingestions both the original and the transformed column.
+
+Additionally, a filter with the following three OR clauses are applied:
+* `animal` values that match "super-mongoose"
+* `triple-number` values that match 300
+* `location` values that match 3
+
Review Comment:
Match the order of the list with the order in the WHERE clause
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
+* `triple-number`: multiplies the `number` column by 3. This stores the results in a new column called `triple-number`. Note that it ingestions both the original and the transformed column.
+
+Additionally, a filter with the following three OR clauses are applied:
Review Comment:
Change to active voice
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
+* `triple-number`: multiplies the `number` column by 3. This stores the results in a new column called `triple-number`. Note that it ingestions both the original and the transformed column.
+
+Additionally, a filter with the following three OR clauses are applied:
+* `animal` values that match "super-mongoose"
+* `triple-number` values that match 300
+* `location` values that match 3
+
+## Query the transformed data
+
+In the web console, open a new tab in the **Query** view. Run the following query to view the ingested data:
+
+```sql
+SELECT * FROM "transform_tutorial"
+```
+
+Returns the following:
+
+| `__time` | `animal` | `location` | `number` | `triple-number` |
+| -- | -- | -- | -- | -- |
+| `2018-01-01T05:01:35.000Z` | `super-mongoose` | `2` | `200` | `600` |
+| `2018-01-01T06:01:35.000Z` | `super-snake` | `3` | `300` | `900` |
+| `2018-01-01T07:01:35.000Z` | `super-octopus` | `1` | `100` | `300` |
+
+Note that the filter applies after the transformations. The only row that isn't ingested from the input data is the "lion" row because it doesn't meet any of the three filter condition listed.
Review Comment:
>Note that the filter applies after the transformations.
This is also kind of ambiguous. You introduce transformations as what happens in the SELECT clause, but it's not taking those transformed values for applying the filter (remember you couldn't use the transformed value directly in the WHERE clause). I would remove this part in favor of describing earlier that you only ingest the rows that meet these filters.
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
+* `triple-number`: multiplies the `number` column by 3. This stores the results in a new column called `triple-number`. Note that it ingestions both the original and the transformed column.
+
+Additionally, a filter with the following three OR clauses are applied:
+* `animal` values that match "super-mongoose"
+* `triple-number` values that match 300
+* `location` values that match 3
+
+## Query the transformed data
+
+In the web console, open a new tab in the **Query** view. Run the following query to view the ingested data:
+
+```sql
+SELECT * FROM "transform_tutorial"
+```
+
+Returns the following:
+
+| `__time` | `animal` | `location` | `number` | `triple-number` |
+| -- | -- | -- | -- | -- |
+| `2018-01-01T05:01:35.000Z` | `super-mongoose` | `2` | `200` | `600` |
+| `2018-01-01T06:01:35.000Z` | `super-snake` | `3` | `300` | `900` |
+| `2018-01-01T07:01:35.000Z` | `super-octopus` | `1` | `100` | `300` |
+
+Note that the filter applies after the transformations. The only row that isn't ingested from the input data is the "lion" row because it doesn't meet any of the three filter condition listed.
+
Review Comment:
The [original tutorial](https://druid.apache.org/docs/latest/tutorials/tutorial-transform-spec) had this concluding sort of sentence -- I suggest adding it back or something similar, since the point of this tutorial is transformations, not filters.
It might even make sense to remove the filters from this doc. The native JSON case let you use the `transformSpec.transforms` column names directly in `transformSpec.filter` but it seems that's not possible with SQL. Thoughts on this @techdocsmith @ektravel?
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
+* `triple-number`: multiplies the `number` column by 3. This stores the results in a new column called `triple-number`. Note that it ingestions both the original and the transformed column.
+
+Additionally, a filter with the following three OR clauses are applied:
+* `animal` values that match "super-mongoose"
+* `triple-number` values that match 300
+* `location` values that match 3
+
+## Query the transformed data
+
+In the web console, open a new tab in the **Query** view. Run the following query to view the ingested data:
+
+```sql
+SELECT * FROM "transform_tutorial"
+```
+
+Returns the following:
+
+| `__time` | `animal` | `location` | `number` | `triple-number` |
+| -- | -- | -- | -- | -- |
+| `2018-01-01T05:01:35.000Z` | `super-mongoose` | `2` | `200` | `600` |
+| `2018-01-01T06:01:35.000Z` | `super-snake` | `3` | `300` | `900` |
+| `2018-01-01T07:01:35.000Z` | `super-octopus` | `1` | `100` | `300` |
+
+Note that the filter applies after the transformations. The only row that isn't ingested from the input data is the "lion" row because it doesn't meet any of the three filter condition listed.
+
+## Learn more
+
+See the following topics for more information:
+
+* [All functions](../querying/sql-functions.md) for a list of functions that can be used to transform data.
+* [Transform spec reference](../ingestion/ingestion-spec.md/#transformspec) to learn more about transforms in JSON-based batch ingestion.
+* [Query filters](../querying/filters.md) to learn about the types of filters available.
+* [WHERE clause](../querying/sql.md#where) to learn how to create filters in SQL.
Review Comment:
Possibly true for this one too, but it's more related in that you do use WHERE to add filters.
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
+* `triple-number`: multiplies the `number` column by 3. This stores the results in a new column called `triple-number`. Note that it ingestions both the original and the transformed column.
+
+Additionally, a filter with the following three OR clauses are applied:
+* `animal` values that match "super-mongoose"
+* `triple-number` values that match 300
+* `location` values that match 3
+
+## Query the transformed data
+
+In the web console, open a new tab in the **Query** view. Run the following query to view the ingested data:
+
+```sql
+SELECT * FROM "transform_tutorial"
+```
+
+Returns the following:
+
+| `__time` | `animal` | `location` | `number` | `triple-number` |
+| -- | -- | -- | -- | -- |
+| `2018-01-01T05:01:35.000Z` | `super-mongoose` | `2` | `200` | `600` |
+| `2018-01-01T06:01:35.000Z` | `super-snake` | `3` | `300` | `900` |
+| `2018-01-01T07:01:35.000Z` | `super-octopus` | `1` | `100` | `300` |
+
+Note that the filter applies after the transformations. The only row that isn't ingested from the input data is the "lion" row because it doesn't meet any of the three filter condition listed.
+
+## Learn more
+
+See the following topics for more information:
+
+* [All functions](../querying/sql-functions.md) for a list of functions that can be used to transform data.
+* [Transform spec reference](../ingestion/ingestion-spec.md/#transformspec) to learn more about transforms in JSON-based batch ingestion.
+* [Query filters](../querying/filters.md) to learn about the types of filters available.
Review Comment:
This topic might be a little too out of scope for this tutorial. Is there a particular reason you chose to add it?
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
+* `triple-number`: multiplies the `number` column by 3. This stores the results in a new column called `triple-number`. Note that it ingestions both the original and the transformed column.
+
+Additionally, a filter with the following three OR clauses are applied:
Review Comment:
Also make it clear that it filters _for_ these values. That is, you only ingest the data that meet any of these conditions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1717239261
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
+* `triple-number`: multiplies the `number` column by 3. This stores the results in a new column called `triple-number`. Note that it ingestions both the original and the transformed column.
+
+Additionally, a filter with the following three OR clauses are applied:
+* `animal` values that match "super-mongoose"
+* `triple-number` values that match 300
+* `location` values that match 3
+
+## Query the transformed data
+
+In the web console, open a new tab in the **Query** view. Run the following query to view the ingested data:
+
+```sql
+SELECT * FROM "transform_tutorial"
+```
+
+Returns the following:
+
+| `__time` | `animal` | `location` | `number` | `triple-number` |
+| -- | -- | -- | -- | -- |
+| `2018-01-01T05:01:35.000Z` | `super-mongoose` | `2` | `200` | `600` |
+| `2018-01-01T06:01:35.000Z` | `super-snake` | `3` | `300` | `900` |
+| `2018-01-01T07:01:35.000Z` | `super-octopus` | `1` | `100` | `300` |
+
+Note that the filter applies after the transformations. The only row that isn't ingested from the input data is the "lion" row because it doesn't meet any of the three filter condition listed.
+
Review Comment:
I think we can keep the filters as long as we explain why the "lion" row was excluded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "edgar2020 (via GitHub)" <gi...@apache.org>.
edgar2020 commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1717369853
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,102 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
Review Comment:
switched to using the same text
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "asdf2014 (via GitHub)" <gi...@apache.org>.
asdf2014 commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1716409133
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
Review Comment:
```suggestion
For this tutorial, you use the following sample data:
```
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z", "animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z", "animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z", "animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z", "animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+In the SELECT clause, you specify the following transformations:
Review Comment:
```suggestion
In the `SELECT` clause, you specify the following transformations:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1717240877
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
+* `triple-number`: multiplies the `number` column by 3. This stores the results in a new column called `triple-number`. Note that it ingestions both the original and the transformed column.
+
+Additionally, a filter with the following three OR clauses are applied:
+* `animal` values that match "super-mongoose"
+* `triple-number` values that match 300
+* `location` values that match 3
+
+## Query the transformed data
+
+In the web console, open a new tab in the **Query** view. Run the following query to view the ingested data:
+
+```sql
+SELECT * FROM "transform_tutorial"
+```
+
+Returns the following:
+
+| `__time` | `animal` | `location` | `number` | `triple-number` |
+| -- | -- | -- | -- | -- |
+| `2018-01-01T05:01:35.000Z` | `super-mongoose` | `2` | `200` | `600` |
+| `2018-01-01T06:01:35.000Z` | `super-snake` | `3` | `300` | `900` |
+| `2018-01-01T07:01:35.000Z` | `super-octopus` | `1` | `100` | `300` |
+
+Note that the filter applies after the transformations. The only row that isn't ingested from the input data is the "lion" row because it doesn't meet any of the three filter condition listed.
+
Review Comment:
We can look at filters as part of the transformation. As in "ingestion-time transformation allows you to filter out irrelevant data before it is ingested."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1718957324
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+Before proceeding, download Druid as described in [Quickstart (local)](index.md) and have it running on your local machine. You don't need to load any data into the Druid cluster.
+
+You should be familiar with data querying in Druid. If you haven't already, go through the [Query data](../tutorials/tutorial-query.md) tutorial first.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+
+```json
+{"timestamp":"2018-01-01T07:01:35Z", "animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z", "animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z", "animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z", "animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" = 100)
+PARTITIONED BY DAY
+```
+
+In the `SELECT` clause, you specify the following transformations:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. Note that it only ingests the transformed data.
+* `triple-number`: multiplies the `number` column by three and stores the results in a column named `triple-number`. Note that the query ingests both the original and the transformed data.
+
+Additionally, the `WHERE` clause applies the following three OR operators so that the query only ingests the rows where at least one of the following conditions is `true`:
+
+* `TEXTCAT('super-', "animal")` matches "super-mongoose"
+* `location` matches 3
+* `number` matches 100
+
+## Query the transformed data
+
+In the web console, open a new tab in the **Query** view. Run the following query to view the ingested data:
+
+```sql
+SELECT * FROM "transform_tutorial"
+```
+
+Returns the following:
+
+| `__time` | `animal` | `location` | `number` | `triple-number` |
+| -- | -- | -- | -- | -- |
+| `2018-01-01T05:01:35.000Z` | `super-mongoose` | `2` | `200` | `600` |
+| `2018-01-01T06:01:35.000Z` | `super-snake` | `3` | `300` | `900` |
+| `2018-01-01T07:01:35.000Z` | `super-octopus` | `1` | `100` | `300` |
+
+Once a row is accepted by the filter, the ingestion job applies the transformations. In this example, the filter selects the first three rows because each row meets at least one of the necessary OR conditions. Note that for the three rows selected, the ingestion job ingests the transformed `animal` column, the `location` column, and both the original `number` and the transformed `triple-number` column. The "lion" row is not accepted by the filter, so it is not ingested or transformed.
Review Comment:
It still works as a closing statement. I would out it at the end of "Transform data during ingestion".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "edgar2020 (via GitHub)" <gi...@apache.org>.
edgar2020 commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1719128053
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+Before proceeding, download Druid as described in [Quickstart (local)](index.md) and have it running on your local machine. You don't need to load any data into the Druid cluster.
Review Comment:
Also adding back the registered symbol: ®
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "edgar2020 (via GitHub)" <gi...@apache.org>.
edgar2020 commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1717385362
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,102 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+
+```json
+{"timestamp":"2018-01-01T07:01:35Z", "animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z", "animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z", "animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z", "animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+In the `SELECT` clause, you specify the following transformations:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. Note that it only ingests the transformed data.
+* `triple-number`: multiplies the `number` column by three and stores the results in a column named `triple-number`. Note that the query ingests both the original and the transformed data.
+
+Additionally, the `WHERE` clause applies the following three OR clauses so that the query only ingests the rows where at least one of the following conditions are `true`:
Review Comment:
Corrected from: clause -> operator
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "edgar2020 (via GitHub)" <gi...@apache.org>.
edgar2020 commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1715897442
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
+* `triple-number`: multiplies the `number` column by 3. This stores the results in a new column called `triple-number`. Note that it ingestions both the original and the transformed column.
+
+Additionally, a filter with the following three OR clauses are applied:
+* `animal` values that match "super-mongoose"
+* `triple-number` values that match 300
+* `location` values that match 3
+
+## Query the transformed data
+
+In the web console, open a new tab in the **Query** view. Run the following query to view the ingested data:
+
+```sql
+SELECT * FROM "transform_tutorial"
+```
+
+Returns the following:
+
+| `__time` | `animal` | `location` | `number` | `triple-number` |
+| -- | -- | -- | -- | -- |
+| `2018-01-01T05:01:35.000Z` | `super-mongoose` | `2` | `200` | `600` |
+| `2018-01-01T06:01:35.000Z` | `super-snake` | `3` | `300` | `900` |
+| `2018-01-01T07:01:35.000Z` | `super-octopus` | `1` | `100` | `300` |
+
+Note that the filter applies after the transformations. The only row that isn't ingested from the input data is the "lion" row because it doesn't meet any of the three filter condition listed.
+
+## Learn more
+
+See the following topics for more information:
+
+* [All functions](../querying/sql-functions.md) for a list of functions that can be used to transform data.
+* [Transform spec reference](../ingestion/ingestion-spec.md/#transformspec) to learn more about transforms in JSON-based batch ingestion.
+* [Query filters](../querying/filters.md) to learn about the types of filters available.
Review Comment:
I included it for the titles. A lot of titles start have what is the equivalent in SQL(ex: "Example: equivalent of `WHERE someArrayColumn = ARRAY[1, 2, 3]`"), so I though that would be useful to see what were some of the possible filters that a customer would be able to create
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1718956478
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+Before proceeding, download Druid as described in [Quickstart (local)](index.md) and have it running on your local machine. You don't need to load any data into the Druid cluster.
+
+You should be familiar with data querying in Druid. If you haven't already, go through the [Query data](../tutorials/tutorial-query.md) tutorial first.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+
+```json
+{"timestamp":"2018-01-01T07:01:35Z", "animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z", "animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z", "animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z", "animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" = 100)
+PARTITIONED BY DAY
+```
+
+In the `SELECT` clause, you specify the following transformations:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. Note that it only ingests the transformed data.
+* `triple-number`: multiplies the `number` column by three and stores the results in a column named `triple-number`. Note that the query ingests both the original and the transformed data.
+
+Additionally, the `WHERE` clause applies the following three OR operators so that the query only ingests the rows where at least one of the following conditions is `true`:
+
+* `TEXTCAT('super-', "animal")` matches "super-mongoose"
+* `location` matches 3
+* `number` matches 100
+
+## Query the transformed data
+
+In the web console, open a new tab in the **Query** view. Run the following query to view the ingested data:
+
+```sql
+SELECT * FROM "transform_tutorial"
+```
+
+Returns the following:
+
+| `__time` | `animal` | `location` | `number` | `triple-number` |
+| -- | -- | -- | -- | -- |
+| `2018-01-01T05:01:35.000Z` | `super-mongoose` | `2` | `200` | `600` |
+| `2018-01-01T06:01:35.000Z` | `super-snake` | `3` | `300` | `900` |
+| `2018-01-01T07:01:35.000Z` | `super-octopus` | `1` | `100` | `300` |
+
+Once a row is accepted by the filter, the ingestion job applies the transformations. In this example, the filter selects the first three rows because each row meets at least one of the necessary OR conditions. Note that for the three rows selected, the ingestion job ingests the transformed `animal` column, the `location` column, and both the original `number` and the transformed `triple-number` column. The "lion" row is not accepted by the filter, so it is not ingested or transformed.
Review Comment:
```suggestion
Once a row passes the filter, the ingestion job applies the transformations. In this example, the filter selects the first three rows because each row meets at least one of the required OR conditions. For the selected rows, the ingestion job ingests the transformed `animal` column, the `location` column, and both the original `number` and the transformed `triple-number` column. The "lion" row doesn't meet any of the conditions, so it is not ingested or transformed.
```
I'm not sure if "is accepted by" is best choice of words here. Also, I think this paragraph is better suited for the "Transform data during ingestion" section since it refers to the ingestion job and not the `SELECT * FROM` query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1717182004
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,102 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+
+```json
+{"timestamp":"2018-01-01T07:01:35Z", "animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z", "animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z", "animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z", "animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+In the `SELECT` clause, you specify the following transformations:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. Note that it only ingests the transformed data.
+* `triple-number`: multiplies the `number` column by three and stores the results in a column named `triple-number`. Note that the query ingests both the original and the transformed data.
+
+Additionally, the `WHERE` clause applies the following three OR clauses so that the query only ingests the rows where at least one of the following conditions are `true`:
Review Comment:
```suggestion
Additionally, the `WHERE` clause applies the following three OR operators so that the query only ingests the rows where at least one of the following conditions is `true`:
```
OR is an operator, not a clause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "edgar2020 (via GitHub)" <gi...@apache.org>.
edgar2020 commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1717499355
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,102 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+
+```json
+{"timestamp":"2018-01-01T07:01:35Z", "animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z", "animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z", "animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z", "animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+In the `SELECT` clause, you specify the following transformations:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. Note that it only ingests the transformed data.
+* `triple-number`: multiplies the `number` column by three and stores the results in a column named `triple-number`. Note that the query ingests both the original and the transformed data.
+
+Additionally, the `WHERE` clause applies the following three OR clauses so that the query only ingests the rows where at least one of the following conditions are `true`:
+
+* `TEXTCAT('super-', "animal")` value matches "super-mongoose"
+* `location` value matches 3
+* `'number' * 3` value matches 300
+
Review Comment:
This conflicts a little with this earlier [comment](https://github.com/apache/druid/pull/16845#discussion_r1712442247).
I've reworded to hopefully add context, and combine your comment with the earlier comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "edgar2020 (via GitHub)" <gi...@apache.org>.
edgar2020 commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1716063495
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
+* `triple-number`: multiplies the `number` column by 3. This stores the results in a new column called `triple-number`. Note that it ingestions both the original and the transformed column.
+
+Additionally, a filter with the following three OR clauses are applied:
+* `animal` values that match "super-mongoose"
+* `triple-number` values that match 300
+* `location` values that match 3
+
+## Query the transformed data
+
+In the web console, open a new tab in the **Query** view. Run the following query to view the ingested data:
+
+```sql
+SELECT * FROM "transform_tutorial"
+```
+
+Returns the following:
+
+| `__time` | `animal` | `location` | `number` | `triple-number` |
+| -- | -- | -- | -- | -- |
+| `2018-01-01T05:01:35.000Z` | `super-mongoose` | `2` | `200` | `600` |
+| `2018-01-01T06:01:35.000Z` | `super-snake` | `3` | `300` | `900` |
+| `2018-01-01T07:01:35.000Z` | `super-octopus` | `1` | `100` | `300` |
+
+Note that the filter applies after the transformations. The only row that isn't ingested from the input data is the "lion" row because it doesn't meet any of the three filter condition listed.
+
+## Learn more
+
+See the following topics for more information:
+
+* [All functions](../querying/sql-functions.md) for a list of functions that can be used to transform data.
+* [Transform spec reference](../ingestion/ingestion-spec.md/#transformspec) to learn more about transforms in JSON-based batch ingestion.
+* [Query filters](../querying/filters.md) to learn about the types of filters available.
+* [WHERE clause](../querying/sql.md#where) to learn how to create filters in SQL.
Review Comment:
I choose to keep this one since it shows two separate examples of `WHERE` clause, and some of the nuance for it, like it can reference a subquery. I will keep it unless it is decided to remove the filter aspect of the tutorial
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "edgar2020 (via GitHub)" <gi...@apache.org>.
edgar2020 commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1716066706
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
+* `triple-number`: multiplies the `number` column by 3. This stores the results in a new column called `triple-number`. Note that it ingestions both the original and the transformed column.
+
+Additionally, a filter with the following three OR clauses are applied:
+* `animal` values that match "super-mongoose"
+* `triple-number` values that match 300
+* `location` values that match 3
+
+## Query the transformed data
+
+In the web console, open a new tab in the **Query** view. Run the following query to view the ingested data:
+
+```sql
+SELECT * FROM "transform_tutorial"
+```
+
+Returns the following:
+
+| `__time` | `animal` | `location` | `number` | `triple-number` |
+| -- | -- | -- | -- | -- |
+| `2018-01-01T05:01:35.000Z` | `super-mongoose` | `2` | `200` | `600` |
+| `2018-01-01T06:01:35.000Z` | `super-snake` | `3` | `300` | `900` |
+| `2018-01-01T07:01:35.000Z` | `super-octopus` | `1` | `100` | `300` |
+
+Note that the filter applies after the transformations. The only row that isn't ingested from the input data is the "lion" row because it doesn't meet any of the three filter condition listed.
+
Review Comment:
Added concluding sentence back(with modification).
Awaiting the response from @techdocsmith or @ektravel to see if it is best to remove the part with filters, or if it should be kept
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "edgar2020 (via GitHub)" <gi...@apache.org>.
edgar2020 commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1716053853
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
+* `triple-number`: multiplies the `number` column by 3. This stores the results in a new column called `triple-number`. Note that it ingestions both the original and the transformed column.
+
+Additionally, a filter with the following three OR clauses are applied:
Review Comment:
Re-worded the paragraph, and included the suggestions about explaining how it only ingests when one of the filter conditions are true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] updating transformation during ingestion tutorial (druid)
Posted by "edgar2020 (via GitHub)" <gi...@apache.org>.
edgar2020 commented on code in PR #16845:
URL: https://github.com/apache/druid/pull/16845#discussion_r1716056807
##########
docs/tutorials/tutorial-transform.md:
##########
@@ -0,0 +1,101 @@
+---
+id: tutorial-transform
+title: Transform input data
+sidebar_label: Transform input data
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This tutorial demonstrates how to transform input data during ingestion.
+
+## Prerequisite
+
+This tutorial assumes you've already downloaded Apache Druid® as described in
+the [single-machine quickstart](./index.md) and have it running on your local machine.
+
+It's helpful to have finished [Load a file](../tutorials/tutorial-batch.md) and [Query data](../tutorials/tutorial-query.md) tutorials.
+
+## Sample data
+
+For this tutorial, you use the following sample data:
+```json
+{"timestamp":"2018-01-01T07:01:35Z","animal":"octopus", "location":1, "number":100}
+{"timestamp":"2018-01-01T05:01:35Z","animal":"mongoose", "location":2,"number":200}
+{"timestamp":"2018-01-01T06:01:35Z","animal":"snake", "location":3, "number":300}
+{"timestamp":"2018-01-01T01:01:35Z","animal":"lion", "location":4, "number":300}
+```
+
+## Transform data during ingestion
+
+Load the sample dataset using the [`INSERT INTO`](../multi-stage-query/reference.md/#insert) statement and the [`EXTERN`](../multi-stage-query/reference.md/#extern-function) function to ingest the data inline. In the [Druid web console](../operations/web-console.md), go to the **Query** view and run the following query:
+
+```sql
+INSERT INTO "transform_tutorial"
+WITH "ext" AS (
+ SELECT *
+ FROM TABLE(EXTERN('{"type":"inline","data":"{\"timestamp\":\"2018-01-01T07:01:35Z\",\"animal\":\"octopus\", \"location\":1, \"number\":100}\n{\"timestamp\":\"2018-01-01T05:01:35Z\",\"animal\":\"mongoose\", \"location\":2,\"number\":200}\n{\"timestamp\":\"2018-01-01T06:01:35Z\",\"animal\":\"snake\", \"location\":3, \"number\":300}\n{\"timestamp\":\"2018-01-01T01:01:35Z\",\"animal\":\"lion\", \"location\":4, \"number\":300}"}', '{"type":"json"}')) EXTEND ("timestamp" VARCHAR, "animal" VARCHAR, "location" BIGINT, "number" BIGINT)
+)
+SELECT
+ TIME_PARSE("timestamp") AS "__time",
+ TEXTCAT('super-', "animal") AS "animal",
+ "location",
+ "number",
+ "number" * 3 AS "triple-number"
+FROM "ext"
+WHERE (TEXTCAT('super-', "animal") = 'super-mongoose' OR "location" = 3 OR "number" * 3 = 300)
+PARTITIONED BY DAY
+```
+
+The following columns have transformations applied on ingestion:
+* `animal`: prepends "super-" to the values in the `animal` column using the [`TEXTCAT`](../querying/sql-functions.md/#textcat) function. This will override the `animal` column with the transformed version.
+* `triple-number`: multiplies the `number` column by 3. This stores the results in a new column called `triple-number`. Note that it ingestions both the original and the transformed column.
+
+Additionally, a filter with the following three OR clauses are applied:
+* `animal` values that match "super-mongoose"
+* `triple-number` values that match 300
+* `location` values that match 3
+
Review Comment:
swapped location and triple-number to match order
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org