You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "skuldshao (via GitHub)" <gi...@apache.org> on 2025/05/21 07:08:41 UTC
[PR] [Docs] add set command examplee to sql api (druid)
skuldshao opened a new pull request, #18027:
URL: https://github.com/apache/druid/pull/18027
Add the Set Command example to sql api doc page
This PR has:
- [X] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] Update and add SQL examples to use SET (druid)
Posted by "skuldshao (via GitHub)" <gi...@apache.org>.
skuldshao commented on code in PR #18027:
URL: https://github.com/apache/druid/pull/18027#discussion_r2168072531
##########
docs/tutorials/tutorial-query-deep-storage.md:
##########
@@ -191,6 +191,20 @@ curl --location 'http://localhost:8888/druid/v2/sql/' \
The response you get back is an empty response cause there are no records on the Historicals that match the query.
+Additionally, instead of specifying the `context` object within the JSON payload, you can also set context parameters inline using SET statements.
+
+For example, to query pages from the `wikipedia` datasource with timestamps before `2016-06-27 00:10:00`, you can use the following API curl request:
+
+```bash
+curl --location 'http://localhost:8888/druid/v2/sql/statements' \
+--header 'Content-Type: application/json' \
+--data '{
+ "query": "SET executionMode = '\''ASYNC'\''; SELECT page FROM wikipedia WHERE __time < TIMESTAMP '\''2016-06-27 00:10:00'\'' LIMIT 10"
+}'
+```
+
+Both methods,using SET statements orspecifying context parameters in the JSON `context` object, return the same response format.
+
Review Comment:
Addressed and updated the section :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] Update and add SQL examples to use SET (druid)
Posted by "skuldshao (via GitHub)" <gi...@apache.org>.
skuldshao commented on code in PR #18027:
URL: https://github.com/apache/druid/pull/18027#discussion_r2136631487
##########
docs/tutorials/tutorial-query-deep-storage.md:
##########
@@ -191,6 +191,25 @@ curl --location 'http://localhost:8888/druid/v2/sql/' \
The response you get back is an empty response cause there are no records on the Historicals that match the query.
+You can also use Set command to enable `executionMode` of the given query. However, the results depends on where you set `executionMode`.
Review Comment:
Updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] Update and add SQL examples to use SET (druid)
Posted by "vtlim (via GitHub)" <gi...@apache.org>.
vtlim commented on code in PR #18027:
URL: https://github.com/apache/druid/pull/18027#discussion_r2167379055
##########
docs/api-reference/sql-api.md:
##########
@@ -229,6 +234,20 @@ Content-Length: 192
</TabItem>
</Tabs>
+You can also specify query-level context parameters directly within the SQL query string using the `SET` command. For more details, see [SET statements](../querying/sql.md#set-statements).
+
+The following request body is functionally equivalent to the previous example and uses SET instead of the `context` parameter:
+
+```JSON
+[
Review Comment:
This should be a curly brace instead of square brace
##########
docs/querying/query-from-deep-storage.md:
##########
@@ -131,10 +131,20 @@ The response for submitting a query includes the query ID along with basic infor
"durationMs": DURATION_IN_MS,
}
```
-:::info
-You can't use `SET` command to set up the value `executionMode`.This context parameter must be specified using the `context` field in the Druid SQL API or Druid Web Console.
-:::
+Additionally, instead of specifying the `context` object within the JSON payload, you can set context parameters inline using SET statements.
+
+While submitting queries through the API, you can include SET statements as part of the SQL query string in the request payload:
+
+```bash
+curl --location 'http://localhost:8888/druid/v2/sql/statements' \
+--header 'Content-Type: application/json' \
+--data '{
+ "query": "SET executionMode = '\''ASYNC'\''; SET selectDestination = '\''durableStorage'\''; SELECT * FROM \"YOUR_DATASOURCE\" WHERE \"__time\" > TIMESTAMP '\''2017-09-01'\'' AND \"__time\" <= TIMESTAMP '\''2017-09-02'\''"
+}'
+```
+
+Both methods, using SET statements or specifying context parameters in the JSON `context` object, return the same response above.
Review Comment:
A few thoughts:
* Suggest moving it up (before "The response for..."), so it's grouped together with the request section
* Since the purpose of this page is to show the ready how to query from deep storage, let's not go into so much detail on SET. A short note should suffice, such as this:
`````
Note that you can also submit context parameters using [SET](link-to-doc). For example:
```
"query": "SET executionMode = '\''ASYNC'\''; SET selectDestination = '\''durableStorage'\''; SELECT * FROM \"YOUR_DATASOURCE\" WHERE \"__time\" > TIMESTAMP '\''2017-09-01'\'' AND \"__time\" <= TIMESTAMP '\''2017-09-02'\''"
```
`````
##########
docs/tutorials/tutorial-query-deep-storage.md:
##########
@@ -191,6 +191,20 @@ curl --location 'http://localhost:8888/druid/v2/sql/' \
The response you get back is an empty response cause there are no records on the Historicals that match the query.
+Additionally, instead of specifying the `context` object within the JSON payload, you can also set context parameters inline using SET statements.
+
+For example, to query pages from the `wikipedia` datasource with timestamps before `2016-06-27 00:10:00`, you can use the following API curl request:
+
+```bash
+curl --location 'http://localhost:8888/druid/v2/sql/statements' \
+--header 'Content-Type: application/json' \
+--data '{
+ "query": "SET executionMode = '\''ASYNC'\''; SELECT page FROM wikipedia WHERE __time < TIMESTAMP '\''2016-06-27 00:10:00'\'' LIMIT 10"
+}'
+```
+
+Both methods,using SET statements orspecifying context parameters in the JSON `context` object, return the same response format.
+
Review Comment:
Since this is a tutorial, let's just set one specific user path rather than giving them many options. My recommendation here is to update the two existing examples to use SET and remove the `context` field entirely.
The point of the tutorial isn't to learn how to use the Query API (including SET/context). Someone who chooses to learn more can click the link to the reference that's already included in the "further reading" section.
##########
docs/querying/using-caching.md:
##########
@@ -83,8 +83,12 @@ As long as the service is set to populate the cache, you can set cache options f
}
}
```
+
In this example the user has set `populateCache` to `false` to avoid filling the result cache with results for segments that are over a year old. For more information, see [Druid SQL client APIs](../api-reference/sql-api.md).
+You can also use the SET command to specify cache options directly within your SQL query string. For more information, see [SET statements](../querying/sql.md#set-statements).
+
+
## Learn more
Review Comment:
```suggestion
You can also use the SET command to specify cache options directly within your SQL query string. For more information, see [SET statements](../querying/sql.md#set-statements).
## Learn more
```
##########
docs/api-reference/sql-ingestion-api.md:
##########
@@ -101,22 +101,23 @@ The `/druid/v2/sql/task` endpoint accepts the following:
### Sample request
The following example shows a query that fetches data from an external JSON source and inserts it into a table named `wikipedia`.
+The example specifies two query context parameters:
+
+- `maxNumTasks=3`: Limits the maximum number of parallel tasks to 3.
+- `finalizeAggregations=false`: Ensures Druid saving intermediate data aggregation during loading. It's useful if you want to work with intermediate data states or control aggregation later. For more information, see [Rollup](../multi-stage-query/concepts.md#rollup).
Review Comment:
```suggestion
- `finalizeAggregations=false`: Ensures that Druid saves the aggregation's intermediate type during ingestion. For more information, see [Rollup](../multi-stage-query/concepts.md#rollup).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] Update and add SQL examples to use SET (druid)
Posted by "vtlim (via GitHub)" <gi...@apache.org>.
vtlim commented on code in PR #18027:
URL: https://github.com/apache/druid/pull/18027#discussion_r2112581296
##########
docs/api-reference/sql-api.md:
##########
@@ -195,6 +200,20 @@ Content-Length: 192
</TabItem>
</Tabs>
+You can also specify query-level context parameters directly within the SQL query string using the `SET` command. For more details on `SET` statements, see [SET statements](../querying/sql.md#set-statements).
+
+The following functionally equivalent JSON object uses `SET` command for `sqlQueryId` directly in the query:
Review Comment:
```suggestion
The following request body is functionally equivalent to the previous example and uses `SET` instead of the `context` parameter:
```
##########
docs/api-reference/sql-api.md:
##########
@@ -195,6 +200,20 @@ Content-Length: 192
</TabItem>
</Tabs>
+You can also specify query-level context parameters directly within the SQL query string using the `SET` command. For more details on `SET` statements, see [SET statements](../querying/sql.md#set-statements).
Review Comment:
```suggestion
You can also specify query-level context parameters directly within the SQL query string using the `SET` command. For more details, see [SET statements](../querying/sql.md#set-statements).
```
##########
docs/api-reference/sql-ingestion-api.md:
##########
@@ -100,7 +100,10 @@ The `/druid/v2/sql/task` endpoint accepts the following:
### Sample request
-The following example shows a query that fetches data from an external JSON source and inserts it into a table named `wikipedia`.
+The following example shows a query that fetches data from an external JSON source and inserts it into a table named `wikipedia`.It specifies two special settings directly in the query before inserting the data:
Review Comment:
```suggestion
The following example shows a query that fetches data from an external JSON source and inserts it into a table named `wikipedia`. It specifies two query context parameters:
```
##########
docs/api-reference/sql-api.md:
##########
@@ -154,7 +154,12 @@ If you detect a truncated response, treat it as an error.
#### Sample request
-The following example retrieves all rows in the `wikipedia` datasource where the `user` is `BlueMoon2662`. The query is assigned the ID `request01` using the `sqlQueryId` context parameter. The optional properties `header`, `typesHeader`, and `sqlTypesHeader` are set to `true` to include type information to the response.
+This sample request demonstrates the following actions:
+- This query retrieves all rows from the `wikipedia` datasource.
+- This query also filters the results where the `user` value is `BlueMoon2662`.
+- The `sqlTimeZone` context parameter sets the query's time zone to `America/Los_Angeles`
+- This response also set its `header`, `typesHeader`, and `sqlTypesHeader` properties as `true`.
Review Comment:
```suggestion
- The query results also include descriptors for `header`, `typesHeader`, and `sqlTypesHeader`.
```
##########
docs/multi-stage-query/examples.md:
##########
@@ -32,7 +32,7 @@ sidebar_label: Examples
These example queries show you some of the things you can do when modifying queries for your use case. Copy the example queries into the **Query** view of the web console and run them to see what they do.
:::tip
-When you insert or replace data with SQL-based ingestion, set the context parameter `finalizeAggregations` to `false`. This context parameter is automatically set for you if you use the Druid console. If you use the API, you must explicitly set it. For more information, see [Rollup](./concepts.md#rollup).
+When you insert or replace data with SQL-based ingestion, set the context parameter `finalizeAggregations` to `false`. This context parameter is automatically set for you if you use the Druid console. If you use the API, you must explicitly set it. For an example on how to set up this, see this [SQL API Sample Request](../api-reference/sql-ingestion-api#sample-request). For more information about Rollup, see [Rollup](./concepts.md#rollup).
Review Comment:
```suggestion
When you insert or replace data with SQL-based ingestion, set the context parameter `finalizeAggregations` to `false`. This context parameter is automatically set for you if you use the Druid console. If you use the API, you must explicitly set it. For an example, see [SQL-based ingestion API](../api-reference/sql-ingestion-api#sample-request). For details on aggregations, see [Rollup](./concepts.md#rollup).
```
##########
docs/api-reference/sql-api.md:
##########
@@ -154,7 +154,12 @@ If you detect a truncated response, treat it as an error.
#### Sample request
-The following example retrieves all rows in the `wikipedia` datasource where the `user` is `BlueMoon2662`. The query is assigned the ID `request01` using the `sqlQueryId` context parameter. The optional properties `header`, `typesHeader`, and `sqlTypesHeader` are set to `true` to include type information to the response.
+This sample request demonstrates the following actions:
+- This query retrieves all rows from the `wikipedia` datasource.
+- This query also filters the results where the `user` value is `BlueMoon2662`.
+- The `sqlTimeZone` context parameter sets the query's time zone to `America/Los_Angeles`
Review Comment:
```suggestion
- The `sqlTimeZone` context parameter sets the time zone to `America/Los_Angeles` for query results.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] Update and add SQL examples to use SET (druid)
Posted by "skuldshao (via GitHub)" <gi...@apache.org>.
skuldshao commented on code in PR #18027:
URL: https://github.com/apache/druid/pull/18027#discussion_r2168066657
##########
docs/api-reference/sql-api.md:
##########
@@ -229,6 +234,20 @@ Content-Length: 192
</TabItem>
</Tabs>
+You can also specify query-level context parameters directly within the SQL query string using the `SET` command. For more details, see [SET statements](../querying/sql.md#set-statements).
+
+The following request body is functionally equivalent to the previous example and uses SET instead of the `context` parameter:
+
+```JSON
+[
Review Comment:
Addressed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] Update and add SQL examples to use SET (druid)
Posted by "vtlim (via GitHub)" <gi...@apache.org>.
vtlim commented on code in PR #18027:
URL: https://github.com/apache/druid/pull/18027#discussion_r2124513003
##########
docs/querying/query-from-deep-storage.md:
##########
Review Comment:
After we confirm this limitation, let's roll back most of these changes here as well as in `tutorial-query-deep-storage.md`. We can just add a note (using the `:::` Docusaurus syntax) that "you can't use SET with the `executionMode` query context parameter ".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] Update and add SQL examples to use SET (druid)
Posted by "skuldshao (via GitHub)" <gi...@apache.org>.
skuldshao commented on code in PR #18027:
URL: https://github.com/apache/druid/pull/18027#discussion_r2136623266
##########
docs/querying/using-caching.md:
##########
@@ -83,8 +83,20 @@ As long as the service is set to populate the cache, you can set cache options f
}
}
```
+
In this example the user has set `populateCache` to `false` to avoid filling the result cache with results for segments that are over a year old. For more information, see [Druid SQL client APIs](../api-reference/sql-api.md).
+You can also use the SET command to specify cache options directly within your SQL query string. For example:
+
+```
+{
+ "query" : "SET useCache=true; SET populateCache=false; SELECT COUNT(*) FROM data_source WHERE foo = 'bar' AND __time > TIMESTAMP '2020-01-01 00:00:00'"
+}
+```
+
+For details about SET, see [SET statements](../querying/sql.md#set-statements).
+
+
Review Comment:
resolved in the commit below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] Update and add SQL examples to use SET (druid)
Posted by "vtlim (via GitHub)" <gi...@apache.org>.
vtlim commented on code in PR #18027:
URL: https://github.com/apache/druid/pull/18027#discussion_r2138488654
##########
docs/api-reference/sql-ingestion-api.md:
##########
@@ -100,23 +100,22 @@ The `/druid/v2/sql/task` endpoint accepts the following:
### Sample request
-The following example shows a query that fetches data from an external JSON source and inserts it into a table named `wikipedia`.
+The following example shows a query that fetches data from an external JSON source and inserts it into a table named `wikipedia`. It specifies two query context parameters:
Review Comment:
```suggestion
The following example shows a query that fetches data from an external JSON source and inserts it into a table named `wikipedia`.
The example specifies two query context parameters:
```
##########
docs/api-reference/sql-api.md:
##########
@@ -188,7 +188,12 @@ If you detect a truncated response, treat it as an error.
#### Sample request
-The following example retrieves all rows in the `wikipedia` datasource where the `user` is `BlueMoon2662`. The query is assigned the ID `request01` using the `sqlQueryId` context parameter. The optional properties `header`, `typesHeader`, and `sqlTypesHeader` are set to `true` to include type information to the response.
+This sample request demonstrates the following actions:
Review Comment:
```suggestion
In the following example, this query demonstrates the following actions:
```
##########
docs/querying/using-caching.md:
##########
@@ -83,8 +83,12 @@ As long as the service is set to populate the cache, you can set cache options f
}
}
```
+
In this example the user has set `populateCache` to `false` to avoid filling the result cache with results for segments that are over a year old. For more information, see [Druid SQL client APIs](../api-reference/sql-api.md).
+You can also use the SET command to specify cache options directly within your SQL query string. For details about SET, see [SET statements](../querying/sql.md#set).
Review Comment:
```suggestion
You can also use the SET command to specify cache options directly within your SQL query string. For more information, see [SET statements](../querying/sql.md#set-statements).
```
##########
docs/querying/query-from-deep-storage.md:
##########
Review Comment:
Discard changes to this file since we won't document the limitation since it'll be fixed
##########
docs/api-reference/sql-api.md:
##########
@@ -188,7 +188,12 @@ If you detect a truncated response, treat it as an error.
#### Sample request
-The following example retrieves all rows in the `wikipedia` datasource where the `user` is `BlueMoon2662`. The query is assigned the ID `request01` using the `sqlQueryId` context parameter. The optional properties `header`, `typesHeader`, and `sqlTypesHeader` are set to `true` to include type information to the response.
+This sample request demonstrates the following actions:
+- This query retrieves all rows from the `wikipedia` datasource.
+- This query also filters the results where the `user` value is `BlueMoon2662`.
+- The `sqlTimeZone` context parameter sets the time zone to `America/Los_Angeles` for query results.
+- The query results also include descriptors for `header`, `typesHeader`, and `sqlTypesHeader`.
Review Comment:
```suggestion
- Retrieves all rows from the `wikipedia` datasource.
- Filters the results where the `user` value is `BlueMoon2662`.
- Applies the `sqlTimeZone` context parameter to set the time zone of results to `America/Los_Angeles`.
- Returns descriptors for `header`, `typesHeader`, and `sqlTypesHeader`.
```
##########
docs/api-reference/sql-ingestion-api.md:
##########
@@ -100,23 +100,22 @@ The `/druid/v2/sql/task` endpoint accepts the following:
### Sample request
-The following example shows a query that fetches data from an external JSON source and inserts it into a table named `wikipedia`.
+The following example shows a query that fetches data from an external JSON source and inserts it into a table named `wikipedia`. It specifies two query context parameters:
+- `maxNumTasks=3`: This limits the maximum number of parallel tasks to 3.
+- `finalizeAggregations=false`: This prevents Druid from performing final data aggregation during loading. It's useful if you want to work with intermediate data states or control aggregation later. For more information, see [Rollup](../multi-stage-query/concepts/#rollup).
Review Comment:
```suggestion
- `maxNumTasks=3`: Limits the maximum number of parallel tasks to 3.
- `finalizeAggregations=false`: Prevents Druid from performing final data aggregation during loading. It's useful if you want to work with intermediate data states or control aggregation later. For more information, see [Rollup](../multi-stage-query/concepts.md#rollup).
```
##########
docs/multi-stage-query/reference.md:
##########
@@ -127,6 +128,10 @@ SELECT
FROM <table>
```
+When the number of rows in the result set exceeds the value of the parameter, Druid splits the output into multiple files.
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] Update and add SQL examples to use SET (druid)
Posted by "vtlim (via GitHub)" <gi...@apache.org>.
vtlim commented on code in PR #18027:
URL: https://github.com/apache/druid/pull/18027#discussion_r2112581296
##########
docs/api-reference/sql-api.md:
##########
@@ -195,6 +200,20 @@ Content-Length: 192
</TabItem>
</Tabs>
+You can also specify query-level context parameters directly within the SQL query string using the `SET` command. For more details on `SET` statements, see [SET statements](../querying/sql.md#set-statements).
+
+The following functionally equivalent JSON object uses `SET` command for `sqlQueryId` directly in the query:
Review Comment:
```suggestion
The following request body is functionally equivalent to the previous example and uses SET instead of the `context` parameter:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] Update and add SQL examples to use SET (druid)
Posted by "skuldshao (via GitHub)" <gi...@apache.org>.
skuldshao commented on code in PR #18027:
URL: https://github.com/apache/druid/pull/18027#discussion_r2136632814
##########
docs/querying/query-from-deep-storage.md:
##########
Review Comment:
Updated in the sections.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] Update and add SQL examples to use SET (druid)
Posted by "skuldshao (via GitHub)" <gi...@apache.org>.
skuldshao commented on code in PR #18027:
URL: https://github.com/apache/druid/pull/18027#discussion_r2136619777
##########
docs/multi-stage-query/reference.md:
##########
@@ -111,14 +111,15 @@ s3://export-bucket/export/query-6564a32f-2194-423a-912e-eead470a37c4-worker0-par
Keep the following in mind when using EXTERN to export rows:
- Only INSERT statements are supported.
- Only `CSV` format is supported as an export format.
-- Partitioning (`PARTITIONED BY`) and clustering (`CLUSTERED BY`) aren't supported with EXTERN statements.
+- Partitioning (PARTITIONED BY) and clustering (CLUSTERED BY) aren't supported with EXTERN statements.
- You can export to Amazon S3, Google GCS, or local storage.
- The destination provided should contain no other files or directories.
-When you export data, use the `rowsPerPage` context parameter to restrict the size of exported files.
-When the number of rows in the result set exceeds the value of the parameter, Druid splits the output into multiple files.
+When you export data, use SET to restrict `rowsPerPage` to control the size of exported files. For example:
Review Comment:
Updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] Update and add SQL examples to use SET (druid)
Posted by "vtlim (via GitHub)" <gi...@apache.org>.
vtlim commented on code in PR #18027:
URL: https://github.com/apache/druid/pull/18027#discussion_r2125151199
##########
docs/api-reference/sql-ingestion-api.md:
##########
@@ -100,23 +100,22 @@ The `/druid/v2/sql/task` endpoint accepts the following:
### Sample request
-The following example shows a query that fetches data from an external JSON source and inserts it into a table named `wikipedia`.
+The following example shows a query that fetches data from an external JSON source and inserts it into a table named `wikipedia`. It specifies two query context parameters:
+- `maxNumTasks=3`: This limits the max number of parallel tasks for this data loading job to 3.
Review Comment:
```suggestion
- `maxNumTasks=3`: This limits the maximum number of parallel tasks to 3.
```
##########
docs/api-reference/sql-ingestion-api.md:
##########
@@ -100,23 +100,22 @@ The `/druid/v2/sql/task` endpoint accepts the following:
### Sample request
-The following example shows a query that fetches data from an external JSON source and inserts it into a table named `wikipedia`.
+The following example shows a query that fetches data from an external JSON source and inserts it into a table named `wikipedia`. It specifies two query context parameters:
+- `maxNumTasks=3`: This limits the max number of parallel tasks for this data loading job to 3.
+- `finalizeAggregations=false`: This prevents Druid from performing final data aggregation during loading. It's useful if you want to work with intermediate data states or control aggregation later. For more information about Rollup, see [Rollup](../multi-stage-query/concepts/#rollup).
Review Comment:
```suggestion
- `finalizeAggregations=false`: This prevents Druid from performing final data aggregation during loading. It's useful if you want to work with intermediate data states or control aggregation later. For more information, see [Rollup](../multi-stage-query/concepts/#rollup).
```
##########
docs/querying/using-caching.md:
##########
@@ -83,8 +83,20 @@ As long as the service is set to populate the cache, you can set cache options f
}
}
```
+
In this example the user has set `populateCache` to `false` to avoid filling the result cache with results for segments that are over a year old. For more information, see [Druid SQL client APIs](../api-reference/sql-api.md).
+You can also use the SET command to specify cache options directly within your SQL query string. For example:
+
+```
+{
+ "query" : "SET useCache=true; SET populateCache=false; SELECT COUNT(*) FROM data_source WHERE foo = 'bar' AND __time > TIMESTAMP '2020-01-01 00:00:00'"
+}
+```
+
+For details about SET, see [SET statements](../querying/sql.md#set-statements).
+
+
Review Comment:
Let's actually not add a duplicate example in this doc. The other docs were more introductory, so it made sense to show multiple ways. Instead, let's just do a short one-liner letting the user know that there's another route to set the parameter. "You can also set the context parameter directly in `query` using the SET command. For details, see <link to the sql-api example>."
##########
docs/tutorials/tutorial-query-deep-storage.md:
##########
@@ -191,6 +191,25 @@ curl --location 'http://localhost:8888/druid/v2/sql/' \
The response you get back is an empty response cause there are no records on the Historicals that match the query.
+You can also use Set command to enable `executionMode` of the given query. However, the results depends on where you set `executionMode`.
Review Comment:
We'll likely roll this back as well (see comment on query-from-deep-storage)
##########
docs/multi-stage-query/reference.md:
##########
@@ -111,14 +111,15 @@ s3://export-bucket/export/query-6564a32f-2194-423a-912e-eead470a37c4-worker0-par
Keep the following in mind when using EXTERN to export rows:
- Only INSERT statements are supported.
- Only `CSV` format is supported as an export format.
-- Partitioning (`PARTITIONED BY`) and clustering (`CLUSTERED BY`) aren't supported with EXTERN statements.
+- Partitioning (PARTITIONED BY) and clustering (CLUSTERED BY) aren't supported with EXTERN statements.
- You can export to Amazon S3, Google GCS, or local storage.
- The destination provided should contain no other files or directories.
-When you export data, use the `rowsPerPage` context parameter to restrict the size of exported files.
-When the number of rows in the result set exceeds the value of the parameter, Druid splits the output into multiple files.
+When you export data, use SET to restrict `rowsPerPage` to control the size of exported files. For example:
Review Comment:
```suggestion
The following statement shows the format of a SQL query using EXTERN to export rows:
```
##########
docs/multi-stage-query/reference.md:
##########
@@ -501,10 +506,11 @@ When using the sort-merge algorithm, keep the following in mind:
- All join types are supported with `sortMerge`: LEFT, RIGHT, INNER, FULL, and CROSS.
-The following example runs using a single sort-merge join stage that receives `eventstream`
-(partitioned on `user_id`) and `users` (partitioned on `id`) as inputs. There is no limit on the size of either input.
+The following example shows a single sort-merge join stage where it explicitly set `sqlJoinAlgorithm` to `sortMerge` using the SET command. This query also takes `eventstream` (partitioned on `user_id`) and `users` (partitioned on id) as `inputs`, with no limit on the size of either input.
```sql
+SET sqlJoinAlgorithm='sortMerge';
+
Review Comment:
```suggestion
```
##########
docs/multi-stage-query/reference.md:
##########
@@ -111,14 +111,15 @@ s3://export-bucket/export/query-6564a32f-2194-423a-912e-eead470a37c4-worker0-par
Keep the following in mind when using EXTERN to export rows:
- Only INSERT statements are supported.
- Only `CSV` format is supported as an export format.
-- Partitioning (`PARTITIONED BY`) and clustering (`CLUSTERED BY`) aren't supported with EXTERN statements.
+- Partitioning (PARTITIONED BY) and clustering (CLUSTERED BY) aren't supported with EXTERN statements.
- You can export to Amazon S3, Google GCS, or local storage.
- The destination provided should contain no other files or directories.
-When you export data, use the `rowsPerPage` context parameter to restrict the size of exported files.
-When the number of rows in the result set exceeds the value of the parameter, Druid splits the output into multiple files.
+When you export data, use SET to restrict `rowsPerPage` to control the size of exported files. For example:
```sql
+SET rowsPerPage=<number_of_rows>;
+
Review Comment:
```suggestion
```
##########
docs/multi-stage-query/reference.md:
##########
@@ -111,14 +111,15 @@ s3://export-bucket/export/query-6564a32f-2194-423a-912e-eead470a37c4-worker0-par
Keep the following in mind when using EXTERN to export rows:
- Only INSERT statements are supported.
- Only `CSV` format is supported as an export format.
-- Partitioning (`PARTITIONED BY`) and clustering (`CLUSTERED BY`) aren't supported with EXTERN statements.
+- Partitioning (PARTITIONED BY) and clustering (CLUSTERED BY) aren't supported with EXTERN statements.
- You can export to Amazon S3, Google GCS, or local storage.
- The destination provided should contain no other files or directories.
-When you export data, use the `rowsPerPage` context parameter to restrict the size of exported files.
-When the number of rows in the result set exceeds the value of the parameter, Druid splits the output into multiple files.
+When you export data, use SET to restrict `rowsPerPage` to control the size of exported files. For example:
Review Comment:
I think the original text is better suited to generally introduce the parameter and why you want to use it (not jumping into how you want to set it)
##########
docs/multi-stage-query/reference.md:
##########
@@ -127,6 +128,10 @@ SELECT
FROM <table>
```
+When the number of rows in the result set exceeds the value of the parameter, Druid splits the output into multiple files.
Review Comment:
Move this line back above (see previous comment)
##########
docs/multi-stage-query/reference.md:
##########
@@ -127,6 +128,10 @@ SELECT
FROM <table>
```
+When the number of rows in the result set exceeds the value of the parameter, Druid splits the output into multiple files.
+For details about SET, see [SET statements](../querying/sql.md#set-statements).
Review Comment:
```suggestion
For details about applying context parameters using SET, see [SET statements](../querying/sql.md#set-statements).
```
##########
docs/multi-stage-query/reference.md:
##########
@@ -501,10 +506,11 @@ When using the sort-merge algorithm, keep the following in mind:
- All join types are supported with `sortMerge`: LEFT, RIGHT, INNER, FULL, and CROSS.
-The following example runs using a single sort-merge join stage that receives `eventstream`
-(partitioned on `user_id`) and `users` (partitioned on `id`) as inputs. There is no limit on the size of either input.
+The following example shows a single sort-merge join stage where it explicitly set `sqlJoinAlgorithm` to `sortMerge` using the SET command. This query also takes `eventstream` (partitioned on `user_id`) and `users` (partitioned on id) as `inputs`, with no limit on the size of either input.
Review Comment:
```suggestion
The following query runs a single sort-merge join stage that takes the following inputs:
* `eventstream` partitioned on `user_id`
* `users` partitioned on `id`
There is no limit on the size of either input.
The SET clause sets the `sqlJoinAlgorithm` context parameter so that Druid applies the sort-merge join algorithm for the query.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] Update and add SQL examples to use SET (druid)
Posted by "vtlim (via GitHub)" <gi...@apache.org>.
vtlim commented on code in PR #18027:
URL: https://github.com/apache/druid/pull/18027#discussion_r2125160299
##########
docs/multi-stage-query/reference.md:
##########
@@ -501,10 +506,11 @@ When using the sort-merge algorithm, keep the following in mind:
- All join types are supported with `sortMerge`: LEFT, RIGHT, INNER, FULL, and CROSS.
-The following example runs using a single sort-merge join stage that receives `eventstream`
-(partitioned on `user_id`) and `users` (partitioned on `id`) as inputs. There is no limit on the size of either input.
+The following example shows a single sort-merge join stage where it explicitly set `sqlJoinAlgorithm` to `sortMerge` using the SET command. This query also takes `eventstream` (partitioned on `user_id`) and `users` (partitioned on id) as `inputs`, with no limit on the size of either input.
Review Comment:
```suggestion
The following query runs a single sort-merge join stage that takes the following inputs:
* `eventstream` partitioned on `user_id`
* `users` partitioned on `id`
There is no limit on the size of either input.
The SET command assigns the `sqlJoinAlgorithm` context parameter so that Druid uses the sort-merge join algorithm for the query.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] Update and add SQL examples to use SET (druid)
Posted by "vtlim (via GitHub)" <gi...@apache.org>.
vtlim commented on code in PR #18027:
URL: https://github.com/apache/druid/pull/18027#discussion_r2125156238
##########
docs/multi-stage-query/reference.md:
##########
@@ -127,6 +128,10 @@ SELECT
FROM <table>
```
+When the number of rows in the result set exceeds the value of the parameter, Druid splits the output into multiple files.
+For details about SET, see [SET statements](../querying/sql.md#set-statements).
Review Comment:
```suggestion
For details on applying context parameters using SET, see [SET statements](../querying/sql.md#set-statements).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [Docs] Update and add SQL examples to use SET (druid)
Posted by "skuldshao (via GitHub)" <gi...@apache.org>.
skuldshao commented on code in PR #18027:
URL: https://github.com/apache/druid/pull/18027#discussion_r2148951111
##########
docs/api-reference/sql-ingestion-api.md:
##########
@@ -101,22 +101,23 @@ The `/druid/v2/sql/task` endpoint accepts the following:
### Sample request
The following example shows a query that fetches data from an external JSON source and inserts it into a table named `wikipedia`.
+The example specifies two query context parameters:
+
+- `maxNumTasks=3`: Limits the maximum number of parallel tasks to 3.
+- `finalizeAggregations=false`: Prevents Druid from performing final data aggregation during loading. It's useful if you want to work with intermediate data states or control aggregation later. For more information, see [Rollup](../multi-stage-query/concepts.md#rollup).
Review Comment:
```suggestion
- `finalizeAggregations=false`: Ensures Druid saving intermediate data aggregation during loading. It's useful if you want to work with intermediate data states or control aggregation later. For more information, see [Rollup](../multi-stage-query/concepts.md#rollup).
```
##########
docs/api-reference/sql-api.md:
##########
@@ -229,6 +234,20 @@ Content-Length: 192
</TabItem>
</Tabs>
+You can also specify query-level context parameters directly within the SQL query string using the `SET` command. For more details, see [SET statements](../querying/sql.md#set).
Review Comment:
```suggestion
You can also specify query-level context parameters directly within the SQL query string using the `SET` command. For more details, see [SET statements](../querying/sql.md#set-statements).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org