You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "petermarshallio (via GitHub)" <gi...@apache.org> on 2023/08/16 14:02:47 UTC
[PR] 202308-notebook-distribution (druid)
petermarshallio opened a new pull request, #14849:
URL: https://github.com/apache/druid/pull/14849
A new notebook on analysing data distribution, including estimating rank, quantiles, and histograms. This has a focus on the Quantiles sketch.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] been tested in a test Druid cluster.
- [ ] been tested on the Jupyter docker container.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] 202308-notebook-distribution (druid)
Posted by "petermarshallio (via GitHub)" <gi...@apache.org>.
petermarshallio closed pull request #14849: 202308-notebook-distribution
URL: https://github.com/apache/druid/pull/14849
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] 202308-notebook-distribution (druid)
Posted by "vtlim (via GitHub)" <gi...@apache.org>.
vtlim commented on PR #14849:
URL: https://github.com/apache/druid/pull/14849#issuecomment-1689039016
Reminder to also add the Apache license to the notebook
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] 202308-notebook-distribution (druid)
Posted by "vtlim (via GitHub)" <gi...@apache.org>.
vtlim commented on code in PR #14849:
URL: https://github.com/apache/druid/pull/14849#discussion_r1302246337
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
Review Comment:
```suggestion
"In this tutorial, you work through some examples of the functions available in the Datasketches extension, a core extension that enables estimation of quantiles, ranks, and histograms."
```
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
Review Comment:
```suggestion
"A structure, also known as a \"sketch,\" holds a representation of the distribution of values. Sketches are a a class of streaming algorithms that include quantiles and count distinct algorithms. To create Quantiles sketches, use the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function and provide the column for which to create the sketch.\n",
```
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
+ "\n",
+ "To see how quantiles sketches look, run the following cell. It uses `DS_QUANTILES_SKETCH` to create a Quantiles sketch for both `Distance` and `Delay`."
Review Comment:
Not sure if this cell has the desired output? It's a really wide cell result.
<img width="1413" alt="image" src="https://github.com/apache/druid/assets/7747997/c7f127da-a86d-4a12-a6c7-3bf61a4ad66f">
If this is the intent, maybe explain that sketches are represented as base64-encoded strings.
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
+ "\n",
+ "To see how quantiles sketches look, run the following cell. It uses `DS_QUANTILES_SKETCH` to create a Quantiles sketch for both `Distance` and `Delay`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bbe82c0d-aa42-4740-8f3e-a79ab7caa227",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT \n",
+ " DS_QUANTILES_SKETCH(Distance) AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e484022e-63eb-4025-b113-d8bb6351778c",
+ "metadata": {},
+ "source": [
+ "This function can be combined with `FILTER` to limit the sketch based on other dimension values.\n",
+ "\n",
+ "Running the cell below shows two Quantiles sketches that are only representative of flights leaving Atlanta."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "63ebe760-d949-48cb-be18-5e09503140e9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_QUANTILES_SKETCH(Distance) FILTER (WHERE Origin='ATL') AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) FILTER (WHERE Origin='ATL') AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0e835dae-1834-4fdd-b6f5-747a88cdb0cd",
+ "metadata": {},
+ "source": [
+ "As with other Datasketches, the `DS_QUANTILES_SKETCH` function (or the [classic equivallent](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles#aggregator), `quantilesDoublesSketch`) can be used at ingestion time to pre-calculate these objects, reducing processing required at query time.\n",
+ "\n",
+ "In the rest of this notebook, these representations will be used to find quantiles (the estimated value at a given position) and ranks (the estimated position of a specific value), and to build histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ae00642f-1266-4c5e-a02c-1789ef57ad62",
+ "metadata": {},
+ "source": [
+ "## Estimating the quantile\n",
+ "\n",
+ "A quantile allows us to conclude that, within a given set of observations, the specified percentage of values falls below a particular value.\n",
+ "\n",
+ "Druid provides functions that allow you to estimate single and multiple quantiles.\n",
+ "\n",
+ "### Estimate a single quantile value\n",
+ "\n",
+ "Run the following cell, which uses the [`DS_GET_QUANTILE`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantile) function to find out the values for each `Reporting_Airline` at the quartile positions - -25%, 50%, and 75%."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f98b55f3-c6ce-4ad0-9c24-b2253d2eefe4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.25) AS \"25percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.5) AS \"50percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.75) AS \"75percent\"\n",
Review Comment:
Include leading zero for readability (https://developers.google.com/style/numbers#numbers-as-numerals)
```suggestion
" Reporting_Airline,\n",
" DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance), 0.25) AS \"25percent\",\n",
" DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance), 0.5) AS \"50percent\",\n",
" DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance), 0.75) AS \"75percent\"\n",
```
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
+ "\n",
+ "To see how quantiles sketches look, run the following cell. It uses `DS_QUANTILES_SKETCH` to create a Quantiles sketch for both `Distance` and `Delay`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bbe82c0d-aa42-4740-8f3e-a79ab7caa227",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT \n",
+ " DS_QUANTILES_SKETCH(Distance) AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e484022e-63eb-4025-b113-d8bb6351778c",
+ "metadata": {},
+ "source": [
+ "This function can be combined with `FILTER` to limit the sketch based on other dimension values.\n",
+ "\n",
+ "Running the cell below shows two Quantiles sketches that are only representative of flights leaving Atlanta."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "63ebe760-d949-48cb-be18-5e09503140e9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_QUANTILES_SKETCH(Distance) FILTER (WHERE Origin='ATL') AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) FILTER (WHERE Origin='ATL') AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0e835dae-1834-4fdd-b6f5-747a88cdb0cd",
+ "metadata": {},
+ "source": [
+ "As with other Datasketches, the `DS_QUANTILES_SKETCH` function (or the [classic equivallent](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles#aggregator), `quantilesDoublesSketch`) can be used at ingestion time to pre-calculate these objects, reducing processing required at query time.\n",
+ "\n",
+ "In the rest of this notebook, these representations will be used to find quantiles (the estimated value at a given position) and ranks (the estimated position of a specific value), and to build histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ae00642f-1266-4c5e-a02c-1789ef57ad62",
+ "metadata": {},
+ "source": [
+ "## Estimating the quantile\n",
+ "\n",
+ "A quantile allows us to conclude that, within a given set of observations, the specified percentage of values falls below a particular value.\n",
+ "\n",
+ "Druid provides functions that allow you to estimate single and multiple quantiles.\n",
+ "\n",
+ "### Estimate a single quantile value\n",
+ "\n",
+ "Run the following cell, which uses the [`DS_GET_QUANTILE`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantile) function to find out the values for each `Reporting_Airline` at the quartile positions - -25%, 50%, and 75%."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f98b55f3-c6ce-4ad0-9c24-b2253d2eefe4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.25) AS \"25percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.5) AS \"50percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "88585044-d2b5-4ddf-b3bb-4203fd021e60",
+ "metadata": {},
+ "source": [
+ "The results show that, for each `Reporting_Airline`, 25% of flights flew a `Distance` below the first value, 50% below the second, and 75 below the third.\n",
+ "\n",
+ "Druid provides a shorthand version of the SQL statement used in the previous cell - [`APPROX_QUANTILE_DS`](https://druid.apache.org/docs/latest/querying/sql-functions#approx_quantile_ds).\n",
+ "\n",
+ "Run the cell below, which gives the same result as the query above but uses this shorthand function. There's no need for the `DS_QUANTILES_SKETCH` function to create a sketch against `Distance`, and instead just provide `Distance`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6a7480f8-0491-4514-b005-bb2e884ee448",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " APPROX_QUANTILE_DS(Distance,.25) AS \"25percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.5) AS \"50percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "669865f3-d612-4c2a-951b-d0f2e52fade3",
+ "metadata": {},
+ "source": [
+ "As with `DS_QUANTILES_SKETCH`, this can be combined with a `FILTER` clause.\n",
+ "\n",
+ "Run the following cell which looks only at flights that were delayed by 3 hours. By using the `APPROX_QUANTILE_DS` function the results will show the distance that 98% of flights in this category flew."
Review Comment:
```suggestion
"Run the following cell which looks only at flights from three airlines that were delayed by 3 hours. The results show the distance that 98% of flights in this category flew."
```
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
+ "\n",
+ "To see how quantiles sketches look, run the following cell. It uses `DS_QUANTILES_SKETCH` to create a Quantiles sketch for both `Distance` and `Delay`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bbe82c0d-aa42-4740-8f3e-a79ab7caa227",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT \n",
+ " DS_QUANTILES_SKETCH(Distance) AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e484022e-63eb-4025-b113-d8bb6351778c",
+ "metadata": {},
+ "source": [
+ "This function can be combined with `FILTER` to limit the sketch based on other dimension values.\n",
+ "\n",
+ "Running the cell below shows two Quantiles sketches that are only representative of flights leaving Atlanta."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "63ebe760-d949-48cb-be18-5e09503140e9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_QUANTILES_SKETCH(Distance) FILTER (WHERE Origin='ATL') AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) FILTER (WHERE Origin='ATL') AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0e835dae-1834-4fdd-b6f5-747a88cdb0cd",
+ "metadata": {},
+ "source": [
+ "As with other Datasketches, the `DS_QUANTILES_SKETCH` function (or the [classic equivallent](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles#aggregator), `quantilesDoublesSketch`) can be used at ingestion time to pre-calculate these objects, reducing processing required at query time.\n",
+ "\n",
+ "In the rest of this notebook, these representations will be used to find quantiles (the estimated value at a given position) and ranks (the estimated position of a specific value), and to build histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ae00642f-1266-4c5e-a02c-1789ef57ad62",
+ "metadata": {},
+ "source": [
+ "## Estimating the quantile\n",
+ "\n",
+ "A quantile allows us to conclude that, within a given set of observations, the specified percentage of values falls below a particular value.\n",
+ "\n",
+ "Druid provides functions that allow you to estimate single and multiple quantiles.\n",
+ "\n",
+ "### Estimate a single quantile value\n",
+ "\n",
+ "Run the following cell, which uses the [`DS_GET_QUANTILE`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantile) function to find out the values for each `Reporting_Airline` at the quartile positions - -25%, 50%, and 75%."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f98b55f3-c6ce-4ad0-9c24-b2253d2eefe4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.25) AS \"25percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.5) AS \"50percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "88585044-d2b5-4ddf-b3bb-4203fd021e60",
+ "metadata": {},
+ "source": [
+ "The results show that, for each `Reporting_Airline`, 25% of flights flew a `Distance` below the first value, 50% below the second, and 75 below the third.\n",
+ "\n",
+ "Druid provides a shorthand version of the SQL statement used in the previous cell - [`APPROX_QUANTILE_DS`](https://druid.apache.org/docs/latest/querying/sql-functions#approx_quantile_ds).\n",
+ "\n",
+ "Run the cell below, which gives the same result as the query above but uses this shorthand function. There's no need for the `DS_QUANTILES_SKETCH` function to create a sketch against `Distance`, and instead just provide `Distance`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6a7480f8-0491-4514-b005-bb2e884ee448",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " APPROX_QUANTILE_DS(Distance,.25) AS \"25percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.5) AS \"50percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "669865f3-d612-4c2a-951b-d0f2e52fade3",
+ "metadata": {},
+ "source": [
+ "As with `DS_QUANTILES_SKETCH`, this can be combined with a `FILTER` clause.\n",
+ "\n",
+ "Run the following cell which looks only at flights that were delayed by 3 hours. By using the `APPROX_QUANTILE_DS` function the results will show the distance that 98% of flights in this category flew."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "42b3ee0a-c901-4be5-b19b-17df69b901ad",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'CO') AS \"Distance_CO\",\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'US') AS \"Distance_US\",\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'AA') AS \"Distance_AA\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE Delay > 180\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cda1c68b-f515-4eba-870c-137234c45633",
+ "metadata": {},
+ "source": [
+ "Run the next cell to produce a plot of the number of delayed flights on a particular day, by airline, that were delayed by more than 75% of flights were in the two-week period before.\n",
+ "\n",
+ "Table `b` lists, for each `Reporting_Airline`, the third-quartile `Delay` within a two-week period.\n",
+ "\n",
+ "This data is then used through a `JOIN` against `example-flights-quantiles` to `COUNT` the number of flights, per `Reporting_Airline`, that are above the third-quartile.\n",
+ "\n",
+ "The sorted results are then put into a pandas Dataframe and plotted."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c3421e46-9157-48a1-bbda-b0904d9d2adc",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "WITH b AS (\n",
+ "SELECT\n",
+ " \"Reporting_Airline\",\n",
+ " APPROX_QUANTILE_DS(Delay,.75) AS \"75delay\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE (TIMESTAMP '2005-11-01' <= \"__time\" AND \"__time\" < TIMESTAMP '2005-11-14')\n",
+ "GROUP BY 1)\n",
+ "\n",
+ "SELECT\n",
+ " a.\"Reporting_Airline\",\n",
+ " COUNT(*) AS \"Flights\"\n",
+ "FROM \"example-flights-quantiles\" a\n",
+ "LEFT JOIN b ON a.Reporting_Airline = b.Reporting_Airline\n",
+ "WHERE TIME_FLOOR(a.\"__time\", 'PT1H') = TIMESTAMP '2005-11-15 11:00:00'\n",
+ "AND a.\"Delay\" > b.\"75delay\"\n",
+ "GROUP BY 1\n",
+ "ORDER BY 2 DESC\n",
+ "'''\n",
+ "\n",
+ "df1 = pd.DataFrame(sql_client.sql(sql))\n",
+ "\n",
+ "df1.plot.bar(x='Reporting_Airline', y='Flights')\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3dedcdbd-aa56-4b75-80a3-327cea3f7f52",
+ "metadata": {},
+ "source": [
+ "A query like the one above could be adapted to work with streaming ingestion of event data, giving a real-time view of the numbers of flights currently delayed that exceed the 75th percentile delay from the previous two weeks."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "53feba1b-64f0-42d2-a8f6-59b57b38b187",
+ "metadata": {},
+ "source": [
+ "### Estimate multiple quantile values\n",
+ "\n",
+ "In the examples above, you used separate `APPROX_QUANTILE_DS` functions to find the values at different positions in the sketch.\n",
+ "\n",
+ "```sql\n",
+ " APPROX_QUANTILE_DS(Distance,.25) AS \"25percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.5) AS \"50percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.75) AS \"75percent\"\n",
+ "```\n",
+ "\n",
+ "Alternatively, the values can be returned in an [`ARRAY`](https://druid.apache.org/docs/latest/querying/sql-array-functions) by using the [`DS_GET_QUANTILES`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantiles) function.\n",
+ "\n",
+ "Run the following cell, which returns quantiles in an `ARRAY` object instead of separate columns - one for quartiles, one for deciles - broken down by `Reporting_Airline` on a particular day."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "88ffd086-0e86-4441-bd60-fdd4a25be6e8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_GET_QUANTILES(DS_QUANTILES_SKETCH(Distance),0, .25, .5, .75, 1) AS \"quartiles\",\n",
+ " DS_GET_QUANTILES(DS_QUANTILES_SKETCH(Distance),0, .1, .2, .3, .4, .5, .6, .7, .8, .9, 1) AS \"deciles\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE TIME_FLOOR(\"__time\", 'P1D') = TIMESTAMP '2005-11-24'\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "aa138aa8-d0b4-47fe-8e6c-873543e52f0b",
+ "metadata": {},
+ "source": [
+ "## Estimating the rank (cumulative distribution functions)\n",
+ "\n",
+ "Druid provides two functions for estimating what the [normalized rank](https://datasketches.apache.org/docs/Quantiles/SketchingQuantilesAndRanksTutorial.html) of a value is within the data. Or, to put it another way, what percentage of other records fall below a given value.\n",
+ "\n",
+ "### Estimate a single rank position\n",
+ "\n",
+ "Use the [`DS_RANK`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_rank) function to estimate where a given value falls in the distribution of the data, returned as a percentage.\n",
+ "\n",
+ "Run the following cell which returns the rank of a particular value in the \"XE\" flights on the 24th November 2005.\n",
Review Comment:
```suggestion
"Run the following cell to obtain the rank of a particular value in the \"XE\" flights on the 24th November 2005.\n",
```
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
Review Comment:
Added details for more information and to transition better into working with Quantiles sketches
https://datasketches.apache.org/docs/Architecture/KeyFeatures.html
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
+ "\n",
+ "To see how quantiles sketches look, run the following cell. It uses `DS_QUANTILES_SKETCH` to create a Quantiles sketch for both `Distance` and `Delay`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bbe82c0d-aa42-4740-8f3e-a79ab7caa227",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT \n",
+ " DS_QUANTILES_SKETCH(Distance) AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e484022e-63eb-4025-b113-d8bb6351778c",
+ "metadata": {},
+ "source": [
+ "This function can be combined with `FILTER` to limit the sketch based on other dimension values.\n",
+ "\n",
+ "Running the cell below shows two Quantiles sketches that are only representative of flights leaving Atlanta."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "63ebe760-d949-48cb-be18-5e09503140e9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_QUANTILES_SKETCH(Distance) FILTER (WHERE Origin='ATL') AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) FILTER (WHERE Origin='ATL') AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0e835dae-1834-4fdd-b6f5-747a88cdb0cd",
+ "metadata": {},
+ "source": [
+ "As with other Datasketches, the `DS_QUANTILES_SKETCH` function (or the [classic equivallent](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles#aggregator), `quantilesDoublesSketch`) can be used at ingestion time to pre-calculate these objects, reducing processing required at query time.\n",
+ "\n",
+ "In the rest of this notebook, these representations will be used to find quantiles (the estimated value at a given position) and ranks (the estimated position of a specific value), and to build histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ae00642f-1266-4c5e-a02c-1789ef57ad62",
+ "metadata": {},
+ "source": [
+ "## Estimating the quantile\n",
+ "\n",
+ "A quantile allows us to conclude that, within a given set of observations, the specified percentage of values falls below a particular value.\n",
+ "\n",
+ "Druid provides functions that allow you to estimate single and multiple quantiles.\n",
+ "\n",
+ "### Estimate a single quantile value\n",
+ "\n",
+ "Run the following cell, which uses the [`DS_GET_QUANTILE`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantile) function to find out the values for each `Reporting_Airline` at the quartile positions - -25%, 50%, and 75%."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f98b55f3-c6ce-4ad0-9c24-b2253d2eefe4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.25) AS \"25percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.5) AS \"50percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "88585044-d2b5-4ddf-b3bb-4203fd021e60",
+ "metadata": {},
+ "source": [
+ "The results show that, for each `Reporting_Airline`, 25% of flights flew a `Distance` below the first value, 50% below the second, and 75 below the third.\n",
+ "\n",
+ "Druid provides a shorthand version of the SQL statement used in the previous cell - [`APPROX_QUANTILE_DS`](https://druid.apache.org/docs/latest/querying/sql-functions#approx_quantile_ds).\n",
Review Comment:
```suggestion
"Druid provides a shorthand version of the SQL statement used in the previous cell with the [`APPROX_QUANTILE_DS`](https://druid.apache.org/docs/latest/querying/sql-functions#approx_quantile_ds) function.\n",
```
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
Review Comment:
```suggestion
"The Apache Druid database provides functions to analyze large distributions thanks to the [Apache Datasketches](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
```
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
Review Comment:
Consider adding a table of contents for easier navigation and to preview the sections in the tutorial
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
+ "\n",
+ "To see how quantiles sketches look, run the following cell. It uses `DS_QUANTILES_SKETCH` to create a Quantiles sketch for both `Distance` and `Delay`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bbe82c0d-aa42-4740-8f3e-a79ab7caa227",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT \n",
+ " DS_QUANTILES_SKETCH(Distance) AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e484022e-63eb-4025-b113-d8bb6351778c",
+ "metadata": {},
+ "source": [
+ "This function can be combined with `FILTER` to limit the sketch based on other dimension values.\n",
+ "\n",
+ "Running the cell below shows two Quantiles sketches that are only representative of flights leaving Atlanta."
Review Comment:
Similar comment as above. A user who simply runs the cell can't tell that anything has been filtered out due to the base64 representation. Consider adding a bit more to the query to make the output human readable.
<img width="1422" alt="image" src="https://github.com/apache/druid/assets/7747997/2fe20f43-be94-45d5-aa48-23ab3f75807c">
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
+ "\n",
+ "To see how quantiles sketches look, run the following cell. It uses `DS_QUANTILES_SKETCH` to create a Quantiles sketch for both `Distance` and `Delay`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bbe82c0d-aa42-4740-8f3e-a79ab7caa227",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT \n",
+ " DS_QUANTILES_SKETCH(Distance) AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e484022e-63eb-4025-b113-d8bb6351778c",
+ "metadata": {},
+ "source": [
+ "This function can be combined with `FILTER` to limit the sketch based on other dimension values.\n",
+ "\n",
+ "Running the cell below shows two Quantiles sketches that are only representative of flights leaving Atlanta."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "63ebe760-d949-48cb-be18-5e09503140e9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_QUANTILES_SKETCH(Distance) FILTER (WHERE Origin='ATL') AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) FILTER (WHERE Origin='ATL') AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0e835dae-1834-4fdd-b6f5-747a88cdb0cd",
+ "metadata": {},
+ "source": [
+ "As with other Datasketches, the `DS_QUANTILES_SKETCH` function (or the [classic equivallent](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles#aggregator), `quantilesDoublesSketch`) can be used at ingestion time to pre-calculate these objects, reducing processing required at query time.\n",
+ "\n",
+ "In the rest of this notebook, these representations will be used to find quantiles (the estimated value at a given position) and ranks (the estimated position of a specific value), and to build histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ae00642f-1266-4c5e-a02c-1789ef57ad62",
+ "metadata": {},
+ "source": [
+ "## Estimating the quantile\n",
+ "\n",
+ "A quantile allows us to conclude that, within a given set of observations, the specified percentage of values falls below a particular value.\n",
+ "\n",
+ "Druid provides functions that allow you to estimate single and multiple quantiles.\n",
+ "\n",
+ "### Estimate a single quantile value\n",
+ "\n",
+ "Run the following cell, which uses the [`DS_GET_QUANTILE`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantile) function to find out the values for each `Reporting_Airline` at the quartile positions - -25%, 50%, and 75%."
Review Comment:
```suggestion
"Run the following cell, which uses the [`DS_GET_QUANTILE`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantile) function to find out the values for each `Reporting_Airline` at the quartile positions of 25%, 50%, and 75%."
```
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
+ "\n",
+ "To see how quantiles sketches look, run the following cell. It uses `DS_QUANTILES_SKETCH` to create a Quantiles sketch for both `Distance` and `Delay`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bbe82c0d-aa42-4740-8f3e-a79ab7caa227",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT \n",
+ " DS_QUANTILES_SKETCH(Distance) AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e484022e-63eb-4025-b113-d8bb6351778c",
+ "metadata": {},
+ "source": [
+ "This function can be combined with `FILTER` to limit the sketch based on other dimension values.\n",
+ "\n",
+ "Running the cell below shows two Quantiles sketches that are only representative of flights leaving Atlanta."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "63ebe760-d949-48cb-be18-5e09503140e9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_QUANTILES_SKETCH(Distance) FILTER (WHERE Origin='ATL') AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) FILTER (WHERE Origin='ATL') AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0e835dae-1834-4fdd-b6f5-747a88cdb0cd",
+ "metadata": {},
+ "source": [
+ "As with other Datasketches, the `DS_QUANTILES_SKETCH` function (or the [classic equivallent](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles#aggregator), `quantilesDoublesSketch`) can be used at ingestion time to pre-calculate these objects, reducing processing required at query time.\n",
Review Comment:
```suggestion
"As with other Datasketches, the `DS_QUANTILES_SKETCH` function (or the [native query equivalent](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles#aggregator), `quantilesDoublesSketch`) can be used at ingestion time to pre-calculate these objects, reducing processing required at query time.\n",
```
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
+ "\n",
+ "To see how quantiles sketches look, run the following cell. It uses `DS_QUANTILES_SKETCH` to create a Quantiles sketch for both `Distance` and `Delay`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bbe82c0d-aa42-4740-8f3e-a79ab7caa227",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT \n",
+ " DS_QUANTILES_SKETCH(Distance) AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e484022e-63eb-4025-b113-d8bb6351778c",
+ "metadata": {},
+ "source": [
+ "This function can be combined with `FILTER` to limit the sketch based on other dimension values.\n",
+ "\n",
+ "Running the cell below shows two Quantiles sketches that are only representative of flights leaving Atlanta."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "63ebe760-d949-48cb-be18-5e09503140e9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_QUANTILES_SKETCH(Distance) FILTER (WHERE Origin='ATL') AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) FILTER (WHERE Origin='ATL') AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0e835dae-1834-4fdd-b6f5-747a88cdb0cd",
+ "metadata": {},
+ "source": [
+ "As with other Datasketches, the `DS_QUANTILES_SKETCH` function (or the [classic equivallent](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles#aggregator), `quantilesDoublesSketch`) can be used at ingestion time to pre-calculate these objects, reducing processing required at query time.\n",
+ "\n",
+ "In the rest of this notebook, these representations will be used to find quantiles (the estimated value at a given position) and ranks (the estimated position of a specific value), and to build histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ae00642f-1266-4c5e-a02c-1789ef57ad62",
+ "metadata": {},
+ "source": [
+ "## Estimating the quantile\n",
+ "\n",
+ "A quantile allows us to conclude that, within a given set of observations, the specified percentage of values falls below a particular value.\n",
+ "\n",
+ "Druid provides functions that allow you to estimate single and multiple quantiles.\n",
+ "\n",
+ "### Estimate a single quantile value\n",
+ "\n",
+ "Run the following cell, which uses the [`DS_GET_QUANTILE`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantile) function to find out the values for each `Reporting_Airline` at the quartile positions - -25%, 50%, and 75%."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f98b55f3-c6ce-4ad0-9c24-b2253d2eefe4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.25) AS \"25percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.5) AS \"50percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "88585044-d2b5-4ddf-b3bb-4203fd021e60",
+ "metadata": {},
+ "source": [
+ "The results show that, for each `Reporting_Airline`, 25% of flights flew a `Distance` below the first value, 50% below the second, and 75 below the third.\n",
+ "\n",
+ "Druid provides a shorthand version of the SQL statement used in the previous cell - [`APPROX_QUANTILE_DS`](https://druid.apache.org/docs/latest/querying/sql-functions#approx_quantile_ds).\n",
+ "\n",
+ "Run the cell below, which gives the same result as the query above but uses this shorthand function. There's no need for the `DS_QUANTILES_SKETCH` function to create a sketch against `Distance`, and instead just provide `Distance`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6a7480f8-0491-4514-b005-bb2e884ee448",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " APPROX_QUANTILE_DS(Distance,.25) AS \"25percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.5) AS \"50percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.75) AS \"75percent\"\n",
Review Comment:
```suggestion
" APPROX_QUANTILE_DS(Distance, 0.25) AS \"25percent\",\n",
" APPROX_QUANTILE_DS(Distance, 0.5) AS \"50percent\",\n",
" APPROX_QUANTILE_DS(Distance, 0.75) AS \"75percent\"\n",
```
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
+ "\n",
+ "To see how quantiles sketches look, run the following cell. It uses `DS_QUANTILES_SKETCH` to create a Quantiles sketch for both `Distance` and `Delay`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bbe82c0d-aa42-4740-8f3e-a79ab7caa227",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT \n",
+ " DS_QUANTILES_SKETCH(Distance) AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e484022e-63eb-4025-b113-d8bb6351778c",
+ "metadata": {},
+ "source": [
+ "This function can be combined with `FILTER` to limit the sketch based on other dimension values.\n",
+ "\n",
+ "Running the cell below shows two Quantiles sketches that are only representative of flights leaving Atlanta."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "63ebe760-d949-48cb-be18-5e09503140e9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_QUANTILES_SKETCH(Distance) FILTER (WHERE Origin='ATL') AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) FILTER (WHERE Origin='ATL') AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0e835dae-1834-4fdd-b6f5-747a88cdb0cd",
+ "metadata": {},
+ "source": [
+ "As with other Datasketches, the `DS_QUANTILES_SKETCH` function (or the [classic equivallent](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles#aggregator), `quantilesDoublesSketch`) can be used at ingestion time to pre-calculate these objects, reducing processing required at query time.\n",
+ "\n",
+ "In the rest of this notebook, these representations will be used to find quantiles (the estimated value at a given position) and ranks (the estimated position of a specific value), and to build histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ae00642f-1266-4c5e-a02c-1789ef57ad62",
+ "metadata": {},
+ "source": [
+ "## Estimating the quantile\n",
+ "\n",
+ "A quantile allows us to conclude that, within a given set of observations, the specified percentage of values falls below a particular value.\n",
+ "\n",
+ "Druid provides functions that allow you to estimate single and multiple quantiles.\n",
+ "\n",
+ "### Estimate a single quantile value\n",
+ "\n",
+ "Run the following cell, which uses the [`DS_GET_QUANTILE`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantile) function to find out the values for each `Reporting_Airline` at the quartile positions - -25%, 50%, and 75%."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f98b55f3-c6ce-4ad0-9c24-b2253d2eefe4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.25) AS \"25percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.5) AS \"50percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "88585044-d2b5-4ddf-b3bb-4203fd021e60",
+ "metadata": {},
+ "source": [
+ "The results show that, for each `Reporting_Airline`, 25% of flights flew a `Distance` below the first value, 50% below the second, and 75 below the third.\n",
+ "\n",
+ "Druid provides a shorthand version of the SQL statement used in the previous cell - [`APPROX_QUANTILE_DS`](https://druid.apache.org/docs/latest/querying/sql-functions#approx_quantile_ds).\n",
+ "\n",
+ "Run the cell below, which gives the same result as the query above but uses this shorthand function. There's no need for the `DS_QUANTILES_SKETCH` function to create a sketch against `Distance`, and instead just provide `Distance`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6a7480f8-0491-4514-b005-bb2e884ee448",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " APPROX_QUANTILE_DS(Distance,.25) AS \"25percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.5) AS \"50percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "669865f3-d612-4c2a-951b-d0f2e52fade3",
+ "metadata": {},
+ "source": [
+ "As with `DS_QUANTILES_SKETCH`, this can be combined with a `FILTER` clause.\n",
Review Comment:
Avoid ambiguous "this"
```suggestion
"As with `DS_QUANTILES_SKETCH`, `APPROX_QUANTILE_DS` can be combined with a `FILTER` clause.\n",
```
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
+ "\n",
+ "To see how quantiles sketches look, run the following cell. It uses `DS_QUANTILES_SKETCH` to create a Quantiles sketch for both `Distance` and `Delay`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bbe82c0d-aa42-4740-8f3e-a79ab7caa227",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT \n",
+ " DS_QUANTILES_SKETCH(Distance) AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e484022e-63eb-4025-b113-d8bb6351778c",
+ "metadata": {},
+ "source": [
+ "This function can be combined with `FILTER` to limit the sketch based on other dimension values.\n",
+ "\n",
+ "Running the cell below shows two Quantiles sketches that are only representative of flights leaving Atlanta."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "63ebe760-d949-48cb-be18-5e09503140e9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_QUANTILES_SKETCH(Distance) FILTER (WHERE Origin='ATL') AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) FILTER (WHERE Origin='ATL') AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0e835dae-1834-4fdd-b6f5-747a88cdb0cd",
+ "metadata": {},
+ "source": [
+ "As with other Datasketches, the `DS_QUANTILES_SKETCH` function (or the [classic equivallent](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles#aggregator), `quantilesDoublesSketch`) can be used at ingestion time to pre-calculate these objects, reducing processing required at query time.\n",
+ "\n",
+ "In the rest of this notebook, these representations will be used to find quantiles (the estimated value at a given position) and ranks (the estimated position of a specific value), and to build histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ae00642f-1266-4c5e-a02c-1789ef57ad62",
+ "metadata": {},
+ "source": [
+ "## Estimating the quantile\n",
+ "\n",
+ "A quantile allows us to conclude that, within a given set of observations, the specified percentage of values falls below a particular value.\n",
+ "\n",
+ "Druid provides functions that allow you to estimate single and multiple quantiles.\n",
+ "\n",
+ "### Estimate a single quantile value\n",
+ "\n",
+ "Run the following cell, which uses the [`DS_GET_QUANTILE`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantile) function to find out the values for each `Reporting_Airline` at the quartile positions - -25%, 50%, and 75%."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f98b55f3-c6ce-4ad0-9c24-b2253d2eefe4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.25) AS \"25percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.5) AS \"50percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "88585044-d2b5-4ddf-b3bb-4203fd021e60",
+ "metadata": {},
+ "source": [
+ "The results show that, for each `Reporting_Airline`, 25% of flights flew a `Distance` below the first value, 50% below the second, and 75 below the third.\n",
+ "\n",
+ "Druid provides a shorthand version of the SQL statement used in the previous cell - [`APPROX_QUANTILE_DS`](https://druid.apache.org/docs/latest/querying/sql-functions#approx_quantile_ds).\n",
+ "\n",
+ "Run the cell below, which gives the same result as the query above but uses this shorthand function. There's no need for the `DS_QUANTILES_SKETCH` function to create a sketch against `Distance`, and instead just provide `Distance`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6a7480f8-0491-4514-b005-bb2e884ee448",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " APPROX_QUANTILE_DS(Distance,.25) AS \"25percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.5) AS \"50percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "669865f3-d612-4c2a-951b-d0f2e52fade3",
+ "metadata": {},
+ "source": [
+ "As with `DS_QUANTILES_SKETCH`, this can be combined with a `FILTER` clause.\n",
+ "\n",
+ "Run the following cell which looks only at flights that were delayed by 3 hours. By using the `APPROX_QUANTILE_DS` function the results will show the distance that 98% of flights in this category flew."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "42b3ee0a-c901-4be5-b19b-17df69b901ad",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'CO') AS \"Distance_CO\",\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'US') AS \"Distance_US\",\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'AA') AS \"Distance_AA\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE Delay > 180\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cda1c68b-f515-4eba-870c-137234c45633",
+ "metadata": {},
+ "source": [
+ "Run the next cell to produce a plot of the number of delayed flights on a particular day, by airline, that were delayed by more than 75% of flights were in the two-week period before.\n",
Review Comment:
```suggestion
"Run the next cell to produce a plot of the number of delayed flights on a particular day, by airline, that were delayed by more than 75% of flights compared to the two week period before.\n",
```
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
+ "\n",
+ "To see how quantiles sketches look, run the following cell. It uses `DS_QUANTILES_SKETCH` to create a Quantiles sketch for both `Distance` and `Delay`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bbe82c0d-aa42-4740-8f3e-a79ab7caa227",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT \n",
+ " DS_QUANTILES_SKETCH(Distance) AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e484022e-63eb-4025-b113-d8bb6351778c",
+ "metadata": {},
+ "source": [
+ "This function can be combined with `FILTER` to limit the sketch based on other dimension values.\n",
+ "\n",
+ "Running the cell below shows two Quantiles sketches that are only representative of flights leaving Atlanta."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "63ebe760-d949-48cb-be18-5e09503140e9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_QUANTILES_SKETCH(Distance) FILTER (WHERE Origin='ATL') AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) FILTER (WHERE Origin='ATL') AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0e835dae-1834-4fdd-b6f5-747a88cdb0cd",
+ "metadata": {},
+ "source": [
+ "As with other Datasketches, the `DS_QUANTILES_SKETCH` function (or the [classic equivallent](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles#aggregator), `quantilesDoublesSketch`) can be used at ingestion time to pre-calculate these objects, reducing processing required at query time.\n",
+ "\n",
+ "In the rest of this notebook, these representations will be used to find quantiles (the estimated value at a given position) and ranks (the estimated position of a specific value), and to build histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ae00642f-1266-4c5e-a02c-1789ef57ad62",
+ "metadata": {},
+ "source": [
+ "## Estimating the quantile\n",
+ "\n",
+ "A quantile allows us to conclude that, within a given set of observations, the specified percentage of values falls below a particular value.\n",
+ "\n",
+ "Druid provides functions that allow you to estimate single and multiple quantiles.\n",
+ "\n",
+ "### Estimate a single quantile value\n",
+ "\n",
+ "Run the following cell, which uses the [`DS_GET_QUANTILE`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantile) function to find out the values for each `Reporting_Airline` at the quartile positions - -25%, 50%, and 75%."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f98b55f3-c6ce-4ad0-9c24-b2253d2eefe4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.25) AS \"25percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.5) AS \"50percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "88585044-d2b5-4ddf-b3bb-4203fd021e60",
+ "metadata": {},
+ "source": [
+ "The results show that, for each `Reporting_Airline`, 25% of flights flew a `Distance` below the first value, 50% below the second, and 75 below the third.\n",
+ "\n",
+ "Druid provides a shorthand version of the SQL statement used in the previous cell - [`APPROX_QUANTILE_DS`](https://druid.apache.org/docs/latest/querying/sql-functions#approx_quantile_ds).\n",
+ "\n",
+ "Run the cell below, which gives the same result as the query above but uses this shorthand function. There's no need for the `DS_QUANTILES_SKETCH` function to create a sketch against `Distance`, and instead just provide `Distance`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6a7480f8-0491-4514-b005-bb2e884ee448",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " APPROX_QUANTILE_DS(Distance,.25) AS \"25percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.5) AS \"50percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "669865f3-d612-4c2a-951b-d0f2e52fade3",
+ "metadata": {},
+ "source": [
+ "As with `DS_QUANTILES_SKETCH`, this can be combined with a `FILTER` clause.\n",
+ "\n",
+ "Run the following cell which looks only at flights that were delayed by 3 hours. By using the `APPROX_QUANTILE_DS` function the results will show the distance that 98% of flights in this category flew."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "42b3ee0a-c901-4be5-b19b-17df69b901ad",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'CO') AS \"Distance_CO\",\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'US') AS \"Distance_US\",\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'AA') AS \"Distance_AA\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE Delay > 180\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cda1c68b-f515-4eba-870c-137234c45633",
+ "metadata": {},
+ "source": [
+ "Run the next cell to produce a plot of the number of delayed flights on a particular day, by airline, that were delayed by more than 75% of flights were in the two-week period before.\n",
+ "\n",
+ "Table `b` lists, for each `Reporting_Airline`, the third-quartile `Delay` within a two-week period.\n",
+ "\n",
+ "This data is then used through a `JOIN` against `example-flights-quantiles` to `COUNT` the number of flights, per `Reporting_Airline`, that are above the third-quartile.\n",
+ "\n",
+ "The sorted results are then put into a pandas Dataframe and plotted."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c3421e46-9157-48a1-bbda-b0904d9d2adc",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "WITH b AS (\n",
+ "SELECT\n",
+ " \"Reporting_Airline\",\n",
+ " APPROX_QUANTILE_DS(Delay,.75) AS \"75delay\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE (TIMESTAMP '2005-11-01' <= \"__time\" AND \"__time\" < TIMESTAMP '2005-11-14')\n",
+ "GROUP BY 1)\n",
+ "\n",
+ "SELECT\n",
+ " a.\"Reporting_Airline\",\n",
+ " COUNT(*) AS \"Flights\"\n",
+ "FROM \"example-flights-quantiles\" a\n",
+ "LEFT JOIN b ON a.Reporting_Airline = b.Reporting_Airline\n",
+ "WHERE TIME_FLOOR(a.\"__time\", 'PT1H') = TIMESTAMP '2005-11-15 11:00:00'\n",
+ "AND a.\"Delay\" > b.\"75delay\"\n",
+ "GROUP BY 1\n",
+ "ORDER BY 2 DESC\n",
+ "'''\n",
+ "\n",
+ "df1 = pd.DataFrame(sql_client.sql(sql))\n",
+ "\n",
+ "df1.plot.bar(x='Reporting_Airline', y='Flights')\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3dedcdbd-aa56-4b75-80a3-327cea3f7f52",
+ "metadata": {},
+ "source": [
+ "A query like the one above could be adapted to work with streaming ingestion of event data, giving a real-time view of the numbers of flights currently delayed that exceed the 75th percentile delay from the previous two weeks."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "53feba1b-64f0-42d2-a8f6-59b57b38b187",
+ "metadata": {},
+ "source": [
+ "### Estimate multiple quantile values\n",
+ "\n",
+ "In the examples above, you used separate `APPROX_QUANTILE_DS` functions to find the values at different positions in the sketch.\n",
+ "\n",
+ "```sql\n",
+ " APPROX_QUANTILE_DS(Distance,.25) AS \"25percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.5) AS \"50percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.75) AS \"75percent\"\n",
+ "```\n",
+ "\n",
+ "Alternatively, the values can be returned in an [`ARRAY`](https://druid.apache.org/docs/latest/querying/sql-array-functions) by using the [`DS_GET_QUANTILES`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantiles) function.\n",
+ "\n",
+ "Run the following cell, which returns quantiles in an `ARRAY` object instead of separate columns - one for quartiles, one for deciles - broken down by `Reporting_Airline` on a particular day."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "88ffd086-0e86-4441-bd60-fdd4a25be6e8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_GET_QUANTILES(DS_QUANTILES_SKETCH(Distance),0, .25, .5, .75, 1) AS \"quartiles\",\n",
+ " DS_GET_QUANTILES(DS_QUANTILES_SKETCH(Distance),0, .1, .2, .3, .4, .5, .6, .7, .8, .9, 1) AS \"deciles\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE TIME_FLOOR(\"__time\", 'P1D') = TIMESTAMP '2005-11-24'\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "aa138aa8-d0b4-47fe-8e6c-873543e52f0b",
+ "metadata": {},
+ "source": [
+ "## Estimating the rank (cumulative distribution functions)\n",
+ "\n",
+ "Druid provides two functions for estimating what the [normalized rank](https://datasketches.apache.org/docs/Quantiles/SketchingQuantilesAndRanksTutorial.html) of a value is within the data. Or, to put it another way, what percentage of other records fall below a given value.\n",
+ "\n",
+ "### Estimate a single rank position\n",
+ "\n",
+ "Use the [`DS_RANK`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_rank) function to estimate where a given value falls in the distribution of the data, returned as a percentage.\n",
+ "\n",
+ "Run the following cell which returns the rank of a particular value in the \"XE\" flights on the 24th November 2005.\n",
+ "\n",
+ "Notice that the `DS_QUANTILES_SKETCH` function is used to provide `DS_RANK` with the sketch representation."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "4114078b-eb95-45c5-8fe9-fcb170b4366a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_RANK(DS_QUANTILES_SKETCH(Distance),950) AS \"rank\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE TIME_FLOOR(\"__time\", 'P1D') = TIMESTAMP '2005-11-24'\n",
+ "AND \"Reporting_Airline\" = 'XE'\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f06cb646-3ab2-4f7d-865c-dabc669390ce",
+ "metadata": {},
+ "source": [
+ "This tells us that this distance is representative of about 90% of flights on that day. The results for \"XE\" in the previous cell, where we showed the deciles for the same flights, corroborates that.\n",
Review Comment:
What distance? 950? Are there units?
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
+ "\n",
+ "To see how quantiles sketches look, run the following cell. It uses `DS_QUANTILES_SKETCH` to create a Quantiles sketch for both `Distance` and `Delay`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bbe82c0d-aa42-4740-8f3e-a79ab7caa227",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT \n",
+ " DS_QUANTILES_SKETCH(Distance) AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e484022e-63eb-4025-b113-d8bb6351778c",
+ "metadata": {},
+ "source": [
+ "This function can be combined with `FILTER` to limit the sketch based on other dimension values.\n",
+ "\n",
+ "Running the cell below shows two Quantiles sketches that are only representative of flights leaving Atlanta."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "63ebe760-d949-48cb-be18-5e09503140e9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_QUANTILES_SKETCH(Distance) FILTER (WHERE Origin='ATL') AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) FILTER (WHERE Origin='ATL') AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0e835dae-1834-4fdd-b6f5-747a88cdb0cd",
+ "metadata": {},
+ "source": [
+ "As with other Datasketches, the `DS_QUANTILES_SKETCH` function (or the [classic equivallent](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles#aggregator), `quantilesDoublesSketch`) can be used at ingestion time to pre-calculate these objects, reducing processing required at query time.\n",
+ "\n",
+ "In the rest of this notebook, these representations will be used to find quantiles (the estimated value at a given position) and ranks (the estimated position of a specific value), and to build histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ae00642f-1266-4c5e-a02c-1789ef57ad62",
+ "metadata": {},
+ "source": [
+ "## Estimating the quantile\n",
+ "\n",
+ "A quantile allows us to conclude that, within a given set of observations, the specified percentage of values falls below a particular value.\n",
+ "\n",
+ "Druid provides functions that allow you to estimate single and multiple quantiles.\n",
+ "\n",
+ "### Estimate a single quantile value\n",
+ "\n",
+ "Run the following cell, which uses the [`DS_GET_QUANTILE`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantile) function to find out the values for each `Reporting_Airline` at the quartile positions - -25%, 50%, and 75%."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f98b55f3-c6ce-4ad0-9c24-b2253d2eefe4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.25) AS \"25percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.5) AS \"50percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "88585044-d2b5-4ddf-b3bb-4203fd021e60",
+ "metadata": {},
+ "source": [
+ "The results show that, for each `Reporting_Airline`, 25% of flights flew a `Distance` below the first value, 50% below the second, and 75 below the third.\n",
+ "\n",
+ "Druid provides a shorthand version of the SQL statement used in the previous cell - [`APPROX_QUANTILE_DS`](https://druid.apache.org/docs/latest/querying/sql-functions#approx_quantile_ds).\n",
+ "\n",
+ "Run the cell below, which gives the same result as the query above but uses this shorthand function. There's no need for the `DS_QUANTILES_SKETCH` function to create a sketch against `Distance`, and instead just provide `Distance`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6a7480f8-0491-4514-b005-bb2e884ee448",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " APPROX_QUANTILE_DS(Distance,.25) AS \"25percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.5) AS \"50percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "669865f3-d612-4c2a-951b-d0f2e52fade3",
+ "metadata": {},
+ "source": [
+ "As with `DS_QUANTILES_SKETCH`, this can be combined with a `FILTER` clause.\n",
+ "\n",
+ "Run the following cell which looks only at flights that were delayed by 3 hours. By using the `APPROX_QUANTILE_DS` function the results will show the distance that 98% of flights in this category flew."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "42b3ee0a-c901-4be5-b19b-17df69b901ad",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'CO') AS \"Distance_CO\",\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'US') AS \"Distance_US\",\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'AA') AS \"Distance_AA\"\n",
Review Comment:
```suggestion
" APPROX_QUANTILE_DS(Distance, 0.98) FILTER (WHERE Reporting_Airline = 'CO') AS \"Distance_CO\",\n",
" APPROX_QUANTILE_DS(Distance, 0.98) FILTER (WHERE Reporting_Airline = 'US') AS \"Distance_US\",\n",
" APPROX_QUANTILE_DS(Distance, 0.98) FILTER (WHERE Reporting_Airline = 'AA') AS \"Distance_AA\"\n",
```
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
+ "\n",
+ "To see how quantiles sketches look, run the following cell. It uses `DS_QUANTILES_SKETCH` to create a Quantiles sketch for both `Distance` and `Delay`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bbe82c0d-aa42-4740-8f3e-a79ab7caa227",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT \n",
+ " DS_QUANTILES_SKETCH(Distance) AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e484022e-63eb-4025-b113-d8bb6351778c",
+ "metadata": {},
+ "source": [
+ "This function can be combined with `FILTER` to limit the sketch based on other dimension values.\n",
+ "\n",
+ "Running the cell below shows two Quantiles sketches that are only representative of flights leaving Atlanta."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "63ebe760-d949-48cb-be18-5e09503140e9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_QUANTILES_SKETCH(Distance) FILTER (WHERE Origin='ATL') AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) FILTER (WHERE Origin='ATL') AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0e835dae-1834-4fdd-b6f5-747a88cdb0cd",
+ "metadata": {},
+ "source": [
+ "As with other Datasketches, the `DS_QUANTILES_SKETCH` function (or the [classic equivallent](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles#aggregator), `quantilesDoublesSketch`) can be used at ingestion time to pre-calculate these objects, reducing processing required at query time.\n",
+ "\n",
+ "In the rest of this notebook, these representations will be used to find quantiles (the estimated value at a given position) and ranks (the estimated position of a specific value), and to build histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ae00642f-1266-4c5e-a02c-1789ef57ad62",
+ "metadata": {},
+ "source": [
+ "## Estimating the quantile\n",
+ "\n",
+ "A quantile allows us to conclude that, within a given set of observations, the specified percentage of values falls below a particular value.\n",
+ "\n",
+ "Druid provides functions that allow you to estimate single and multiple quantiles.\n",
+ "\n",
+ "### Estimate a single quantile value\n",
+ "\n",
+ "Run the following cell, which uses the [`DS_GET_QUANTILE`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantile) function to find out the values for each `Reporting_Airline` at the quartile positions - -25%, 50%, and 75%."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f98b55f3-c6ce-4ad0-9c24-b2253d2eefe4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.25) AS \"25percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.5) AS \"50percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "88585044-d2b5-4ddf-b3bb-4203fd021e60",
+ "metadata": {},
+ "source": [
+ "The results show that, for each `Reporting_Airline`, 25% of flights flew a `Distance` below the first value, 50% below the second, and 75 below the third.\n",
+ "\n",
+ "Druid provides a shorthand version of the SQL statement used in the previous cell - [`APPROX_QUANTILE_DS`](https://druid.apache.org/docs/latest/querying/sql-functions#approx_quantile_ds).\n",
+ "\n",
+ "Run the cell below, which gives the same result as the query above but uses this shorthand function. There's no need for the `DS_QUANTILES_SKETCH` function to create a sketch against `Distance`, and instead just provide `Distance`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6a7480f8-0491-4514-b005-bb2e884ee448",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " APPROX_QUANTILE_DS(Distance,.25) AS \"25percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.5) AS \"50percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "669865f3-d612-4c2a-951b-d0f2e52fade3",
+ "metadata": {},
+ "source": [
+ "As with `DS_QUANTILES_SKETCH`, this can be combined with a `FILTER` clause.\n",
+ "\n",
+ "Run the following cell which looks only at flights that were delayed by 3 hours. By using the `APPROX_QUANTILE_DS` function the results will show the distance that 98% of flights in this category flew."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "42b3ee0a-c901-4be5-b19b-17df69b901ad",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'CO') AS \"Distance_CO\",\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'US') AS \"Distance_US\",\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'AA') AS \"Distance_AA\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE Delay > 180\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cda1c68b-f515-4eba-870c-137234c45633",
+ "metadata": {},
+ "source": [
+ "Run the next cell to produce a plot of the number of delayed flights on a particular day, by airline, that were delayed by more than 75% of flights were in the two-week period before.\n",
+ "\n",
+ "Table `b` lists, for each `Reporting_Airline`, the third-quartile `Delay` within a two-week period.\n",
+ "\n",
+ "This data is then used through a `JOIN` against `example-flights-quantiles` to `COUNT` the number of flights, per `Reporting_Airline`, that are above the third-quartile.\n",
+ "\n",
+ "The sorted results are then put into a pandas Dataframe and plotted."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c3421e46-9157-48a1-bbda-b0904d9d2adc",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "WITH b AS (\n",
+ "SELECT\n",
+ " \"Reporting_Airline\",\n",
+ " APPROX_QUANTILE_DS(Delay,.75) AS \"75delay\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE (TIMESTAMP '2005-11-01' <= \"__time\" AND \"__time\" < TIMESTAMP '2005-11-14')\n",
+ "GROUP BY 1)\n",
+ "\n",
+ "SELECT\n",
+ " a.\"Reporting_Airline\",\n",
+ " COUNT(*) AS \"Flights\"\n",
+ "FROM \"example-flights-quantiles\" a\n",
+ "LEFT JOIN b ON a.Reporting_Airline = b.Reporting_Airline\n",
+ "WHERE TIME_FLOOR(a.\"__time\", 'PT1H') = TIMESTAMP '2005-11-15 11:00:00'\n",
+ "AND a.\"Delay\" > b.\"75delay\"\n",
+ "GROUP BY 1\n",
+ "ORDER BY 2 DESC\n",
+ "'''\n",
+ "\n",
+ "df1 = pd.DataFrame(sql_client.sql(sql))\n",
+ "\n",
+ "df1.plot.bar(x='Reporting_Airline', y='Flights')\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3dedcdbd-aa56-4b75-80a3-327cea3f7f52",
+ "metadata": {},
+ "source": [
+ "A query like the one above could be adapted to work with streaming ingestion of event data, giving a real-time view of the numbers of flights currently delayed that exceed the 75th percentile delay from the previous two weeks."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "53feba1b-64f0-42d2-a8f6-59b57b38b187",
+ "metadata": {},
+ "source": [
+ "### Estimate multiple quantile values\n",
+ "\n",
+ "In the examples above, you used separate `APPROX_QUANTILE_DS` functions to find the values at different positions in the sketch.\n",
+ "\n",
+ "```sql\n",
+ " APPROX_QUANTILE_DS(Distance,.25) AS \"25percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.5) AS \"50percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.75) AS \"75percent\"\n",
Review Comment:
```suggestion
" APPROX_QUANTILE_DS(Distance, 0.25) AS \"25percent\",\n",
" APPROX_QUANTILE_DS(Distance, 0.5) AS \"50percent\",\n",
" APPROX_QUANTILE_DS(Distance, 0.75) AS \"75percent\"\n",
```
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
+ "\n",
+ "To see how quantiles sketches look, run the following cell. It uses `DS_QUANTILES_SKETCH` to create a Quantiles sketch for both `Distance` and `Delay`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bbe82c0d-aa42-4740-8f3e-a79ab7caa227",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT \n",
+ " DS_QUANTILES_SKETCH(Distance) AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e484022e-63eb-4025-b113-d8bb6351778c",
+ "metadata": {},
+ "source": [
+ "This function can be combined with `FILTER` to limit the sketch based on other dimension values.\n",
+ "\n",
+ "Running the cell below shows two Quantiles sketches that are only representative of flights leaving Atlanta."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "63ebe760-d949-48cb-be18-5e09503140e9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_QUANTILES_SKETCH(Distance) FILTER (WHERE Origin='ATL') AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) FILTER (WHERE Origin='ATL') AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0e835dae-1834-4fdd-b6f5-747a88cdb0cd",
+ "metadata": {},
+ "source": [
+ "As with other Datasketches, the `DS_QUANTILES_SKETCH` function (or the [classic equivallent](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles#aggregator), `quantilesDoublesSketch`) can be used at ingestion time to pre-calculate these objects, reducing processing required at query time.\n",
+ "\n",
+ "In the rest of this notebook, these representations will be used to find quantiles (the estimated value at a given position) and ranks (the estimated position of a specific value), and to build histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ae00642f-1266-4c5e-a02c-1789ef57ad62",
+ "metadata": {},
+ "source": [
+ "## Estimating the quantile\n",
+ "\n",
+ "A quantile allows us to conclude that, within a given set of observations, the specified percentage of values falls below a particular value.\n",
+ "\n",
+ "Druid provides functions that allow you to estimate single and multiple quantiles.\n",
+ "\n",
+ "### Estimate a single quantile value\n",
+ "\n",
+ "Run the following cell, which uses the [`DS_GET_QUANTILE`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantile) function to find out the values for each `Reporting_Airline` at the quartile positions - -25%, 50%, and 75%."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f98b55f3-c6ce-4ad0-9c24-b2253d2eefe4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.25) AS \"25percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.5) AS \"50percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "88585044-d2b5-4ddf-b3bb-4203fd021e60",
+ "metadata": {},
+ "source": [
+ "The results show that, for each `Reporting_Airline`, 25% of flights flew a `Distance` below the first value, 50% below the second, and 75 below the third.\n",
+ "\n",
+ "Druid provides a shorthand version of the SQL statement used in the previous cell - [`APPROX_QUANTILE_DS`](https://druid.apache.org/docs/latest/querying/sql-functions#approx_quantile_ds).\n",
+ "\n",
+ "Run the cell below, which gives the same result as the query above but uses this shorthand function. There's no need for the `DS_QUANTILES_SKETCH` function to create a sketch against `Distance`, and instead just provide `Distance`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6a7480f8-0491-4514-b005-bb2e884ee448",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " APPROX_QUANTILE_DS(Distance,.25) AS \"25percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.5) AS \"50percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "669865f3-d612-4c2a-951b-d0f2e52fade3",
+ "metadata": {},
+ "source": [
+ "As with `DS_QUANTILES_SKETCH`, this can be combined with a `FILTER` clause.\n",
+ "\n",
+ "Run the following cell which looks only at flights that were delayed by 3 hours. By using the `APPROX_QUANTILE_DS` function the results will show the distance that 98% of flights in this category flew."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "42b3ee0a-c901-4be5-b19b-17df69b901ad",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'CO') AS \"Distance_CO\",\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'US') AS \"Distance_US\",\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'AA') AS \"Distance_AA\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE Delay > 180\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cda1c68b-f515-4eba-870c-137234c45633",
+ "metadata": {},
+ "source": [
+ "Run the next cell to produce a plot of the number of delayed flights on a particular day, by airline, that were delayed by more than 75% of flights were in the two-week period before.\n",
+ "\n",
+ "Table `b` lists, for each `Reporting_Airline`, the third-quartile `Delay` within a two-week period.\n",
+ "\n",
+ "This data is then used through a `JOIN` against `example-flights-quantiles` to `COUNT` the number of flights, per `Reporting_Airline`, that are above the third-quartile.\n",
Review Comment:
```suggestion
"Table `b` lists, for each `Reporting_Airline`, the third quartile `Delay` within a two week period.\n",
"\n",
"This data is then used through a `JOIN` against `example-flights-quantiles` to `COUNT` the number of flights, per `Reporting_Airline`, that are above the third quartile.\n",
```
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
Review Comment:
```suggestion
"# Analyzing data distributions\n",
```
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
+ "\n",
+ "To see how quantiles sketches look, run the following cell. It uses `DS_QUANTILES_SKETCH` to create a Quantiles sketch for both `Distance` and `Delay`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bbe82c0d-aa42-4740-8f3e-a79ab7caa227",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT \n",
+ " DS_QUANTILES_SKETCH(Distance) AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e484022e-63eb-4025-b113-d8bb6351778c",
+ "metadata": {},
+ "source": [
+ "This function can be combined with `FILTER` to limit the sketch based on other dimension values.\n",
+ "\n",
+ "Running the cell below shows two Quantiles sketches that are only representative of flights leaving Atlanta."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "63ebe760-d949-48cb-be18-5e09503140e9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_QUANTILES_SKETCH(Distance) FILTER (WHERE Origin='ATL') AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) FILTER (WHERE Origin='ATL') AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0e835dae-1834-4fdd-b6f5-747a88cdb0cd",
+ "metadata": {},
+ "source": [
+ "As with other Datasketches, the `DS_QUANTILES_SKETCH` function (or the [classic equivallent](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles#aggregator), `quantilesDoublesSketch`) can be used at ingestion time to pre-calculate these objects, reducing processing required at query time.\n",
+ "\n",
+ "In the rest of this notebook, these representations will be used to find quantiles (the estimated value at a given position) and ranks (the estimated position of a specific value), and to build histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ae00642f-1266-4c5e-a02c-1789ef57ad62",
+ "metadata": {},
+ "source": [
+ "## Estimating the quantile\n",
+ "\n",
+ "A quantile allows us to conclude that, within a given set of observations, the specified percentage of values falls below a particular value.\n",
+ "\n",
+ "Druid provides functions that allow you to estimate single and multiple quantiles.\n",
+ "\n",
+ "### Estimate a single quantile value\n",
+ "\n",
+ "Run the following cell, which uses the [`DS_GET_QUANTILE`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantile) function to find out the values for each `Reporting_Airline` at the quartile positions - -25%, 50%, and 75%."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f98b55f3-c6ce-4ad0-9c24-b2253d2eefe4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.25) AS \"25percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.5) AS \"50percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "88585044-d2b5-4ddf-b3bb-4203fd021e60",
+ "metadata": {},
+ "source": [
+ "The results show that, for each `Reporting_Airline`, 25% of flights flew a `Distance` below the first value, 50% below the second, and 75 below the third.\n",
+ "\n",
+ "Druid provides a shorthand version of the SQL statement used in the previous cell - [`APPROX_QUANTILE_DS`](https://druid.apache.org/docs/latest/querying/sql-functions#approx_quantile_ds).\n",
+ "\n",
+ "Run the cell below, which gives the same result as the query above but uses this shorthand function. There's no need for the `DS_QUANTILES_SKETCH` function to create a sketch against `Distance`, and instead just provide `Distance`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6a7480f8-0491-4514-b005-bb2e884ee448",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " APPROX_QUANTILE_DS(Distance,.25) AS \"25percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.5) AS \"50percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "669865f3-d612-4c2a-951b-d0f2e52fade3",
+ "metadata": {},
+ "source": [
+ "As with `DS_QUANTILES_SKETCH`, this can be combined with a `FILTER` clause.\n",
+ "\n",
+ "Run the following cell which looks only at flights that were delayed by 3 hours. By using the `APPROX_QUANTILE_DS` function the results will show the distance that 98% of flights in this category flew."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "42b3ee0a-c901-4be5-b19b-17df69b901ad",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'CO') AS \"Distance_CO\",\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'US') AS \"Distance_US\",\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'AA') AS \"Distance_AA\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE Delay > 180\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cda1c68b-f515-4eba-870c-137234c45633",
+ "metadata": {},
+ "source": [
+ "Run the next cell to produce a plot of the number of delayed flights on a particular day, by airline, that were delayed by more than 75% of flights were in the two-week period before.\n",
+ "\n",
+ "Table `b` lists, for each `Reporting_Airline`, the third-quartile `Delay` within a two-week period.\n",
+ "\n",
+ "This data is then used through a `JOIN` against `example-flights-quantiles` to `COUNT` the number of flights, per `Reporting_Airline`, that are above the third-quartile.\n",
+ "\n",
+ "The sorted results are then put into a pandas Dataframe and plotted."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c3421e46-9157-48a1-bbda-b0904d9d2adc",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "WITH b AS (\n",
+ "SELECT\n",
+ " \"Reporting_Airline\",\n",
+ " APPROX_QUANTILE_DS(Delay,.75) AS \"75delay\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE (TIMESTAMP '2005-11-01' <= \"__time\" AND \"__time\" < TIMESTAMP '2005-11-14')\n",
+ "GROUP BY 1)\n",
+ "\n",
+ "SELECT\n",
+ " a.\"Reporting_Airline\",\n",
+ " COUNT(*) AS \"Flights\"\n",
+ "FROM \"example-flights-quantiles\" a\n",
+ "LEFT JOIN b ON a.Reporting_Airline = b.Reporting_Airline\n",
+ "WHERE TIME_FLOOR(a.\"__time\", 'PT1H') = TIMESTAMP '2005-11-15 11:00:00'\n",
+ "AND a.\"Delay\" > b.\"75delay\"\n",
+ "GROUP BY 1\n",
+ "ORDER BY 2 DESC\n",
+ "'''\n",
+ "\n",
+ "df1 = pd.DataFrame(sql_client.sql(sql))\n",
+ "\n",
+ "df1.plot.bar(x='Reporting_Airline', y='Flights')\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3dedcdbd-aa56-4b75-80a3-327cea3f7f52",
+ "metadata": {},
+ "source": [
+ "A query like the one above could be adapted to work with streaming ingestion of event data, giving a real-time view of the numbers of flights currently delayed that exceed the 75th percentile delay from the previous two weeks."
Review Comment:
```suggestion
"You can adapt a query like the one above to work with streaming ingestion of event data, giving a real-time view of the numbers of flights currently delayed that exceed the 75th percentile delay from the previous two weeks."
```
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
+ "\n",
+ "To see how quantiles sketches look, run the following cell. It uses `DS_QUANTILES_SKETCH` to create a Quantiles sketch for both `Distance` and `Delay`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bbe82c0d-aa42-4740-8f3e-a79ab7caa227",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT \n",
+ " DS_QUANTILES_SKETCH(Distance) AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e484022e-63eb-4025-b113-d8bb6351778c",
+ "metadata": {},
+ "source": [
+ "This function can be combined with `FILTER` to limit the sketch based on other dimension values.\n",
+ "\n",
+ "Running the cell below shows two Quantiles sketches that are only representative of flights leaving Atlanta."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "63ebe760-d949-48cb-be18-5e09503140e9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_QUANTILES_SKETCH(Distance) FILTER (WHERE Origin='ATL') AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) FILTER (WHERE Origin='ATL') AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0e835dae-1834-4fdd-b6f5-747a88cdb0cd",
+ "metadata": {},
+ "source": [
+ "As with other Datasketches, the `DS_QUANTILES_SKETCH` function (or the [classic equivallent](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles#aggregator), `quantilesDoublesSketch`) can be used at ingestion time to pre-calculate these objects, reducing processing required at query time.\n",
+ "\n",
+ "In the rest of this notebook, these representations will be used to find quantiles (the estimated value at a given position) and ranks (the estimated position of a specific value), and to build histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ae00642f-1266-4c5e-a02c-1789ef57ad62",
+ "metadata": {},
+ "source": [
+ "## Estimating the quantile\n",
+ "\n",
+ "A quantile allows us to conclude that, within a given set of observations, the specified percentage of values falls below a particular value.\n",
+ "\n",
+ "Druid provides functions that allow you to estimate single and multiple quantiles.\n",
+ "\n",
+ "### Estimate a single quantile value\n",
+ "\n",
+ "Run the following cell, which uses the [`DS_GET_QUANTILE`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantile) function to find out the values for each `Reporting_Airline` at the quartile positions - -25%, 50%, and 75%."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f98b55f3-c6ce-4ad0-9c24-b2253d2eefe4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.25) AS \"25percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.5) AS \"50percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "88585044-d2b5-4ddf-b3bb-4203fd021e60",
+ "metadata": {},
+ "source": [
+ "The results show that, for each `Reporting_Airline`, 25% of flights flew a `Distance` below the first value, 50% below the second, and 75 below the third.\n",
+ "\n",
+ "Druid provides a shorthand version of the SQL statement used in the previous cell - [`APPROX_QUANTILE_DS`](https://druid.apache.org/docs/latest/querying/sql-functions#approx_quantile_ds).\n",
+ "\n",
+ "Run the cell below, which gives the same result as the query above but uses this shorthand function. There's no need for the `DS_QUANTILES_SKETCH` function to create a sketch against `Distance`, and instead just provide `Distance`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6a7480f8-0491-4514-b005-bb2e884ee448",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " APPROX_QUANTILE_DS(Distance,.25) AS \"25percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.5) AS \"50percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "669865f3-d612-4c2a-951b-d0f2e52fade3",
+ "metadata": {},
+ "source": [
+ "As with `DS_QUANTILES_SKETCH`, this can be combined with a `FILTER` clause.\n",
+ "\n",
+ "Run the following cell which looks only at flights that were delayed by 3 hours. By using the `APPROX_QUANTILE_DS` function the results will show the distance that 98% of flights in this category flew."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "42b3ee0a-c901-4be5-b19b-17df69b901ad",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'CO') AS \"Distance_CO\",\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'US') AS \"Distance_US\",\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'AA') AS \"Distance_AA\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE Delay > 180\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cda1c68b-f515-4eba-870c-137234c45633",
+ "metadata": {},
+ "source": [
+ "Run the next cell to produce a plot of the number of delayed flights on a particular day, by airline, that were delayed by more than 75% of flights were in the two-week period before.\n",
+ "\n",
+ "Table `b` lists, for each `Reporting_Airline`, the third-quartile `Delay` within a two-week period.\n",
+ "\n",
+ "This data is then used through a `JOIN` against `example-flights-quantiles` to `COUNT` the number of flights, per `Reporting_Airline`, that are above the third-quartile.\n",
+ "\n",
+ "The sorted results are then put into a pandas Dataframe and plotted."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c3421e46-9157-48a1-bbda-b0904d9d2adc",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "WITH b AS (\n",
+ "SELECT\n",
+ " \"Reporting_Airline\",\n",
+ " APPROX_QUANTILE_DS(Delay,.75) AS \"75delay\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE (TIMESTAMP '2005-11-01' <= \"__time\" AND \"__time\" < TIMESTAMP '2005-11-14')\n",
+ "GROUP BY 1)\n",
+ "\n",
+ "SELECT\n",
+ " a.\"Reporting_Airline\",\n",
+ " COUNT(*) AS \"Flights\"\n",
+ "FROM \"example-flights-quantiles\" a\n",
+ "LEFT JOIN b ON a.Reporting_Airline = b.Reporting_Airline\n",
+ "WHERE TIME_FLOOR(a.\"__time\", 'PT1H') = TIMESTAMP '2005-11-15 11:00:00'\n",
+ "AND a.\"Delay\" > b.\"75delay\"\n",
+ "GROUP BY 1\n",
+ "ORDER BY 2 DESC\n",
+ "'''\n",
+ "\n",
+ "df1 = pd.DataFrame(sql_client.sql(sql))\n",
+ "\n",
+ "df1.plot.bar(x='Reporting_Airline', y='Flights')\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3dedcdbd-aa56-4b75-80a3-327cea3f7f52",
+ "metadata": {},
+ "source": [
+ "A query like the one above could be adapted to work with streaming ingestion of event data, giving a real-time view of the numbers of flights currently delayed that exceed the 75th percentile delay from the previous two weeks."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "53feba1b-64f0-42d2-a8f6-59b57b38b187",
+ "metadata": {},
+ "source": [
+ "### Estimate multiple quantile values\n",
+ "\n",
+ "In the examples above, you used separate `APPROX_QUANTILE_DS` functions to find the values at different positions in the sketch.\n",
+ "\n",
+ "```sql\n",
+ " APPROX_QUANTILE_DS(Distance,.25) AS \"25percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.5) AS \"50percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.75) AS \"75percent\"\n",
+ "```\n",
+ "\n",
+ "Alternatively, the values can be returned in an [`ARRAY`](https://druid.apache.org/docs/latest/querying/sql-array-functions) by using the [`DS_GET_QUANTILES`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantiles) function.\n",
+ "\n",
+ "Run the following cell, which returns quantiles in an `ARRAY` object instead of separate columns - one for quartiles, one for deciles - broken down by `Reporting_Airline` on a particular day."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "88ffd086-0e86-4441-bd60-fdd4a25be6e8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_GET_QUANTILES(DS_QUANTILES_SKETCH(Distance),0, .25, .5, .75, 1) AS \"quartiles\",\n",
+ " DS_GET_QUANTILES(DS_QUANTILES_SKETCH(Distance),0, .1, .2, .3, .4, .5, .6, .7, .8, .9, 1) AS \"deciles\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE TIME_FLOOR(\"__time\", 'P1D') = TIMESTAMP '2005-11-24'\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "aa138aa8-d0b4-47fe-8e6c-873543e52f0b",
+ "metadata": {},
+ "source": [
+ "## Estimating the rank (cumulative distribution functions)\n",
+ "\n",
+ "Druid provides two functions for estimating what the [normalized rank](https://datasketches.apache.org/docs/Quantiles/SketchingQuantilesAndRanksTutorial.html) of a value is within the data. Or, to put it another way, what percentage of other records fall below a given value.\n",
Review Comment:
```suggestion
"Druid provides two functions for estimating the [normalized rank](https://datasketches.apache.org/docs/Quantiles/SketchingQuantilesAndRanksTutorial.html) of a value within the data. Or, to put it another way, what percentage of other records fall below a given value.\n",
```
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
+ "\n",
+ "To see how quantiles sketches look, run the following cell. It uses `DS_QUANTILES_SKETCH` to create a Quantiles sketch for both `Distance` and `Delay`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bbe82c0d-aa42-4740-8f3e-a79ab7caa227",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT \n",
+ " DS_QUANTILES_SKETCH(Distance) AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e484022e-63eb-4025-b113-d8bb6351778c",
+ "metadata": {},
+ "source": [
+ "This function can be combined with `FILTER` to limit the sketch based on other dimension values.\n",
+ "\n",
+ "Running the cell below shows two Quantiles sketches that are only representative of flights leaving Atlanta."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "63ebe760-d949-48cb-be18-5e09503140e9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_QUANTILES_SKETCH(Distance) FILTER (WHERE Origin='ATL') AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) FILTER (WHERE Origin='ATL') AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0e835dae-1834-4fdd-b6f5-747a88cdb0cd",
+ "metadata": {},
+ "source": [
+ "As with other Datasketches, the `DS_QUANTILES_SKETCH` function (or the [classic equivallent](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles#aggregator), `quantilesDoublesSketch`) can be used at ingestion time to pre-calculate these objects, reducing processing required at query time.\n",
+ "\n",
+ "In the rest of this notebook, these representations will be used to find quantiles (the estimated value at a given position) and ranks (the estimated position of a specific value), and to build histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ae00642f-1266-4c5e-a02c-1789ef57ad62",
+ "metadata": {},
+ "source": [
+ "## Estimating the quantile\n",
+ "\n",
+ "A quantile allows us to conclude that, within a given set of observations, the specified percentage of values falls below a particular value.\n",
+ "\n",
+ "Druid provides functions that allow you to estimate single and multiple quantiles.\n",
+ "\n",
+ "### Estimate a single quantile value\n",
+ "\n",
+ "Run the following cell, which uses the [`DS_GET_QUANTILE`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantile) function to find out the values for each `Reporting_Airline` at the quartile positions - -25%, 50%, and 75%."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f98b55f3-c6ce-4ad0-9c24-b2253d2eefe4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.25) AS \"25percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.5) AS \"50percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "88585044-d2b5-4ddf-b3bb-4203fd021e60",
+ "metadata": {},
+ "source": [
+ "The results show that, for each `Reporting_Airline`, 25% of flights flew a `Distance` below the first value, 50% below the second, and 75 below the third.\n",
+ "\n",
+ "Druid provides a shorthand version of the SQL statement used in the previous cell - [`APPROX_QUANTILE_DS`](https://druid.apache.org/docs/latest/querying/sql-functions#approx_quantile_ds).\n",
+ "\n",
+ "Run the cell below, which gives the same result as the query above but uses this shorthand function. There's no need for the `DS_QUANTILES_SKETCH` function to create a sketch against `Distance`, and instead just provide `Distance`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6a7480f8-0491-4514-b005-bb2e884ee448",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " APPROX_QUANTILE_DS(Distance,.25) AS \"25percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.5) AS \"50percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "669865f3-d612-4c2a-951b-d0f2e52fade3",
+ "metadata": {},
+ "source": [
+ "As with `DS_QUANTILES_SKETCH`, this can be combined with a `FILTER` clause.\n",
+ "\n",
+ "Run the following cell which looks only at flights that were delayed by 3 hours. By using the `APPROX_QUANTILE_DS` function the results will show the distance that 98% of flights in this category flew."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "42b3ee0a-c901-4be5-b19b-17df69b901ad",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'CO') AS \"Distance_CO\",\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'US') AS \"Distance_US\",\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'AA') AS \"Distance_AA\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE Delay > 180\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cda1c68b-f515-4eba-870c-137234c45633",
+ "metadata": {},
+ "source": [
+ "Run the next cell to produce a plot of the number of delayed flights on a particular day, by airline, that were delayed by more than 75% of flights were in the two-week period before.\n",
+ "\n",
+ "Table `b` lists, for each `Reporting_Airline`, the third-quartile `Delay` within a two-week period.\n",
+ "\n",
+ "This data is then used through a `JOIN` against `example-flights-quantiles` to `COUNT` the number of flights, per `Reporting_Airline`, that are above the third-quartile.\n",
+ "\n",
+ "The sorted results are then put into a pandas Dataframe and plotted."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c3421e46-9157-48a1-bbda-b0904d9d2adc",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "WITH b AS (\n",
+ "SELECT\n",
+ " \"Reporting_Airline\",\n",
+ " APPROX_QUANTILE_DS(Delay,.75) AS \"75delay\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE (TIMESTAMP '2005-11-01' <= \"__time\" AND \"__time\" < TIMESTAMP '2005-11-14')\n",
+ "GROUP BY 1)\n",
+ "\n",
+ "SELECT\n",
+ " a.\"Reporting_Airline\",\n",
+ " COUNT(*) AS \"Flights\"\n",
+ "FROM \"example-flights-quantiles\" a\n",
+ "LEFT JOIN b ON a.Reporting_Airline = b.Reporting_Airline\n",
+ "WHERE TIME_FLOOR(a.\"__time\", 'PT1H') = TIMESTAMP '2005-11-15 11:00:00'\n",
+ "AND a.\"Delay\" > b.\"75delay\"\n",
+ "GROUP BY 1\n",
+ "ORDER BY 2 DESC\n",
+ "'''\n",
+ "\n",
+ "df1 = pd.DataFrame(sql_client.sql(sql))\n",
+ "\n",
+ "df1.plot.bar(x='Reporting_Airline', y='Flights')\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3dedcdbd-aa56-4b75-80a3-327cea3f7f52",
+ "metadata": {},
+ "source": [
+ "A query like the one above could be adapted to work with streaming ingestion of event data, giving a real-time view of the numbers of flights currently delayed that exceed the 75th percentile delay from the previous two weeks."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "53feba1b-64f0-42d2-a8f6-59b57b38b187",
+ "metadata": {},
+ "source": [
+ "### Estimate multiple quantile values\n",
+ "\n",
+ "In the examples above, you used separate `APPROX_QUANTILE_DS` functions to find the values at different positions in the sketch.\n",
+ "\n",
+ "```sql\n",
+ " APPROX_QUANTILE_DS(Distance,.25) AS \"25percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.5) AS \"50percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.75) AS \"75percent\"\n",
+ "```\n",
+ "\n",
+ "Alternatively, the values can be returned in an [`ARRAY`](https://druid.apache.org/docs/latest/querying/sql-array-functions) by using the [`DS_GET_QUANTILES`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantiles) function.\n",
+ "\n",
+ "Run the following cell, which returns quantiles in an `ARRAY` object instead of separate columns - one for quartiles, one for deciles - broken down by `Reporting_Airline` on a particular day."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "88ffd086-0e86-4441-bd60-fdd4a25be6e8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_GET_QUANTILES(DS_QUANTILES_SKETCH(Distance),0, .25, .5, .75, 1) AS \"quartiles\",\n",
+ " DS_GET_QUANTILES(DS_QUANTILES_SKETCH(Distance),0, .1, .2, .3, .4, .5, .6, .7, .8, .9, 1) AS \"deciles\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE TIME_FLOOR(\"__time\", 'P1D') = TIMESTAMP '2005-11-24'\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "aa138aa8-d0b4-47fe-8e6c-873543e52f0b",
+ "metadata": {},
+ "source": [
+ "## Estimating the rank (cumulative distribution functions)\n",
+ "\n",
+ "Druid provides two functions for estimating what the [normalized rank](https://datasketches.apache.org/docs/Quantiles/SketchingQuantilesAndRanksTutorial.html) of a value is within the data. Or, to put it another way, what percentage of other records fall below a given value.\n",
+ "\n",
+ "### Estimate a single rank position\n",
+ "\n",
+ "Use the [`DS_RANK`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_rank) function to estimate where a given value falls in the distribution of the data, returned as a percentage.\n",
+ "\n",
+ "Run the following cell which returns the rank of a particular value in the \"XE\" flights on the 24th November 2005.\n",
+ "\n",
+ "Notice that the `DS_QUANTILES_SKETCH` function is used to provide `DS_RANK` with the sketch representation."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "4114078b-eb95-45c5-8fe9-fcb170b4366a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_RANK(DS_QUANTILES_SKETCH(Distance),950) AS \"rank\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE TIME_FLOOR(\"__time\", 'P1D') = TIMESTAMP '2005-11-24'\n",
+ "AND \"Reporting_Airline\" = 'XE'\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f06cb646-3ab2-4f7d-865c-dabc669390ce",
+ "metadata": {},
+ "source": [
+ "This tells us that this distance is representative of about 90% of flights on that day. The results for \"XE\" in the previous cell, where we showed the deciles for the same flights, corroborates that.\n",
+ "\n",
+ "Run the cell below to see how far up the rankings a delay of 30-minutes would be when looking at each delayed flight posted by a `Reporting_Airline` in the week commencing 31st October 2005."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f864f0bc-3c82-4f0e-9470-332e9672fce4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_RANK(DS_QUANTILES_SKETCH(Delay),30) AS \"rank\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE TIME_FLOOR(\"__time\", 'P1W') = TIMESTAMP '2005-10-31'\n",
+ "AND Delay > 0\n",
+ "GROUP BY 1\n",
+ "ORDER BY 2 DESC\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ea9192f0-8a5f-4954-b63c-133964cc4b28",
+ "metadata": {},
+ "source": [
+ "### Estimate multiple rank positions\n",
+ "\n",
+ "In order to estimate multiple rank positions, the [`DS_CDF`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_cdf) function is available.\n",
+ "\n",
+ "Run the cell below to use this function to return an `ARRAY` of ranking positions for 6 different delays per `Reporting_Airline`, from 30 minutes through to 16 hours. The results show what percentage of reported delays in the dataset were under each of the lengths of time we have specified.\n",
+ "\n",
+ "The `MAX` is added as a sorting criteria."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "95127870-05f0-434a-927a-044205b7104b",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_CDF(DS_QUANTILES_SKETCH(Delay), 30, 60, 120, 240, 480, 960) AS \"cdf\",\n",
+ " MAX(Delay) AS \"Longest Delay\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE TIME_FLOOR(\"__time\", 'P1W') = TIMESTAMP '2005-11-07'\n",
+ "AND Delay > 0\n",
+ "GROUP BY 1\n",
+ "ORDER BY 3 DESC\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ecc3b60a-54f8-4e91-9618-8afc635a7b02",
+ "metadata": {},
+ "source": [
+ "## Estimating histograms\n",
+ "\n",
+ "When wanting to understand the shape of the distribution, Druid provides the [`DS_HISTOGRAM`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_histogram) function, which also acts on an underlying Quantiles sketch.\n",
+ "\n",
+ "Run the following cell to see how the `DS_HISTOGRAM` function estimates the buckets of `Distance` of flights taking off from Los Angeles."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "efaca521-f5c3-4d61-8fb6-f5ed77b1760c",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT Origin,\n",
+ " DS_HISTOGRAM(DS_QUANTILES_SKETCH(Distance), 250, 500, 750, 1000, 1250, 1500, 1750, 2000) AS Histogram\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE \"Origin\" = 'LAX'\n",
+ "AND TIME_FLOOR(\"__time\", 'P1W') = TIMESTAMP '2005-10-31'\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ea9b2ef0-b323-4b1c-b8dc-755e6fff32d7",
+ "metadata": {},
+ "source": [
+ "The results show an estimated count of the number of flights in each of the `Distance` buckets specified, plus one final bucket for the remainder.\n",
+ "\n",
+ "Combining `DS_QUANTILES_SKETCH` with `FILTER` focuses the distribution being analysed.\n",
+ "\n",
+ "Run the next cell which produces histograms for two specific `Reporting_Airline` with flights from Los Angeles."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "7dc74d23-0d5a-4ded-aa62-10d93d39d461",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_HISTOGRAM(DS_QUANTILES_SKETCH(Distance) FILTER (WHERE \"Reporting_Airline\" = 'AA'), 250, 500, 750, 1000, 1250, 1500, 1750, 2000) AS \"Histogram-AA\",\n",
+ " DS_HISTOGRAM(DS_QUANTILES_SKETCH(Distance) FILTER (WHERE \"Reporting_Airline\" = 'AS'), 250, 500, 750, 1000, 1250, 1500, 1750, 2000) AS \"Histogram-AS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE \"Origin\" = 'LAX'\n",
+ "AND TIME_FLOOR(\"__time\", 'P1W') = TIMESTAMP '2005-10-31'\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f58a1846-5072-4495-b840-a620de3c0442",
+ "metadata": {},
+ "source": [
+ "## Summary\n",
+ "\n",
+ "* Apache Datasketch \"Quantile\" sketches are supported in Apache Druid\n",
+ "* SQL functions exist for estimating individual ranks (`DS_RANK`) and quantiles (`DS_GET_QUANTILE`)\n",
+ "* Druid can return an `ARRAY` of ranks (`DS_CDF`) and quantiles (`DS_GET_QUANTILES`)\n",
+ "* Histograms can be estimated using the `DS_HISTOGRAM` function\n",
+ "\n",
+ "## Learn more\n",
+ "\n",
+ "* Read [quantiles extension](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles) documentation\n",
Review Comment:
```suggestion
"* Read the [quantiles extension](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles) documentation\n",
```
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
+ "\n",
+ "To see how quantiles sketches look, run the following cell. It uses `DS_QUANTILES_SKETCH` to create a Quantiles sketch for both `Distance` and `Delay`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bbe82c0d-aa42-4740-8f3e-a79ab7caa227",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT \n",
+ " DS_QUANTILES_SKETCH(Distance) AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e484022e-63eb-4025-b113-d8bb6351778c",
+ "metadata": {},
+ "source": [
+ "This function can be combined with `FILTER` to limit the sketch based on other dimension values.\n",
+ "\n",
+ "Running the cell below shows two Quantiles sketches that are only representative of flights leaving Atlanta."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "63ebe760-d949-48cb-be18-5e09503140e9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_QUANTILES_SKETCH(Distance) FILTER (WHERE Origin='ATL') AS \"Distance_QS\",\n",
+ " DS_QUANTILES_SKETCH(Delay) FILTER (WHERE Origin='ATL') AS \"Delay_QS\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0e835dae-1834-4fdd-b6f5-747a88cdb0cd",
+ "metadata": {},
+ "source": [
+ "As with other Datasketches, the `DS_QUANTILES_SKETCH` function (or the [classic equivallent](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles#aggregator), `quantilesDoublesSketch`) can be used at ingestion time to pre-calculate these objects, reducing processing required at query time.\n",
+ "\n",
+ "In the rest of this notebook, these representations will be used to find quantiles (the estimated value at a given position) and ranks (the estimated position of a specific value), and to build histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ae00642f-1266-4c5e-a02c-1789ef57ad62",
+ "metadata": {},
+ "source": [
+ "## Estimating the quantile\n",
+ "\n",
+ "A quantile allows us to conclude that, within a given set of observations, the specified percentage of values falls below a particular value.\n",
+ "\n",
+ "Druid provides functions that allow you to estimate single and multiple quantiles.\n",
+ "\n",
+ "### Estimate a single quantile value\n",
+ "\n",
+ "Run the following cell, which uses the [`DS_GET_QUANTILE`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantile) function to find out the values for each `Reporting_Airline` at the quartile positions - -25%, 50%, and 75%."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f98b55f3-c6ce-4ad0-9c24-b2253d2eefe4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.25) AS \"25percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.5) AS \"50percent\",\n",
+ " DS_GET_QUANTILE(DS_QUANTILES_SKETCH(Distance),.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "88585044-d2b5-4ddf-b3bb-4203fd021e60",
+ "metadata": {},
+ "source": [
+ "The results show that, for each `Reporting_Airline`, 25% of flights flew a `Distance` below the first value, 50% below the second, and 75 below the third.\n",
+ "\n",
+ "Druid provides a shorthand version of the SQL statement used in the previous cell - [`APPROX_QUANTILE_DS`](https://druid.apache.org/docs/latest/querying/sql-functions#approx_quantile_ds).\n",
+ "\n",
+ "Run the cell below, which gives the same result as the query above but uses this shorthand function. There's no need for the `DS_QUANTILES_SKETCH` function to create a sketch against `Distance`, and instead just provide `Distance`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6a7480f8-0491-4514-b005-bb2e884ee448",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " APPROX_QUANTILE_DS(Distance,.25) AS \"25percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.5) AS \"50percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.75) AS \"75percent\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "669865f3-d612-4c2a-951b-d0f2e52fade3",
+ "metadata": {},
+ "source": [
+ "As with `DS_QUANTILES_SKETCH`, this can be combined with a `FILTER` clause.\n",
+ "\n",
+ "Run the following cell which looks only at flights that were delayed by 3 hours. By using the `APPROX_QUANTILE_DS` function the results will show the distance that 98% of flights in this category flew."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "42b3ee0a-c901-4be5-b19b-17df69b901ad",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'CO') AS \"Distance_CO\",\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'US') AS \"Distance_US\",\n",
+ " APPROX_QUANTILE_DS(Distance,.98) FILTER (WHERE Reporting_Airline = 'AA') AS \"Distance_AA\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE Delay > 180\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cda1c68b-f515-4eba-870c-137234c45633",
+ "metadata": {},
+ "source": [
+ "Run the next cell to produce a plot of the number of delayed flights on a particular day, by airline, that were delayed by more than 75% of flights were in the two-week period before.\n",
+ "\n",
+ "Table `b` lists, for each `Reporting_Airline`, the third-quartile `Delay` within a two-week period.\n",
+ "\n",
+ "This data is then used through a `JOIN` against `example-flights-quantiles` to `COUNT` the number of flights, per `Reporting_Airline`, that are above the third-quartile.\n",
+ "\n",
+ "The sorted results are then put into a pandas Dataframe and plotted."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c3421e46-9157-48a1-bbda-b0904d9d2adc",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "WITH b AS (\n",
+ "SELECT\n",
+ " \"Reporting_Airline\",\n",
+ " APPROX_QUANTILE_DS(Delay,.75) AS \"75delay\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE (TIMESTAMP '2005-11-01' <= \"__time\" AND \"__time\" < TIMESTAMP '2005-11-14')\n",
+ "GROUP BY 1)\n",
+ "\n",
+ "SELECT\n",
+ " a.\"Reporting_Airline\",\n",
+ " COUNT(*) AS \"Flights\"\n",
+ "FROM \"example-flights-quantiles\" a\n",
+ "LEFT JOIN b ON a.Reporting_Airline = b.Reporting_Airline\n",
+ "WHERE TIME_FLOOR(a.\"__time\", 'PT1H') = TIMESTAMP '2005-11-15 11:00:00'\n",
+ "AND a.\"Delay\" > b.\"75delay\"\n",
+ "GROUP BY 1\n",
+ "ORDER BY 2 DESC\n",
+ "'''\n",
+ "\n",
+ "df1 = pd.DataFrame(sql_client.sql(sql))\n",
+ "\n",
+ "df1.plot.bar(x='Reporting_Airline', y='Flights')\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3dedcdbd-aa56-4b75-80a3-327cea3f7f52",
+ "metadata": {},
+ "source": [
+ "A query like the one above could be adapted to work with streaming ingestion of event data, giving a real-time view of the numbers of flights currently delayed that exceed the 75th percentile delay from the previous two weeks."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "53feba1b-64f0-42d2-a8f6-59b57b38b187",
+ "metadata": {},
+ "source": [
+ "### Estimate multiple quantile values\n",
+ "\n",
+ "In the examples above, you used separate `APPROX_QUANTILE_DS` functions to find the values at different positions in the sketch.\n",
+ "\n",
+ "```sql\n",
+ " APPROX_QUANTILE_DS(Distance,.25) AS \"25percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.5) AS \"50percent\",\n",
+ " APPROX_QUANTILE_DS(Distance,.75) AS \"75percent\"\n",
+ "```\n",
+ "\n",
+ "Alternatively, the values can be returned in an [`ARRAY`](https://druid.apache.org/docs/latest/querying/sql-array-functions) by using the [`DS_GET_QUANTILES`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_get_quantiles) function.\n",
+ "\n",
+ "Run the following cell, which returns quantiles in an `ARRAY` object instead of separate columns - one for quartiles, one for deciles - broken down by `Reporting_Airline` on a particular day."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "88ffd086-0e86-4441-bd60-fdd4a25be6e8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_GET_QUANTILES(DS_QUANTILES_SKETCH(Distance),0, .25, .5, .75, 1) AS \"quartiles\",\n",
+ " DS_GET_QUANTILES(DS_QUANTILES_SKETCH(Distance),0, .1, .2, .3, .4, .5, .6, .7, .8, .9, 1) AS \"deciles\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE TIME_FLOOR(\"__time\", 'P1D') = TIMESTAMP '2005-11-24'\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "aa138aa8-d0b4-47fe-8e6c-873543e52f0b",
+ "metadata": {},
+ "source": [
+ "## Estimating the rank (cumulative distribution functions)\n",
+ "\n",
+ "Druid provides two functions for estimating what the [normalized rank](https://datasketches.apache.org/docs/Quantiles/SketchingQuantilesAndRanksTutorial.html) of a value is within the data. Or, to put it another way, what percentage of other records fall below a given value.\n",
+ "\n",
+ "### Estimate a single rank position\n",
+ "\n",
+ "Use the [`DS_RANK`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_rank) function to estimate where a given value falls in the distribution of the data, returned as a percentage.\n",
+ "\n",
+ "Run the following cell which returns the rank of a particular value in the \"XE\" flights on the 24th November 2005.\n",
+ "\n",
+ "Notice that the `DS_QUANTILES_SKETCH` function is used to provide `DS_RANK` with the sketch representation."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "4114078b-eb95-45c5-8fe9-fcb170b4366a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " DS_RANK(DS_QUANTILES_SKETCH(Distance),950) AS \"rank\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE TIME_FLOOR(\"__time\", 'P1D') = TIMESTAMP '2005-11-24'\n",
+ "AND \"Reporting_Airline\" = 'XE'\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f06cb646-3ab2-4f7d-865c-dabc669390ce",
+ "metadata": {},
+ "source": [
+ "This tells us that this distance is representative of about 90% of flights on that day. The results for \"XE\" in the previous cell, where we showed the deciles for the same flights, corroborates that.\n",
+ "\n",
+ "Run the cell below to see how far up the rankings a delay of 30-minutes would be when looking at each delayed flight posted by a `Reporting_Airline` in the week commencing 31st October 2005."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f864f0bc-3c82-4f0e-9470-332e9672fce4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_RANK(DS_QUANTILES_SKETCH(Delay),30) AS \"rank\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE TIME_FLOOR(\"__time\", 'P1W') = TIMESTAMP '2005-10-31'\n",
+ "AND Delay > 0\n",
+ "GROUP BY 1\n",
+ "ORDER BY 2 DESC\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ea9192f0-8a5f-4954-b63c-133964cc4b28",
+ "metadata": {},
+ "source": [
+ "### Estimate multiple rank positions\n",
+ "\n",
+ "In order to estimate multiple rank positions, the [`DS_CDF`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_cdf) function is available.\n",
+ "\n",
+ "Run the cell below to use this function to return an `ARRAY` of ranking positions for 6 different delays per `Reporting_Airline`, from 30 minutes through to 16 hours. The results show what percentage of reported delays in the dataset were under each of the lengths of time we have specified.\n",
+ "\n",
+ "The `MAX` is added as a sorting criteria."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "95127870-05f0-434a-927a-044205b7104b",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT\n",
+ " Reporting_Airline,\n",
+ " DS_CDF(DS_QUANTILES_SKETCH(Delay), 30, 60, 120, 240, 480, 960) AS \"cdf\",\n",
+ " MAX(Delay) AS \"Longest Delay\"\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE TIME_FLOOR(\"__time\", 'P1W') = TIMESTAMP '2005-11-07'\n",
+ "AND Delay > 0\n",
+ "GROUP BY 1\n",
+ "ORDER BY 3 DESC\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ecc3b60a-54f8-4e91-9618-8afc635a7b02",
+ "metadata": {},
+ "source": [
+ "## Estimating histograms\n",
+ "\n",
+ "When wanting to understand the shape of the distribution, Druid provides the [`DS_HISTOGRAM`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_histogram) function, which also acts on an underlying Quantiles sketch.\n",
+ "\n",
+ "Run the following cell to see how the `DS_HISTOGRAM` function estimates the buckets of `Distance` of flights taking off from Los Angeles."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "efaca521-f5c3-4d61-8fb6-f5ed77b1760c",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "SELECT Origin,\n",
+ " DS_HISTOGRAM(DS_QUANTILES_SKETCH(Distance), 250, 500, 750, 1000, 1250, 1500, 1750, 2000) AS Histogram\n",
+ "FROM \"example-flights-quantiles\"\n",
+ "WHERE \"Origin\" = 'LAX'\n",
+ "AND TIME_FLOOR(\"__time\", 'P1W') = TIMESTAMP '2005-10-31'\n",
+ "GROUP BY 1\n",
+ "'''\n",
+ "\n",
+ "display.sql(sql)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ea9b2ef0-b323-4b1c-b8dc-755e6fff32d7",
+ "metadata": {},
+ "source": [
+ "The results show an estimated count of the number of flights in each of the `Distance` buckets specified, plus one final bucket for the remainder.\n",
+ "\n",
+ "Combining `DS_QUANTILES_SKETCH` with `FILTER` focuses the distribution being analysed.\n",
+ "\n",
+ "Run the next cell which produces histograms for two specific `Reporting_Airline` with flights from Los Angeles."
Review Comment:
For this cell and the previous one, could be cool to see them in histogram form: https://matplotlib.org/stable/gallery/statistics/hist.html#sphx-glr-gallery-statistics-hist-py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] 202308-notebook-distribution (druid)
Posted by "petermarshallio (via GitHub)" <gi...@apache.org>.
petermarshallio commented on code in PR #14849:
URL: https://github.com/apache/druid/pull/14849#discussion_r1309767006
##########
examples/quickstart/jupyter-notebooks/notebooks/03-query/03-dataDistribution.ipynb:
##########
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4188e2c5-4ab0-45e3-9371-52d510a97413",
+ "metadata": {},
+ "source": [
+ "# Analysing data distributions\n",
+ "\n",
+ "The Druid database provides functions to analyse large distributions thanks to the Apache [Datasketch](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) project's [Quantiles](https://datasketches.apache.org/docs/Quantiles/QuantilesOverview.html) sketch, as well as [t-digest](https://druid.apache.org/docs/latest/development/extensions-contrib/tdigestsketch-quantiles/), [momentSketch](https://druid.apache.org/docs/latest/development/extensions-contrib/momentsketch-quantiles/), and [approxHistogram](https://druid.apache.org/docs/latest/development/extensions-core/approximate-histograms/).\n",
+ "\n",
+ "In this tutorial, work through some examples of the functions available in the Apache Datasketches ezxtension, a core extension that enables estimation of quantiles, ranks, and histograms."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "557e06e8-9b35-4b34-8322-8a8ede6de709",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "This tutorial works with Druid 26.0.0 or later.\n",
+ "\n",
+ "#### Run using Docker\n",
+ "\n",
+ "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see [Docker for Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-docker.html).\n",
+ " \n",
+ "#### Run without using Docker\n",
+ "\n",
+ "If you do not use the Docker Compose environment, you need the following:\n",
+ "\n",
+ "* A running Apache Druid instance, with a `DRUID_HOST` local environment variable containing the server name of your Druid router.\n",
+ "* [druidapi](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/druidapi/README.md), a Python client for Apache Druid. Follow the instructions in the Install section of the README file.\n",
+ "* [matplotlib](https://matplotlib.org/), a library for creating visualizations in Python.\n",
+ "* [pandas](https://pandas.pydata.org/), a data analysis and manipulation tool."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3317c51-627c-4a44-ad73-0860a5f4c937",
+ "metadata": {},
+ "source": [
+ "### Initialize Python\n",
+ "\n",
+ "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
+ "\n",
+ "If successful, the Druid version number will be shown in the output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a19226-6abc-436d-ac3c-9c04d6026707",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import druidapi\n",
+ "import os\n",
+ "\n",
+ "if 'DRUID_HOST' not in os.environ.keys():\n",
+ " druid_host=f\"http://localhost:8888\"\n",
+ "else:\n",
+ " druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
+ " \n",
+ "print(f\"Opening a connection to {druid_host}.\")\n",
+ "druid = druidapi.jupyter_client(druid_host)\n",
+ "\n",
+ "display = druid.display\n",
+ "sql_client = druid.sql\n",
+ "status_client = druid.status\n",
+ "\n",
+ "status_client.version"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fee22cea-6176-4119-b4ea-24ed76cfa8fe",
+ "metadata": {},
+ "source": [
+ "### Load example data\n",
+ "\n",
+ "Once your Druid environment is up and running, ingest the sample data for this tutorial.\n",
+ "\n",
+ "Run the following cell to create a table called `example-flights-quantiles`. When completed, you'll see a description of the final table.\n",
+ "\n",
+ "Monitor the ingestion task process in the Druid console."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "926a6366-61e5-4aaf-acab-ad90fbc5a994",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql='''\n",
+ "REPLACE INTO \"example-flights-quantiles\" OVERWRITE ALL\n",
+ "WITH \"ext\" AS (SELECT *\n",
+ "FROM TABLE(\n",
+ " EXTERN(\n",
+ " '{\"type\":\"http\",\"uris\":[\"https://static.imply.io/example-data/flight_on_time/flights/On_Time_Reporting_Carrier_On_Time_Performance_(1987_present)_2005_11.csv.zip\"]}',\n",
+ " '{\"type\":\"csv\",\"findColumnsFromHeader\":true}'\n",
+ " )\n",
+ ") EXTEND (\"depaturetime\" VARCHAR, \"arrivalime\" VARCHAR, \"Year\" BIGINT, \"Quarter\" BIGINT, \"Month\" BIGINT, \"DayofMonth\" BIGINT, \"DayOfWeek\" BIGINT, \"FlightDate\" VARCHAR, \"Reporting_Airline\" VARCHAR, \"DOT_ID_Reporting_Airline\" BIGINT, \"IATA_CODE_Reporting_Airline\" VARCHAR, \"Tail_Number\" VARCHAR, \"Flight_Number_Reporting_Airline\" BIGINT, \"OriginAirportID\" BIGINT, \"OriginAirportSeqID\" BIGINT, \"OriginCityMarketID\" BIGINT, \"Origin\" VARCHAR, \"OriginCityName\" VARCHAR, \"OriginState\" VARCHAR, \"OriginStateFips\" BIGINT, \"OriginStateName\" VARCHAR, \"OriginWac\" BIGINT, \"DestAirportID\" BIGINT, \"DestAirportSeqID\" BIGINT, \"DestCityMarketID\" BIGINT, \"Dest\" VARCHAR, \"DestCityName\" VARCHAR, \"DestState\" VARCHAR, \"DestStateFips\" BIGINT, \"DestStateName\" VARCHAR, \"DestWac\" BIGINT, \"CRSDepTime\" BIGINT, \"DepTime\" BIGINT, \"DepDelay\" BIGINT, \"DepDelayMinutes\" BIGINT, \"DepDel15\" BIGINT, \"DepartureDelayGroups\" BIGINT, \"DepTimeBlk\" VAR
CHAR, \"TaxiOut\" BIGINT, \"WheelsOff\" BIGINT, \"WheelsOn\" BIGINT, \"TaxiIn\" BIGINT, \"CRSArrTime\" BIGINT, \"ArrTime\" BIGINT, \"ArrDelay\" BIGINT, \"ArrDelayMinutes\" BIGINT, \"ArrDel15\" BIGINT, \"ArrivalDelayGroups\" BIGINT, \"ArrTimeBlk\" VARCHAR, \"Cancelled\" BIGINT, \"CancellationCode\" VARCHAR, \"Diverted\" BIGINT, \"CRSElapsedTime\" BIGINT, \"ActualElapsedTime\" BIGINT, \"AirTime\" BIGINT, \"Flights\" BIGINT, \"Distance\" BIGINT, \"DistanceGroup\" BIGINT, \"CarrierDelay\" BIGINT, \"WeatherDelay\" BIGINT, \"NASDelay\" BIGINT, \"SecurityDelay\" BIGINT, \"LateAircraftDelay\" BIGINT, \"FirstDepTime\" VARCHAR, \"TotalAddGTime\" VARCHAR, \"LongestAddGTime\" VARCHAR, \"DivAirportLandings\" VARCHAR, \"DivReachedDest\" VARCHAR, \"DivActualElapsedTime\" VARCHAR, \"DivArrDelay\" VARCHAR, \"DivDistance\" VARCHAR, \"Div1Airport\" VARCHAR, \"Div1AirportID\" VARCHAR, \"Div1AirportSeqID\" VARCHAR, \"Div1WheelsOn\" VARCHAR, \"Div1TotalGTime\" VARCHAR, \"Div1LongestGTime\" VARCHAR, \"Div
1WheelsOff\" VARCHAR, \"Div1TailNum\" VARCHAR, \"Div2Airport\" VARCHAR, \"Div2AirportID\" VARCHAR, \"Div2AirportSeqID\" VARCHAR, \"Div2WheelsOn\" VARCHAR, \"Div2TotalGTime\" VARCHAR, \"Div2LongestGTime\" VARCHAR, \"Div2WheelsOff\" VARCHAR, \"Div2TailNum\" VARCHAR, \"Div3Airport\" VARCHAR, \"Div3AirportID\" VARCHAR, \"Div3AirportSeqID\" VARCHAR, \"Div3WheelsOn\" VARCHAR, \"Div3TotalGTime\" VARCHAR, \"Div3LongestGTime\" VARCHAR, \"Div3WheelsOff\" VARCHAR, \"Div3TailNum\" VARCHAR, \"Div4Airport\" VARCHAR, \"Div4AirportID\" VARCHAR, \"Div4AirportSeqID\" VARCHAR, \"Div4WheelsOn\" VARCHAR, \"Div4TotalGTime\" VARCHAR, \"Div4LongestGTime\" VARCHAR, \"Div4WheelsOff\" VARCHAR, \"Div4TailNum\" VARCHAR, \"Div5Airport\" VARCHAR, \"Div5AirportID\" VARCHAR, \"Div5AirportSeqID\" VARCHAR, \"Div5WheelsOn\" VARCHAR, \"Div5TotalGTime\" VARCHAR, \"Div5LongestGTime\" VARCHAR, \"Div5WheelsOff\" VARCHAR, \"Div5TailNum\" VARCHAR, \"Unnamed: 109\" VARCHAR))\n",
+ "SELECT\n",
+ " TIME_PARSE(\"depaturetime\") AS \"__time\",\n",
+ " \"Reporting_Airline\",\n",
+ " \"Tail_Number\",\n",
+ " \"Distance\",\n",
+ " \"Origin\",\n",
+ " DepDelayMinutes + ArrDelayMinutes AS \"Delay\"\n",
+ "FROM \"ext\"\n",
+ "PARTITIONED BY DAY\n",
+ "'''\n",
+ "\n",
+ "sql_client.run_task(sql)\n",
+ "sql_client.wait_until_ready('example-flights-quantiles')\n",
+ "display.table('example-flights-quantiles')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6f0d1f7-2f34-44fe-9912-4017ed10893c",
+ "metadata": {},
+ "source": [
+ "Finally, run the following cell to import additional Python modules that you will use."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4cfd50d-5ef9-4fd1-8077-bf03758c35df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import matplotlib\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6f687b0a-517a-4df3-a7cb-b7095a11bb4b",
+ "metadata": {},
+ "source": [
+ "## Understanding the Quantiles sketch\n",
+ "\n",
+ "A structure known as a \"sketch\" holds a representation of the distribution of values that we will use throughout this notebook. These are created by using the [`DS_QUANTILES_SKETCH`](https://druid.apache.org/docs/latest/querying/sql-functions#ds_quantiles_sketch) function, providing it with the data set that we want to create the representation of.\n",
+ "\n",
+ "To see how quantiles sketches look, run the following cell. It uses `DS_QUANTILES_SKETCH` to create a Quantiles sketch for both `Distance` and `Delay`."
Review Comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org