You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by ca...@apache.org on 2025/07/28 21:21:41 UTC
(druid-website-src) 01/01: proposed 34.0.0 changes
This is an automated email from the ASF dual-hosted git repository.
capistrant pushed a commit to branch 34.0.0-staging
in repository https://gitbox.apache.org/repos/asf/druid-website-src.git
commit 2c39012ff9118e93db8a6d24a2964832bd8dfc79
Author: capistrant <lu...@imply.io>
AuthorDate: Mon Jul 28 16:20:32 2025 -0500
proposed 34.0.0 changes
---
docs/34.0.0/api-reference/api-reference.md | 44 +
.../api-reference/automatic-compaction-api.md | 2 +-
.../api-reference/data-management-api.md | 59 +-
.../api-reference/dynamic-configuration-api.md | 10 +-
docs/34.0.0/api-reference/json-querying-api.md | 925 +++++
docs/34.0.0/api-reference/legacy-metadata-api.md | 344 ++
docs/34.0.0/api-reference/lookups-api.md | 279 ++
docs/34.0.0/api-reference/retention-rules-api.md | 562 +++
.../api-reference/service-status-api.md | 134 +-
docs/{latest => 34.0.0}/api-reference/sql-api.md | 114 +-
.../api-reference/sql-ingestion-api.md | 29 +-
docs/34.0.0/api-reference/sql-jdbc.md | 251 ++
docs/34.0.0/api-reference/supervisor-api.md | 3622 ++++++++++++++++++++
docs/{latest => 34.0.0}/api-reference/tasks-api.md | 6 +-
docs/34.0.0/assets/compaction-config.png | Bin 0 -> 103224 bytes
docs/34.0.0/assets/datasources-action-button.png | Bin 0 -> 800 bytes
docs/34.0.0/assets/druid-architecture.png | Bin 0 -> 134117 bytes
docs/34.0.0/assets/druid-architecture.svg | 19 +
docs/34.0.0/assets/druid-column-types.png | Bin 0 -> 93363 bytes
docs/34.0.0/assets/druid-dataflow-2x.png | Bin 0 -> 130160 bytes
docs/34.0.0/assets/druid-dataflow-3.png | Bin 0 -> 71425 bytes
docs/34.0.0/assets/druid-manage-1.png | Bin 0 -> 80415 bytes
docs/34.0.0/assets/druid-timeline.png | Bin 0 -> 24160 bytes
.../34.0.0/assets/files/kttm-kafka-supervisor.json | 66 +
docs/34.0.0/assets/indexing_service.png | Bin 0 -> 22490 bytes
.../msq-ui-download-query-results.png | Bin 0 -> 62538 bytes
.../multi-stage-query/tutorial-msq-convert.png | Bin 0 -> 80379 bytes
.../assets/multi-stage-query/ui-annotated.png | Bin 0 -> 227968 bytes
docs/34.0.0/assets/multi-stage-query/ui-empty.png | Bin 0 -> 77306 bytes
docs/34.0.0/assets/native-queries-01.png | Bin 0 -> 259507 bytes
docs/34.0.0/assets/nested-combined-json.png | Bin 0 -> 272858 bytes
docs/34.0.0/assets/nested-display-data-types.png | Bin 0 -> 254224 bytes
docs/34.0.0/assets/nested-examine-schema.png | Bin 0 -> 251706 bytes
docs/34.0.0/assets/nested-extract-as-type.png | Bin 0 -> 288918 bytes
docs/34.0.0/assets/nested-extract-elements.png | Bin 0 -> 317905 bytes
docs/34.0.0/assets/nested-group-aggregate.png | Bin 0 -> 249178 bytes
.../assets/nested-msq-ingestion-transform.png | Bin 0 -> 370419 bytes
docs/34.0.0/assets/nested-msq-ingestion.png | Bin 0 -> 302033 bytes
docs/34.0.0/assets/nested-parse-deserialize.png | Bin 0 -> 213791 bytes
docs/34.0.0/assets/nested-retrieve-json.png | Bin 0 -> 295566 bytes
docs/34.0.0/assets/nested-return-json.png | Bin 0 -> 349144 bytes
docs/34.0.0/assets/retention-rules.png | Bin 0 -> 148273 bytes
docs/34.0.0/assets/security-model-1.png | Bin 0 -> 85098 bytes
docs/34.0.0/assets/security-model-2.png | Bin 0 -> 29613 bytes
docs/34.0.0/assets/segmentPropagation.png | Bin 0 -> 30569 bytes
docs/34.0.0/assets/services-overview.png | Bin 0 -> 112702 bytes
.../assets/spectator-histogram-size-comparison.png | Bin 0 -> 45331 bytes
docs/34.0.0/assets/supervisor-actions.png | Bin 0 -> 77894 bytes
docs/34.0.0/assets/supervisor-info-dialog.png | Bin 0 -> 107805 bytes
docs/34.0.0/assets/supervisor-view.png | Bin 0 -> 63532 bytes
.../assets/tutorial-batch-data-loader-00.png | Bin 0 -> 2904 bytes
.../assets/tutorial-batch-data-loader-01.png | Bin 0 -> 227941 bytes
.../assets/tutorial-batch-data-loader-015.png | Bin 0 -> 206331 bytes
.../assets/tutorial-batch-data-loader-02.png | Bin 0 -> 1193472 bytes
.../assets/tutorial-batch-data-loader-03.png | Bin 0 -> 405035 bytes
.../assets/tutorial-batch-data-loader-04.png | Bin 0 -> 488906 bytes
.../assets/tutorial-batch-data-loader-05.png | Bin 0 -> 447046 bytes
.../assets/tutorial-batch-data-loader-06.png | Bin 0 -> 191383 bytes
.../assets/tutorial-batch-data-loader-07.png | Bin 0 -> 173618 bytes
.../assets/tutorial-batch-data-loader-08.png | Bin 0 -> 325806 bytes
.../assets/tutorial-batch-data-loader-09.png | Bin 0 -> 187695 bytes
.../assets/tutorial-batch-data-loader-10.png | Bin 0 -> 219732 bytes
.../assets/tutorial-batch-data-loader-11.png | Bin 0 -> 316540 bytes
.../assets/tutorial-batch-data-loader-12.png | Bin 0 -> 31752 bytes
.../assets/tutorial-batch-submit-task-01.png | Bin 0 -> 170537 bytes
.../assets/tutorial-batch-submit-task-02.png | Bin 0 -> 207377 bytes
docs/34.0.0/assets/tutorial-compaction-01.png | Bin 0 -> 35710 bytes
docs/34.0.0/assets/tutorial-compaction-02.png | Bin 0 -> 166571 bytes
docs/34.0.0/assets/tutorial-compaction-03.png | Bin 0 -> 26755 bytes
docs/34.0.0/assets/tutorial-compaction-04.png | Bin 0 -> 184365 bytes
docs/34.0.0/assets/tutorial-compaction-05.png | Bin 0 -> 26588 bytes
docs/34.0.0/assets/tutorial-compaction-06.png | Bin 0 -> 206717 bytes
docs/34.0.0/assets/tutorial-compaction-07.png | Bin 0 -> 26683 bytes
docs/34.0.0/assets/tutorial-compaction-08.png | Bin 0 -> 28751 bytes
docs/34.0.0/assets/tutorial-deletion-01.png | Bin 0 -> 163245 bytes
docs/34.0.0/assets/tutorial-deletion-02.png | Bin 0 -> 159346 bytes
docs/34.0.0/assets/tutorial-deletion-03.png | Bin 0 -> 153770 bytes
.../assets/tutorial-kafka-data-loader-01.png | Bin 0 -> 75787 bytes
.../assets/tutorial-kafka-data-loader-02.png | Bin 0 -> 420744 bytes
.../assets/tutorial-kafka-data-loader-03.png | Bin 0 -> 192051 bytes
.../assets/tutorial-kafka-data-loader-04.png | Bin 0 -> 244021 bytes
.../assets/tutorial-kafka-data-loader-05.png | Bin 0 -> 506701 bytes
.../assets/tutorial-kafka-data-loader-06.png | Bin 0 -> 95274 bytes
.../assets/tutorial-kafka-data-loader-07.png | Bin 0 -> 114038 bytes
.../assets/tutorial-kafka-data-loader-08.png | Bin 0 -> 81750 bytes
.../assets/tutorial-kafka-data-loader-09.png | Bin 0 -> 141501 bytes
.../assets/tutorial-kafka-data-loader-10.png | Bin 0 -> 80127 bytes
.../assets/tutorial-kafka-data-loader-11.png | Bin 0 -> 74913 bytes
.../assets/tutorial-kafka-data-loader-12.png | Bin 0 -> 134001 bytes
.../assets/tutorial-kafka-submit-supervisor-01.png | Bin 0 -> 130690 bytes
docs/34.0.0/assets/tutorial-query-01.png | Bin 0 -> 78666 bytes
docs/34.0.0/assets/tutorial-query-02.png | Bin 0 -> 90958 bytes
docs/34.0.0/assets/tutorial-query-03.png | Bin 0 -> 118994 bytes
docs/34.0.0/assets/tutorial-query-04.png | Bin 0 -> 136945 bytes
docs/34.0.0/assets/tutorial-query-05.png | Bin 0 -> 135626 bytes
docs/34.0.0/assets/tutorial-query-06.png | Bin 0 -> 135743 bytes
docs/34.0.0/assets/tutorial-query-07.png | Bin 0 -> 148492 bytes
.../tutorial-query-deepstorage-retention-rule.png | Bin 0 -> 99106 bytes
docs/34.0.0/assets/tutorial-quickstart-01.png | Bin 0 -> 68216 bytes
docs/34.0.0/assets/tutorial-quickstart-02.png | Bin 0 -> 115511 bytes
docs/34.0.0/assets/tutorial-quickstart-03.png | Bin 0 -> 171269 bytes
docs/34.0.0/assets/tutorial-quickstart-04.png | Bin 0 -> 129369 bytes
docs/34.0.0/assets/tutorial-quickstart-05.png | Bin 0 -> 110367 bytes
docs/34.0.0/assets/tutorial-retention-00.png | Bin 0 -> 77704 bytes
docs/34.0.0/assets/tutorial-retention-01.png | Bin 0 -> 35171 bytes
docs/34.0.0/assets/tutorial-retention-02.png | Bin 0 -> 240310 bytes
docs/34.0.0/assets/tutorial-retention-03.png | Bin 0 -> 30029 bytes
docs/34.0.0/assets/tutorial-retention-04.png | Bin 0 -> 44617 bytes
docs/34.0.0/assets/tutorial-retention-05.png | Bin 0 -> 38992 bytes
docs/34.0.0/assets/tutorial-retention-06.png | Bin 0 -> 137570 bytes
.../34.0.0/assets/tutorial-sql-aggregate-query.png | Bin 0 -> 99042 bytes
docs/34.0.0/assets/tutorial-sql-auto-queries.png | Bin 0 -> 178642 bytes
.../assets/tutorial-sql-count-distinct-help.png | Bin 0 -> 190668 bytes
docs/34.0.0/assets/tutorial-sql-count-distinct.png | Bin 0 -> 109726 bytes
docs/34.0.0/assets/tutorial-sql-demo-queries.png | Bin 0 -> 158884 bytes
docs/34.0.0/assets/tutorial-sql-query-plan.png | Bin 0 -> 152371 bytes
.../assets/tutorial-sql-result-column-actions.png | Bin 0 -> 113313 bytes
docs/34.0.0/assets/tutorial-theta-01.png | Bin 0 -> 160923 bytes
docs/34.0.0/assets/tutorial-theta-02.png | Bin 0 -> 149124 bytes
docs/34.0.0/assets/tutorial-theta-03.png | Bin 0 -> 164755 bytes
docs/34.0.0/assets/tutorial-theta-04.png | Bin 0 -> 141568 bytes
docs/34.0.0/assets/tutorial-theta-05.png | Bin 0 -> 88165 bytes
docs/34.0.0/assets/tutorial-theta-06.png | Bin 0 -> 105328 bytes
docs/34.0.0/assets/tutorial-theta-07.png | Bin 0 -> 105225 bytes
docs/34.0.0/assets/tutorial-theta-08.png | Bin 0 -> 104913 bytes
docs/34.0.0/assets/web-console-0.7-tasks.png | Bin 0 -> 83141 bytes
docs/34.0.0/assets/web-console-01-home-view.png | Bin 0 -> 87581 bytes
.../34.0.0/assets/web-console-02-data-loader-1.png | Bin 0 -> 80007 bytes
.../34.0.0/assets/web-console-03-data-loader-2.png | Bin 0 -> 268671 bytes
docs/34.0.0/assets/web-console-04-datasources.png | Bin 0 -> 97639 bytes
docs/34.0.0/assets/web-console-05-retention.png | Bin 0 -> 86941 bytes
docs/34.0.0/assets/web-console-06-segments.png | Bin 0 -> 103333 bytes
docs/34.0.0/assets/web-console-07-supervisors.png | Bin 0 -> 51193 bytes
.../assets/web-console-08-supervisor-status.png | Bin 0 -> 89501 bytes
docs/34.0.0/assets/web-console-09-task-status.png | Bin 0 -> 201632 bytes
docs/34.0.0/assets/web-console-10-servers.png | Bin 0 -> 87782 bytes
docs/34.0.0/assets/web-console-11-query-sql.png | Bin 0 -> 131799 bytes
docs/34.0.0/assets/web-console-12-query-rune.png | Bin 0 -> 106246 bytes
docs/34.0.0/assets/web-console-13-lookups.png | Bin 0 -> 66934 bytes
docs/34.0.0/comparisons/druid-vs-elasticsearch.md | 39 +
docs/34.0.0/comparisons/druid-vs-key-value.md | 46 +
docs/34.0.0/comparisons/druid-vs-kudu.md | 39 +
docs/34.0.0/comparisons/druid-vs-redshift.md | 62 +
docs/34.0.0/comparisons/druid-vs-spark.md | 42 +
docs/34.0.0/comparisons/druid-vs-sql-on-hadoop.md | 82 +
docs/34.0.0/configuration/extensions.md | 177 +
docs/34.0.0/configuration/human-readable-byte.md | 98 +
docs/{latest => 34.0.0}/configuration/index.md | 386 ++-
docs/{latest => 34.0.0}/configuration/logging.md | 4 +-
.../data-management/automatic-compaction.md | 6 +-
docs/34.0.0/data-management/compaction.md | 113 +
docs/{latest => 34.0.0}/data-management/delete.md | 46 +-
docs/34.0.0/data-management/index.md | 34 +
.../data-management/manual-compaction.md | 2 +-
docs/34.0.0/data-management/schema-changes.md | 39 +
docs/34.0.0/data-management/update.md | 78 +
docs/34.0.0/design/architecture.md | 185 +
docs/34.0.0/design/broker.md | 54 +
docs/34.0.0/design/coordinator.md | 176 +
docs/34.0.0/design/deep-storage.md | 88 +
.../34.0.0/design/extensions-contrib/dropwizard.md | 94 +
docs/34.0.0/design/historical.md | 73 +
docs/34.0.0/design/index.md | 104 +
docs/34.0.0/design/indexer.md | 95 +
docs/34.0.0/design/indexing-service.md | 51 +
docs/34.0.0/design/metadata-storage.md | 175 +
docs/34.0.0/design/middlemanager.md | 43 +
docs/34.0.0/design/overlord.md | 59 +
docs/34.0.0/design/peons.md | 48 +
docs/34.0.0/design/router.md | 230 ++
docs/34.0.0/design/segments.md | 217 ++
docs/{latest => 34.0.0}/design/storage.md | 2 +-
docs/34.0.0/design/zookeeper.md | 76 +
docs/34.0.0/development/build.md | 110 +
docs/34.0.0/development/docs-contribute.md | 227 ++
docs/34.0.0/development/experimental.md | 37 +
.../extensions-contrib/aliyun-oss-extensions.md | 236 ++
.../extensions-contrib/ambari-metrics-emitter.md | 98 +
.../development/extensions-contrib/cassandra.md | 30 +
.../development/extensions-contrib/cloudfiles.md | 42 +
.../extensions-contrib/compressed-big-decimal.md | 47 +-
.../extensions-contrib/ddsketch-quantiles.md | 139 +
.../development/extensions-contrib/delta-lake.md | 54 +
.../extensions-contrib/distinctcount.md | 99 +
.../extensions-contrib/druid-ranger-security.md | 130 +
.../extensions-contrib/gce-extensions.md | 103 +
.../development/extensions-contrib/graphite.md | 117 +
.../development/extensions-contrib/iceberg.md | 149 +
.../development/extensions-contrib/influx.md | 67 +
.../extensions-contrib/influxdb-emitter.md | 78 +
.../extensions-contrib/kafka-emitter.md | 65 +
.../extensions-contrib/materialized-view.md | 136 +
.../extensions-contrib/momentsketch-quantiles.md | 121 +
.../extensions-contrib/moving-average-query.md | 364 ++
.../extensions-contrib/opentsdb-emitter.md | 62 +
.../development/extensions-contrib/prometheus.md | 117 +
.../extensions-contrib/rabbit-stream-ingestion.md | 238 ++
.../development/extensions-contrib/redis-cache.md | 111 +
.../extensions-contrib/spectator-histogram.md | 457 +++
.../development/extensions-contrib/sqlserver.md | 56 +
.../development/extensions-contrib/statsd.md | 75 +
.../extensions-contrib/tdigestsketch-quantiles.md | 175 +
.../development/extensions-contrib/thrift.md | 87 +
.../development/extensions-contrib/time-min-max.md | 104 +
.../extensions-core/approximate-histograms.md | 315 ++
docs/34.0.0/development/extensions-core/avro.md | 64 +
docs/34.0.0/development/extensions-core/azure.md | 96 +
.../development/extensions-core/bloom-filter.md | 175 +
docs/34.0.0/development/extensions-core/catalog.md | 456 +++
.../extensions-core/datasketches-extension.md | 40 +
.../extensions-core/datasketches-hll.md | 155 +
.../extensions-core/datasketches-kll.md | 2 +-
.../extensions-core/datasketches-quantiles.md | 4 +-
.../extensions-core/datasketches-theta.md | 333 ++
.../extensions-core/datasketches-tuple.md | 252 ++
.../development/extensions-core/druid-aws-rds.md | 38 +
.../extensions-core/druid-basic-security.md | 735 ++++
.../development/extensions-core/druid-kerberos.md | 125 +
.../development/extensions-core/druid-lookups.md | 223 ++
.../development/extensions-core/druid-pac4j.md | 63 +
.../34.0.0/development/extensions-core/examples.md | 26 +
docs/34.0.0/development/extensions-core/google.md | 56 +
docs/34.0.0/development/extensions-core/hdfs.md | 166 +
.../development/extensions-core/k8s-jobs.md | 8 +-
.../development/extensions-core/kubernetes.md | 86 +
docs/34.0.0/development/extensions-core/mysql.md | 213 ++
docs/34.0.0/development/extensions-core/orc.md | 84 +
docs/34.0.0/development/extensions-core/parquet.md | 36 +
.../development/extensions-core/postgresql.md | 155 +
.../34.0.0/development/extensions-core/protobuf.md | 321 ++
docs/34.0.0/development/extensions-core/s3.md | 153 +
.../extensions-core/simple-client-sslcontext.md | 52 +
docs/34.0.0/development/extensions-core/stats.md | 236 ++
.../development/extensions-core/test-stats.md | 122 +
docs/{latest => 34.0.0}/development/javascript.md | 2 +-
docs/34.0.0/development/modules.md | 409 +++
docs/34.0.0/development/overview.md | 85 +
docs/34.0.0/development/versioning.md | 46 +
docs/34.0.0/ingestion/concurrent-append-replace.md | 153 +
docs/34.0.0/ingestion/data-formats.md | 1927 +++++++++++
docs/34.0.0/ingestion/faq.md | 84 +
docs/{latest => 34.0.0}/ingestion/hadoop.md | 2 +-
docs/{latest => 34.0.0}/ingestion/index.md | 4 +-
docs/34.0.0/ingestion/ingestion-spec.md | 565 +++
docs/{latest => 34.0.0}/ingestion/input-sources.md | 16 +-
.../ingestion/kafka-ingestion.md | 12 +-
docs/34.0.0/ingestion/kinesis-ingestion.md | 335 ++
docs/34.0.0/ingestion/native-batch-firehose.md | 28 +
docs/34.0.0/ingestion/native-batch-simple-task.md | 185 +
docs/34.0.0/ingestion/native-batch.md | 771 +++++
docs/{latest => 34.0.0}/ingestion/partitioning.md | 6 +-
docs/{latest => 34.0.0}/ingestion/rollup.md | 6 +-
docs/34.0.0/ingestion/schema-design.md | 316 ++
docs/34.0.0/ingestion/schema-model.md | 58 +
docs/34.0.0/ingestion/standalone-realtime.md | 45 +
docs/34.0.0/ingestion/streaming.md | 35 +
docs/{latest => 34.0.0}/ingestion/supervisor.md | 18 +-
docs/{latest => 34.0.0}/ingestion/tasks.md | 1 +
docs/34.0.0/ingestion/tranquility.md | 30 +
docs/34.0.0/misc/papers-and-talks.md | 43 +
.../multi-stage-query/concepts.md | 8 +-
.../multi-stage-query/examples.md | 2 +-
docs/{latest => 34.0.0}/multi-stage-query/index.md | 2 +-
.../multi-stage-query/known-issues.md | 4 +-
.../multi-stage-query/reference.md | 118 +-
docs/34.0.0/multi-stage-query/security.md | 86 +
docs/34.0.0/operations/alerts.md | 37 +
docs/34.0.0/operations/auth-ldap.md | 313 ++
docs/34.0.0/operations/auth.md | 200 ++
.../operations/basic-cluster-tuning.md | 7 +-
.../operations/clean-metadata-store.md | 11 +-
docs/34.0.0/operations/deep-storage-migration.md | 66 +
docs/34.0.0/operations/dump-segment.md | 320 ++
docs/34.0.0/operations/durable-storage.md | 122 +
docs/34.0.0/operations/dynamic-config-provider.md | 78 +
docs/34.0.0/operations/export-metadata.md | 202 ++
docs/34.0.0/operations/high-availability.md | 38 +
docs/34.0.0/operations/http-compression.md | 31 +
docs/34.0.0/operations/insert-segment-to-db.md | 48 +
docs/34.0.0/operations/java.md | 101 +
docs/34.0.0/operations/kubernetes.md | 34 +
docs/34.0.0/operations/metadata-migration.md | 97 +
docs/{latest => 34.0.0}/operations/metrics.md | 124 +-
.../operations/migrate-from-firehose-ingestion.md | 207 ++
docs/34.0.0/operations/mixed-workloads.md | 214 ++
docs/34.0.0/operations/other-hadoop.md | 299 ++
docs/34.0.0/operations/password-provider.md | 55 +
docs/{latest => 34.0.0}/operations/pull-deps.md | 12 +-
docs/34.0.0/operations/request-logging.md | 249 ++
docs/34.0.0/operations/reset-cluster.md | 79 +
docs/34.0.0/operations/rolling-updates.md | 103 +
docs/34.0.0/operations/rule-configuration.md | 357 ++
docs/34.0.0/operations/security-overview.md | 304 ++
.../operations/security-user-auth.md | 6 +-
docs/34.0.0/operations/segment-optimization.md | 106 +
docs/34.0.0/operations/single-server.md | 57 +
docs/34.0.0/operations/tls-support.md | 107 +
docs/34.0.0/operations/use_sbt_to_build_fat_jar.md | 127 +
docs/34.0.0/operations/web-console.md | 182 +
docs/34.0.0/querying/aggregations.md | 647 ++++
docs/34.0.0/querying/arrays.md | 292 ++
docs/34.0.0/querying/caching.md | 119 +
docs/34.0.0/querying/datasource.md | 501 +++
docs/34.0.0/querying/datasourcemetadataquery.md | 62 +
docs/34.0.0/querying/dimensionspecs.md | 550 +++
docs/34.0.0/querying/filters.md | 901 +++++
docs/34.0.0/querying/geo.md | 156 +
docs/34.0.0/querying/granularities.md | 468 +++
docs/{latest => 34.0.0}/querying/groupbyquery.md | 4 +-
docs/34.0.0/querying/having.md | 266 ++
docs/34.0.0/querying/hll-old.md | 141 +
docs/34.0.0/querying/joins.md | 42 +
docs/34.0.0/querying/kafka-extraction-namespace.md | 96 +
docs/34.0.0/querying/limitspec.md | 72 +
docs/34.0.0/querying/lookups-cached-global.md | 387 +++
docs/34.0.0/querying/lookups.md | 340 ++
docs/34.0.0/querying/math-expr.md | 339 ++
docs/34.0.0/querying/multi-value-dimensions.md | 522 +++
docs/34.0.0/querying/multitenancy.md | 91 +
docs/{latest => 34.0.0}/querying/nested-columns.md | 10 +-
docs/34.0.0/querying/post-aggregations.md | 297 ++
docs/{latest => 34.0.0}/querying/query-context.md | 1 +
docs/34.0.0/querying/query-execution.md | 123 +
.../querying/query-from-deep-storage.md | 9 +-
docs/34.0.0/querying/query-processing.md | 48 +
docs/34.0.0/querying/querying.md | 154 +
docs/34.0.0/querying/scan-query.md | 214 ++
docs/34.0.0/querying/searchquery.md | 191 ++
.../querying/segmentmetadataquery.md | 9 +-
docs/34.0.0/querying/select-query.md | 27 +
docs/34.0.0/querying/sorting-orders.md | 59 +
.../querying/sql-aggregations.md | 2 +-
docs/34.0.0/querying/sql-array-functions.md | 66 +
docs/34.0.0/querying/sql-data-types.md | 162 +
docs/{latest => 34.0.0}/querying/sql-functions.md | 21 +-
.../querying/sql-json-functions.md | 9 +-
.../querying/sql-metadata-tables.md | 1 +
.../querying/sql-multivalue-string-functions.md | 66 +
docs/34.0.0/querying/sql-operators.md | 117 +
.../querying/sql-query-context.md | 28 +-
docs/34.0.0/querying/sql-scalar.md | 282 ++
.../{latest => 34.0.0}/querying/sql-translation.md | 5 +-
docs/34.0.0/querying/sql-window-functions.md | 423 +++
docs/{latest => 34.0.0}/querying/sql.md | 27 +
docs/34.0.0/querying/timeboundaryquery.md | 63 +
docs/34.0.0/querying/timeseriesquery.md | 171 +
docs/34.0.0/querying/tips-good-queries.md | 203 ++
docs/34.0.0/querying/topnmetricspec.md | 91 +
docs/34.0.0/querying/topnquery.md | 264 ++
docs/34.0.0/querying/troubleshooting.md | 68 +
docs/{latest => 34.0.0}/querying/using-caching.md | 3 +-
docs/34.0.0/querying/virtual-columns.md | 193 ++
docs/34.0.0/release-info/migr-ansi-sql-null.md | 376 ++
docs/34.0.0/release-info/migr-front-coded-dict.md | 122 +
docs/34.0.0/release-info/migr-mvd-array.md | 260 ++
.../release-info/migr-subquery-limit.md | 2 +-
docs/34.0.0/release-info/migration-guide.md | 47 +
docs/34.0.0/release-info/release-notes.md | 119 +
.../release-info/upgrade-notes.md | 48 +-
docs/{latest => 34.0.0}/tutorials/cluster.md | 10 +-
docs/{latest => 34.0.0}/tutorials/docker.md | 8 +-
docs/{latest => 34.0.0}/tutorials/index.md | 26 +-
docs/34.0.0/tutorials/tutorial-append-data.md | 133 +
.../tutorials/tutorial-batch-hadoop.md | 2 +-
docs/34.0.0/tutorials/tutorial-batch-native.md | 159 +
.../{latest => 34.0.0}/tutorials/tutorial-batch.md | 2 +-
docs/34.0.0/tutorials/tutorial-compaction.md | 190 +
docs/34.0.0/tutorials/tutorial-delete-data.md | 187 +
docs/34.0.0/tutorials/tutorial-extern.md | 206 ++
.../tutorials/tutorial-ingestion-spec.md | 2 +-
docs/34.0.0/tutorials/tutorial-jdbc.md | 34 +
docs/34.0.0/tutorials/tutorial-kafka.md | 289 ++
docs/34.0.0/tutorials/tutorial-kerberos-hadoop.md | 123 +
docs/34.0.0/tutorials/tutorial-latest-by.md | 233 ++
docs/34.0.0/tutorials/tutorial-msq-convert-spec.md | 176 +
docs/34.0.0/tutorials/tutorial-msq-extern.md | 151 +
.../tutorials/tutorial-query-deep-storage.md | 21 +-
docs/34.0.0/tutorials/tutorial-query.md | 246 ++
docs/34.0.0/tutorials/tutorial-retention.md | 115 +
docs/34.0.0/tutorials/tutorial-rollup.md | 162 +
docs/34.0.0/tutorials/tutorial-sketches-theta.md | 262 ++
docs/34.0.0/tutorials/tutorial-sql-null.md | 216 ++
docs/34.0.0/tutorials/tutorial-sql-query-view.md | 197 ++
docs/34.0.0/tutorials/tutorial-transform.md | 103 +
docs/34.0.0/tutorials/tutorial-unnest-arrays.md | 655 ++++
docs/34.0.0/tutorials/tutorial-update-data.md | 248 ++
.../api-reference/automatic-compaction-api.md | 2 +-
docs/latest/api-reference/data-management-api.md | 59 +-
.../api-reference/dynamic-configuration-api.md | 10 +-
docs/latest/api-reference/service-status-api.md | 134 +-
docs/latest/api-reference/sql-api.md | 114 +-
docs/latest/api-reference/sql-ingestion-api.md | 29 +-
docs/latest/api-reference/tasks-api.md | 6 +-
docs/latest/configuration/index.md | 386 ++-
docs/latest/configuration/logging.md | 4 +-
.../latest/data-management/automatic-compaction.md | 6 +-
docs/latest/data-management/delete.md | 46 +-
docs/latest/data-management/manual-compaction.md | 2 +-
.../latest/design/extensions-contrib/dropwizard.md | 571 +--
docs/latest/design/storage.md | 2 +-
.../extensions-contrib/compressed-big-decimal.md | 47 +-
.../extensions-core/datasketches-kll.md | 2 +-
.../extensions-core/datasketches-quantiles.md | 4 +-
.../latest/development/extensions-core/k8s-jobs.md | 8 +-
docs/latest/development/javascript.md | 2 +-
docs/latest/ingestion/hadoop.md | 2 +-
docs/latest/ingestion/index.md | 4 +-
docs/latest/ingestion/input-sources.md | 16 +-
docs/latest/ingestion/kafka-ingestion.md | 12 +-
docs/latest/ingestion/partitioning.md | 6 +-
docs/latest/ingestion/rollup.md | 6 +-
docs/latest/ingestion/supervisor.md | 18 +-
docs/latest/ingestion/tasks.md | 1 +
docs/latest/multi-stage-query/concepts.md | 8 +-
docs/latest/multi-stage-query/examples.md | 2 +-
docs/latest/multi-stage-query/index.md | 2 +-
docs/latest/multi-stage-query/known-issues.md | 4 +-
docs/latest/multi-stage-query/reference.md | 118 +-
docs/latest/operations/basic-cluster-tuning.md | 7 +-
docs/latest/operations/clean-metadata-store.md | 11 +-
docs/latest/operations/metrics.md | 124 +-
docs/latest/operations/pull-deps.md | 12 +-
docs/latest/operations/security-user-auth.md | 6 +-
docs/latest/querying/groupbyquery.md | 4 +-
docs/latest/querying/nested-columns.md | 10 +-
docs/latest/querying/query-context.md | 1 +
docs/latest/querying/query-from-deep-storage.md | 9 +-
docs/latest/querying/segmentmetadataquery.md | 9 +-
docs/latest/querying/sql-aggregations.md | 2 +-
docs/latest/querying/sql-functions.md | 21 +-
docs/latest/querying/sql-json-functions.md | 9 +-
docs/latest/querying/sql-metadata-tables.md | 1 +
docs/latest/querying/sql-query-context.md | 28 +-
docs/latest/querying/sql-translation.md | 5 +-
docs/latest/querying/sql.md | 27 +
docs/latest/querying/using-caching.md | 3 +-
docs/latest/release-info/migr-subquery-limit.md | 2 +-
docs/latest/release-info/release-notes.md | 264 +-
docs/latest/release-info/upgrade-notes.md | 48 +-
docs/latest/tutorials/cluster.md | 10 +-
docs/latest/tutorials/docker.md | 8 +-
docs/latest/tutorials/index.md | 26 +-
docs/latest/tutorials/tutorial-batch-hadoop.md | 2 +-
docs/latest/tutorials/tutorial-batch.md | 2 +-
docs/latest/tutorials/tutorial-ingestion-spec.md | 2 +-
.../tutorials/tutorial-query-deep-storage.md | 21 +-
published_versions/data/example-manifests-v2.tsv | 2 +
published_versions/data/example-manifests.tsv | 2 +
published_versions/data/wikipedia.json.gz | Bin 0 -> 2016532 bytes
static/js/version.js | 8 +-
450 files changed, 42856 insertions(+), 1941 deletions(-)
diff --git a/docs/34.0.0/api-reference/api-reference.md b/docs/34.0.0/api-reference/api-reference.md
new file mode 100644
index 000000000..dd4a4ab63
--- /dev/null
+++ b/docs/34.0.0/api-reference/api-reference.md
@@ -0,0 +1,44 @@
+---
+id: api-reference
+title: API reference
+sidebar_label: Overview
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This topic is an index to the Apache Druid API documentation.
+
+## HTTP APIs
+* [Druid SQL queries](./sql-api.md) to submit SQL queries using the Druid SQL API.
+* [SQL-based ingestion](./sql-ingestion-api.md) to submit SQL-based batch ingestion requests.
+* [JSON querying](./json-querying-api.md) to submit JSON-based native queries.
+* [Tasks](./tasks-api.md) to manage data ingestion operations.
+* [Supervisors](./supervisor-api.md) to manage supervisors for data ingestion lifecycle and data processing.
+* [Retention rules](./retention-rules-api.md) to define and manage data retention rules across datasources.
+* [Data management](./data-management-api.md) to manage data segments.
+* [Automatic compaction](./automatic-compaction-api.md) to optimize segment sizes after ingestion.
+* [Lookups](./lookups-api.md) to manage and modify key-value datasources.
+* [Service status](./service-status-api.md) to monitor components within the Druid cluster.
+* [Dynamic configuration](./dynamic-configuration-api.md) to configure the behavior of the Coordinator and Overlord processes.
+* [Legacy metadata](./legacy-metadata-api.md) to retrieve datasource metadata.
+
+## Java APIs
+* [SQL JDBC driver](./sql-jdbc.md) to connect to Druid and make Druid SQL queries using the Avatica JDBC driver.
\ No newline at end of file
diff --git a/docs/latest/api-reference/automatic-compaction-api.md b/docs/34.0.0/api-reference/automatic-compaction-api.md
similarity index 99%
copy from docs/latest/api-reference/automatic-compaction-api.md
copy to docs/34.0.0/api-reference/automatic-compaction-api.md
index ae0595b00..f3744a45f 100644
--- a/docs/latest/api-reference/automatic-compaction-api.md
+++ b/docs/34.0.0/api-reference/automatic-compaction-api.md
@@ -29,7 +29,7 @@ import TabItem from '@theme/TabItem';
This topic describes the status and configuration API endpoints for [automatic compaction using Coordinator duties](../data-management/automatic-compaction.md#auto-compaction-using-coordinator-duties) in Apache Druid. You can configure automatic compaction in the Druid web console or API.
-:::info Experimental
+:::info[Experimental]
Instead of the automatic compaction API, you can use the supervisor API to submit auto-compaction jobs using compaction supervisors. For more information, see [Auto-compaction using compaction supervisors](../data-management/automatic-compaction.md#auto-compaction-using-compaction-supervisors).
diff --git a/docs/latest/api-reference/data-management-api.md b/docs/34.0.0/api-reference/data-management-api.md
similarity index 81%
copy from docs/latest/api-reference/data-management-api.md
copy to docs/34.0.0/api-reference/data-management-api.md
index d7156376c..fe37c6a81 100644
--- a/docs/latest/api-reference/data-management-api.md
+++ b/docs/34.0.0/api-reference/data-management-api.md
@@ -34,7 +34,8 @@ Replace it with the information for your deployment.
For example, use `http://localhost:8888` for quickstart deployments.
:::info
-Avoid using indexing or kill tasks and these APIs at the same time for the same datasource and time chunk.
+- Coordinator APIs for data management are now deprecated. Use new APIs served by the Overlord instead.
+- Do not use these APIs while an indexing task or kill task is in progress for the same datasource and interval.
:::
## Segment management
@@ -46,6 +47,9 @@ When you use these APIs concurrently with an indexing task or a kill task, the b
Druid terminates some segments and marks others as used.
Furthermore, it is possible that all segments could be unused, yet an indexing task might still be able to read data from these segments and complete successfully.
+All of the following APIs, except [Segment deletion](#segment-deletion) are served by the Overlord as it is the service responsible for performing actions on segment metadata on behalf of indexing tasks.
+This makes it the single source of truth for segment metadata, thus ensuring a consistent view across the Druid cluster and allowing the Overlord to cache metadata to improve performance.
+
### Segment IDs
You must provide segment IDs when using many of the endpoints described in this topic.
@@ -59,10 +63,11 @@ This is a "soft delete" of the segment from Historicals.
To undo this action, [mark the segment used](#mark-a-single-segment-as-used).
Note that this endpoint returns an HTTP `200 OK` response code even if the segment ID or datasource doesn't exist.
+Check the response payload to confirm if any segment was actually updated.
#### URL
-`DELETE` `/druid/coordinator/v1/datasources/{datasource}/segments/{segmentId}`
+`DELETE` `/druid/indexer/v1/datasources/{datasource}/segments/{segmentId}`
#### Header
@@ -97,7 +102,7 @@ The following example updates the segment `wikipedia_hour_2015-09-12T16:00:00.00
```shell
-curl --request DELETE "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/datasources/wikipedia_hour/segments/wikipedia_hour_2015-09-12T16:00:00.000Z_2015-09-12T17:00:00.000Z_2023-08-10T04:12:03.860Z" \
+curl --request DELETE "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/datasources/wikipedia_hour/segments/wikipedia_hour_2015-09-12T16:00:00.000Z_2015-09-12T17:00:00.000Z_2023-08-10T04:12:03.860Z" \
--header 'Content-Type: application/json' \
--header 'Accept: application/json, text/plain'
```
@@ -107,7 +112,7 @@ curl --request DELETE "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/datasou
```HTTP
-DELETE /druid/coordinator/v1/datasources/wikipedia_hour/segments/wikipedia_hour_2015-09-12T16:00:00.000Z_2015-09-12T17:00:00.000Z_2023-08-10T04:12:03.860Z HTTP/1.1
+DELETE /druid/indexer/v1/datasources/wikipedia_hour/segments/wikipedia_hour_2015-09-12T16:00:00.000Z_2015-09-12T17:00:00.000Z_2023-08-10T04:12:03.860Z HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
Content-Type: application/json
Accept: application/json, text/plain
@@ -123,7 +128,8 @@ Accept: application/json, text/plain
```json
{
- "segmentStateChanged": true
+ "segmentStateChanged": true,
+ "numChangedSegments": 1
}
```
</details>
@@ -134,7 +140,7 @@ Marks the state of a segment as used, using the segment ID.
#### URL
-`POST` `/druid/coordinator/v1/datasources/segments/{segmentId}`
+`POST` `/druid/indexer/v1/datasources/{datasource}/segments/{segmentId}`
#### Header
@@ -169,7 +175,7 @@ The following example updates the segment with ID `wikipedia_hour_2015-09-12T18:
```shell
-curl --request POST "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/datasources/wikipedia_hour/segments/wikipedia_hour_2015-09-12T18:00:00.000Z_2015-09-12T19:00:00.000Z_2023-08-10T04:12:03.860Z" \
+curl --request POST "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/datasources/wikipedia_hour/segments/wikipedia_hour_2015-09-12T18:00:00.000Z_2015-09-12T19:00:00.000Z_2023-08-10T04:12:03.860Z" \
--header 'Content-Type: application/json' \
--header 'Accept: application/json, text/plain'
```
@@ -179,7 +185,7 @@ curl --request POST "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/datasourc
```HTTP
-POST /druid/coordinator/v1/datasources/wikipedia_hour/segments/wikipedia_hour_2015-09-12T18:00:00.000Z_2015-09-12T19:00:00.000Z_2023-08-10T04:12:03.860Z HTTP/1.1
+POST /druid/indexer/v1/datasources/wikipedia_hour/segments/wikipedia_hour_2015-09-12T18:00:00.000Z_2015-09-12T19:00:00.000Z_2023-08-10T04:12:03.860Z HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
Content-Type: application/json
Accept: application/json, text/plain
@@ -195,7 +201,8 @@ Accept: application/json, text/plain
```json
{
- "segmentStateChanged": true
+ "segmentStateChanged": true,
+ "numChangedSegments": 1
}
```
</details>
@@ -211,14 +218,14 @@ within the specified interval that match the optional list of versions; partiall
#### URL
-`POST` `/druid/coordinator/v1/datasources/{datasource}/markUnused`
+`POST` `/druid/indexer/v1/datasources/{datasource}/markUnused`
#### Request body
The group of segments is sent as a JSON request payload that accepts the following properties:
|Property|Description|Required|Example|
-|----------|-------------|---------|---------|
+|--------|-----------|--------|-------|
|`interval`|ISO 8601 segments interval.|Yes, if `segmentIds` is not specified.|`"2015-09-12T03:00:00.000Z/2015-09-12T05:00:00.000Z"`|
|`segmentIds`|List of segment IDs.|Yes, if `interval` is not specified.|`["segmentId1", "segmentId2"]`|
|`versions`|List of segment versions. Must be provided with `interval`.|No.|`["2024-03-14T16:00:04.086Z", ""2024-03-12T16:00:04.086Z"]`|
@@ -259,7 +266,7 @@ The following example marks two segments from the `wikipedia_hour` datasource un
```shell
-curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/datasources/wikipedia_hour/markUnused" \
+curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/datasources/wikipedia_hour/markUnused" \
--header 'Content-Type: application/json' \
--data '{
"segmentIds": [
@@ -274,7 +281,7 @@ curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/datasources/wikipedia_ho
```HTTP
-POST /druid/coordinator/v1/datasources/wikipedia_hour/markUnused HTTP/1.1
+POST /druid/indexer/v1/datasources/wikipedia_hour/markUnused HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
Content-Type: application/json
Content-Length: 230
@@ -313,14 +320,14 @@ within the specified interval that match the optional list of versions; partiall
#### URL
-`POST` `/druid/coordinator/v1/datasources/{datasource}/markUsed`
+`POST` `/druid/indexer/v1/datasources/{datasource}/markUsed`
#### Request body
The group of segments is sent as a JSON request payload that accepts the following properties:
|Property|Description|Required|Example|
-|----------|-------------|---------|---------|
+|--------|-----------|--------|-------|
|`interval`|ISO 8601 segments interval.|Yes, if `segmentIds` is not specified.|`"2015-09-12T03:00:00.000Z/2015-09-12T05:00:00.000Z"`|
|`segmentIds`|List of segment IDs.|Yes, if `interval` is not specified.|`["segmentId1", "segmentId2"]`|
|`versions`|List of segment versions. Must be provided with `interval`.|No.|`["2024-03-14T16:00:04.086Z", ""2024-03-12T16:00:04.086Z"]`|
@@ -361,7 +368,7 @@ The following example marks two segments from the `wikipedia_hour` datasource us
```shell
-curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/datasources/wikipedia_hour/markUsed" \
+curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/datasources/wikipedia_hour/markUsed" \
--header 'Content-Type: application/json' \
--data '{
"segmentIds": [
@@ -376,7 +383,7 @@ curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/datasources/wikipedia_ho
```HTTP
-POST /druid/coordinator/v1/datasources/wikipedia_hour/markUsed HTTP/1.1
+POST /druid/indexer/v1/datasources/wikipedia_hour/markUsed HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
Content-Type: application/json
Content-Length: 230
@@ -410,10 +417,11 @@ Marks the state of all segments of a datasource as unused.
This action performs a "soft delete" of the segments from Historicals.
Note that this endpoint returns an HTTP `200 OK` response code even if the datasource doesn't exist.
+Check the response payload to confirm if any segment was actually updated.
#### URL
-`DELETE` `/druid/coordinator/v1/datasources/{datasource}`
+`DELETE` `/druid/indexer/v1/datasources/{datasource}`
#### Responses
@@ -437,7 +445,7 @@ Note that this endpoint returns an HTTP `200 OK` response code even if the datas
```shell
-curl --request DELETE "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/datasources/wikipedia_hour"
+curl --request DELETE "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/datasources/wikipedia_hour"
```
</TabItem>
@@ -445,7 +453,7 @@ curl --request DELETE "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/datasou
```HTTP
-DELETE /druid/coordinator/v1/datasources/wikipedia_hour HTTP/1.1
+DELETE /druid/indexer/v1/datasources/wikipedia_hour HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
```
@@ -464,16 +472,17 @@ Host: http://ROUTER_IP:ROUTER_PORT
```
</details>
-### Mark all segments used
+### Mark all non-overshadowed segments used
-Marks the state of all unused segments of a datasource as used.
+Marks the state of all unused segments of a datasource as used given that they are not already overshadowed by other segments.
The endpoint returns the number of changed segments.
Note that this endpoint returns an HTTP `200 OK` response code even if the datasource doesn't exist.
+Check the response payload to get the number of segments actually updated.
#### URL
-`POST` `/druid/coordinator/v1/datasources/{datasource}`
+`POST` `/druid/indexer/v1/datasources/{datasource}`
#### Header
@@ -509,7 +518,7 @@ The following example updates all unused segments of `wikipedia_hour` to used.
```shell
-curl --request POST "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/datasources/wikipedia_hour" \
+curl --request POST "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/datasources/wikipedia_hour" \
--header 'Content-Type: application/json' \
--header 'Accept: application/json, text/plain'
```
@@ -519,7 +528,7 @@ curl --request POST "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/datasourc
```HTTP
-POST /druid/coordinator/v1/datasources/wikipedia_hour HTTP/1.1
+POST /druid/indexer/v1/datasources/wikipedia_hour HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
Content-Type: application/json
Accept: application/json, text/plain
diff --git a/docs/latest/api-reference/dynamic-configuration-api.md b/docs/34.0.0/api-reference/dynamic-configuration-api.md
similarity index 98%
copy from docs/latest/api-reference/dynamic-configuration-api.md
copy to docs/34.0.0/api-reference/dynamic-configuration-api.md
index 971aa81d2..cad61e4b8 100644
--- a/docs/latest/api-reference/dynamic-configuration-api.md
+++ b/docs/34.0.0/api-reference/dynamic-configuration-api.md
@@ -106,7 +106,9 @@ Host: http://ROUTER_IP:ROUTER_PORT
"useRoundRobinSegmentAssignment": true,
"smartSegmentLoading": true,
"debugDimensions": null,
- "turboLoadingNodes": []
+ "turboLoadingNodes": [],
+ "cloneServers": {}
+
}
```
@@ -174,7 +176,8 @@ curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config" \
"replicateAfterLoadTimeout": false,
"maxNonPrimaryReplicantsToLoad": 2147483647,
"useRoundRobinSegmentAssignment": true,
- "turboLoadingNodes": []
+ "turboLoadingNodes": [],
+ "cloneServers": {}
}'
```
@@ -206,7 +209,8 @@ Content-Length: 683
"replicateAfterLoadTimeout": false,
"maxNonPrimaryReplicantsToLoad": 2147483647,
"useRoundRobinSegmentAssignment": true,
- "turboLoadingNodes": []
+ "turboLoadingNodes": [],
+ "cloneServers": {}
}
```
diff --git a/docs/34.0.0/api-reference/json-querying-api.md b/docs/34.0.0/api-reference/json-querying-api.md
new file mode 100644
index 000000000..5d03ec8b3
--- /dev/null
+++ b/docs/34.0.0/api-reference/json-querying-api.md
@@ -0,0 +1,925 @@
+---
+id: json-querying-api
+title: JSON querying API
+sidebar_label: JSON querying
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
+<!--
+
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+This topic describes the API endpoints to submit JSON-based [native queries](../querying/querying.md) to Apache Druid.
+
+In this topic, `http://SERVICE_IP:SERVICE_PORT` is a placeholder for the server address of deployment and the service port. For example, on the quickstart configuration, replace `http://ROUTER_IP:ROUTER_PORT` with `http://localhost:8888`.
+
+
+## Submit a query
+
+Submits a JSON-based native query. The body of the request is the native query itself.
+
+Druid supports different types of queries for different use cases. All queries require the following properties:
+* `queryType`: A string representing the type of query. Druid supports the following native query types: `timeseries`, `topN`, `groupBy`, `timeBoundaries`, `segmentMetadata`, `datasourceMetadata`, `scan`, and `search`.
+* `dataSource`: A string or object defining the source of data to query. The most common value is the name of the datasource to query. For more information, see [Datasources](../querying/datasource.md).
+
+For additional properties based on your query type or use case, see [available native queries](../querying/querying.md#available-queries).
+
+### URL
+
+`POST` `/druid/v2`
+
+### Query parameters
+
+* `pretty` (optional)
+ * Druid returns the response in a pretty-printed format using indentation and line breaks.
+
+### Responses
+
+<Tabs>
+
+<TabItem value="1" label="200 SUCCESS">
+
+
+*Successfully submitted query*
+
+</TabItem>
+<TabItem value="2" label="400 BAD REQUEST">
+
+
+*Error thrown due to bad query. Returns a JSON object detailing the error with the following format:*
+
+```json
+{
+ "error": "A well-defined error code.",
+ "errorMessage": "A message with additional details about the error.",
+ "errorClass": "Class of exception that caused this error.",
+ "host": "The host on which the error occurred."
+}
+```
+For more information on possible error messages, see [query execution failures](../querying/querying.md#query-execution-failures).
+
+</TabItem>
+</Tabs>
+
+---
+
+### Example query: `topN`
+
+The following example shows a `topN` query. The query analyzes the `social_media` datasource to return the top five users from the `username` dimension with the highest number of views from the `views` metric.
+
+<Tabs>
+
+<TabItem value="3" label="cURL">
+
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/v2?pretty=null" \
+--header 'Content-Type: application/json' \
+--data '{
+ "queryType": "topN",
+ "dataSource": "social_media",
+ "dimension": "username",
+ "threshold": 5,
+ "metric": "views",
+ "granularity": "all",
+ "aggregations": [
+ {
+ "type": "longSum",
+ "name": "views",
+ "fieldName": "views"
+ }
+ ],
+ "intervals": [
+ "2022-01-01T00:00:00.000/2024-01-01T00:00:00.000"
+ ]
+}'
+```
+</TabItem>
+<TabItem value="4" label="HTTP">
+
+
+```HTTP
+POST /druid/v2?pretty=null HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+Content-Type: application/json
+Content-Length: 336
+
+{
+ "queryType": "topN",
+ "dataSource": "social_media",
+ "dimension": "username",
+ "threshold": 5,
+ "metric": "views",
+ "granularity": "all",
+ "aggregations": [
+ {
+ "type": "longSum",
+ "name": "views",
+ "fieldName": "views"
+ }
+ ],
+ "intervals": [
+ "2022-01-01T00:00:00.000/2024-01-01T00:00:00.000"
+ ]
+}
+```
+
+</TabItem>
+</Tabs>
+
+#### Example response: `topN`
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+[
+ {
+ "timestamp": "2023-07-03T18:49:54.848Z",
+ "result": [
+ {
+ "views": 11591218026,
+ "username": "gus"
+ },
+ {
+ "views": 11578638578,
+ "username": "miette"
+ },
+ {
+ "views": 11561618880,
+ "username": "leon"
+ },
+ {
+ "views": 11552609824,
+ "username": "mia"
+ },
+ {
+ "views": 11551537517,
+ "username": "milton"
+ }
+ ]
+ }
+]
+ ```
+</details>
+
+### Example query: `groupBy`
+
+The following example submits a JSON query of the `groupBy` type to retrieve the `username` with the highest votes to posts ratio from the `social_media` datasource.
+
+In this query:
+* The `upvoteSum` aggregation calculates the sum of the `upvotes` for each user.
+* The `postCount` aggregation calculates the sum of posts for each user.
+* The `upvoteToPostRatio` is a post-aggregation of the `upvoteSum` and the `postCount`, divided to calculate the ratio.
+* The result is sorted based on the `upvoteToPostRatio` in descending order.
+
+<Tabs>
+
+<TabItem value="5" label="cURL">
+
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/v2" \
+--header 'Content-Type: application/json' \
+--data '{
+ "queryType": "groupBy",
+ "dataSource": "social_media",
+ "dimensions": ["username"],
+ "granularity": "all",
+ "aggregations": [
+ { "type": "doubleSum", "name": "upvoteSum", "fieldName": "upvotes" },
+ { "type": "count", "name": "postCount", "fieldName": "post_title" }
+ ],
+ "postAggregations": [
+ {
+ "type": "arithmetic",
+ "name": "upvoteToPostRatio",
+ "fn": "/",
+ "fields": [
+ { "type": "fieldAccess", "name": "upvoteSum", "fieldName": "upvoteSum" },
+ { "type": "fieldAccess", "name": "postCount", "fieldName": "postCount" }
+ ]
+ }
+ ],
+ "intervals": ["2022-01-01T00:00:00.000/2024-01-01T00:00:00.000"],
+ "limitSpec": {
+ "type": "default",
+ "limit": 1,
+ "columns": [
+ { "dimension": "upvoteToPostRatio", "direction": "descending" }
+ ]
+ }
+}'
+```
+
+</TabItem>
+
+<TabItem value="6" label="HTTP">
+
+```HTTP
+POST /druid/v2?pretty=null HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+Content-Type: application/json
+Content-Length: 817
+
+{
+ "queryType": "groupBy",
+ "dataSource": "social_media",
+ "dimensions": ["username"],
+ "granularity": "all",
+ "aggregations": [
+ { "type": "doubleSum", "name": "upvoteSum", "fieldName": "upvotes" },
+ { "type": "count", "name": "postCount", "fieldName": "post_title" }
+ ],
+ "postAggregations": [
+ {
+ "type": "arithmetic",
+ "name": "upvoteToPostRatio",
+ "fn": "/",
+ "fields": [
+ { "type": "fieldAccess", "name": "upvoteSum", "fieldName": "upvoteSum" },
+ { "type": "fieldAccess", "name": "postCount", "fieldName": "postCount" }
+ ]
+ }
+ ],
+ "intervals": ["2022-01-01T00:00:00.000/2024-01-01T00:00:00.000"],
+ "limitSpec": {
+ "type": "default",
+ "limit": 1,
+ "columns": [
+ { "dimension": "upvoteToPostRatio", "direction": "descending" }
+ ]
+ }
+}
+```
+
+</TabItem>
+</Tabs>
+
+#### Example response: `groupBy`
+
+<details>
+ <summary>View the response</summary>
+
+```json
+[
+ {
+ "version": "v1",
+ "timestamp": "2022-01-01T00:00:00.000Z",
+ "event": {
+ "upvoteSum": 8.0419541E7,
+ "upvoteToPostRatio": 69.53014661762697,
+ "postCount": 1156614,
+ "username": "miette"
+ }
+ }
+]
+```
+</details>
+
+## Get segment information for query
+
+Retrieves an array that contains objects with segment information, including the server locations associated with the query provided in the request body.
+
+### URL
+
+`POST` `/druid/v2/candidates`
+
+### Query parameters
+
+* `pretty` (optional)
+ * Druid returns the response in a pretty-printed format using indentation and line breaks.
+
+### Responses
+
+<Tabs>
+
+<TabItem value="7" label="200 SUCCESS">
+
+
+*Successfully retrieved segment information*
+
+</TabItem>
+<TabItem value="8" label="400 BAD REQUEST">
+
+
+*Error thrown due to bad query. Returns a JSON object detailing the error with the following format:*
+
+```json
+{
+ "error": "A well-defined error code.",
+ "errorMessage": "A message with additional details about the error.",
+ "errorClass": "Class of exception that caused this error.",
+ "host": "The host on which the error occurred."
+}
+```
+
+For more information on possible error messages, see [query execution failures](../querying/querying.md#query-execution-failures).
+
+</TabItem>
+</Tabs>
+
+---
+
+### Sample request
+
+<Tabs>
+
+<TabItem value="9" label="cURL">
+
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/v2/candidates" \
+--header 'Content-Type: application/json' \
+--data '{
+ "queryType": "topN",
+ "dataSource": "social_media",
+ "dimension": "username",
+ "threshold": 5,
+ "metric": "views",
+ "granularity": "all",
+ "aggregations": [
+ {
+ "type": "longSum",
+ "name": "views",
+ "fieldName": "views"
+ }
+ ],
+ "intervals": [
+ "2022-01-01T00:00:00.000/2024-01-01T00:00:00.000"
+ ]
+}'
+```
+
+</TabItem>
+<TabItem value="10" label="HTTP">
+
+
+```HTTP
+POST /druid/v2/candidates HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+Content-Type: application/json
+Content-Length: 336
+
+{
+ "queryType": "topN",
+ "dataSource": "social_media",
+ "dimension": "username",
+ "threshold": 5,
+ "metric": "views",
+ "granularity": "all",
+
+ "aggregations": [
+ {
+ "type": "longSum",
+ "name": "views",
+ "fieldName": "views"
+ }
+ ],
+ "intervals": [
+ "2020-01-01T00:00:00.000/2024-01-01T00:00:00.000"
+ ]
+}
+```
+
+</TabItem>
+</Tabs>
+
+### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+[
+ {
+ "interval": "2023-07-03T18:00:00.000Z/2023-07-03T19:00:00.000Z",
+ "version": "2023-07-03T18:51:18.905Z",
+ "partitionNumber": 0,
+ "size": 21563693,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-03T19:00:00.000Z/2023-07-03T20:00:00.000Z",
+ "version": "2023-07-03T19:00:00.657Z",
+ "partitionNumber": 0,
+ "size": 6057236,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-05T21:00:00.000Z/2023-07-05T22:00:00.000Z",
+ "version": "2023-07-05T21:09:58.102Z",
+ "partitionNumber": 0,
+ "size": 223926186,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-05T21:00:00.000Z/2023-07-05T22:00:00.000Z",
+ "version": "2023-07-05T21:09:58.102Z",
+ "partitionNumber": 1,
+ "size": 20244827,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-05T22:00:00.000Z/2023-07-05T23:00:00.000Z",
+ "version": "2023-07-05T22:00:00.524Z",
+ "partitionNumber": 0,
+ "size": 104628051,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-05T22:00:00.000Z/2023-07-05T23:00:00.000Z",
+ "version": "2023-07-05T22:00:00.524Z",
+ "partitionNumber": 1,
+ "size": 1603995,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-05T23:00:00.000Z/2023-07-06T00:00:00.000Z",
+ "version": "2023-07-05T23:21:55.242Z",
+ "partitionNumber": 0,
+ "size": 181506843,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T00:00:00.000Z/2023-07-06T01:00:00.000Z",
+ "version": "2023-07-06T00:02:08.498Z",
+ "partitionNumber": 0,
+ "size": 9170974,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T00:00:00.000Z/2023-07-06T01:00:00.000Z",
+ "version": "2023-07-06T00:02:08.498Z",
+ "partitionNumber": 1,
+ "size": 23969632,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T01:00:00.000Z/2023-07-06T02:00:00.000Z",
+ "version": "2023-07-06T01:13:53.982Z",
+ "partitionNumber": 0,
+ "size": 599895,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T01:00:00.000Z/2023-07-06T02:00:00.000Z",
+ "version": "2023-07-06T01:13:53.982Z",
+ "partitionNumber": 1,
+ "size": 1627041,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T02:00:00.000Z/2023-07-06T03:00:00.000Z",
+ "version": "2023-07-06T02:55:50.701Z",
+ "partitionNumber": 0,
+ "size": 629753,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T02:00:00.000Z/2023-07-06T03:00:00.000Z",
+ "version": "2023-07-06T02:55:50.701Z",
+ "partitionNumber": 1,
+ "size": 1342360,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T04:00:00.000Z/2023-07-06T05:00:00.000Z",
+ "version": "2023-07-06T04:02:36.562Z",
+ "partitionNumber": 0,
+ "size": 2131434,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T05:00:00.000Z/2023-07-06T06:00:00.000Z",
+ "version": "2023-07-06T05:23:27.856Z",
+ "partitionNumber": 0,
+ "size": 797161,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T05:00:00.000Z/2023-07-06T06:00:00.000Z",
+ "version": "2023-07-06T05:23:27.856Z",
+ "partitionNumber": 1,
+ "size": 1176858,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T06:00:00.000Z/2023-07-06T07:00:00.000Z",
+ "version": "2023-07-06T06:46:34.638Z",
+ "partitionNumber": 0,
+ "size": 2148760,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T07:00:00.000Z/2023-07-06T08:00:00.000Z",
+ "version": "2023-07-06T07:38:28.050Z",
+ "partitionNumber": 0,
+ "size": 2040748,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T08:00:00.000Z/2023-07-06T09:00:00.000Z",
+ "version": "2023-07-06T08:27:31.407Z",
+ "partitionNumber": 0,
+ "size": 678723,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T08:00:00.000Z/2023-07-06T09:00:00.000Z",
+ "version": "2023-07-06T08:27:31.407Z",
+ "partitionNumber": 1,
+ "size": 1437866,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T10:00:00.000Z/2023-07-06T11:00:00.000Z",
+ "version": "2023-07-06T10:02:42.079Z",
+ "partitionNumber": 0,
+ "size": 1671296,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T11:00:00.000Z/2023-07-06T12:00:00.000Z",
+ "version": "2023-07-06T11:27:23.902Z",
+ "partitionNumber": 0,
+ "size": 574893,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T11:00:00.000Z/2023-07-06T12:00:00.000Z",
+ "version": "2023-07-06T11:27:23.902Z",
+ "partitionNumber": 1,
+ "size": 1427384,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T12:00:00.000Z/2023-07-06T13:00:00.000Z",
+ "version": "2023-07-06T12:52:00.846Z",
+ "partitionNumber": 0,
+ "size": 2115172,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T14:00:00.000Z/2023-07-06T15:00:00.000Z",
+ "version": "2023-07-06T14:32:33.926Z",
+ "partitionNumber": 0,
+ "size": 589108,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T14:00:00.000Z/2023-07-06T15:00:00.000Z",
+ "version": "2023-07-06T14:32:33.926Z",
+ "partitionNumber": 1,
+ "size": 1392649,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T15:00:00.000Z/2023-07-06T16:00:00.000Z",
+ "version": "2023-07-06T15:53:25.467Z",
+ "partitionNumber": 0,
+ "size": 2037851,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T16:00:00.000Z/2023-07-06T17:00:00.000Z",
+ "version": "2023-07-06T16:02:26.568Z",
+ "partitionNumber": 0,
+ "size": 230400650,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T16:00:00.000Z/2023-07-06T17:00:00.000Z",
+ "version": "2023-07-06T16:02:26.568Z",
+ "partitionNumber": 1,
+ "size": 38209056,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ },
+ {
+ "interval": "2023-07-06T17:00:00.000Z/2023-07-06T18:00:00.000Z",
+ "version": "2023-07-06T17:00:02.391Z",
+ "partitionNumber": 0,
+ "size": 211099463,
+ "locations": [
+ {
+ "name": "localhost:8083",
+ "host": "localhost:8083",
+ "hostAndTlsPort": null,
+ "maxSize": 300000000000,
+ "type": "historical",
+ "tier": "_default_tier",
+ "priority": 0
+ }
+ ]
+ }
+]
+ ```
+</details>
diff --git a/docs/34.0.0/api-reference/legacy-metadata-api.md b/docs/34.0.0/api-reference/legacy-metadata-api.md
new file mode 100644
index 000000000..d22be18a7
--- /dev/null
+++ b/docs/34.0.0/api-reference/legacy-metadata-api.md
@@ -0,0 +1,344 @@
+---
+id: legacy-metadata-api
+title: Legacy metadata API
+sidebar_label: Legacy metadata
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+This document describes the legacy API endpoints to retrieve datasource metadata from Apache Druid. Use the [SQL metadata tables](../querying/sql-metadata-tables.md) to retrieve datasource metadata instead.
+
+## Segment loading
+
+`GET /druid/coordinator/v1/loadstatus`
+
+Returns the percentage of segments actually loaded in the cluster versus segments that should be loaded in the cluster.
+
+`GET /druid/coordinator/v1/loadstatus?simple`
+
+Returns the number of segments left to load until segments that should be loaded in the cluster are available for queries. This does not include segment replication counts.
+
+`GET /druid/coordinator/v1/loadstatus?full`
+
+Returns the number of segments left to load in each tier until segments that should be loaded in the cluster are all available. This includes segment replication counts.
+
+`GET /druid/coordinator/v1/loadstatus?full&computeUsingClusterView`
+
+Returns the number of segments not yet loaded for each tier until all segments loading in the cluster are available.
+The result includes segment replication counts. It also factors in the number of available nodes that are of a service type that can load the segment when computing the number of segments remaining to load.
+A segment is considered fully loaded when:
+- Druid has replicated it the number of times configured in the corresponding load rule.
+- Or the number of replicas for the segment in each tier where it is configured to be replicated equals the available nodes of a service type that are currently allowed to load the segment in the tier.
+
+`GET /druid/coordinator/v1/loadqueue`
+
+Returns the ids of segments to load and drop for each Historical process.
+
+`GET /druid/coordinator/v1/loadqueue?simple`
+
+Returns the number of segments to load and drop, as well as the total segment load and drop size in bytes for each Historical process.
+
+`GET /druid/coordinator/v1/loadqueue?full`
+
+Returns the serialized JSON of segments to load and drop for each Historical process.
+
+## Segment loading by datasource
+
+Note that all _interval_ query parameters are ISO 8601 strings—for example, 2016-06-27/2016-06-28.
+Also note that these APIs only guarantees that the segments are available at the time of the call.
+Segments can still become missing because of historical process failures or any other reasons afterward.
+
+`GET /druid/coordinator/v1/datasources/{dataSourceName}/loadstatus?forceMetadataRefresh={boolean}&interval={myInterval}`
+
+Returns the percentage of segments actually loaded in the cluster versus segments that should be loaded in the cluster for the given
+datasource over the given interval (or last 2 weeks if interval is not given). `forceMetadataRefresh` is required to be set.
+* Setting `forceMetadataRefresh` to true will force the coordinator to poll latest segment metadata from the metadata store
+(Note: `forceMetadataRefresh=true` refreshes Coordinator's metadata cache of all datasources. This can be a heavy operation in terms
+of the load on the metadata store but can be necessary to make sure that we verify all the latest segments' load status)
+* Setting `forceMetadataRefresh` to false will use the metadata cached on the coordinator from the last force/periodic refresh.
+If no used segments are found for the given inputs, this API returns `204 No Content`
+
+`GET /druid/coordinator/v1/datasources/{dataSourceName}/loadstatus?simple&forceMetadataRefresh={boolean}&interval={myInterval}`
+
+Returns the number of segments left to load until segments that should be loaded in the cluster are available for the given datasource
+over the given interval (or last 2 weeks if interval is not given). This does not include segment replication counts. `forceMetadataRefresh` is required to be set.
+* Setting `forceMetadataRefresh` to true will force the coordinator to poll latest segment metadata from the metadata store
+(Note: `forceMetadataRefresh=true` refreshes Coordinator's metadata cache of all datasources. This can be a heavy operation in terms
+of the load on the metadata store but can be necessary to make sure that we verify all the latest segments' load status)
+* Setting `forceMetadataRefresh` to false will use the metadata cached on the coordinator from the last force/periodic refresh.
+If no used segments are found for the given inputs, this API returns `204 No Content`
+
+`GET /druid/coordinator/v1/datasources/{dataSourceName}/loadstatus?full&forceMetadataRefresh={boolean}&interval={myInterval}`
+
+Returns the number of segments left to load in each tier until segments that should be loaded in the cluster are all available for the given datasource over the given interval (or last 2 weeks if interval is not given). This includes segment replication counts. `forceMetadataRefresh` is required to be set.
+* Setting `forceMetadataRefresh` to true will force the coordinator to poll latest segment metadata from the metadata store
+(Note: `forceMetadataRefresh=true` refreshes Coordinator's metadata cache of all datasources. This can be a heavy operation in terms
+of the load on the metadata store but can be necessary to make sure that we verify all the latest segments' load status)
+* Setting `forceMetadataRefresh` to false will use the metadata cached on the coordinator from the last force/periodic refresh.
+
+You can pass the optional query parameter `computeUsingClusterView` to factor in the available cluster services when calculating
+the segments left to load. See [Coordinator Segment Loading](#segment-loading) for details.
+If no used segments are found for the given inputs, this API returns `204 No Content`
+
+## Metadata store information
+
+:::info
+ Note: Much of this information is available in a simpler, easier-to-use form through the Druid SQL
+ [`sys.segments`](../querying/sql-metadata-tables.md#segments-table) table.
+:::
+
+`GET /druid/coordinator/v1/metadata/segments`
+
+Returns a list of all segments for each datasource enabled in the cluster.
+
+`GET /druid/coordinator/v1/metadata/segments?datasources={dataSourceName1}&datasources={dataSourceName2}`
+
+Returns a list of all segments for one or more specific datasources enabled in the cluster.
+
+`GET /druid/coordinator/v1/metadata/segments?includeOvershadowedStatus`
+
+Returns a list of all segments for each datasource with the full segment metadata and an extra field `overshadowed`.
+
+`GET /druid/coordinator/v1/metadata/segments?includeOvershadowedStatus&includeRealtimeSegments`
+
+Returns a list of all published and realtime segments for each datasource with the full segment metadata and extra fields `overshadowed`,`realtime` & `numRows`. Realtime segments are returned only when `druid.centralizedDatasourceSchema.enabled` is set on the Coordinator.
+
+`GET /druid/coordinator/v1/metadata/segments?includeOvershadowedStatus&datasources={dataSourceName1}&datasources={dataSourceName2}`
+
+Returns a list of all segments for one or more specific datasources with the full segment metadata and an extra field `overshadowed`.
+
+`GET /druid/coordinator/v1/metadata/segments?includeOvershadowedStatus&includeRealtimeSegments&datasources={dataSourceName1}&datasources={dataSourceName2}`
+
+Returns a list of all published and realtime segments for the specified datasources with the full segment metadata and extra fields `overshadwed`,`realtime` & `numRows`. Realtime segments are returned only when `druid.centralizedDatasourceSchema.enabled` is set on the Coordinator.
+
+`GET /druid/coordinator/v1/metadata/datasources`
+
+Returns a list of the names of datasources with at least one used segment in the cluster, retrieved from the metadata database. Users should call this API to get the eventual state that the system will be in.
+
+`GET /druid/coordinator/v1/metadata/datasources?includeUnused`
+
+Returns a list of the names of datasources, regardless of whether there are used segments belonging to those datasources in the cluster or not.
+
+`GET /druid/coordinator/v1/metadata/datasources?includeDisabled`
+
+Returns a list of the names of datasources, regardless of whether the datasource is disabled or not.
+
+`GET /druid/coordinator/v1/metadata/datasources?full`
+
+Returns a list of all datasources with at least one used segment in the cluster. Returns all metadata about those datasources as stored in the metadata store.
+
+`GET /druid/coordinator/v1/metadata/datasources/{dataSourceName}`
+
+Returns full metadata for a datasource as stored in the metadata store.
+
+`GET /druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments`
+
+Returns a list of all segments for a datasource as stored in the metadata store.
+
+`GET /druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments?full`
+
+Returns a list of all segments for a datasource with the full segment metadata as stored in the metadata store.
+
+`GET /druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments/{segmentId}`
+
+Returns full segment metadata for a specific segment as stored in the metadata store, if the segment is used. If the
+segment is unused, or is unknown, a 404 response is returned.
+
+`GET /druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments/{segmentId}?includeUnused=true`
+
+Returns full segment metadata for a specific segment as stored in the metadata store. If it is unknown, a 404 response
+is returned.
+
+`GET /druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments`
+
+Returns a list of all segments, overlapping with any of given intervals, for a datasource as stored in the metadata store. Request body is array of string IS0 8601 intervals like `[interval1, interval2,...]`—for example, `["2012-01-01T00:00:00.000/2012-01-03T00:00:00.000", "2012-01-05T00:00:00.000/2012-01-07T00:00:00.000"]`.
+
+`GET /druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments?full`
+
+Returns a list of all segments, overlapping with any of given intervals, for a datasource with the full segment metadata as stored in the metadata store. Request body is array of string ISO 8601 intervals like `[interval1, interval2,...]`—for example, `["2012-01-01T00:00:00.000/2012-01-03T00:00:00.000", "2012-01-05T00:00:00.000/2012-01-07T00:00:00.000"]`.
+
+`POST /druid/coordinator/v1/metadata/dataSourceInformation`
+
+Returns information about the specified datasources, including the datasource schema.
+
+`POST /druid/coordinator/v1/metadata/bootstrapSegments`
+
+Returns information about bootstrap segments for all datasources. The returned set includes all broadcast segments if broadcast rules are configured.
+
+<a name="coordinator-datasources"></a>
+
+## Datasources
+
+Note that all _interval_ URL parameters are ISO 8601 strings delimited by a `_` instead of a `/`—for example, `2016-06-27_2016-06-28`.
+
+`GET /druid/coordinator/v1/datasources`
+
+Returns a list of datasource names found in the cluster as seen by the coordinator. This view is updated every [`druid.coordinator.period`](../configuration/index.md#coordinator-operation).
+
+`GET /druid/coordinator/v1/datasources?simple`
+
+Returns a list of JSON objects containing the name and properties of datasources found in the cluster. Properties include segment count, total segment byte size, replicated total segment byte size, minTime, and maxTime.
+
+`GET /druid/coordinator/v1/datasources?full`
+
+Returns a list of datasource names found in the cluster with all metadata about those datasources.
+
+`GET /druid/coordinator/v1/datasources/{dataSourceName}`
+
+Returns a JSON object containing the name and properties of a datasource. Properties include segment count, total segment byte size, replicated total segment byte size, minTime, and maxTime.
+
+`GET /druid/coordinator/v1/datasources/{dataSourceName}?full`
+
+Returns full metadata for a datasource.
+
+`GET /druid/coordinator/v1/datasources/{dataSourceName}/intervals`
+
+Returns a set of segment intervals.
+
+`GET /druid/coordinator/v1/datasources/{dataSourceName}/intervals?simple`
+
+Returns a map of an interval to a JSON object containing the total byte size of segments and number of segments for that interval.
+
+`GET /druid/coordinator/v1/datasources/{dataSourceName}/intervals?full`
+
+Returns a map of an interval to a map of segment metadata to a set of server names that contain the segment for that interval.
+
+`GET /druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}`
+
+Returns a set of segment ids for an interval.
+
+`GET /druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}?simple`
+
+Returns a map of segment intervals contained within the specified interval to a JSON object containing the total byte size of segments and number of segments for an interval.
+
+`GET /druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}?full`
+
+Returns a map of segment intervals contained within the specified interval to a map of segment metadata to a set of server names that contain the segment for an interval.
+
+`GET /druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}/serverview`
+
+Returns a map of segment intervals contained within the specified interval to information about the servers that contain the segment for an interval.
+
+`GET /druid/coordinator/v1/datasources/{dataSourceName}/segments`
+
+Returns a list of all segments for a datasource in the cluster.
+
+`GET /druid/coordinator/v1/datasources/{dataSourceName}/segments?full`
+
+Returns a list of all segments for a datasource in the cluster with the full segment metadata.
+
+`GET /druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}`
+
+Returns full segment metadata for a specific segment in the cluster.
+
+`GET /druid/coordinator/v1/datasources/{dataSourceName}/tiers`
+
+Return the tiers that a datasource exists in.
+
+## Intervals
+
+Note that all _interval_ URL parameters are ISO 8601 strings delimited by a `_` instead of a `/` as in `2016-06-27_2016-06-28`.
+
+`GET /druid/coordinator/v1/intervals`
+
+Returns all intervals for all datasources with total size and count.
+
+`GET /druid/coordinator/v1/intervals/{interval}`
+
+Returns aggregated total size and count for all intervals that intersect given ISO interval.
+
+`GET /druid/coordinator/v1/intervals/{interval}?simple`
+
+Returns total size and count for each interval within given ISO interval.
+
+`GET /druid/coordinator/v1/intervals/{interval}?full`
+
+Returns total size and count for each datasource for each interval within given ISO interval.
+
+## Server information
+
+`GET /druid/coordinator/v1/servers`
+
+Returns a list of servers URLs using the format `{hostname}:{port}`. Note that
+processes that run with different types will appear multiple times with different
+ports.
+
+`GET /druid/coordinator/v1/servers?simple`
+
+Returns a list of server data objects in which each object has the following keys:
+* `host`: host URL include (`{hostname}:{port}`)
+* `type`: process type (`indexer-executor`, `historical`)
+* `currSize`: storage size currently used
+* `maxSize`: maximum storage size
+* `priority`
+* `tier`
+
+
+## Query server
+
+This section documents the API endpoints for the services that reside on Query servers (Brokers) in the suggested [three-server configuration](../design/architecture.md#druid-servers).
+
+### Broker
+
+#### Datasource information
+
+Note that all _interval_ URL parameters are ISO 8601 strings delimited by a `_` instead of a `/`
+as in `2016-06-27_2016-06-28`.
+
+:::info
+ Note: Much of this information is available in a simpler, easier-to-use form through the Druid SQL
+ [`INFORMATION_SCHEMA.TABLES`](../querying/sql-metadata-tables.md#tables-table),
+ [`INFORMATION_SCHEMA.COLUMNS`](../querying/sql-metadata-tables.md#columns-table), and
+ [`sys.segments`](../querying/sql-metadata-tables.md#segments-table) tables.
+:::
+
+`GET /druid/v2/datasources`
+
+Returns a list of queryable datasources.
+
+`GET /druid/v2/datasources/{dataSourceName}`
+
+Returns the dimensions and metrics of the datasource. Optionally, you can provide request parameter "full" to get list of served intervals with dimensions and metrics being served for those intervals. You can also provide request param "interval" explicitly to refer to a particular interval.
+
+If no interval is specified, a default interval spanning a configurable period before the current time will be used. The default duration of this interval is specified in ISO 8601 duration format via: `druid.query.segmentMetadata.defaultHistory`
+
+`GET /druid/v2/datasources/{dataSourceName}/dimensions`
+
+:::info
+ This API is deprecated and will be removed in future releases. Please use [SegmentMetadataQuery](../querying/segmentmetadataquery.md) instead
+ which provides more comprehensive information and supports all dataSource types including streaming dataSources. It's also encouraged to use [INFORMATION_SCHEMA tables](../querying/sql-metadata-tables.md)
+ if you're using SQL.
+:::
+
+Returns the dimensions of the datasource.
+
+`GET /druid/v2/datasources/{dataSourceName}/metrics`
+
+:::info
+ This API is deprecated and will be removed in future releases. Please use [SegmentMetadataQuery](../querying/segmentmetadataquery.md) instead
+ which provides more comprehensive information and supports all dataSource types including streaming dataSources. It's also encouraged to use [INFORMATION_SCHEMA tables](../querying/sql-metadata-tables.md)
+ if you're using SQL.
+:::
+
+Returns the metrics of the datasource.
+
+`GET /druid/v2/datasources/{dataSourceName}/candidates?intervals={comma-separated-intervals}&numCandidates={numCandidates}`
+
+Returns segment information lists including server locations for the given datasource and intervals. If "numCandidates" is not specified, it will return all servers for each interval.
diff --git a/docs/34.0.0/api-reference/lookups-api.md b/docs/34.0.0/api-reference/lookups-api.md
new file mode 100644
index 000000000..4a122917b
--- /dev/null
+++ b/docs/34.0.0/api-reference/lookups-api.md
@@ -0,0 +1,279 @@
+---
+id: lookups-api
+title: Lookups API
+sidebar_label: Lookups
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+This document describes the API endpoints to configure, update, retrieve, and manage lookups for Apache Druid.
+
+## Configure lookups
+
+### Bulk update
+
+Lookups can be updated in bulk by posting a JSON object to `/druid/coordinator/v1/lookups/config`. The format of the json object is as follows:
+
+```json
+{
+ "<tierName>": {
+ "<lookupName>": {
+ "version": "<version>",
+ "lookupExtractorFactory": {
+ "type": "<someExtractorFactoryType>",
+ "<someExtractorField>": "<someExtractorValue>"
+ }
+ }
+ }
+}
+```
+
+Note that "version" is an arbitrary string assigned by the user, when making updates to existing lookup then user would need to specify a lexicographically higher version.
+
+For example, a config might look something like:
+
+```json
+{
+ "__default": {
+ "country_code": {
+ "version": "v0",
+ "lookupExtractorFactory": {
+ "type": "map",
+ "map": {
+ "77483": "United States"
+ }
+ }
+ },
+ "site_id": {
+ "version": "v0",
+ "lookupExtractorFactory": {
+ "type": "cachedNamespace",
+ "extractionNamespace": {
+ "type": "jdbc",
+ "connectorConfig": {
+ "createTables": true,
+ "connectURI": "jdbc:mysql:\/\/localhost:3306\/druid",
+ "user": "druid",
+ "password": "diurd"
+ },
+ "table": "lookupTable",
+ "keyColumn": "country_id",
+ "valueColumn": "country_name",
+ "tsColumn": "timeColumn"
+ },
+ "firstCacheTimeout": 120000,
+ "injective": true
+ }
+ },
+ "site_id_customer1": {
+ "version": "v0",
+ "lookupExtractorFactory": {
+ "type": "map",
+ "map": {
+ "847632": "Internal Use Only"
+ }
+ }
+ },
+ "site_id_customer2": {
+ "version": "v0",
+ "lookupExtractorFactory": {
+ "type": "map",
+ "map": {
+ "AHF77": "Home"
+ }
+ }
+ }
+ },
+ "realtime_customer1": {
+ "country_code": {
+ "version": "v0",
+ "lookupExtractorFactory": {
+ "type": "map",
+ "map": {
+ "77483": "United States"
+ }
+ }
+ },
+ "site_id_customer1": {
+ "version": "v0",
+ "lookupExtractorFactory": {
+ "type": "map",
+ "map": {
+ "847632": "Internal Use Only"
+ }
+ }
+ }
+ },
+ "realtime_customer2": {
+ "country_code": {
+ "version": "v0",
+ "lookupExtractorFactory": {
+ "type": "map",
+ "map": {
+ "77483": "United States"
+ }
+ }
+ },
+ "site_id_customer2": {
+ "version": "v0",
+ "lookupExtractorFactory": {
+ "type": "map",
+ "map": {
+ "AHF77": "Home"
+ }
+ }
+ }
+ }
+}
+```
+
+All entries in the map will UPDATE existing entries. No entries will be deleted.
+
+### Update lookup
+
+A `POST` to a particular lookup extractor factory via `/druid/coordinator/v1/lookups/config/{tier}/{id}` creates or updates that specific extractor factory.
+
+For example, a post to `/druid/coordinator/v1/lookups/config/realtime_customer1/site_id_customer1` might contain the following:
+
+```json
+{
+ "version": "v1",
+ "lookupExtractorFactory": {
+ "type": "map",
+ "map": {
+ "847632": "Internal Use Only"
+ }
+ }
+}
+```
+
+This will replace the `site_id_customer1` lookup in the `realtime_customer1` with the definition above.
+
+Assign a unique version identifier each time you update a lookup extractor factory. Otherwise the call will fail.
+
+### Get all lookups
+
+A `GET` to `/druid/coordinator/v1/lookups/config/all` will return all known lookup specs for all tiers.
+
+### Get lookup
+
+A `GET` to a particular lookup extractor factory is accomplished via `/druid/coordinator/v1/lookups/config/{tier}/{id}`
+
+Using the prior example, a `GET` to `/druid/coordinator/v1/lookups/config/realtime_customer2/site_id_customer2` should return
+
+```json
+{
+ "version": "v1",
+ "lookupExtractorFactory": {
+ "type": "map",
+ "map": {
+ "AHF77": "Home"
+ }
+ }
+}
+```
+
+### Delete lookup
+
+A `DELETE` to `/druid/coordinator/v1/lookups/config/{tier}/{id}` will remove that lookup from the cluster. If it was last lookup in the tier, then tier is deleted as well.
+
+### Delete tier
+
+A `DELETE` to `/druid/coordinator/v1/lookups/config/{tier}` will remove that tier from the cluster.
+
+### List tier names
+
+A `GET` to `/druid/coordinator/v1/lookups/config` will return a list of known tier names in the dynamic configuration.
+To discover a list of tiers currently active in the cluster in addition to ones known in the dynamic configuration, the parameter `discover=true` can be added as per `/druid/coordinator/v1/lookups/config?discover=true`.
+
+### List lookup names
+
+A `GET` to `/druid/coordinator/v1/lookups/config/{tier}` will return a list of known lookup names for that tier.
+
+These end points can be used to get the propagation status of configured lookups to processes using lookups such as Historicals.
+
+## Lookup status
+
+### List load status of all lookups
+
+`GET` `/druid/coordinator/v1/lookups/status` with optional query parameter `detailed`.
+
+### List load status of lookups in a tier
+
+`GET` `/druid/coordinator/v1/lookups/status/{tier}` with optional query parameter `detailed`.
+
+### List load status of single lookup
+
+`GET` `/druid/coordinator/v1/lookups/status/{tier}/{lookup}` with optional query parameter `detailed`.
+
+### List lookup state of all processes
+
+`GET` `/druid/coordinator/v1/lookups/nodeStatus` with optional query parameter `discover` to discover tiers advertised by other Druid nodes, or by default, returning all configured lookup tiers. The default response will also include the lookups which are loaded, being loaded, or being dropped on each node, for each tier, including the complete lookup spec. Add the optional query parameter `detailed=false` to only include the 'version' of the lookup instead of the complete spec.
+
+### List lookup state of processes in a tier
+
+`GET` `/druid/coordinator/v1/lookups/nodeStatus/{tier}`
+
+### List lookup state of single process
+
+`GET` `/druid/coordinator/v1/lookups/nodeStatus/{tier}/{host:port}`
+
+## Internal API
+
+The Peon, Router, Broker, and Historical processes all have the ability to consume lookup configuration.
+There is an internal API these processes use to list/load/drop their lookups starting at `/druid/listen/v1/lookups`.
+These follow the same convention for return values as the cluster wide dynamic configuration. Following endpoints
+can be used for debugging purposes but not otherwise.
+
+### Get lookups
+
+A `GET` to the process at `/druid/listen/v1/lookups` will return a json map of all the lookups currently active on the process.
+The return value will be a json map of the lookups to their extractor factories.
+
+```json
+{
+ "site_id_customer2": {
+ "version": "v1",
+ "lookupExtractorFactory": {
+ "type": "map",
+ "map": {
+ "AHF77": "Home"
+ }
+ }
+ }
+}
+```
+
+### Get lookup
+
+A `GET` to the process at `/druid/listen/v1/lookups/some_lookup_name` will return the LookupExtractorFactory for the lookup identified by `some_lookup_name`.
+The return value will be the json representation of the factory.
+
+```json
+{
+ "version": "v1",
+ "lookupExtractorFactory": {
+ "type": "map",
+ "map": {
+ "AHF77": "Home"
+ }
+ }
+}
+```
\ No newline at end of file
diff --git a/docs/34.0.0/api-reference/retention-rules-api.md b/docs/34.0.0/api-reference/retention-rules-api.md
new file mode 100644
index 000000000..c21e546ab
--- /dev/null
+++ b/docs/34.0.0/api-reference/retention-rules-api.md
@@ -0,0 +1,562 @@
+---
+id: retention-rules-api
+title: Retention rules API
+sidebar_label: Retention rules
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
+<!--
+
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+This topic describes the API endpoints for managing retention rules in Apache Druid. You can configure retention rules in the Druid web console or API.
+
+Druid uses retention rules to determine what data is retained in the cluster. Druid supports load, drop, and broadcast rules. For more information, see [Using rules to drop and retain data](../operations/rule-configuration.md).
+
+In this topic, `http://ROUTER_IP:ROUTER_PORT` is a placeholder for your Router service address and port. Replace it with the information for your deployment. For example, use `http://localhost:8888` for quickstart deployments.
+
+## Update retention rules for a datasource
+
+Updates one or more retention rules for a datasource. The request body takes an array of retention rule objects. For details on defining retention rules, see the following sources:
+
+* [Load rules](../operations/rule-configuration.md#load-rules)
+* [Drop rules](../operations/rule-configuration.md#drop-rules)
+* [Broadcast rules](../operations/rule-configuration.md#broadcast-rules)
+
+This request overwrites any existing rules for the datasource.
+Druid reads rules in the order in which they appear; for more information, see [rule structure](../operations/rule-configuration.md).
+
+Note that this endpoint returns an HTTP `200 OK` even if the datasource does not exist.
+
+### URL
+
+`POST` `/druid/coordinator/v1/rules/{dataSource}`
+
+### Header parameters
+
+The endpoint supports a set of optional header parameters to populate the `author` and `comment` fields in the `auditInfo` property for audit history.
+
+* `X-Druid-Author` (optional)
+ * Type: String
+ * A string representing the author making the configuration change.
+* `X-Druid-Comment` (optional)
+ * Type: String
+ * A string describing the update.
+
+### Responses
+
+<Tabs>
+
+<TabItem value="1" label="200 SUCCESS">
+
+
+*Successfully updated retention rules for specified datasource*
+
+</TabItem>
+</Tabs>
+
+---
+
+### Sample request
+
+The following example sets a set of broadcast, load, and drop retention rules for the `kttm1` datasource.
+
+<Tabs>
+
+<TabItem value="2" label="cURL">
+
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/rules/kttm1" \
+--header 'X-Druid-Author: doc intern' \
+--header 'X-Druid-Comment: submitted via api' \
+--header 'Content-Type: application/json' \
+--data '[
+ {
+ "type": "broadcastForever"
+ },
+ {
+ "type": "loadForever",
+ "tieredReplicants": {
+ "_default_tier": 2
+ },
+ "useDefaultTierForNull": true
+ },
+ {
+ "type": "dropByPeriod",
+ "period": "P1M"
+ }
+]'
+```
+
+</TabItem>
+<TabItem value="3" label="HTTP">
+
+
+```HTTP
+POST /druid/coordinator/v1/rules/kttm1 HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+X-Druid-Author: doc intern
+X-Druid-Comment: submitted via api
+Content-Type: application/json
+Content-Length: 273
+
+[
+ {
+ "type": "broadcastForever"
+ },
+ {
+ "type": "loadForever",
+ "tieredReplicants": {
+ "_default_tier": 1
+ },
+ "useDefaultTierForNull": true
+ },
+ {
+ "type": "dropByPeriod",
+ "period": "P1M"
+ }
+]
+```
+
+</TabItem>
+</Tabs>
+
+### Sample response
+
+A successful request returns an HTTP `200 OK` message code and an empty response body.
+
+## Update default retention rules for all datasources
+
+Updates one or more default retention rules for all datasources. Submit retention rules as an array of objects in the request body. For details on defining retention rules, see the following sources:
+
+* [Load rules](../operations/rule-configuration.md#load-rules)
+* [Drop rules](../operations/rule-configuration.md#drop-rules)
+* [Broadcast rules](../operations/rule-configuration.md#broadcast-rules)
+
+This request overwrites any existing rules for all datasources. To remove default retention rules for all datasources, submit an empty rule array in the request body. Rules are read in the order in which they appear; for more information, see [rule structure](../operations/rule-configuration.md).
+
+### URL
+
+`POST` `/druid/coordinator/v1/rules/_default`
+
+### Header parameters
+
+The endpoint supports a set of optional header parameters to populate the `author` and `comment` fields in the `auditInfo` property for audit history.
+
+* `X-Druid-Author` (optional)
+ * Type: String
+ * A string representing the author making the configuration change.
+* `X-Druid-Comment` (optional)
+ * Type: String
+ * A string describing the update.
+
+### Responses
+
+<Tabs>
+
+<TabItem value="4" label="200 SUCCESS">
+
+
+*Successfully updated default retention rules*
+
+</TabItem>
+<TabItem value="5" label="500 SERVER ERROR">
+
+
+*Error with request body*
+
+</TabItem>
+</Tabs>
+
+---
+
+### Sample request
+
+The following example updates the default retention rule for all datasources with a `loadByInterval` rule.
+
+<Tabs>
+
+<TabItem value="6" label="cURL">
+
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/rules/_default" \
+--header 'Content-Type: application/json' \
+--data '[
+ {
+ "type": "loadByInterval",
+ "tieredReplicants": {},
+ "useDefaultTierForNull": false,
+ "interval": "2010-01-01/2020-01-01"
+ }
+]'
+```
+
+</TabItem>
+<TabItem value="7" label="HTTP">
+
+
+```HTTP
+POST /druid/coordinator/v1/rules/_default HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+Content-Type: application/json
+Content-Length: 205
+
+[
+ {
+ "type": "loadByInterval",
+ "tieredReplicants": {},
+ "useDefaultTierForNull": false,
+ "interval": "2010-01-01/2020-01-01"
+ }
+]
+```
+
+</TabItem>
+</Tabs>
+
+### Sample response
+
+A successful request returns an HTTP `200 OK` message code and an empty response body.
+
+## Get an array of all retention rules
+
+Retrieves all current retention rules in the cluster including the default retention rule. Returns an array of objects for each datasource and their associated retention rules.
+
+### URL
+
+`GET` `/druid/coordinator/v1/rules`
+
+### Responses
+
+<Tabs>
+
+<TabItem value="8" label="200 SUCCESS">
+
+
+*Successfully retrieved retention rules*
+
+</TabItem>
+</Tabs>
+
+---
+
+### Sample request
+
+<Tabs>
+
+<TabItem value="9" label="cURL">
+
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/rules"
+```
+
+</TabItem>
+<TabItem value="10" label="HTTP">
+
+
+```HTTP
+GET /druid/coordinator/v1/rules HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+</TabItem>
+</Tabs>
+
+### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+{
+ "_default": [
+ {
+ "tieredReplicants": {
+ "_default_tier": 2
+ },
+ "type": "loadForever"
+ }
+ ],
+ "social_media": [
+ {
+ "interval": "2023-01-01T00:00:00.000Z/2023-02-01T00:00:00.000Z",
+ "type": "dropByInterval"
+ }
+ ],
+ "wikipedia_api": [],
+}
+ ```
+</details>
+
+## Get an array of retention rules for a datasource
+
+Retrieves an array of rule objects for a single datasource. Returns an empty array if there are no retention rules.
+
+Note that this endpoint returns an HTTP `200 OK` message code even if the datasource doesn't exist.
+
+### URL
+
+`GET` `/druid/coordinator/v1/rules/{dataSource}`
+
+### Query parameters
+
+* `full` (optional)
+ * Includes the default retention rule for the datasource in the response.
+
+### Responses
+
+<Tabs>
+
+<TabItem value="11" label="200 SUCCESS">
+
+
+*Successfully retrieved retention rules*
+
+</TabItem>
+</Tabs>
+
+---
+
+### Sample request
+
+The following example retrieves the custom retention rules and default retention rules for datasource with the name `social_media`.
+
+<Tabs>
+
+<TabItem value="12" label="cURL">
+
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/rules/social_media?full=null"
+```
+
+</TabItem>
+<TabItem value="13" label="HTTP">
+
+
+```HTTP
+GET /druid/coordinator/v1/rules/social_media?full=null HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+</TabItem>
+</Tabs>
+
+### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+[
+ {
+ "interval": "2020-01-01T00:00:00.000Z/2022-02-01T00:00:00.000Z",
+ "type": "dropByInterval"
+ },
+ {
+ "interval": "2010-01-01T00:00:00.000Z/2020-01-01T00:00:00.000Z",
+ "tieredReplicants": {
+ "_default_tier": 2
+ },
+ "type": "loadByInterval"
+ },
+ {
+ "tieredReplicants": {
+ "_default_tier": 2
+ },
+ "type": "loadForever"
+ }
+]
+ ```
+
+</details>
+
+## Get audit history for all datasources
+
+Retrieves the audit history of rules for all datasources over an interval of time. The default interval is 1 week. You can change this period by setting `druid.audit.manager.auditHistoryMillis` in the `runtime.properties` file for the Coordinator.
+
+### URL
+
+`GET` `/druid/coordinator/v1/rules/history`
+
+### Query parameters
+
+Note that the following query parameters cannot be chained.
+
+* `interval` (optional)
+ * Type: ISO 8601.
+ * Limits the number of results to the specified time interval. Delimit with `/`. For example, `2023-07-13/2023-07-19`.
+* `count` (optional)
+ * Type: Int
+ * Limits the number of results to the last `n` entries.
+
+### Responses
+
+<Tabs>
+
+<TabItem value="14" label="200 SUCCESS">
+
+
+*Successfully retrieved audit history*
+
+</TabItem>
+<TabItem value="15" label="400 BAD REQUEST">
+
+
+*Request in the incorrect format*
+
+</TabItem>
+<TabItem value="16" label="404 NOT FOUND">
+
+
+*`count` query parameter too large*
+
+</TabItem>
+</Tabs>
+
+---
+
+### Sample request
+
+The following example retrieves the audit history for all datasources from `2023-07-13` to `2023-07-19`.
+
+<Tabs>
+
+<TabItem value="17" label="cURL">
+
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/rules/history?interval=2023-07-13%2F2023-07-19"
+```
+
+</TabItem>
+<TabItem value="18" label="HTTP">
+
+
+```HTTP
+GET /druid/coordinator/v1/rules/history?interval=2023-07-13/2023-07-19 HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+</TabItem>
+</Tabs>
+
+### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+[
+ {
+ "key": "social_media",
+ "type": "rules",
+ "auditInfo": {
+ "author": "console",
+ "comment": "test",
+ "ip": "127.0.0.1"
+ },
+ "payload": "[{\"interval\":\"2023-01-01T00:00:00.000Z/2023-02-01T00:00:00.000Z\",\"type\":\"dropByInterval\"}]",
+ "auditTime": "2023-07-13T18:05:33.066Z"
+ },
+ {
+ "key": "social_media",
+ "type": "rules",
+ "auditInfo": {
+ "author": "console",
+ "comment": "test",
+ "ip": "127.0.0.1"
+ },
+ "payload": "[]",
+ "auditTime": "2023-07-18T18:10:21.203Z"
+ },
+ {
+ "key": "wikipedia_api",
+ "type": "rules",
+ "auditInfo": {
+ "author": "console",
+ "comment": "test",
+ "ip": "127.0.0.1"
+ },
+ "payload": "[{\"tieredReplicants\":{\"_default_tier\":2},\"type\":\"loadForever\"}]",
+ "auditTime": "2023-07-18T18:10:44.519Z"
+ },
+ {
+ "key": "wikipedia_api",
+ "type": "rules",
+ "auditInfo": {
+ "author": "console",
+ "comment": "test",
+ "ip": "127.0.0.1"
+ },
+ "payload": "[]",
+ "auditTime": "2023-07-18T18:11:02.110Z"
+ },
+ {
+ "key": "social_media",
+ "type": "rules",
+ "auditInfo": {
+ "author": "console",
+ "comment": "test",
+ "ip": "127.0.0.1"
+ },
+ "payload": "[{\"interval\":\"2023-07-03T18:49:54.848Z/2023-07-03T18:49:55.861Z\",\"type\":\"dropByInterval\"}]",
+ "auditTime": "2023-07-18T18:32:50.060Z"
+ },
+ {
+ "key": "social_media",
+ "type": "rules",
+ "auditInfo": {
+ "author": "console",
+ "comment": "test",
+ "ip": "127.0.0.1"
+ },
+ "payload": "[{\"interval\":\"2020-01-01T00:00:00.000Z/2022-02-01T00:00:00.000Z\",\"type\":\"dropByInterval\"}]",
+ "auditTime": "2023-07-18T18:34:09.657Z"
+ },
+ {
+ "key": "social_media",
+ "type": "rules",
+ "auditInfo": {
+ "author": "console",
+ "comment": "test",
+ "ip": "127.0.0.1"
+ },
+ "payload": "[{\"interval\":\"2020-01-01T00:00:00.000Z/2022-02-01T00:00:00.000Z\",\"type\":\"dropByInterval\"},{\"tieredReplicants\":{\"_default_tier\":2},\"type\":\"loadForever\"}]",
+ "auditTime": "2023-07-18T18:38:37.223Z"
+ },
+ {
+ "key": "social_media",
+ "type": "rules",
+ "auditInfo": {
+ "author": "console",
+ "comment": "test",
+ "ip": "127.0.0.1"
+ },
+ "payload": "[{\"interval\":\"2020-01-01T00:00:00.000Z/2022-02-01T00:00:00.000Z\",\"type\":\"dropByInterval\"},{\"interval\":\"2010-01-01T00:00:00.000Z/2020-01-01T00:00:00.000Z\",\"tieredReplicants\":{\"_default_tier\":2},\"type\":\"loadByInterval\"}]",
+ "auditTime": "2023-07-18T18:49:43.964Z"
+ }
+]
+ ```
+</details>
diff --git a/docs/latest/api-reference/service-status-api.md b/docs/34.0.0/api-reference/service-status-api.md
similarity index 93%
copy from docs/latest/api-reference/service-status-api.md
copy to docs/34.0.0/api-reference/service-status-api.md
index 6e801ae63..47d2a5a6d 100644
--- a/docs/latest/api-reference/service-status-api.md
+++ b/docs/34.0.0/api-reference/service-status-api.md
@@ -658,6 +658,138 @@ Host: http://COORDINATOR_IP:COORDINATOR_PORT
</details>
+
+### Get Historical Cloning Status
+
+Retrieves the current status of Historical cloning from the Coordinator.
+
+#### URL
+
+`GET` `/druid/coordinator/v1/config/cloneStatus`
+
+#### Responses
+
+<Tabs>
+
+<TabItem value="56" label="200 SUCCESS">
+
+
+<br/>
+
+*Successfully retrieved cloning status*
+
+</TabItem>
+</Tabs>
+
+#### Sample request
+
+<Tabs>
+
+<TabItem value="58" label="cURL">
+
+
+```shell
+curl "http://COORDINATOR_IP:COORDINATOR_PORT/druid/coordinator/v1/config/cloneStatus"
+```
+
+</TabItem>
+<TabItem value="59" label="HTTP">
+
+
+```http
+GET /druid/coordinator/v1/config/cloneStatus HTTP/1.1
+Host: http://COORDINATOR_IP:COORDINATOR_PORT
+```
+
+</TabItem>
+</Tabs>
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+```json
+{
+ "cloneStatus": [
+ {
+ "sourceServer": "localhost:8089",
+ "targetServer": "localhost:8083",
+ "state": "IN_PROGRESS",
+ "segmentLoadsRemaining": 0,
+ "segmentDropsRemaining": 0,
+ "bytesToLoad": 0
+ }
+ ]
+}
+```
+
+</details>
+
+### Get Broker dynamic configuration view
+
+Retrieves the list of Brokers which have an up-to-date view of Coordinator dynamic configuration.
+
+#### URL
+
+`GET` `/druid/coordinator/v1/config/syncedBrokers`
+
+#### Responses
+
+<Tabs>
+
+<TabItem value="56" label="200 SUCCESS">
+
+
+<br/>
+
+*Successfully retrieved Broker Configuration view*
+
+</TabItem>
+</Tabs>
+
+#### Sample request
+
+<Tabs>
+
+<TabItem value="58" label="cURL">
+
+
+```shell
+curl "http://COORDINATOR_IP:COORDINATOR_PORT/druid/coordinator/v1/config/syncedBrokers"
+```
+
+</TabItem>
+<TabItem value="59" label="HTTP">
+
+
+```http
+GET /druid/coordinator/v1/config/syncedBrokers HTTP/1.1
+Host: http://COORDINATOR_IP:COORDINATOR_PORT
+```
+
+</TabItem>
+</Tabs>
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+```json
+{
+ "syncedBrokers": [
+ {
+ "host": "localhost",
+ "port": 8082,
+ "lastSyncTimestampMillis": 1745756337472
+ }
+ ]
+}
+```
+
+</details>
+
## Overlord
### Get Overlord leader address
@@ -1334,4 +1466,4 @@ Host: http://BROKER_IP:BROKER_PORT
#### Sample response
-A successful response to this endpoint results in an empty response body.
\ No newline at end of file
+A successful response to this endpoint results in an empty response body.
diff --git a/docs/latest/api-reference/sql-api.md b/docs/34.0.0/api-reference/sql-api.md
similarity index 92%
copy from docs/latest/api-reference/sql-api.md
copy to docs/34.0.0/api-reference/sql-api.md
index ccba5b154..af60cee4c 100644
--- a/docs/latest/api-reference/sql-api.md
+++ b/docs/34.0.0/api-reference/sql-api.md
@@ -38,7 +38,8 @@ In this topic, `http://ROUTER_IP:ROUTER_PORT` is a placeholder for your Router s
### Submit a query
-Submits a SQL-based query in the JSON request body. Returns a JSON object with the query results and optional metadata for the results. You can also use this endpoint to query [metadata tables](../querying/sql-metadata-tables.md).
+Submits a SQL-based query in the JSON or text format request body.
+Returns a JSON object with the query results and optional metadata for the results. You can also use this endpoint to query [metadata tables](../querying/sql-metadata-tables.md).
Each query has an associated SQL query ID. You can set this ID manually using the SQL context parameter `sqlQueryId`. If not set, Druid automatically generates `sqlQueryId` and returns it in the response header for `X-Druid-SQL-Query-Id`. Note that you need the `sqlQueryId` to [cancel a query](#cancel-a-query).
@@ -46,12 +47,14 @@ Each query has an associated SQL query ID. You can set this ID manually using th
`POST` `/druid/v2/sql`
-#### Request body
+#### JSON Format Request body
-The request body takes the following properties:
+To send queries in JSON format, the `Content-Type` in the HTTP request MUST be `application/json`.
+If there are multiple `Content-Type` headers, the **first** one is used.
-* `query`: SQL query string.
+The request body takes the following properties:
+* `query`: SQL query string. HTTP requests are permitted to include multiple `SET` statements to assign [SQL query context parameter](../querying/sql-query-context.md) values to apply to the query statement, see [SET](../querying/sql.md#set) for details. Context parameters set by `SET` statements take priority over values set in `context`.
* `resultFormat`: String that indicates the format to return query results. Select one of the following formats:
* `object`: Returns a JSON array of JSON objects with the HTTP response header `Content-Type: application/json`.
Object field names match the columns returned by the SQL query in the same order as the SQL query.
@@ -83,7 +86,7 @@ The request body takes the following properties:
* The `header` property is set to true.
* The version of Druid supports `typesHeader` and `sqlTypesHeader`, regardless of whether either property is set.
-* `context`: JSON object containing optional [SQL query context parameters](../querying/sql-query-context.md), such as to set the query ID, time zone, and whether to use an approximation algorithm for distinct count.
+* `context`: JSON object containing optional [SQL query context parameters](../querying/sql-query-context.md), such as to set the query ID, time zone, and whether to use an approximation algorithm for distinct count. You can also set the context through the SQL SET command. For more information, see [Druid SQL overview](../querying/sql.md#set).
* `parameters`: List of query parameters for parameterized queries. Each parameter in the array should be a JSON object containing the parameter's SQL data type and parameter value. For more information on using dynamic parameters, see [Dynamic parameters](../querying/sql.md#dynamic-parameters). For a list of supported SQL types, see [Data types](../querying/sql-data-types.md).
@@ -99,6 +102,36 @@ The request body takes the following properties:
}
```
+##### Text Format Request body
+
+Druid also allows you to submit SQL queries in text format which is simpler than above JSON format.
+To do this, just set the `Content-Type` request header to `text/plain` or `application/x-www-form-urlencoded`, and pass SQL via the HTTP Body.
+
+If `application/x-www-form-urlencoded` is used, make sure the SQL query is URL-encoded.
+
+If there are multiple `Content-Type` headers, the **first** one is used.
+
+For response, the `resultFormat` is always `object` with the HTTP response header `Content-Type: application/json`.
+If you want more control over the query context or response format, use the above JSON format request body instead.
+
+The following example demonstrates how to submit a SQL query in text format:
+
+```commandline
+echo 'SELECT 1' | curl -H 'Content-Type: text/plain' http://ROUTER_IP:ROUTER_PORT/druid/v2/sql --data @-
+```
+
+We can also use `application/x-www-form-urlencoded` to submit URL-encoded SQL queries as shown by the following examples:
+
+```commandline
+echo 'SELECT%20%31' | curl http://ROUTER_IP:ROUTER_PORT/druid/v2/sql --data @-
+echo 'SELECT 1' | curl http://ROUTER_IP:ROUTER_PORT/druid/v2/sql --data-urlencode @-
+```
+
+The `curl` tool uses `application/x-www-form-urlencoded` as Content-Type header if the header is not given.
+
+The first example pass the URL-encoded query `SELECT%20%31`, which is `SELECT 1`, to the `curl` and `curl` will directly sends it to the server.
+While the second example passes the raw query `SELECT 1` to `curl` and the `curl` encodes the query to `SELECT%20%31` because of `--data-urlencode` option and sends the encoded text to the server.
+
#### Responses
<Tabs>
@@ -155,7 +188,12 @@ If you detect a truncated response, treat it as an error.
#### Sample request
-The following example retrieves all rows in the `wikipedia` datasource where the `user` is `BlueMoon2662`. The query is assigned the ID `request01` using the `sqlQueryId` context parameter. The optional properties `header`, `typesHeader`, and `sqlTypesHeader` are set to `true` to include type information to the response.
+In the following example, this query demonstrates the following actions:
+- Retrieves all rows from the `wikipedia` datasource.
+- Filters the results where the `user` value is `BlueMoon2662`.
+- Applies the `sqlTimeZone` context parameter to set the time zone of results to `America/Los_Angeles`.
+- Returns descriptors for `header`, `typesHeader`, and `sqlTypesHeader`.
+
<Tabs>
@@ -167,7 +205,7 @@ curl "http://ROUTER_IP:ROUTER_PORT/druid/v2/sql" \
--header 'Content-Type: application/json' \
--data '{
"query": "SELECT * FROM wikipedia WHERE user='\''BlueMoon2662'\''",
- "context" : {"sqlQueryId" : "request01"},
+ "context" : {"sqlTimeZone" : "America/Los_Angeles"},
"header" : true,
"typesHeader" : true,
"sqlTypesHeader" : true
@@ -182,11 +220,11 @@ curl "http://ROUTER_IP:ROUTER_PORT/druid/v2/sql" \
POST /druid/v2/sql HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
Content-Type: application/json
-Content-Length: 192
+Content-Length: 201
{
"query": "SELECT * FROM wikipedia WHERE user='BlueMoon2662'",
- "context" : {"sqlQueryId" : "request01"},
+ "context" : {"sqlTimeZone" : "America/Los_Angeles"},
"header" : true,
"typesHeader" : true,
"sqlTypesHeader" : true
@@ -196,6 +234,20 @@ Content-Length: 192
</TabItem>
</Tabs>
+You can also specify query-level context parameters directly within the SQL query string using the `SET` command. For more details, see [SET](../querying/sql.md#set).
+
+The following request body is functionally equivalent to the previous example and uses SET instead of the `context` parameter:
+
+```JSON
+{
+ "query": "SET sqlTimeZone='America/Los_Angeles'; SELECT * FROM wikipedia WHERE user='BlueMoon2662'",
+ "header": true,
+ "typesHeader": true,
+ "sqlTypesHeader": true
+}
+```
+
+
#### Sample response
<details>
@@ -229,24 +281,24 @@ Content-Length: 192
"sqlType": "VARCHAR"
},
"isAnonymous": {
- "type": "LONG",
- "sqlType": "BIGINT"
+ "type": "STRING",
+ "sqlType": "VARCHAR"
},
"isMinor": {
- "type": "LONG",
- "sqlType": "BIGINT"
+ "type": "STRING",
+ "sqlType": "VARCHAR"
},
"isNew": {
- "type": "LONG",
- "sqlType": "BIGINT"
+ "type": "STRING",
+ "sqlType": "VARCHAR"
},
"isRobot": {
- "type": "LONG",
- "sqlType": "BIGINT"
+ "type": "STRING",
+ "sqlType": "VARCHAR"
},
"isUnpatrolled": {
- "type": "LONG",
- "sqlType": "BIGINT"
+ "type": "STRING",
+ "sqlType": "VARCHAR"
},
"metroCode": {
"type": "LONG",
@@ -286,22 +338,22 @@ Content-Length: 192
}
},
{
- "__time": "2015-09-12T00:47:53.259Z",
+ "__time": "2015-09-11T17:47:53.259-07:00",
"channel": "#ja.wikipedia",
- "cityName": "",
+ "cityName": null,
"comment": "/* 対戦通算成績と得失点 */",
- "countryIsoCode": "",
- "countryName": "",
- "isAnonymous": 0,
- "isMinor": 1,
- "isNew": 0,
- "isRobot": 0,
- "isUnpatrolled": 0,
- "metroCode": 0,
+ "countryIsoCode": null,
+ "countryName": null,
+ "isAnonymous": "false",
+ "isMinor": "true",
+ "isNew": "false",
+ "isRobot": "false",
+ "isUnpatrolled": "false",
+ "metroCode": null,
"namespace": "Main",
"page": "アルビレックス新潟の年度別成績一覧",
- "regionIsoCode": "",
- "regionName": "",
+ "regionIsoCode": null,
+ "regionName": null,
"user": "BlueMoon2662",
"delta": 14,
"added": 14,
diff --git a/docs/latest/api-reference/sql-ingestion-api.md b/docs/34.0.0/api-reference/sql-ingestion-api.md
similarity index 84%
copy from docs/latest/api-reference/sql-ingestion-api.md
copy to docs/34.0.0/api-reference/sql-ingestion-api.md
index 8fe4e01ba..59942aff8 100644
--- a/docs/latest/api-reference/sql-ingestion-api.md
+++ b/docs/34.0.0/api-reference/sql-ingestion-api.md
@@ -101,22 +101,23 @@ The `/druid/v2/sql/task` endpoint accepts the following:
### Sample request
The following example shows a query that fetches data from an external JSON source and inserts it into a table named `wikipedia`.
+The example specifies two query context parameters:
+
+- `maxNumTasks=3`: Limits the maximum number of parallel tasks to 3.
+- `finalizeAggregations=false`: Ensures that Druid saves the aggregation's intermediate type during ingestion. For more information, see [Rollup](../multi-stage-query/concepts.md#rollup).
+
<Tabs>
<TabItem value="4" label="HTTP">
-
```HTTP
POST /druid/v2/sql/task HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
Content-Type: application/json
{
- "query": "INSERT INTO wikipedia\nSELECT\n TIME_PARSE(\"timestamp\") AS __time,\n *\nFROM TABLE(\n EXTERN(\n '{\"type\": \"http\", \"uris\": [\"https://druid.apache.org/data/wikipedia.json.gz\"]}',\n '{\"type\": \"json\"}',\n '[{\"name\": \"added\", \"type\": \"long\"}, {\"name\": \"channel\", \"type\": \"string\"}, {\"name\": \"cityName\", \"type\": \"string\"}, {\"name\": \"comment\", \"type\": \"string\"}, {\"name\": \"commentLength\", \"type\": \"long\"}, {\"name\": \"co [...]
- "context": {
- "maxNumTasks": 3
- }
+ "query": "SET maxNumTasks=3;\nSET finalizeAggregations=false;\nINSERT INTO wikipedia\nSELECT\n TIME_PARSE(\"timestamp\") AS __time,\n *\nFROM TABLE(\n EXTERN(\n '{\"type\": \"http\", \"uris\": [\"https://druid.apache.org/data/wikipedia.json.gz\"]}',\n '{\"type\": \"json\"}',\n '[{\"name\": \"added\", \"type\": \"long\"}, {\"name\": \"channel\", \"type\": \"string\"}, {\"name\": \"cityName\", \"type\": \"string\"}, {\"name\": \"comment\", \"type\": \"string\"}, {\"name\": \" [...]
}
```
@@ -128,12 +129,9 @@ Content-Type: application/json
```shell
curl --location --request POST 'http://ROUTER_IP:ROUTER_PORT/druid/v2/sql/task' \
--header 'Content-Type: application/json' \
- --data-raw '{
- "query": "INSERT INTO wikipedia\nSELECT\n TIME_PARSE(\"timestamp\") AS __time,\n *\nFROM TABLE(\n EXTERN(\n '\''{\"type\": \"http\", \"uris\": [\"https://druid.apache.org/data/wikipedia.json.gz\"]}'\'',\n '\''{\"type\": \"json\"}'\'',\n '\''[{\"name\": \"added\", \"type\": \"long\"}, {\"name\": \"channel\", \"type\": \"string\"}, {\"name\": \"cityName\", \"type\": \"string\"}, {\"name\": \"comment\", \"type\": \"string\"}, {\"name\": \"commentLength\", \"type\": \"long\"} [...]
- "context": {
- "maxNumTasks": 3
- }
- }'
+ --data '{
+ "query": "SET maxNumTasks=3;\nSET finalizeAggregations=false;\nINSERT INTO wikipedia\nSELECT\n TIME_PARSE(\"timestamp\") AS __time,\n *\nFROM TABLE(\n EXTERN(\n '\''{\"type\": \"http\", \"uris\": [\"https://druid.apache.org/data/wikipedia.json.gz\"]}'\'',\n '\''{\"type\": \"json\"}'\'',\n '\''[{\"name\": \"added\", \"type\": \"long\"}, {\"name\": \"channel\", \"type\": \"string\"}, {\"name\": \"cityName\", \"type\": \"string\"}, {\"name\": \"comment\", \"type\": \"string\"} [...]
+}'
```
</TabItem>
@@ -148,10 +146,7 @@ import requests
url = "http://ROUTER_IP:ROUTER_PORT/druid/v2/sql/task"
payload = json.dumps({
- "query": "INSERT INTO wikipedia\nSELECT\n TIME_PARSE(\"timestamp\") AS __time,\n *\nFROM TABLE(\n EXTERN(\n '{\"type\": \"http\", \"uris\": [\"https://druid.apache.org/data/wikipedia.json.gz\"]}',\n '{\"type\": \"json\"}',\n '[{\"name\": \"added\", \"type\": \"long\"}, {\"name\": \"channel\", \"type\": \"string\"}, {\"name\": \"cityName\", \"type\": \"string\"}, {\"name\": \"comment\", \"type\": \"string\"}, {\"name\": \"commentLength\", \"type\": \"long\"}, {\"name\": \"co [...]
- "context": {
- "maxNumTasks": 3
- }
+ "query": "SET maxNumTasks=3;\nSET finalizeAggregations=false;\nINSERT INTO wikipedia\nSELECT\n TIME_PARSE(\"timestamp\") AS __time,\n *\nFROM TABLE(\n EXTERN(\n '{\"type\": \"http\", \"uris\": [\"https://druid.apache.org/data/wikipedia.json.gz\"]}',\n '{\"type\": \"json\"}',\n '[{\"name\": \"added\", \"type\": \"long\"}, {\"name\": \"channel\", \"type\": \"string\"}, {\"name\": \"cityName\", \"type\": \"string\"}, {\"name\": \"comment\", \"type\": \"string\"}, {\"name\": \" [...]
})
headers = {
'Content-Type': 'application/json'
@@ -174,8 +169,8 @@ print(response.text)
```json
{
- "taskId": "query-f795a235-4dc7-4fef-abac-3ae3f9686b79",
- "state": "RUNNING",
+ "taskId": "query-431c4a18-9dde-4ec8-ab82-ec7fd17d5a4e",
+ "state": "RUNNING"
}
```
</details>
diff --git a/docs/34.0.0/api-reference/sql-jdbc.md b/docs/34.0.0/api-reference/sql-jdbc.md
new file mode 100644
index 000000000..affe9ea73
--- /dev/null
+++ b/docs/34.0.0/api-reference/sql-jdbc.md
@@ -0,0 +1,251 @@
+---
+id: sql-jdbc
+title: SQL JDBC driver API
+sidebar_label: SQL JDBC driver
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+:::info
+ Apache Druid supports two query languages: Druid SQL and [native queries](../querying/querying.md).
+ This document describes the SQL language.
+:::
+
+You can make [Druid SQL](../querying/sql.md) queries using the [Avatica JDBC driver](https://calcite.apache.org/avatica/downloads/).
+We recommend using Avatica JDBC driver version 1.23.0 or later. Note that starting with Avatica 1.21.0, you may need to set the [`transparent_reconnection`](https://calcite.apache.org/avatica/docs/client_reference.html#transparent_reconnection) property to `true` if you notice intermittent query failures.
+
+Once you've downloaded the Avatica client jar, add it to your classpath.
+
+Example connection string:
+
+```
+jdbc:avatica:remote:url=http://localhost:8888/druid/v2/sql/avatica/;transparent_reconnection=true
+```
+
+Or, to use the protobuf protocol instead of JSON:
+
+```
+jdbc:avatica:remote:url=http://localhost:8888/druid/v2/sql/avatica-protobuf/;transparent_reconnection=true;serialization=protobuf
+```
+
+The `url` is the `/druid/v2/sql/avatica/` endpoint on the Router, which routes JDBC connections to a consistent Broker.
+For more information, see [Connection stickiness](#connection-stickiness).
+
+Set `transparent_reconnection` to `true` so your connection is not interrupted if the pool of Brokers changes membership,
+or if a Broker is restarted.
+
+Set `serialization` to `protobuf` if using the protobuf endpoint.
+
+Note that as of the time of this writing, Avatica 1.23.0, the latest version, does not support passing
+[connection context parameters](../querying/sql-query-context.md) from the JDBC connection string to Druid. These context parameters
+must be passed using a `Properties` object instead. Refer to the Java code below for an example.
+
+Example Java code:
+
+```java
+// Connect to /druid/v2/sql/avatica/ on your Broker.
+String url = "jdbc:avatica:remote:url=http://localhost:8888/druid/v2/sql/avatica/;transparent_reconnection=true";
+
+// Set any connection context parameters you need here.
+// Any property from https://druid.apache.org/docs/latest/querying/sql-query-context.html can go here.
+Properties connectionProperties = new Properties();
+connectionProperties.setProperty("sqlTimeZone", "Etc/UTC");
+//To connect to a Druid deployment protected by basic authentication,
+//you can incorporate authentication details from https://druid.apache.org/docs/latest/operations/security-overview
+connectionProperties.setProperty("user", "admin");
+connectionProperties.setProperty("password", "password1");
+
+try (Connection connection = DriverManager.getConnection(url, connectionProperties)) {
+ try (
+ final Statement statement = connection.createStatement();
+ final ResultSet resultSet = statement.executeQuery(query)
+ ) {
+ while (resultSet.next()) {
+ // process result set
+ }
+ }
+}
+```
+
+For a runnable example that includes a query that you might run, see [Examples](#examples).
+
+It is also possible to use a protocol buffers JDBC connection with Druid, this offer reduced bloat and potential performance
+improvements for larger result sets. To use it apply the following connection URL instead, everything else remains the same
+
+```
+String url = "jdbc:avatica:remote:url=http://localhost:8888/druid/v2/sql/avatica-protobuf/;transparent_reconnection=true;serialization=protobuf";
+```
+
+:::info
+ The protobuf endpoint is also known to work with the official [Golang Avatica driver](https://github.com/apache/calcite-avatica-go)
+:::
+
+Table metadata is available over JDBC using `connection.getMetaData()` or by querying the
+[INFORMATION_SCHEMA tables](../querying/sql-metadata-tables.md). For an example of this, see [Get the metadata for a datasource](#get-the-metadata-for-a-datasource).
+
+## Connection stickiness
+
+Druid's JDBC server does not share connection state between Brokers. This means that if you're using JDBC and have
+multiple Druid Brokers, you should either connect to a specific Broker or use a load balancer with sticky sessions
+enabled. The Druid Router process provides connection stickiness when balancing JDBC requests, and can be used to achieve
+the necessary stickiness even with a normal non-sticky load balancer. Please see the
+[Router](../design/router.md) documentation for more details.
+
+Note that the non-JDBC [JSON over HTTP](sql-api.md#submit-a-query) API is stateless and does not require stickiness.
+
+## Dynamic parameters
+
+You can use [parameterized queries](../querying/sql.md#dynamic-parameters) in JDBC code, as in this example:
+
+```java
+PreparedStatement statement = connection.prepareStatement("SELECT COUNT(*) AS cnt FROM druid.foo WHERE dim1 = ? OR dim1 = ?");
+statement.setString(1, "abc");
+statement.setString(2, "def");
+final ResultSet resultSet = statement.executeQuery();
+```
+
+Sample code where dynamic parameters replace arrays using STRING_TO_ARRAY:
+```java
+PreparedStatement statement = connection.prepareStatement("select l1 from numfoo where SCALAR_IN_ARRAY(l1, STRING_TO_ARRAY(CAST(? as varchar),','))");
+List<Integer> li = ImmutableList.of(0, 7);
+String sqlArg = Joiner.on(",").join(li);
+statement.setString(1, sqlArg);
+statement.executeQuery();
+```
+
+Sample code using native array:
+```java
+PreparedStatement statement = connection.prepareStatement("select l1 from numfoo where SCALAR_IN_ARRAY(l1, ?)");
+Iterable<Object> list = ImmutableList.of(0, 7);
+ArrayFactoryImpl arrayFactoryImpl = new ArrayFactoryImpl(TimeZone.getDefault());
+AvaticaType type = ColumnMetaData.scalar(Types.INTEGER, SqlType.INTEGER.name(), Rep.INTEGER);
+Array array = arrayFactoryImpl.createArray(type, list);
+statement.setArray(1, array);
+statement.executeQuery();
+```
+
+## Examples
+
+<!-- docs/tutorial-jdbc.md redirects here -->
+
+The following section contains two complete samples that use the JDBC connector:
+
+- [Get the metadata for a datasource](#get-the-metadata-for-a-datasource) shows you how to query the `INFORMATION_SCHEMA` to get metadata like column names.
+- [Query data](#query-data) runs a select query against the datasource.
+
+You can try out these examples after verifying that you meet the [prerequisites](#prerequisites).
+
+For more information about the connection options, see [Client Reference](https://calcite.apache.org/avatica/docs/client_reference.html).
+
+### Prerequisites
+
+Make sure you meet the following requirements before trying these examples:
+
+- A supported [Java version](../operations/java.md)
+
+- [Avatica JDBC driver](https://calcite.apache.org/avatica/downloads/). You can add the JAR to your `CLASSPATH` directly or manage it externally, such as through Maven and a `pom.xml` file.
+
+- An available Druid instance. You can use the `micro-quickstart` configuration described in [Quickstart (local)](../tutorials/index.md). The examples assume that you are using the quickstart, so no authentication or authorization is expected unless explicitly mentioned.
+
+- The example `wikipedia` datasource from the quickstart is loaded on your Druid instance. If you have a different datasource loaded, you can still try these examples. You'll have to update the table name and column names to match your datasource.
+
+### Get the metadata for a datasource
+
+Metadata, such as column names, is available either through the [`INFORMATION_SCHEMA`](../querying/sql-metadata-tables.md) table or through `connection.getMetaData()`. The following example uses the `INFORMATION_SCHEMA` table to retrieve and print the list of column names for the `wikipedia` datasource that you loaded during a previous tutorial.
+
+```java
+import java.sql.*;
+import java.util.Properties;
+
+public class JdbcListColumns {
+
+ public static void main(String[] args)
+ {
+ // Connect to /druid/v2/sql/avatica/ on your Router.
+ // You can connect to a Broker but must configure connection stickiness if you do.
+ String url = "jdbc:avatica:remote:url=http://localhost:8888/druid/v2/sql/avatica/;transparent_reconnection=true";
+
+ String query = "SELECT COLUMN_NAME,* FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = 'wikipedia' and TABLE_SCHEMA='druid'";
+
+ // Set any connection context parameters you need here.
+ // Any property from https://druid.apache.org/docs/latest/querying/sql-query-context.html can go here.
+ Properties connectionProperties = new Properties();
+
+ try (Connection connection = DriverManager.getConnection(url, connectionProperties)) {
+ try (
+ final Statement statement = connection.createStatement();
+ final ResultSet rs = statement.executeQuery(query)
+ ) {
+ while (rs.next()) {
+ String columnName = rs.getString("COLUMN_NAME");
+ System.out.println(columnName);
+ }
+ }
+ } catch (SQLException e) {
+ throw new RuntimeException(e);
+ }
+
+ }
+}
+```
+
+### Query data
+
+Now that you know what columns are available, you can start querying the data. The following example queries the datasource named `wikipedia` for the timestamps and comments from Japan. It also sets the [query context parameter](../querying/sql-query-context.md) `sqlTimeZone`. Optionally, you can also parameterize queries by using [dynamic parameters](#dynamic-parameters).
+
+```java
+import java.sql.*;
+import java.util.Properties;
+
+public class JdbcCountryAndTime {
+
+ public static void main(String[] args)
+ {
+ // Connect to /druid/v2/sql/avatica/ on your Router.
+ // You can connect to a Broker but must configure connection stickiness if you do.
+ String url = "jdbc:avatica:remote:url=http://localhost:8888/druid/v2/sql/avatica/;transparent_reconnection=true";
+
+ //The query you want to run.
+ String query = "SELECT __time, isRobot, countryName, comment FROM wikipedia WHERE countryName='Japan'";
+
+ // Set any connection context parameters you need here.
+ // Any property from https://druid.apache.org/docs/latest/querying/sql-query-context.html can go here.
+ Properties connectionProperties = new Properties();
+ connectionProperties.setProperty("sqlTimeZone", "America/Los_Angeles");
+
+ try (Connection connection = DriverManager.getConnection(url, connectionProperties)) {
+ try (
+ final Statement statement = connection.createStatement();
+ final ResultSet rs = statement.executeQuery(query)
+ ) {
+ while (rs.next()) {
+ Timestamp timeStamp = rs.getTimestamp("__time");
+ String comment = rs.getString("comment");
+ System.out.println(timeStamp);
+ System.out.println(comment);
+ }
+ }
+ } catch (SQLException e) {
+ throw new RuntimeException(e);
+ }
+
+ }
+}
+```
diff --git a/docs/34.0.0/api-reference/supervisor-api.md b/docs/34.0.0/api-reference/supervisor-api.md
new file mode 100644
index 000000000..e43470f0e
--- /dev/null
+++ b/docs/34.0.0/api-reference/supervisor-api.md
@@ -0,0 +1,3622 @@
+---
+id: supervisor-api
+title: Supervisor API
+sidebar_label: Supervisors
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+This topic describes the API endpoints to manage and monitor supervisors for Apache Druid.
+The topic uses the Apache Kafka term offset to refer to the identifier for records in a partition. If you are using Amazon Kinesis, the equivalent is sequence number.
+
+In this topic, `http://ROUTER_IP:ROUTER_PORT` is a placeholder for your Router service address and port. Replace it with the information for your deployment. For example, use `http://localhost:8888` for quickstart deployments.
+
+## Supervisor information
+
+The following table lists the properties of a supervisor object:
+
+|Property|Type|Description|
+|---|---|---|
+|`id`|String|Unique identifier.|
+|`state`|String|Generic state of the supervisor. Available states:`UNHEALTHY_SUPERVISOR`, `UNHEALTHY_TASKS`, `PENDING`, `RUNNING`, `SUSPENDED`, `STOPPING`. See [Supervisor reference](../ingestion/supervisor.md#status-report) for more information.|
+|`detailedState`|String|Detailed state of the supervisor. This property contains a more descriptive, implementation-specific state that may provide more insight into the supervisor's activities than the `state` property. See [Apache Kafka ingestion](../ingestion/kafka-ingestion.md) and [Amazon Kinesis ingestion](../ingestion/kinesis-ingestion.md) for supervisor-specific states.|
+|`healthy`|Boolean|Supervisor health indicator.|
+|`spec`|Object|Container object for the supervisor configuration.|
+|`suspended`|Boolean|Indicates whether the supervisor is in a suspended state.|
+
+### Get an array of active supervisor IDs
+
+Returns an array of strings representing the names of active supervisors. If there are no active supervisors, it returns an empty array.
+
+#### URL
+
+`GET` `/druid/indexer/v1/supervisor`
+
+#### Responses
+
+<Tabs>
+
+<TabItem value="1" label="200 SUCCESS">
+
+
+*Successfully retrieved array of active supervisor IDs*
+
+</TabItem>
+</Tabs>
+
+---
+
+#### Sample request
+
+<Tabs>
+
+<TabItem value="2" label="cURL">
+
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor"
+```
+
+</TabItem>
+<TabItem value="3" label="HTTP">
+
+
+```HTTP
+GET /druid/indexer/v1/supervisor HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+</TabItem>
+</Tabs>
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+ [
+ "wikipedia_stream",
+ "social_media"
+ ]
+ ```
+</details>
+
+### Get an array of active supervisor objects
+
+Retrieves an array of active supervisor objects. If there are no active supervisors, it returns an empty array. For reference on the supervisor object properties, see the preceding [table](#supervisor-information).
+
+#### URL
+
+`GET` `/druid/indexer/v1/supervisor?full`
+
+#### Responses
+
+<Tabs>
+
+<TabItem value="4" label="200 SUCCESS">
+
+
+*Successfully retrieved supervisor objects*
+
+</TabItem>
+</Tabs>
+
+---
+
+#### Sample request
+
+<Tabs>
+
+<TabItem value="5" label="cURL">
+
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor?full=null"
+```
+
+</TabItem>
+<TabItem value="6" label="HTTP">
+
+
+```HTTP
+GET /druid/indexer/v1/supervisor?full=null HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+</TabItem>
+</Tabs>
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+ [
+ {
+ "id": "wikipedia_stream",
+ "state": "RUNNING",
+ "detailedState": "CONNECTING_TO_STREAM",
+ "healthy": true,
+ "spec": {
+ "type": "kafka",
+ "spec": {
+ "dataSchema": {
+ "dataSource": "wikipedia_stream",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "iso",
+ "missingValue": null
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ {
+ "type": "string",
+ "name": "username",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "string",
+ "name": "post_title",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "long",
+ "name": "views",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "upvotes",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "comments",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "string",
+ "name": "edited",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ }
+ ],
+ "dimensionExclusions": [
+ "__time"
+ ],
+ "includeAllDimensions": false,
+ "useSchemaDiscovery": false
+ },
+ "metricsSpec": [],
+ "granularitySpec": {
+ "type": "uniform",
+ "segmentGranularity": "HOUR",
+ "queryGranularity": {
+ "type": "none"
+ },
+ "rollup": false,
+ "intervals": []
+ },
+ "transformSpec": {
+ "filter": null,
+ "transforms": []
+ }
+ },
+ "ioConfig": {
+ "topic": "social_media",
+ "inputFormat": {
+ "type": "json"
+ },
+ "replicas": 1,
+ "taskCount": 1,
+ "taskDuration": "PT3600S",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9042"
+ },
+ "autoScalerConfig": null,
+ "pollTimeout": 100,
+ "startDelay": "PT5S",
+ "period": "PT30S",
+ "useEarliestOffset": true,
+ "completionTimeout": "PT1800S",
+ "lateMessageRejectionPeriod": null,
+ "earlyMessageRejectionPeriod": null,
+ "lateMessageRejectionStartDateTime": null,
+ "configOverrides": null,
+ "idleConfig": null,
+ "stream": "social_media",
+ "useEarliestSequenceNumber": true
+ },
+ "tuningConfig": {
+ "type": "kafka",
+ "appendableIndexSpec": {
+ "type": "onheap",
+ "preserveExistingMetrics": false
+ },
+ "maxRowsInMemory": 150000,
+ "maxBytesInMemory": 0,
+ "skipBytesInMemoryOverheadCheck": false,
+ "maxRowsPerSegment": 5000000,
+ "maxTotalRows": null,
+ "intermediatePersistPeriod": "PT10M",
+ "maxPendingPersists": 0,
+ "indexSpec": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "indexSpecForIntermediatePersists": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "reportParseExceptions": false,
+ "handoffConditionTimeout": 0,
+ "resetOffsetAutomatically": false,
+ "segmentWriteOutMediumFactory": null,
+ "workerThreads": null,
+ "chatRetries": 8,
+ "httpTimeout": "PT10S",
+ "shutdownTimeout": "PT80S",
+ "offsetFetchPeriod": "PT30S",
+ "intermediateHandoffPeriod": "P2147483647D",
+ "logParseExceptions": false,
+ "maxParseExceptions": 2147483647,
+ "maxSavedParseExceptions": 0,
+ "skipSequenceNumberAvailabilityCheck": false,
+ "repartitionTransitionDuration": "PT120S"
+ }
+ },
+ "dataSchema": {
+ "dataSource": "wikipedia_stream",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "iso",
+ "missingValue": null
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ {
+ "type": "string",
+ "name": "username",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "string",
+ "name": "post_title",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "long",
+ "name": "views",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "upvotes",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "comments",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "string",
+ "name": "edited",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ }
+ ],
+ "dimensionExclusions": [
+ "__time"
+ ],
+ "includeAllDimensions": false,
+ "useSchemaDiscovery": false
+ },
+ "metricsSpec": [],
+ "granularitySpec": {
+ "type": "uniform",
+ "segmentGranularity": "HOUR",
+ "queryGranularity": {
+ "type": "none"
+ },
+ "rollup": false,
+ "intervals": []
+ },
+ "transformSpec": {
+ "filter": null,
+ "transforms": []
+ }
+ },
+ "tuningConfig": {
+ "type": "kafka",
+ "appendableIndexSpec": {
+ "type": "onheap",
+ "preserveExistingMetrics": false
+ },
+ "maxRowsInMemory": 150000,
+ "maxBytesInMemory": 0,
+ "skipBytesInMemoryOverheadCheck": false,
+ "maxRowsPerSegment": 5000000,
+ "maxTotalRows": null,
+ "intermediatePersistPeriod": "PT10M",
+ "maxPendingPersists": 0,
+ "indexSpec": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "indexSpecForIntermediatePersists": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "reportParseExceptions": false,
+ "handoffConditionTimeout": 0,
+ "resetOffsetAutomatically": false,
+ "segmentWriteOutMediumFactory": null,
+ "workerThreads": null,
+ "chatRetries": 8,
+ "httpTimeout": "PT10S",
+ "shutdownTimeout": "PT80S",
+ "offsetFetchPeriod": "PT30S",
+ "intermediateHandoffPeriod": "P2147483647D",
+ "logParseExceptions": false,
+ "maxParseExceptions": 2147483647,
+ "maxSavedParseExceptions": 0,
+ "skipSequenceNumberAvailabilityCheck": false,
+ "repartitionTransitionDuration": "PT120S"
+ },
+ "ioConfig": {
+ "topic": "social_media",
+ "inputFormat": {
+ "type": "json"
+ },
+ "replicas": 1,
+ "taskCount": 1,
+ "taskDuration": "PT3600S",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9042"
+ },
+ "autoScalerConfig": null,
+ "pollTimeout": 100,
+ "startDelay": "PT5S",
+ "period": "PT30S",
+ "useEarliestOffset": true,
+ "completionTimeout": "PT1800S",
+ "lateMessageRejectionPeriod": null,
+ "earlyMessageRejectionPeriod": null,
+ "lateMessageRejectionStartDateTime": null,
+ "configOverrides": null,
+ "idleConfig": null,
+ "stream": "social_media",
+ "useEarliestSequenceNumber": true
+ },
+ "context": null,
+ "suspended": false
+ },
+ "suspended": false
+ },
+ {
+ "id": "social_media",
+ "state": "RUNNING",
+ "detailedState": "RUNNING",
+ "healthy": true,
+ "spec": {
+ "type": "kafka",
+ "spec": {
+ "dataSchema": {
+ "dataSource": "social_media",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "iso",
+ "missingValue": null
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ {
+ "type": "string",
+ "name": "username",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "string",
+ "name": "post_title",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "long",
+ "name": "views",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "upvotes",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "comments",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "string",
+ "name": "edited",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ }
+ ],
+ "dimensionExclusions": [
+ "__time"
+ ],
+ "includeAllDimensions": false,
+ "useSchemaDiscovery": false
+ },
+ "metricsSpec": [],
+ "granularitySpec": {
+ "type": "uniform",
+ "segmentGranularity": "HOUR",
+ "queryGranularity": {
+ "type": "none"
+ },
+ "rollup": false,
+ "intervals": []
+ },
+ "transformSpec": {
+ "filter": null,
+ "transforms": []
+ }
+ },
+ "ioConfig": {
+ "topic": "social_media",
+ "inputFormat": {
+ "type": "json"
+ },
+ "replicas": 1,
+ "taskCount": 1,
+ "taskDuration": "PT3600S",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9094"
+ },
+ "autoScalerConfig": null,
+ "pollTimeout": 100,
+ "startDelay": "PT5S",
+ "period": "PT30S",
+ "useEarliestOffset": true,
+ "completionTimeout": "PT1800S",
+ "lateMessageRejectionPeriod": null,
+ "earlyMessageRejectionPeriod": null,
+ "lateMessageRejectionStartDateTime": null,
+ "configOverrides": null,
+ "idleConfig": null,
+ "stream": "social_media",
+ "useEarliestSequenceNumber": true
+ },
+ "tuningConfig": {
+ "type": "kafka",
+ "appendableIndexSpec": {
+ "type": "onheap",
+ "preserveExistingMetrics": false
+ },
+ "maxRowsInMemory": 150000,
+ "maxBytesInMemory": 0,
+ "skipBytesInMemoryOverheadCheck": false,
+ "maxRowsPerSegment": 5000000,
+ "maxTotalRows": null,
+ "intermediatePersistPeriod": "PT10M",
+ "maxPendingPersists": 0,
+ "indexSpec": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "indexSpecForIntermediatePersists": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "reportParseExceptions": false,
+ "handoffConditionTimeout": 0,
+ "resetOffsetAutomatically": false,
+ "segmentWriteOutMediumFactory": null,
+ "workerThreads": null,
+ "chatRetries": 8,
+ "httpTimeout": "PT10S",
+ "shutdownTimeout": "PT80S",
+ "offsetFetchPeriod": "PT30S",
+ "intermediateHandoffPeriod": "P2147483647D",
+ "logParseExceptions": false,
+ "maxParseExceptions": 2147483647,
+ "maxSavedParseExceptions": 0,
+ "skipSequenceNumberAvailabilityCheck": false,
+ "repartitionTransitionDuration": "PT120S"
+ }
+ },
+ "dataSchema": {
+ "dataSource": "social_media",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "iso",
+ "missingValue": null
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ {
+ "type": "string",
+ "name": "username",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "string",
+ "name": "post_title",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "long",
+ "name": "views",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "upvotes",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "comments",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "string",
+ "name": "edited",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ }
+ ],
+ "dimensionExclusions": [
+ "__time"
+ ],
+ "includeAllDimensions": false,
+ "useSchemaDiscovery": false
+ },
+ "metricsSpec": [],
+ "granularitySpec": {
+ "type": "uniform",
+ "segmentGranularity": "HOUR",
+ "queryGranularity": {
+ "type": "none"
+ },
+ "rollup": false,
+ "intervals": []
+ },
+ "transformSpec": {
+ "filter": null,
+ "transforms": []
+ }
+ },
+ "tuningConfig": {
+ "type": "kafka",
+ "appendableIndexSpec": {
+ "type": "onheap",
+ "preserveExistingMetrics": false
+ },
+ "maxRowsInMemory": 150000,
+ "maxBytesInMemory": 0,
+ "skipBytesInMemoryOverheadCheck": false,
+ "maxRowsPerSegment": 5000000,
+ "maxTotalRows": null,
+ "intermediatePersistPeriod": "PT10M",
+ "maxPendingPersists": 0,
+ "indexSpec": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "indexSpecForIntermediatePersists": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "reportParseExceptions": false,
+ "handoffConditionTimeout": 0,
+ "resetOffsetAutomatically": false,
+ "segmentWriteOutMediumFactory": null,
+ "workerThreads": null,
+ "chatRetries": 8,
+ "httpTimeout": "PT10S",
+ "shutdownTimeout": "PT80S",
+ "offsetFetchPeriod": "PT30S",
+ "intermediateHandoffPeriod": "P2147483647D",
+ "logParseExceptions": false,
+ "maxParseExceptions": 2147483647,
+ "maxSavedParseExceptions": 0,
+ "skipSequenceNumberAvailabilityCheck": false,
+ "repartitionTransitionDuration": "PT120S"
+ },
+ "ioConfig": {
+ "topic": "social_media",
+ "inputFormat": {
+ "type": "json"
+ },
+ "replicas": 1,
+ "taskCount": 1,
+ "taskDuration": "PT3600S",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9094"
+ },
+ "autoScalerConfig": null,
+ "pollTimeout": 100,
+ "startDelay": "PT5S",
+ "period": "PT30S",
+ "useEarliestOffset": true,
+ "completionTimeout": "PT1800S",
+ "lateMessageRejectionPeriod": null,
+ "earlyMessageRejectionPeriod": null,
+ "lateMessageRejectionStartDateTime": null,
+ "configOverrides": null,
+ "idleConfig": null,
+ "stream": "social_media",
+ "useEarliestSequenceNumber": true
+ },
+ "context": null,
+ "suspended": false
+ },
+ "suspended": false
+ }
+ ]
+ ```
+</details>
+
+### Get an array of supervisor states
+
+Retrieves an array of objects representing active supervisors and their current state. If there are no active supervisors, it returns an empty array. For reference on the supervisor object properties, see the preceding [table](#supervisor-information).
+
+#### URL
+
+`GET` `/druid/indexer/v1/supervisor?state=true`
+
+#### Responses
+
+<Tabs>
+
+<TabItem value="7" label="200 SUCCESS">
+
+
+*Successfully retrieved supervisor state objects*
+
+</TabItem>
+</Tabs>
+
+---
+
+#### Sample request
+
+<Tabs>
+
+<TabItem value="8" label="cURL">
+
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor?state=true"
+```
+
+</TabItem>
+<TabItem value="9" label="HTTP">
+
+
+```HTTP
+GET /druid/indexer/v1/supervisor?state=true HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+</TabItem>
+</Tabs>
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+ [
+ {
+ "id": "wikipedia_stream",
+ "state": "UNHEALTHY_SUPERVISOR",
+ "detailedState": "UNABLE_TO_CONNECT_TO_STREAM",
+ "healthy": false,
+ "suspended": false
+ },
+ {
+ "id": "social_media",
+ "state": "RUNNING",
+ "detailedState": "RUNNING",
+ "healthy": true,
+ "suspended": false
+ }
+ ]
+ ```
+
+</details>
+
+### Get supervisor specification
+
+Retrieves the specification for a single supervisor. The returned specification includes the `dataSchema`, `ioConfig`, and `tuningConfig` objects.
+
+#### URL
+
+`GET` `/druid/indexer/v1/supervisor/{supervisorId}`
+
+#### Responses
+
+<Tabs>
+
+<TabItem value="10" label="200 SUCCESS">
+
+
+*Successfully retrieved supervisor spec*
+
+</TabItem>
+<TabItem value="11" label="404 NOT FOUND">
+
+
+*Invalid supervisor ID*
+
+</TabItem>
+</Tabs>
+
+---
+
+#### Sample request
+
+The following example shows how to retrieve the specification of a supervisor with the name `wikipedia_stream`.
+
+<Tabs>
+
+<TabItem value="12" label="cURL">
+
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor/wikipedia_stream"
+```
+
+</TabItem>
+<TabItem value="13" label="HTTP">
+
+
+```HTTP
+GET /druid/indexer/v1/supervisor/wikipedia_stream HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+</TabItem>
+</Tabs>
+
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+{
+ "type": "kafka",
+ "spec": {
+ "dataSchema": {
+ "dataSource": "social_media",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "iso",
+ "missingValue": null
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ {
+ "type": "string",
+ "name": "username",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "string",
+ "name": "post_title",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "long",
+ "name": "views",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "upvotes",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "comments",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "string",
+ "name": "edited",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ }
+ ],
+ "dimensionExclusions": [
+ "__time"
+ ],
+ "includeAllDimensions": false,
+ "useSchemaDiscovery": false
+ },
+ "metricsSpec": [],
+ "granularitySpec": {
+ "type": "uniform",
+ "segmentGranularity": "HOUR",
+ "queryGranularity": {
+ "type": "none"
+ },
+ "rollup": false,
+ "intervals": []
+ },
+ "transformSpec": {
+ "filter": null,
+ "transforms": []
+ }
+ },
+ "ioConfig": {
+ "topic": "social_media",
+ "inputFormat": {
+ "type": "json"
+ },
+ "replicas": 1,
+ "taskCount": 1,
+ "taskDuration": "PT3600S",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9094"
+ },
+ "autoScalerConfig": null,
+ "pollTimeout": 100,
+ "startDelay": "PT5S",
+ "period": "PT30S",
+ "useEarliestOffset": true,
+ "completionTimeout": "PT1800S",
+ "lateMessageRejectionPeriod": null,
+ "earlyMessageRejectionPeriod": null,
+ "lateMessageRejectionStartDateTime": null,
+ "configOverrides": null,
+ "idleConfig": null,
+ "stream": "social_media",
+ "useEarliestSequenceNumber": true
+ },
+ "tuningConfig": {
+ "type": "kafka",
+ "appendableIndexSpec": {
+ "type": "onheap",
+ "preserveExistingMetrics": false
+ },
+ "maxRowsInMemory": 150000,
+ "maxBytesInMemory": 0,
+ "skipBytesInMemoryOverheadCheck": false,
+ "maxRowsPerSegment": 5000000,
+ "maxTotalRows": null,
+ "intermediatePersistPeriod": "PT10M",
+ "maxPendingPersists": 0,
+ "indexSpec": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "indexSpecForIntermediatePersists": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "reportParseExceptions": false,
+ "handoffConditionTimeout": 0,
+ "resetOffsetAutomatically": false,
+ "segmentWriteOutMediumFactory": null,
+ "workerThreads": null,
+ "chatRetries": 8,
+ "httpTimeout": "PT10S",
+ "shutdownTimeout": "PT80S",
+ "offsetFetchPeriod": "PT30S",
+ "intermediateHandoffPeriod": "P2147483647D",
+ "logParseExceptions": false,
+ "maxParseExceptions": 2147483647,
+ "maxSavedParseExceptions": 0,
+ "skipSequenceNumberAvailabilityCheck": false,
+ "repartitionTransitionDuration": "PT120S"
+ }
+ },
+ "dataSchema": {
+ "dataSource": "social_media",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "iso",
+ "missingValue": null
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ {
+ "type": "string",
+ "name": "username",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "string",
+ "name": "post_title",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "long",
+ "name": "views",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "upvotes",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "comments",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "string",
+ "name": "edited",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ }
+ ],
+ "dimensionExclusions": [
+ "__time"
+ ],
+ "includeAllDimensions": false,
+ "useSchemaDiscovery": false
+ },
+ "metricsSpec": [],
+ "granularitySpec": {
+ "type": "uniform",
+ "segmentGranularity": "HOUR",
+ "queryGranularity": {
+ "type": "none"
+ },
+ "rollup": false,
+ "intervals": []
+ },
+ "transformSpec": {
+ "filter": null,
+ "transforms": []
+ }
+ },
+ "tuningConfig": {
+ "type": "kafka",
+ "appendableIndexSpec": {
+ "type": "onheap",
+ "preserveExistingMetrics": false
+ },
+ "maxRowsInMemory": 150000,
+ "maxBytesInMemory": 0,
+ "skipBytesInMemoryOverheadCheck": false,
+ "maxRowsPerSegment": 5000000,
+ "maxTotalRows": null,
+ "intermediatePersistPeriod": "PT10M",
+ "maxPendingPersists": 0,
+ "indexSpec": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "indexSpecForIntermediatePersists": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "reportParseExceptions": false,
+ "handoffConditionTimeout": 0,
+ "resetOffsetAutomatically": false,
+ "segmentWriteOutMediumFactory": null,
+ "workerThreads": null,
+ "chatRetries": 8,
+ "httpTimeout": "PT10S",
+ "shutdownTimeout": "PT80S",
+ "offsetFetchPeriod": "PT30S",
+ "intermediateHandoffPeriod": "P2147483647D",
+ "logParseExceptions": false,
+ "maxParseExceptions": 2147483647,
+ "maxSavedParseExceptions": 0,
+ "skipSequenceNumberAvailabilityCheck": false,
+ "repartitionTransitionDuration": "PT120S"
+ },
+ "ioConfig": {
+ "topic": "social_media",
+ "inputFormat": {
+ "type": "json"
+ },
+ "replicas": 1,
+ "taskCount": 1,
+ "taskDuration": "PT3600S",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9094"
+ },
+ "autoScalerConfig": null,
+ "pollTimeout": 100,
+ "startDelay": "PT5S",
+ "period": "PT30S",
+ "useEarliestOffset": true,
+ "completionTimeout": "PT1800S",
+ "lateMessageRejectionPeriod": null,
+ "earlyMessageRejectionPeriod": null,
+ "lateMessageRejectionStartDateTime": null,
+ "configOverrides": null,
+ "idleConfig": null,
+ "stream": "social_media",
+ "useEarliestSequenceNumber": true
+ },
+ "context": null,
+ "suspended": false
+}
+ ```
+</details>
+
+### Get supervisor status
+
+Retrieves the current status report for a single supervisor. The report contains the state of the supervisor tasks and an array of recently thrown exceptions.
+
+For additional information about the status report, see [Supervisor reference](../ingestion/supervisor.md#status-report).
+
+#### URL
+
+`GET` `/druid/indexer/v1/supervisor/{supervisorId}/status`
+
+#### Responses
+
+<Tabs>
+
+<TabItem value="14" label="200 SUCCESS">
+
+
+*Successfully retrieved supervisor status*
+
+</TabItem>
+<TabItem value="15" label="404 NOT FOUND">
+
+
+*Invalid supervisor ID*
+
+</TabItem>
+</Tabs>
+
+---
+
+#### Sample request
+
+The following example shows how to retrieve the status of a supervisor with the name `social_media`.
+
+<Tabs>
+
+<TabItem value="16" label="cURL">
+
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor/social_media/status"
+```
+
+</TabItem>
+<TabItem value="17" label="HTTP">
+
+
+```HTTP
+GET /druid/indexer/v1/supervisor/social_media/status HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+</TabItem>
+</Tabs>
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+ {
+ "id": "social_media",
+ "generationTime": "2023-07-05T23:24:43.934Z",
+ "payload": {
+ "dataSource": "social_media",
+ "stream": "social_media",
+ "partitions": 1,
+ "replicas": 1,
+ "durationSeconds": 3600,
+ "activeTasks": [
+ {
+ "id": "index_kafka_social_media_ab72ae4127c591c_flcbhdlh",
+ "startingOffsets": {
+ "0": 3176381
+ },
+ "startTime": "2023-07-05T23:21:39.321Z",
+ "remainingSeconds": 3415,
+ "type": "ACTIVE",
+ "currentOffsets": {
+ "0": 3296632
+ },
+ "lag": {
+ "0": 3
+ }
+ }
+ ],
+ "publishingTasks": [],
+ "latestOffsets": {
+ "0": 3296635
+ },
+ "minimumLag": {
+ "0": 3
+ },
+ "aggregateLag": 3,
+ "offsetsLastUpdated": "2023-07-05T23:24:30.212Z",
+ "suspended": false,
+ "healthy": true,
+ "state": "RUNNING",
+ "detailedState": "RUNNING",
+ "recentErrors": []
+ }
+ }
+ ```
+</details>
+
+### Get supervisor health
+
+Retrieves the current health report for a single supervisor. The health of a supervisor is determined by the supervisor's `state` (as returned by the `/status` endpoint) and the `druid.supervisor.*` Overlord configuration thresholds.
+
+#### URL
+
+`GET` `/druid/indexer/v1/supervisor/{supervisorId}/health`
+
+#### Responses
+
+<Tabs>
+
+<TabItem value="18" label="200 SUCCESS">
+
+*Supervisor is healthy*
+
+</TabItem>
+
+<TabItem value="19" label="404 NOT FOUND">
+
+*Invalid supervisor ID*
+
+</TabItem>
+
+<TabItem value="20" label="503 SERVICE UNAVAILABLE">
+
+*Supervisor is unhealthy*
+
+</TabItem>
+
+</Tabs>
+
+---
+
+#### Sample request
+
+The following example shows how to retrieve the health report for a supervisor with the name `social_media`.
+
+<Tabs>
+
+<TabItem value="21" label="cURL">
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor/social_media/health"
+```
+</TabItem>
+
+<TabItem value="22" label="HTTP">
+
+```HTTP
+GET /druid/indexer/v1/supervisor/social_media/health HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+</TabItem>
+
+</Tabs>
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+ {
+ "healthy": false
+ }
+ ```
+</details>
+
+### Get supervisor ingestion stats
+
+Returns a snapshot of the current ingestion row counters for each task being managed by the supervisor, along with moving averages for the row counters. See [Row stats](../ingestion/tasks.md#row-stats) for more information.
+
+#### URL
+
+`GET` `/druid/indexer/v1/supervisor/{supervisorId}/stats`
+
+#### Responses
+
+<Tabs>
+
+<TabItem value="23" label="200 SUCCESS">
+
+*Successfully retrieved supervisor stats*
+
+</TabItem>
+
+<TabItem value="24" label="404 NOT FOUND">
+
+*Invalid supervisor ID*
+
+</TabItem>
+
+</Tabs>
+
+---
+
+#### Sample request
+
+The following example shows how to retrieve the current ingestion row counters for a supervisor with the name `custom_data`.
+
+<Tabs>
+
+<TabItem value="25" label="cURL">
+
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor/custom_data/stats"
+```
+
+</TabItem>
+<TabItem value="26" label="HTTP">
+
+
+```HTTP
+GET /druid/indexer/v1/supervisor/custom_data/stats HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+</TabItem>
+</Tabs>
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+ {
+ "0": {
+ "index_kafka_custom_data_881d621078f6b7c_ccplchbi": {
+ "movingAverages": {
+ "buildSegments": {
+ "5m": {
+ "processed": 53.401225142603316,
+ "processedBytes": 5226.400757148808,
+ "unparseable": 0.0,
+ "thrownAway": 0.0,
+ "processedWithError": 0.0
+ },
+ "15m": {
+ "processed": 56.92994990102502,
+ "processedBytes": 5571.772059828217,
+ "unparseable": 0.0,
+ "thrownAway": 0.0,
+ "processedWithError": 0.0
+ },
+ "1m": {
+ "processed": 37.134921285556636,
+ "processedBytes": 3634.2766230628677,
+ "unparseable": 0.0,
+ "thrownAway": 0.0,
+ "processedWithError": 0.0
+ }
+ }
+ },
+ "totals": {
+ "buildSegments": {
+ "processed": 665,
+ "processedBytes": 65079,
+ "processedWithError": 0,
+ "thrownAway": 0,
+ "unparseable": 0
+ }
+ }
+ }
+ }
+ }
+ ```
+</details>
+
+## Audit history
+
+An audit history provides a comprehensive log of events, including supervisor configuration, creation, suspension, and modification history.
+
+### Get audit history for all supervisors
+
+Retrieves an audit history of specs for all supervisors.
+
+#### URL
+
+`GET` `/druid/indexer/v1/supervisor/history`
+
+#### Responses
+
+<Tabs>
+
+<TabItem value="27" label="200 SUCCESS">
+
+
+*Successfully retrieved audit history*
+
+</TabItem>
+</Tabs>
+
+---
+
+#### Sample request
+
+<Tabs>
+
+<TabItem value="28" label="cURL">
+
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor/history"
+```
+
+</TabItem>
+<TabItem value="29" label="HTTP">
+
+
+```HTTP
+GET /druid/indexer/v1/supervisor/history HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+</TabItem>
+</Tabs>
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+{
+ "social_media": [
+ {
+ "spec": {
+ "type": "kafka",
+ "spec": {
+ "dataSchema": {
+ "dataSource": "social_media",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "iso",
+ "missingValue": null
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ {
+ "type": "string",
+ "name": "username",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "string",
+ "name": "post_title",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "long",
+ "name": "views",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "upvotes",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "comments",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "string",
+ "name": "edited",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ }
+ ],
+ "dimensionExclusions": [
+ "__time"
+ ],
+ "includeAllDimensions": false,
+ "useSchemaDiscovery": false
+ },
+ "metricsSpec": [],
+ "granularitySpec": {
+ "type": "uniform",
+ "segmentGranularity": "HOUR",
+ "queryGranularity": {
+ "type": "none"
+ },
+ "rollup": false,
+ "intervals": []
+ },
+ "transformSpec": {
+ "filter": null,
+ "transforms": []
+ }
+ },
+ "ioConfig": {
+ "topic": "social_media",
+ "inputFormat": {
+ "type": "json"
+ },
+ "replicas": 1,
+ "taskCount": 1,
+ "taskDuration": "PT3600S",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9094"
+ },
+ "autoScalerConfig": null,
+ "pollTimeout": 100,
+ "startDelay": "PT5S",
+ "period": "PT30S",
+ "useEarliestOffset": true,
+ "completionTimeout": "PT1800S",
+ "lateMessageRejectionPeriod": null,
+ "earlyMessageRejectionPeriod": null,
+ "lateMessageRejectionStartDateTime": null,
+ "configOverrides": null,
+ "idleConfig": null,
+ "stream": "social_media",
+ "useEarliestSequenceNumber": true
+ },
+ "tuningConfig": {
+ "type": "kafka",
+ "appendableIndexSpec": {
+ "type": "onheap",
+ "preserveExistingMetrics": false
+ },
+ "maxRowsInMemory": 150000,
+ "maxBytesInMemory": 0,
+ "skipBytesInMemoryOverheadCheck": false,
+ "maxRowsPerSegment": 5000000,
+ "maxTotalRows": null,
+ "intermediatePersistPeriod": "PT10M",
+ "maxPendingPersists": 0,
+ "indexSpec": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "indexSpecForIntermediatePersists": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "reportParseExceptions": false,
+ "handoffConditionTimeout": 0,
+ "resetOffsetAutomatically": false,
+ "segmentWriteOutMediumFactory": null,
+ "workerThreads": null,
+ "chatRetries": 8,
+ "httpTimeout": "PT10S",
+ "shutdownTimeout": "PT80S",
+ "offsetFetchPeriod": "PT30S",
+ "intermediateHandoffPeriod": "P2147483647D",
+ "logParseExceptions": false,
+ "maxParseExceptions": 2147483647,
+ "maxSavedParseExceptions": 0,
+ "skipSequenceNumberAvailabilityCheck": false,
+ "repartitionTransitionDuration": "PT120S"
+ }
+ },
+ "dataSchema": {
+ "dataSource": "social_media",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "iso",
+ "missingValue": null
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ {
+ "type": "string",
+ "name": "username",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "string",
+ "name": "post_title",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "long",
+ "name": "views",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "upvotes",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "comments",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "string",
+ "name": "edited",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ }
+ ],
+ "dimensionExclusions": [
+ "__time"
+ ],
+ "includeAllDimensions": false,
+ "useSchemaDiscovery": false
+ },
+ "metricsSpec": [],
+ "granularitySpec": {
+ "type": "uniform",
+ "segmentGranularity": "HOUR",
+ "queryGranularity": {
+ "type": "none"
+ },
+ "rollup": false,
+ "intervals": []
+ },
+ "transformSpec": {
+ "filter": null,
+ "transforms": []
+ }
+ },
+ "tuningConfig": {
+ "type": "kafka",
+ "appendableIndexSpec": {
+ "type": "onheap",
+ "preserveExistingMetrics": false
+ },
+ "maxRowsInMemory": 150000,
+ "maxBytesInMemory": 0,
+ "skipBytesInMemoryOverheadCheck": false,
+ "maxRowsPerSegment": 5000000,
+ "maxTotalRows": null,
+ "intermediatePersistPeriod": "PT10M",
+ "maxPendingPersists": 0,
+ "indexSpec": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "indexSpecForIntermediatePersists": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "reportParseExceptions": false,
+ "handoffConditionTimeout": 0,
+ "resetOffsetAutomatically": false,
+ "segmentWriteOutMediumFactory": null,
+ "workerThreads": null,
+ "chatRetries": 8,
+ "httpTimeout": "PT10S",
+ "shutdownTimeout": "PT80S",
+ "offsetFetchPeriod": "PT30S",
+ "intermediateHandoffPeriod": "P2147483647D",
+ "logParseExceptions": false,
+ "maxParseExceptions": 2147483647,
+ "maxSavedParseExceptions": 0,
+ "skipSequenceNumberAvailabilityCheck": false,
+ "repartitionTransitionDuration": "PT120S"
+ },
+ "ioConfig": {
+ "topic": "social_media",
+ "inputFormat": {
+ "type": "json"
+ },
+ "replicas": 1,
+ "taskCount": 1,
+ "taskDuration": "PT3600S",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9094"
+ },
+ "autoScalerConfig": null,
+ "pollTimeout": 100,
+ "startDelay": "PT5S",
+ "period": "PT30S",
+ "useEarliestOffset": true,
+ "completionTimeout": "PT1800S",
+ "lateMessageRejectionPeriod": null,
+ "earlyMessageRejectionPeriod": null,
+ "lateMessageRejectionStartDateTime": null,
+ "configOverrides": null,
+ "idleConfig": null,
+ "stream": "social_media",
+ "useEarliestSequenceNumber": true
+ },
+ "context": null,
+ "suspended": false
+ },
+ "version": "2023-07-03T18:51:02.970Z"
+ }
+ ]
+}
+ ```
+</details>
+
+### Get audit history for a specific supervisor
+
+Retrieves an audit history of specs for a single supervisor.
+
+#### URL
+
+`GET` `/druid/indexer/v1/supervisor/{supervisorId}/history`
+
+
+#### Responses
+
+<Tabs>
+
+<TabItem value="30" label="200 SUCCESS">
+
+
+*Successfully retrieved supervisor audit history*
+
+</TabItem>
+<TabItem value="31" label="404 NOT FOUND">
+
+
+*Invalid supervisor ID*
+
+</TabItem>
+</Tabs>
+
+---
+
+#### Sample request
+
+The following example shows how to retrieve the audit history of a supervisor with the name `wikipedia_stream`.
+
+<Tabs>
+
+<TabItem value="23" label="cURL">
+
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor/wikipedia_stream/history"
+```
+
+</TabItem>
+<TabItem value="32" label="HTTP">
+
+
+```HTTP
+GET /druid/indexer/v1/supervisor/wikipedia_stream/history HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+</TabItem>
+</Tabs>
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+[
+ {
+ "spec": {
+ "type": "kafka",
+ "spec": {
+ "dataSchema": {
+ "dataSource": "wikipedia_stream",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "iso",
+ "missingValue": null
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ {
+ "type": "string",
+ "name": "username",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "string",
+ "name": "post_title",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "long",
+ "name": "views",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "upvotes",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "comments",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "string",
+ "name": "edited",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ }
+ ],
+ "dimensionExclusions": [
+ "__time"
+ ],
+ "includeAllDimensions": false,
+ "useSchemaDiscovery": false
+ },
+ "metricsSpec": [],
+ "granularitySpec": {
+ "type": "uniform",
+ "segmentGranularity": "HOUR",
+ "queryGranularity": {
+ "type": "none"
+ },
+ "rollup": false,
+ "intervals": []
+ },
+ "transformSpec": {
+ "filter": null,
+ "transforms": []
+ }
+ },
+ "ioConfig": {
+ "topic": "social_media",
+ "inputFormat": {
+ "type": "json"
+ },
+ "replicas": 1,
+ "taskCount": 1,
+ "taskDuration": "PT3600S",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9042"
+ },
+ "autoScalerConfig": null,
+ "pollTimeout": 100,
+ "startDelay": "PT5S",
+ "period": "PT30S",
+ "useEarliestOffset": true,
+ "completionTimeout": "PT1800S",
+ "lateMessageRejectionPeriod": null,
+ "earlyMessageRejectionPeriod": null,
+ "lateMessageRejectionStartDateTime": null,
+ "configOverrides": null,
+ "idleConfig": null,
+ "stream": "social_media",
+ "useEarliestSequenceNumber": true
+ },
+ "tuningConfig": {
+ "type": "kafka",
+ "appendableIndexSpec": {
+ "type": "onheap",
+ "preserveExistingMetrics": false
+ },
+ "maxRowsInMemory": 150000,
+ "maxBytesInMemory": 0,
+ "skipBytesInMemoryOverheadCheck": false,
+ "maxRowsPerSegment": 5000000,
+ "maxTotalRows": null,
+ "intermediatePersistPeriod": "PT10M",
+ "maxPendingPersists": 0,
+ "indexSpec": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "indexSpecForIntermediatePersists": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "reportParseExceptions": false,
+ "handoffConditionTimeout": 0,
+ "resetOffsetAutomatically": false,
+ "segmentWriteOutMediumFactory": null,
+ "workerThreads": null,
+ "chatRetries": 8,
+ "httpTimeout": "PT10S",
+ "shutdownTimeout": "PT80S",
+ "offsetFetchPeriod": "PT30S",
+ "intermediateHandoffPeriod": "P2147483647D",
+ "logParseExceptions": false,
+ "maxParseExceptions": 2147483647,
+ "maxSavedParseExceptions": 0,
+ "skipSequenceNumberAvailabilityCheck": false,
+ "repartitionTransitionDuration": "PT120S"
+ }
+ },
+ "dataSchema": {
+ "dataSource": "wikipedia_stream",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "iso",
+ "missingValue": null
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ {
+ "type": "string",
+ "name": "username",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "string",
+ "name": "post_title",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "long",
+ "name": "views",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "upvotes",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "comments",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "string",
+ "name": "edited",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ }
+ ],
+ "dimensionExclusions": [
+ "__time"
+ ],
+ "includeAllDimensions": false,
+ "useSchemaDiscovery": false
+ },
+ "metricsSpec": [],
+ "granularitySpec": {
+ "type": "uniform",
+ "segmentGranularity": "HOUR",
+ "queryGranularity": {
+ "type": "none"
+ },
+ "rollup": false,
+ "intervals": []
+ },
+ "transformSpec": {
+ "filter": null,
+ "transforms": []
+ }
+ },
+ "tuningConfig": {
+ "type": "kafka",
+ "appendableIndexSpec": {
+ "type": "onheap",
+ "preserveExistingMetrics": false
+ },
+ "maxRowsInMemory": 150000,
+ "maxBytesInMemory": 0,
+ "skipBytesInMemoryOverheadCheck": false,
+ "maxRowsPerSegment": 5000000,
+ "maxTotalRows": null,
+ "intermediatePersistPeriod": "PT10M",
+ "maxPendingPersists": 0,
+ "indexSpec": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "indexSpecForIntermediatePersists": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "reportParseExceptions": false,
+ "handoffConditionTimeout": 0,
+ "resetOffsetAutomatically": false,
+ "segmentWriteOutMediumFactory": null,
+ "workerThreads": null,
+ "chatRetries": 8,
+ "httpTimeout": "PT10S",
+ "shutdownTimeout": "PT80S",
+ "offsetFetchPeriod": "PT30S",
+ "intermediateHandoffPeriod": "P2147483647D",
+ "logParseExceptions": false,
+ "maxParseExceptions": 2147483647,
+ "maxSavedParseExceptions": 0,
+ "skipSequenceNumberAvailabilityCheck": false,
+ "repartitionTransitionDuration": "PT120S"
+ },
+ "ioConfig": {
+ "topic": "social_media",
+ "inputFormat": {
+ "type": "json"
+ },
+ "replicas": 1,
+ "taskCount": 1,
+ "taskDuration": "PT3600S",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9042"
+ },
+ "autoScalerConfig": null,
+ "pollTimeout": 100,
+ "startDelay": "PT5S",
+ "period": "PT30S",
+ "useEarliestOffset": true,
+ "completionTimeout": "PT1800S",
+ "lateMessageRejectionPeriod": null,
+ "earlyMessageRejectionPeriod": null,
+ "lateMessageRejectionStartDateTime": null,
+ "configOverrides": null,
+ "idleConfig": null,
+ "stream": "social_media",
+ "useEarliestSequenceNumber": true
+ },
+ "context": null,
+ "suspended": false
+ },
+ "version": "2023-07-05T20:59:16.872Z"
+ }
+]
+ ```
+</details>
+
+## Manage supervisors
+
+### Create or update a supervisor
+
+Creates a new supervisor spec or updates an existing one with new configuration and schema information. When updating a supervisor spec, the datasource must remain the same as the previous supervisor.
+
+You can define a supervisor spec for [Apache Kafka](../ingestion/kafka-ingestion.md) or [Amazon Kinesis](../ingestion/kinesis-ingestion.md) streaming ingestion methods.
+
+The following table lists the properties of a supervisor spec:
+
+|Property|Type|Description|Required|
+|--------|----|-----------|--------|
+|`type`|String|The supervisor type. One of`kafka` or `kinesis`.|Yes|
+|`spec`|Object|The container object for the supervisor configuration.|Yes|
+|`ioConfig`|Object|The I/O configuration object to define the connection and I/O-related settings for the supervisor and indexing task.|Yes|
+|`dataSchema`|Object|The schema for the indexing task to use during ingestion. See [`dataSchema`](../ingestion/ingestion-spec.md#dataschema) for more information.|Yes|
+|`tuningConfig`|Object|The tuning configuration object to define performance-related settings for the supervisor and indexing tasks.|No|
+
+When you call this endpoint on an existing supervisor, the running supervisor signals its tasks to stop reading and begin publishing, exiting itself. Druid then uses the provided configuration from the request body to create a new supervisor. Druid submits a new schema while retaining existing publishing tasks and starts new tasks at the previous task offsets.
+This way, you can apply configuration changes without a pause in ingestion.
+
+#### URL
+
+`POST` `/druid/indexer/v1/supervisor`
+
+#### Responses
+
+<Tabs>
+
+<TabItem value="33" label="200 SUCCESS">
+
+
+*Successfully created a new supervisor or updated an existing supervisor*
+
+</TabItem>
+<TabItem value="34" label="415 UNSUPPORTED MEDIA TYPE">
+
+
+*Request body content type is not in JSON format*
+
+</TabItem>
+</Tabs>
+
+---
+
+#### Sample request
+
+The following example uses JSON input format to create a supervisor spec for Kafka with a `social_media` datasource and `social_media` topic.
+
+<Tabs>
+
+<TabItem value="35" label="cURL">
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor" \
+--header 'Content-Type: application/json' \
+--data '{
+ "type": "kafka",
+ "spec": {
+ "ioConfig": {
+ "type": "kafka",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9094"
+ },
+ "topic": "social_media",
+ "inputFormat": {
+ "type": "json"
+ },
+ "useEarliestOffset": true
+ },
+ "tuningConfig": {
+ "type": "kafka"
+ },
+ "dataSchema": {
+ "dataSource": "social_media",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "iso"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "username",
+ "post_title",
+ {
+ "type": "long",
+ "name": "views"
+ },
+ {
+ "type": "long",
+ "name": "upvotes"
+ },
+ {
+ "type": "long",
+ "name": "comments"
+ },
+ "edited"
+ ]
+ },
+ "granularitySpec": {
+ "queryGranularity": "none",
+ "rollup": false,
+ "segmentGranularity": "hour"
+ }
+ }
+ }
+}'
+```
+
+</TabItem>
+
+<TabItem value="36" label="HTTP">
+
+```HTTP
+POST /druid/indexer/v1/supervisor HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+Content-Type: application/json
+Content-Length: 1359
+
+{
+ "type": "kafka",
+ "spec": {
+ "ioConfig": {
+ "type": "kafka",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9094"
+ },
+ "topic": "social_media",
+ "inputFormat": {
+ "type": "json"
+ },
+ "useEarliestOffset": true
+ },
+ "tuningConfig": {
+ "type": "kafka"
+ },
+ "dataSchema": {
+ "dataSource": "social_media",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "iso"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "username",
+ "post_title",
+ {
+ "type": "long",
+ "name": "views"
+ },
+ {
+ "type": "long",
+ "name": "upvotes"
+ },
+ {
+ "type": "long",
+ "name": "comments"
+ },
+ "edited"
+ ]
+ },
+ "granularitySpec": {
+ "queryGranularity": "none",
+ "rollup": false,
+ "segmentGranularity": "hour"
+ }
+ }
+ }
+}
+```
+
+</TabItem>
+</Tabs>
+
+#### Sample request with `skipRestartIfUnmodified`
+
+The following example sets the `skipRestartIfUnmodified` flag to true. With this flag set to true, the Supervisor will only restart if there has been a modification to the SupervisorSpec. If left unset, the flag defaults to false.
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor?skipRestartIfUnmodified=true" \
+--header 'Content-Type: application/json' \
+--data '{
+ "type": "kafka",
+ "spec": {
+ "ioConfig": {
+ "type": "kafka",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9094"
+ },
+ "topic": "social_media",
+ "inputFormat": {
+ "type": "json"
+ },
+ "useEarliestOffset": true
+ },
+ "tuningConfig": {
+ "type": "kafka"
+ },
+ "dataSchema": {
+ "dataSource": "social_media",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "iso"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "username",
+ "post_title",
+ {
+ "type": "long",
+ "name": "views"
+ },
+ {
+ "type": "long",
+ "name": "upvotes"
+ },
+ {
+ "type": "long",
+ "name": "comments"
+ },
+ "edited"
+ ]
+ },
+ "granularitySpec": {
+ "queryGranularity": "none",
+ "rollup": false,
+ "segmentGranularity": "hour"
+ }
+ }
+ }
+}'
+```
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+{
+ "id": "social_media"
+}
+ ```
+</details>
+
+### Suspend a running supervisor
+
+Suspends a single running supervisor. Returns the updated supervisor spec, where the `suspended` property is set to `true`. The suspended supervisor continues to emit logs and metrics.
+Indexing tasks remain suspended until you [resume the supervisor](#resume-a-supervisor).
+
+#### URL
+
+`POST` `/druid/indexer/v1/supervisor/{supervisorId}/suspend`
+
+#### Responses
+
+<Tabs>
+
+<TabItem value="37" label="200 SUCCESS">
+
+
+*Successfully shut down supervisor*
+
+</TabItem>
+<TabItem value="38" label="400 BAD REQUEST">
+
+
+*Supervisor already suspended*
+
+</TabItem>
+<TabItem value="39" label="404 NOT FOUND">
+
+
+*Invalid supervisor ID*
+
+</TabItem>
+</Tabs>
+
+---
+
+#### Sample request
+
+The following example shows how to suspend a running supervisor with the name `social_media`.
+
+<Tabs>
+
+<TabItem value="40" label="cURL">
+
+
+```shell
+curl --request POST "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor/social_media/suspend"
+```
+
+</TabItem>
+<TabItem value="41" label="HTTP">
+
+
+```HTTP
+POST /druid/indexer/v1/supervisor/social_media/suspend HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+</TabItem>
+</Tabs>
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+{
+ "type": "kafka",
+ "spec": {
+ "dataSchema": {
+ "dataSource": "social_media",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "iso",
+ "missingValue": null
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ {
+ "type": "string",
+ "name": "username",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "string",
+ "name": "post_title",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "long",
+ "name": "views",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "upvotes",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "comments",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "string",
+ "name": "edited",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ }
+ ],
+ "dimensionExclusions": [
+ "__time"
+ ],
+ "includeAllDimensions": false,
+ "useSchemaDiscovery": false
+ },
+ "metricsSpec": [],
+ "granularitySpec": {
+ "type": "uniform",
+ "segmentGranularity": "HOUR",
+ "queryGranularity": {
+ "type": "none"
+ },
+ "rollup": false,
+ "intervals": []
+ },
+ "transformSpec": {
+ "filter": null,
+ "transforms": []
+ }
+ },
+ "ioConfig": {
+ "topic": "social_media",
+ "inputFormat": {
+ "type": "json"
+ },
+ "replicas": 1,
+ "taskCount": 1,
+ "taskDuration": "PT3600S",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9094"
+ },
+ "autoScalerConfig": null,
+ "pollTimeout": 100,
+ "startDelay": "PT5S",
+ "period": "PT30S",
+ "useEarliestOffset": true,
+ "completionTimeout": "PT1800S",
+ "lateMessageRejectionPeriod": null,
+ "earlyMessageRejectionPeriod": null,
+ "lateMessageRejectionStartDateTime": null,
+ "configOverrides": null,
+ "idleConfig": null,
+ "stream": "social_media",
+ "useEarliestSequenceNumber": true
+ },
+ "tuningConfig": {
+ "type": "kafka",
+ "appendableIndexSpec": {
+ "type": "onheap",
+ "preserveExistingMetrics": false
+ },
+ "maxRowsInMemory": 150000,
+ "maxBytesInMemory": 0,
+ "skipBytesInMemoryOverheadCheck": false,
+ "maxRowsPerSegment": 5000000,
+ "maxTotalRows": null,
+ "intermediatePersistPeriod": "PT10M",
+ "maxPendingPersists": 0,
+ "indexSpec": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "indexSpecForIntermediatePersists": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "reportParseExceptions": false,
+ "handoffConditionTimeout": 0,
+ "resetOffsetAutomatically": false,
+ "segmentWriteOutMediumFactory": null,
+ "workerThreads": null,
+ "chatRetries": 8,
+ "httpTimeout": "PT10S",
+ "shutdownTimeout": "PT80S",
+ "offsetFetchPeriod": "PT30S",
+ "intermediateHandoffPeriod": "P2147483647D",
+ "logParseExceptions": false,
+ "maxParseExceptions": 2147483647,
+ "maxSavedParseExceptions": 0,
+ "skipSequenceNumberAvailabilityCheck": false,
+ "repartitionTransitionDuration": "PT120S"
+ }
+ },
+ "dataSchema": {
+ "dataSource": "social_media",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "iso",
+ "missingValue": null
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ {
+ "type": "string",
+ "name": "username",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "string",
+ "name": "post_title",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "long",
+ "name": "views",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "upvotes",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "comments",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "string",
+ "name": "edited",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ }
+ ],
+ "dimensionExclusions": [
+ "__time"
+ ],
+ "includeAllDimensions": false,
+ "useSchemaDiscovery": false
+ },
+ "metricsSpec": [],
+ "granularitySpec": {
+ "type": "uniform",
+ "segmentGranularity": "HOUR",
+ "queryGranularity": {
+ "type": "none"
+ },
+ "rollup": false,
+ "intervals": []
+ },
+ "transformSpec": {
+ "filter": null,
+ "transforms": []
+ }
+ },
+ "tuningConfig": {
+ "type": "kafka",
+ "appendableIndexSpec": {
+ "type": "onheap",
+ "preserveExistingMetrics": false
+ },
+ "maxRowsInMemory": 150000,
+ "maxBytesInMemory": 0,
+ "skipBytesInMemoryOverheadCheck": false,
+ "maxRowsPerSegment": 5000000,
+ "maxTotalRows": null,
+ "intermediatePersistPeriod": "PT10M",
+ "maxPendingPersists": 0,
+ "indexSpec": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "indexSpecForIntermediatePersists": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "reportParseExceptions": false,
+ "handoffConditionTimeout": 0,
+ "resetOffsetAutomatically": false,
+ "segmentWriteOutMediumFactory": null,
+ "workerThreads": null,
+ "chatRetries": 8,
+ "httpTimeout": "PT10S",
+ "shutdownTimeout": "PT80S",
+ "offsetFetchPeriod": "PT30S",
+ "intermediateHandoffPeriod": "P2147483647D",
+ "logParseExceptions": false,
+ "maxParseExceptions": 2147483647,
+ "maxSavedParseExceptions": 0,
+ "skipSequenceNumberAvailabilityCheck": false,
+ "repartitionTransitionDuration": "PT120S"
+ },
+ "ioConfig": {
+ "topic": "social_media",
+ "inputFormat": {
+ "type": "json"
+ },
+ "replicas": 1,
+ "taskCount": 1,
+ "taskDuration": "PT3600S",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9094"
+ },
+ "autoScalerConfig": null,
+ "pollTimeout": 100,
+ "startDelay": "PT5S",
+ "period": "PT30S",
+ "useEarliestOffset": true,
+ "completionTimeout": "PT1800S",
+ "lateMessageRejectionPeriod": null,
+ "earlyMessageRejectionPeriod": null,
+ "lateMessageRejectionStartDateTime": null,
+ "configOverrides": null,
+ "idleConfig": null,
+ "stream": "social_media",
+ "useEarliestSequenceNumber": true
+ },
+ "context": null,
+ "suspended": true
+}
+ ```
+</details>
+
+### Suspend all supervisors
+
+Suspends all supervisors. Note that this endpoint returns an HTTP `200 Success` code message even if there are no supervisors or running supervisors to suspend.
+
+#### URL
+
+`POST` `/druid/indexer/v1/supervisor/suspendAll`
+
+#### Responses
+
+<Tabs>
+
+<TabItem value="42" label="200 SUCCESS">
+
+
+*Successfully suspended all supervisors*
+
+</TabItem>
+</Tabs>
+
+---
+
+#### Sample request
+
+<Tabs>
+
+<TabItem value="43" label="cURL">
+
+
+```shell
+curl --request POST "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor/suspendAll"
+```
+
+</TabItem>
+<TabItem value="44" label="HTTP">
+
+
+```HTTP
+POST /druid/indexer/v1/supervisor/suspendAll HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+</TabItem>
+</Tabs>
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+{
+ "status": "success"
+}
+ ```
+</details>
+
+### Resume a supervisor
+
+Resumes indexing tasks for a supervisor. Returns an updated supervisor spec with the `suspended` property set to `false`.
+
+#### URL
+
+`POST` `/druid/indexer/v1/supervisor/{supervisorId}/resume`
+
+#### Responses
+
+<Tabs>
+
+<TabItem value="45" label="200 SUCCESS">
+
+
+*Successfully resumed supervisor*
+
+</TabItem>
+<TabItem value="46" label="400 BAD REQUEST">
+
+
+*Supervisor already running*
+
+</TabItem>
+<TabItem value="47" label="404 NOT FOUND">
+
+
+*Invalid supervisor ID*
+
+</TabItem>
+</Tabs>
+
+---
+
+#### Sample request
+
+The following example resumes a previously suspended supervisor with name `social_media`.
+
+<Tabs>
+
+<TabItem value="48" label="cURL">
+
+
+```shell
+curl --request POST "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor/social_media/resume"
+```
+
+</TabItem>
+<TabItem value="49" label="HTTP">
+
+
+```HTTP
+POST /druid/indexer/v1/supervisor/social_media/resume HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+</TabItem>
+</Tabs>
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+{
+ "type": "kafka",
+ "spec": {
+ "dataSchema": {
+ "dataSource": "social_media",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "iso",
+ "missingValue": null
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ {
+ "type": "string",
+ "name": "username",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "string",
+ "name": "post_title",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "long",
+ "name": "views",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "upvotes",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "comments",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "string",
+ "name": "edited",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ }
+ ],
+ "dimensionExclusions": [
+ "__time"
+ ],
+ "includeAllDimensions": false,
+ "useSchemaDiscovery": false
+ },
+ "metricsSpec": [],
+ "granularitySpec": {
+ "type": "uniform",
+ "segmentGranularity": "HOUR",
+ "queryGranularity": {
+ "type": "none"
+ },
+ "rollup": false,
+ "intervals": []
+ },
+ "transformSpec": {
+ "filter": null,
+ "transforms": []
+ }
+ },
+ "ioConfig": {
+ "topic": "social_media",
+ "inputFormat": {
+ "type": "json"
+ },
+ "replicas": 1,
+ "taskCount": 1,
+ "taskDuration": "PT3600S",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9094"
+ },
+ "autoScalerConfig": null,
+ "pollTimeout": 100,
+ "startDelay": "PT5S",
+ "period": "PT30S",
+ "useEarliestOffset": true,
+ "completionTimeout": "PT1800S",
+ "lateMessageRejectionPeriod": null,
+ "earlyMessageRejectionPeriod": null,
+ "lateMessageRejectionStartDateTime": null,
+ "configOverrides": null,
+ "idleConfig": null,
+ "stream": "social_media",
+ "useEarliestSequenceNumber": true
+ },
+ "tuningConfig": {
+ "type": "kafka",
+ "appendableIndexSpec": {
+ "type": "onheap",
+ "preserveExistingMetrics": false
+ },
+ "maxRowsInMemory": 150000,
+ "maxBytesInMemory": 0,
+ "skipBytesInMemoryOverheadCheck": false,
+ "maxRowsPerSegment": 5000000,
+ "maxTotalRows": null,
+ "intermediatePersistPeriod": "PT10M",
+ "maxPendingPersists": 0,
+ "indexSpec": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "indexSpecForIntermediatePersists": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "reportParseExceptions": false,
+ "handoffConditionTimeout": 0,
+ "resetOffsetAutomatically": false,
+ "segmentWriteOutMediumFactory": null,
+ "workerThreads": null,
+ "chatRetries": 8,
+ "httpTimeout": "PT10S",
+ "shutdownTimeout": "PT80S",
+ "offsetFetchPeriod": "PT30S",
+ "intermediateHandoffPeriod": "P2147483647D",
+ "logParseExceptions": false,
+ "maxParseExceptions": 2147483647,
+ "maxSavedParseExceptions": 0,
+ "skipSequenceNumberAvailabilityCheck": false,
+ "repartitionTransitionDuration": "PT120S"
+ }
+ },
+ "dataSchema": {
+ "dataSource": "social_media",
+ "timestampSpec": {
+ "column": "__time",
+ "format": "iso",
+ "missingValue": null
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ {
+ "type": "string",
+ "name": "username",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "string",
+ "name": "post_title",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ },
+ {
+ "type": "long",
+ "name": "views",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "upvotes",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "long",
+ "name": "comments",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": false
+ },
+ {
+ "type": "string",
+ "name": "edited",
+ "multiValueHandling": "SORTED_ARRAY",
+ "createBitmapIndex": true
+ }
+ ],
+ "dimensionExclusions": [
+ "__time"
+ ],
+ "includeAllDimensions": false,
+ "useSchemaDiscovery": false
+ },
+ "metricsSpec": [],
+ "granularitySpec": {
+ "type": "uniform",
+ "segmentGranularity": "HOUR",
+ "queryGranularity": {
+ "type": "none"
+ },
+ "rollup": false,
+ "intervals": []
+ },
+ "transformSpec": {
+ "filter": null,
+ "transforms": []
+ }
+ },
+ "tuningConfig": {
+ "type": "kafka",
+ "appendableIndexSpec": {
+ "type": "onheap",
+ "preserveExistingMetrics": false
+ },
+ "maxRowsInMemory": 150000,
+ "maxBytesInMemory": 0,
+ "skipBytesInMemoryOverheadCheck": false,
+ "maxRowsPerSegment": 5000000,
+ "maxTotalRows": null,
+ "intermediatePersistPeriod": "PT10M",
+ "maxPendingPersists": 0,
+ "indexSpec": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "indexSpecForIntermediatePersists": {
+ "bitmap": {
+ "type": "roaring"
+ },
+ "dimensionCompression": "lz4",
+ "stringDictionaryEncoding": {
+ "type": "utf8"
+ },
+ "metricCompression": "lz4",
+ "longEncoding": "longs"
+ },
+ "reportParseExceptions": false,
+ "handoffConditionTimeout": 0,
+ "resetOffsetAutomatically": false,
+ "segmentWriteOutMediumFactory": null,
+ "workerThreads": null,
+ "chatRetries": 8,
+ "httpTimeout": "PT10S",
+ "shutdownTimeout": "PT80S",
+ "offsetFetchPeriod": "PT30S",
+ "intermediateHandoffPeriod": "P2147483647D",
+ "logParseExceptions": false,
+ "maxParseExceptions": 2147483647,
+ "maxSavedParseExceptions": 0,
+ "skipSequenceNumberAvailabilityCheck": false,
+ "repartitionTransitionDuration": "PT120S"
+ },
+ "ioConfig": {
+ "topic": "social_media",
+ "inputFormat": {
+ "type": "json"
+ },
+ "replicas": 1,
+ "taskCount": 1,
+ "taskDuration": "PT3600S",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9094"
+ },
+ "autoScalerConfig": null,
+ "pollTimeout": 100,
+ "startDelay": "PT5S",
+ "period": "PT30S",
+ "useEarliestOffset": true,
+ "completionTimeout": "PT1800S",
+ "lateMessageRejectionPeriod": null,
+ "earlyMessageRejectionPeriod": null,
+ "lateMessageRejectionStartDateTime": null,
+ "configOverrides": null,
+ "idleConfig": null,
+ "stream": "social_media",
+ "useEarliestSequenceNumber": true
+ },
+ "context": null,
+ "suspended": false
+}
+ ```
+</details>
+
+### Resume all supervisors
+
+Resumes all supervisors. Note that this endpoint returns an HTTP `200 Success` code even if there are no supervisors or suspended supervisors to resume.
+
+#### URL
+
+`POST` `/druid/indexer/v1/supervisor/resumeAll`
+
+#### Responses
+
+<Tabs>
+
+<TabItem value="50" label="200 SUCCESS">
+
+
+*Successfully resumed all supervisors*
+
+</TabItem>
+</Tabs>
+
+---
+
+#### Sample request
+
+<Tabs>
+
+<TabItem value="51" label="cURL">
+
+
+```shell
+curl --request POST "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor/resumeAll"
+```
+
+</TabItem>
+<TabItem value="52" label="HTTP">
+
+
+```HTTP
+POST /druid/indexer/v1/supervisor/resumeAll HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+</TabItem>
+</Tabs>
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+{
+ "status": "success"
+}
+ ```
+</details>
+
+### Reset a supervisor
+
+The supervisor must be running for this endpoint to be available.
+
+Resets the specified supervisor. This endpoint clears supervisor metadata, prompting the supervisor to resume data reading. The supervisor restarts from the earliest or latest available position, depending on the value of the `useEarliestOffset` property.
+After clearing all stored offsets, the supervisor kills and recreates active tasks,
+so that tasks begin reading from valid positions.
+
+Use this endpoint to recover from a stopped state due to missing offsets. Use this endpoint with caution as it may result in skipped messages and lead to data loss or duplicate data.
+
+The indexing service keeps track of the latest persisted offsets to provide exactly-once ingestion guarantees across tasks. Subsequent tasks must start reading from where the previous task completed for Druid to accept the generated segments. If the messages at the expected starting offsets are no longer available, the supervisor refuses to start and in-flight tasks fail. Possible causes for missing messages include the message retention period elapsing or the topic being removed and re- [...]
+
+#### URL
+
+`POST` `/druid/indexer/v1/supervisor/{supervisorId}/reset`
+
+#### Responses
+
+<Tabs>
+
+<TabItem value="53" label="200 SUCCESS">
+
+
+*Successfully reset supervisor*
+
+</TabItem>
+<TabItem value="54" label="404 NOT FOUND">
+
+
+*Invalid supervisor ID*
+
+</TabItem>
+</Tabs>
+
+---
+
+#### Sample request
+
+The following example shows how to reset a supervisor with the name `social_media`.
+
+<Tabs>
+
+<TabItem value="55" label="cURL">
+
+
+```shell
+curl --request POST "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor/social_media/reset"
+```
+
+</TabItem>
+<TabItem value="56" label="HTTP">
+
+
+```HTTP
+POST /druid/indexer/v1/supervisor/social_media/reset HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+</TabItem>
+</Tabs>
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+{
+ "id": "social_media"
+}
+ ```
+</details>
+
+### Reset offsets for a supervisor
+
+The supervisor must be running for this endpoint to be available.
+
+Resets the specified offsets for partitions without resetting the entire set.
+
+This endpoint clears only the stored offsets, prompting the supervisor to resume reading data from the specified offsets.
+If there are no stored offsets, the specified offsets are set in the metadata store.
+
+After resetting stored offsets, the supervisor kills and recreates any active tasks pertaining to the specified partitions,
+so that tasks begin reading specified offsets. For partitions that are not specified in this operation, the supervisor resumes from the last stored offset.
+
+Use this endpoint with caution. It can cause skipped messages, leading to data loss or duplicate data.
+
+#### URL
+
+`POST` `/druid/indexer/v1/supervisor/{supervisorId}/resetOffsets`
+
+#### Responses
+
+<Tabs>
+
+<TabItem value="1" label="200 SUCCESS">
+
+
+*Successfully reset offsets*
+
+</TabItem>
+<TabItem value="2" label="404 NOT FOUND">
+
+
+*Invalid supervisor ID*
+
+</TabItem>
+</Tabs>
+
+---
+#### Reset Offsets Metadata
+
+This section presents the structure and details of the reset offsets metadata payload.
+
+| Field | Type | Description | Required |
+|---------|---------|---------|---------|
+| `type` | String | The type of reset offsets metadata payload. It must match the supervisor's `type`. Possible values: `kafka` or `kinesis`. | Yes |
+| `partitions` | Object | An object representing the reset metadata. See below for details. | Yes |
+
+#### Partitions
+
+The following table defines the fields within the `partitions` object in the reset offsets metadata payload.
+
+| Field | Type | Description | Required |
+|---------|---------|---------|---------|
+| `type` | String | Must be set as `end`. Indicates the end sequence numbers for the reset offsets. | Yes |
+| `stream` | String | The stream to be reset. It must be a valid stream consumed by the supervisor. | Yes |
+| `partitionOffsetMap` | Object | A map of partitions to corresponding offsets for the stream to be reset.| Yes |
+
+#### Sample request
+
+The following example shows how to reset offsets for a Kafka supervisor with the name `social_media`. For example, the supervisor is reading from a Kafka topic `ads_media_stream` and has the stored offsets: `{"0": 0, "1": 10, "2": 20, "3": 40}`.
+
+<Tabs>
+
+<TabItem value="3" label="cURL">
+
+
+```shell
+curl --request POST "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor/social_media/resetOffsets"
+--header 'Content-Type: application/json'
+--data-raw '{"type":"kafka","partitions":{"type":"end","stream":"ads_media_stream","partitionOffsetMap":{"0":100, "2": 650}}}'
+```
+
+</TabItem>
+<TabItem value="4" label="HTTP">
+
+
+```HTTP
+POST /druid/indexer/v1/supervisor/social_media/resetOffsets HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+Content-Type: application/json
+
+{
+ "type": "kafka",
+ "partitions": {
+ "type": "end",
+ "stream": "ads_media_stream",
+ "partitionOffsetMap": {
+ "0": 100,
+ "2": 650
+ }
+ }
+}
+```
+
+The example operation resets offsets only for partitions `0` and `2` to 100 and 650 respectively. After a successful reset,
+when the supervisor's tasks restart, they resume reading from `{"0": 100, "1": 10, "2": 650, "3": 40}`.
+
+</TabItem>
+</Tabs>
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+{
+ "id": "social_media"
+}
+ ```
+</details>
+
+### Terminate a supervisor
+
+Terminates a supervisor and its associated indexing tasks, triggering the publishing of their segments. When you terminate a supervisor, Druid places a tombstone marker in the metadata store to prevent reloading on restart.
+
+The terminated supervisor still exists in the metadata store and its history can be retrieved.
+
+#### URL
+
+`POST` `/druid/indexer/v1/supervisor/{supervisorId}/terminate`
+
+#### Responses
+
+<Tabs>
+
+<TabItem value="49" label="200 SUCCESS">
+
+
+*Successfully terminated a supervisor*
+
+</TabItem>
+<TabItem value="50" label="404 NOT FOUND">
+
+
+*Invalid supervisor ID or supervisor not running*
+
+</TabItem>
+</Tabs>
+
+---
+
+#### Sample request
+
+<Tabs>
+
+<TabItem value="51" label="cURL">
+
+
+```shell
+curl --request POST "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor/social_media/terminate"
+```
+
+</TabItem>
+<TabItem value="52" label="HTTP">
+
+
+```HTTP
+POST /druid/indexer/v1/supervisor/social_media/terminate HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+</TabItem>
+</Tabs>
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+{
+ "id": "social_media"
+}
+ ```
+</details>
+
+### Terminate all supervisors
+
+Terminates all supervisors. Terminated supervisors still exist in the metadata store and their history can be retrieved. Note that this endpoint returns an HTTP `200 Success` code even if there are no supervisors or running supervisors to terminate.
+
+#### URL
+
+`POST` `/druid/indexer/v1/supervisor/terminateAll`
+
+#### Responses
+
+<Tabs>
+
+<TabItem value="53" label="200 SUCCESS">
+
+
+*Successfully terminated all supervisors*
+
+</TabItem>
+</Tabs>
+
+---
+
+#### Sample request
+
+<Tabs>
+
+<TabItem value="54" label="cURL">
+
+
+```shell
+curl --request POST "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor/terminateAll"
+```
+
+</TabItem>
+<TabItem value="55" label="HTTP">
+
+
+```HTTP
+POST /druid/indexer/v1/supervisor/terminateAll HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+</TabItem>
+</Tabs>
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+
+ ```json
+{
+ "status": "success"
+}
+ ```
+</details>
+
+### Handoff task groups for a supervisor early
+
+Trigger handoff for specified task groups of a supervisor early. This is a best effort API and makes no guarantees of handoff execution
+
+#### URL
+
+`POST` `/druid/indexer/v1/supervisor/{supervisorId}/taskGroups/handoff`
+
+#### Sample request
+
+The following example shows how to handoff task groups for a supervisor with the name `social_media` and has the task groups: `1,2,3`.
+
+<Tabs>
+
+<TabItem value="3" label="cURL">
+
+
+```shell
+curl --request POST "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor/social_media/taskGroups/handoff"
+--header 'Content-Type: application/json'
+--data-raw '{"taskGroupIds": [1, 2, 3]}'
+```
+
+</TabItem>
+<TabItem value="4" label="HTTP">
+
+
+```HTTP
+POST /druid/indexer/v1/supervisor/social_media/taskGroups/handoff HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+Content-Type: application/json
+
+{
+ "taskGroupIds": [1, 2, 3],
+}
+```
+
+</TabItem>
+</Tabs>
+
+#### Sample response
+
+<details>
+ <summary>View the response</summary>
+(empty response)
+</details>
+
+### Shut down a supervisor
+
+Shuts down a supervisor. This endpoint is deprecated and will be removed in future releases. Use the equivalent [terminate](#terminate-a-supervisor) endpoint instead.
+
+#### URL
+
+`POST` `/druid/indexer/v1/supervisor/{supervisorId}/shutdown`
diff --git a/docs/latest/api-reference/tasks-api.md b/docs/34.0.0/api-reference/tasks-api.md
similarity index 99%
copy from docs/latest/api-reference/tasks-api.md
copy to docs/34.0.0/api-reference/tasks-api.md
index d94be5b0c..f53037f84 100644
--- a/docs/latest/api-reference/tasks-api.md
+++ b/docs/34.0.0/api-reference/tasks-api.md
@@ -1059,9 +1059,9 @@ Host: http://ROUTER_IP:ROUTER_PORT
2023-07-03T22:11:17,933 INFO [task-runner-0-priority-0] org.apache.kafka.common.metrics.Metrics - Closing reporter org.apache.kafka.common.metrics.JmxReporter
2023-07-03T22:11:17,933 INFO [task-runner-0-priority-0] org.apache.kafka.common.metrics.Metrics - Metrics reporters closed
2023-07-03T22:11:17,935 INFO [task-runner-0-priority-0] org.apache.kafka.common.utils.AppInfoParser - App info kafka.consumer for consumer-kafka-supervisor-dcanhmig-1 unregistered
- 2023-07-03T22:11:17,936 INFO [task-runner-0-priority-0] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/internal-discovery/PEON/localhost:8100]
+ 2023-07-03T22:11:17,936 INFO [task-runner-0-priority-0] org.apache.druid.curator.announcement.PathChildrenAnnouncer - Unannouncing [/druid/internal-discovery/PEON/localhost:8100]
2023-07-03T22:11:17,972 INFO [task-runner-0-priority-0] org.apache.druid.curator.discovery.CuratorDruidNodeAnnouncer - Unannounced self [{"druidNode":{"service":"druid/middleManager","host":"localhost","bindOnHost":false,"plaintextPort":8100,"port":-1,"tlsPort":-1,"enablePlaintextPort":true,"enableTlsPort":false},"nodeType":"peon","services":{"dataNodeService":{"type":"dataNodeService","tier":"_default_tier","maxSize":0,"type":"indexer-executor","serverType":"indexer-executor","prior [...]
- 2023-07-03T22:11:17,972 INFO [task-runner-0-priority-0] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/announcements/localhost:8100]
+ 2023-07-03T22:11:17,972 INFO [task-runner-0-priority-0] org.apache.druid.curator.announcement.PathChildrenAnnouncer - Unannouncing [/druid/announcements/localhost:8100]
2023-07-03T22:11:17,996 INFO [task-runner-0-priority-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
"id" : "index_kafka_social_media_0e905aa31037879_nommnaeg",
"status" : "SUCCESS",
@@ -1601,7 +1601,7 @@ Content-Length: 134
Manually clean up pending segments table in metadata storage for `datasource`. It returns a JSON object response with
`numDeleted` for the number of rows deleted from the pending segments table. This API is used by the
-`druid.coordinator.kill.pendingSegments.on` [Coordinator setting](../configuration/index.md#coordinator-operation)
+`druid.coordinator.kill.pendingSegments.on` [Coordinator setting](../configuration/index.md#data-management)
which automates this operation to perform periodically.
#### URL
diff --git a/docs/34.0.0/assets/compaction-config.png b/docs/34.0.0/assets/compaction-config.png
new file mode 100644
index 000000000..9dbcfefa8
Binary files /dev/null and b/docs/34.0.0/assets/compaction-config.png differ
diff --git a/docs/34.0.0/assets/datasources-action-button.png b/docs/34.0.0/assets/datasources-action-button.png
new file mode 100644
index 000000000..6a52b8444
Binary files /dev/null and b/docs/34.0.0/assets/datasources-action-button.png differ
diff --git a/docs/34.0.0/assets/druid-architecture.png b/docs/34.0.0/assets/druid-architecture.png
new file mode 100644
index 000000000..954a87bc1
Binary files /dev/null and b/docs/34.0.0/assets/druid-architecture.png differ
diff --git a/docs/34.0.0/assets/druid-architecture.svg b/docs/34.0.0/assets/druid-architecture.svg
new file mode 100644
index 000000000..9d0e67188
--- /dev/null
+++ b/docs/34.0.0/assets/druid-architecture.svg
@@ -0,0 +1,19 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+ <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:lucid="lucid" width="1967.24" height="872.83"><g transform="translate(5341 -19.5)" lucid:page-tab-id="0_0"><path d="M-5320 166a6 6 0 0 1 6-6h508a6 6 0 0 1 6 6v308a6 6 0 0 1-6 6h-508a6 6 0 0 1-6-6zM-4640 166a6 6 0 0 1 6-6h508a6 6 0 0 1 6 6v308a6 6 0 0 1-6 6h-508a6 6 0 0 1-6-6zM-3960 166a6 6 0 0 1 6-6h508a6 6 0 0 1 6 6v308a6 6 0 0 1-6 6h-508a6 6 0 0 1-6-6z" stroke="#cfe4ff" stroke-width="2" fill="#cf [...]
\ No newline at end of file
diff --git a/docs/34.0.0/assets/druid-column-types.png b/docs/34.0.0/assets/druid-column-types.png
new file mode 100644
index 000000000..9db56c068
Binary files /dev/null and b/docs/34.0.0/assets/druid-column-types.png differ
diff --git a/docs/34.0.0/assets/druid-dataflow-2x.png b/docs/34.0.0/assets/druid-dataflow-2x.png
new file mode 100644
index 000000000..ab1c583e4
Binary files /dev/null and b/docs/34.0.0/assets/druid-dataflow-2x.png differ
diff --git a/docs/34.0.0/assets/druid-dataflow-3.png b/docs/34.0.0/assets/druid-dataflow-3.png
new file mode 100644
index 000000000..355215cbc
Binary files /dev/null and b/docs/34.0.0/assets/druid-dataflow-3.png differ
diff --git a/docs/34.0.0/assets/druid-manage-1.png b/docs/34.0.0/assets/druid-manage-1.png
new file mode 100644
index 000000000..0d10c6e7b
Binary files /dev/null and b/docs/34.0.0/assets/druid-manage-1.png differ
diff --git a/docs/34.0.0/assets/druid-timeline.png b/docs/34.0.0/assets/druid-timeline.png
new file mode 100644
index 000000000..40380e279
Binary files /dev/null and b/docs/34.0.0/assets/druid-timeline.png differ
diff --git a/docs/34.0.0/assets/files/kttm-kafka-supervisor.json b/docs/34.0.0/assets/files/kttm-kafka-supervisor.json
new file mode 100644
index 000000000..2096f9c7c
--- /dev/null
+++ b/docs/34.0.0/assets/files/kttm-kafka-supervisor.json
@@ -0,0 +1,66 @@
+{
+ "type": "kafka",
+ "spec": {
+ "ioConfig": {
+ "type": "kafka",
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9092"
+ },
+ "topic": "kttm",
+ "inputFormat": {
+ "type": "json"
+ },
+ "useEarliestOffset": true
+ },
+ "tuningConfig": {
+ "type": "kafka"
+ },
+ "dataSchema": {
+ "dataSource": "kttm-kafka-supervisor-api",
+ "timestampSpec": {
+ "column": "timestamp",
+ "format": "iso"
+ },
+ "dimensionsSpec": {
+ "dimensions": [
+ "session",
+ "number",
+ "client_ip",
+ "language",
+ "adblock_list",
+ "app_version",
+ "path",
+ "loaded_image",
+ "referrer",
+ "referrer_host",
+ "server_ip",
+ "screen",
+ "window",
+ {
+ "type": "long",
+ "name": "session_length"
+ },
+ "timezone",
+ "timezone_offset",
+ {
+ "type": "json",
+ "name": "event"
+ },
+ {
+ "type": "json",
+ "name": "agent"
+ },
+ {
+ "type": "json",
+ "name": "geo_ip"
+ }
+ ]
+ },
+ "granularitySpec": {
+ "queryGranularity": "none",
+ "rollup": false,
+ "segmentGranularity": "day"
+ }
+ }
+ }
+}
\ No newline at end of file
diff --git a/docs/34.0.0/assets/indexing_service.png b/docs/34.0.0/assets/indexing_service.png
new file mode 100644
index 000000000..a4462a413
Binary files /dev/null and b/docs/34.0.0/assets/indexing_service.png differ
diff --git a/docs/34.0.0/assets/multi-stage-query/msq-ui-download-query-results.png b/docs/34.0.0/assets/multi-stage-query/msq-ui-download-query-results.png
new file mode 100644
index 000000000..e428cb2df
Binary files /dev/null and b/docs/34.0.0/assets/multi-stage-query/msq-ui-download-query-results.png differ
diff --git a/docs/34.0.0/assets/multi-stage-query/tutorial-msq-convert.png b/docs/34.0.0/assets/multi-stage-query/tutorial-msq-convert.png
new file mode 100644
index 000000000..f16941af6
Binary files /dev/null and b/docs/34.0.0/assets/multi-stage-query/tutorial-msq-convert.png differ
diff --git a/docs/34.0.0/assets/multi-stage-query/ui-annotated.png b/docs/34.0.0/assets/multi-stage-query/ui-annotated.png
new file mode 100644
index 000000000..5a98c00d1
Binary files /dev/null and b/docs/34.0.0/assets/multi-stage-query/ui-annotated.png differ
diff --git a/docs/34.0.0/assets/multi-stage-query/ui-empty.png b/docs/34.0.0/assets/multi-stage-query/ui-empty.png
new file mode 100644
index 000000000..7c30d5a67
Binary files /dev/null and b/docs/34.0.0/assets/multi-stage-query/ui-empty.png differ
diff --git a/docs/34.0.0/assets/native-queries-01.png b/docs/34.0.0/assets/native-queries-01.png
new file mode 100644
index 000000000..27fd29b63
Binary files /dev/null and b/docs/34.0.0/assets/native-queries-01.png differ
diff --git a/docs/34.0.0/assets/nested-combined-json.png b/docs/34.0.0/assets/nested-combined-json.png
new file mode 100644
index 000000000..f98bfcf53
Binary files /dev/null and b/docs/34.0.0/assets/nested-combined-json.png differ
diff --git a/docs/34.0.0/assets/nested-display-data-types.png b/docs/34.0.0/assets/nested-display-data-types.png
new file mode 100644
index 000000000..2776068ee
Binary files /dev/null and b/docs/34.0.0/assets/nested-display-data-types.png differ
diff --git a/docs/34.0.0/assets/nested-examine-schema.png b/docs/34.0.0/assets/nested-examine-schema.png
new file mode 100644
index 000000000..11769a162
Binary files /dev/null and b/docs/34.0.0/assets/nested-examine-schema.png differ
diff --git a/docs/34.0.0/assets/nested-extract-as-type.png b/docs/34.0.0/assets/nested-extract-as-type.png
new file mode 100644
index 000000000..c54a5eeb6
Binary files /dev/null and b/docs/34.0.0/assets/nested-extract-as-type.png differ
diff --git a/docs/34.0.0/assets/nested-extract-elements.png b/docs/34.0.0/assets/nested-extract-elements.png
new file mode 100644
index 000000000..9f7076b50
Binary files /dev/null and b/docs/34.0.0/assets/nested-extract-elements.png differ
diff --git a/docs/34.0.0/assets/nested-group-aggregate.png b/docs/34.0.0/assets/nested-group-aggregate.png
new file mode 100644
index 000000000..2d1907fe6
Binary files /dev/null and b/docs/34.0.0/assets/nested-group-aggregate.png differ
diff --git a/docs/34.0.0/assets/nested-msq-ingestion-transform.png b/docs/34.0.0/assets/nested-msq-ingestion-transform.png
new file mode 100644
index 000000000..b46fde859
Binary files /dev/null and b/docs/34.0.0/assets/nested-msq-ingestion-transform.png differ
diff --git a/docs/34.0.0/assets/nested-msq-ingestion.png b/docs/34.0.0/assets/nested-msq-ingestion.png
new file mode 100644
index 000000000..0487ee188
Binary files /dev/null and b/docs/34.0.0/assets/nested-msq-ingestion.png differ
diff --git a/docs/34.0.0/assets/nested-parse-deserialize.png b/docs/34.0.0/assets/nested-parse-deserialize.png
new file mode 100644
index 000000000..881a67164
Binary files /dev/null and b/docs/34.0.0/assets/nested-parse-deserialize.png differ
diff --git a/docs/34.0.0/assets/nested-retrieve-json.png b/docs/34.0.0/assets/nested-retrieve-json.png
new file mode 100644
index 000000000..4f5fa0f96
Binary files /dev/null and b/docs/34.0.0/assets/nested-retrieve-json.png differ
diff --git a/docs/34.0.0/assets/nested-return-json.png b/docs/34.0.0/assets/nested-return-json.png
new file mode 100644
index 000000000..9a67aaa71
Binary files /dev/null and b/docs/34.0.0/assets/nested-return-json.png differ
diff --git a/docs/34.0.0/assets/retention-rules.png b/docs/34.0.0/assets/retention-rules.png
new file mode 100644
index 000000000..59061d551
Binary files /dev/null and b/docs/34.0.0/assets/retention-rules.png differ
diff --git a/docs/34.0.0/assets/security-model-1.png b/docs/34.0.0/assets/security-model-1.png
new file mode 100644
index 000000000..55c7f24c5
Binary files /dev/null and b/docs/34.0.0/assets/security-model-1.png differ
diff --git a/docs/34.0.0/assets/security-model-2.png b/docs/34.0.0/assets/security-model-2.png
new file mode 100644
index 000000000..dcb256bac
Binary files /dev/null and b/docs/34.0.0/assets/security-model-2.png differ
diff --git a/docs/34.0.0/assets/segmentPropagation.png b/docs/34.0.0/assets/segmentPropagation.png
new file mode 100644
index 000000000..e1ec82029
Binary files /dev/null and b/docs/34.0.0/assets/segmentPropagation.png differ
diff --git a/docs/34.0.0/assets/services-overview.png b/docs/34.0.0/assets/services-overview.png
new file mode 100644
index 000000000..157ce608e
Binary files /dev/null and b/docs/34.0.0/assets/services-overview.png differ
diff --git a/docs/34.0.0/assets/spectator-histogram-size-comparison.png b/docs/34.0.0/assets/spectator-histogram-size-comparison.png
new file mode 100644
index 000000000..306f45abd
Binary files /dev/null and b/docs/34.0.0/assets/spectator-histogram-size-comparison.png differ
diff --git a/docs/34.0.0/assets/supervisor-actions.png b/docs/34.0.0/assets/supervisor-actions.png
new file mode 100644
index 000000000..2797cf69e
Binary files /dev/null and b/docs/34.0.0/assets/supervisor-actions.png differ
diff --git a/docs/34.0.0/assets/supervisor-info-dialog.png b/docs/34.0.0/assets/supervisor-info-dialog.png
new file mode 100644
index 000000000..3be424a41
Binary files /dev/null and b/docs/34.0.0/assets/supervisor-info-dialog.png differ
diff --git a/docs/34.0.0/assets/supervisor-view.png b/docs/34.0.0/assets/supervisor-view.png
new file mode 100644
index 000000000..e3100cdd3
Binary files /dev/null and b/docs/34.0.0/assets/supervisor-view.png differ
diff --git a/docs/34.0.0/assets/tutorial-batch-data-loader-00.png b/docs/34.0.0/assets/tutorial-batch-data-loader-00.png
new file mode 100644
index 000000000..793b6c123
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-batch-data-loader-00.png differ
diff --git a/docs/34.0.0/assets/tutorial-batch-data-loader-01.png b/docs/34.0.0/assets/tutorial-batch-data-loader-01.png
new file mode 100644
index 000000000..2ff1d6398
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-batch-data-loader-01.png differ
diff --git a/docs/34.0.0/assets/tutorial-batch-data-loader-015.png b/docs/34.0.0/assets/tutorial-batch-data-loader-015.png
new file mode 100644
index 000000000..fd588caea
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-batch-data-loader-015.png differ
diff --git a/docs/34.0.0/assets/tutorial-batch-data-loader-02.png b/docs/34.0.0/assets/tutorial-batch-data-loader-02.png
new file mode 100644
index 000000000..736188cb1
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-batch-data-loader-02.png differ
diff --git a/docs/34.0.0/assets/tutorial-batch-data-loader-03.png b/docs/34.0.0/assets/tutorial-batch-data-loader-03.png
new file mode 100644
index 000000000..74bb8c88f
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-batch-data-loader-03.png differ
diff --git a/docs/34.0.0/assets/tutorial-batch-data-loader-04.png b/docs/34.0.0/assets/tutorial-batch-data-loader-04.png
new file mode 100644
index 000000000..e4237cda8
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-batch-data-loader-04.png differ
diff --git a/docs/34.0.0/assets/tutorial-batch-data-loader-05.png b/docs/34.0.0/assets/tutorial-batch-data-loader-05.png
new file mode 100644
index 000000000..d245dde67
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-batch-data-loader-05.png differ
diff --git a/docs/34.0.0/assets/tutorial-batch-data-loader-06.png b/docs/34.0.0/assets/tutorial-batch-data-loader-06.png
new file mode 100644
index 000000000..285fd57ba
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-batch-data-loader-06.png differ
diff --git a/docs/34.0.0/assets/tutorial-batch-data-loader-07.png b/docs/34.0.0/assets/tutorial-batch-data-loader-07.png
new file mode 100644
index 000000000..481838d78
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-batch-data-loader-07.png differ
diff --git a/docs/34.0.0/assets/tutorial-batch-data-loader-08.png b/docs/34.0.0/assets/tutorial-batch-data-loader-08.png
new file mode 100644
index 000000000..b64c5a4e0
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-batch-data-loader-08.png differ
diff --git a/docs/34.0.0/assets/tutorial-batch-data-loader-09.png b/docs/34.0.0/assets/tutorial-batch-data-loader-09.png
new file mode 100644
index 000000000..bec3085f6
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-batch-data-loader-09.png differ
diff --git a/docs/34.0.0/assets/tutorial-batch-data-loader-10.png b/docs/34.0.0/assets/tutorial-batch-data-loader-10.png
new file mode 100644
index 000000000..857a5a5c4
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-batch-data-loader-10.png differ
diff --git a/docs/34.0.0/assets/tutorial-batch-data-loader-11.png b/docs/34.0.0/assets/tutorial-batch-data-loader-11.png
new file mode 100644
index 000000000..bf7e304b8
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-batch-data-loader-11.png differ
diff --git a/docs/34.0.0/assets/tutorial-batch-data-loader-12.png b/docs/34.0.0/assets/tutorial-batch-data-loader-12.png
new file mode 100644
index 000000000..f195b9ca5
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-batch-data-loader-12.png differ
diff --git a/docs/34.0.0/assets/tutorial-batch-submit-task-01.png b/docs/34.0.0/assets/tutorial-batch-submit-task-01.png
new file mode 100644
index 000000000..01b91427f
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-batch-submit-task-01.png differ
diff --git a/docs/34.0.0/assets/tutorial-batch-submit-task-02.png b/docs/34.0.0/assets/tutorial-batch-submit-task-02.png
new file mode 100644
index 000000000..ba7caeb22
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-batch-submit-task-02.png differ
diff --git a/docs/34.0.0/assets/tutorial-compaction-01.png b/docs/34.0.0/assets/tutorial-compaction-01.png
new file mode 100644
index 000000000..aeb9bf36f
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-compaction-01.png differ
diff --git a/docs/34.0.0/assets/tutorial-compaction-02.png b/docs/34.0.0/assets/tutorial-compaction-02.png
new file mode 100644
index 000000000..836d8a7a7
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-compaction-02.png differ
diff --git a/docs/34.0.0/assets/tutorial-compaction-03.png b/docs/34.0.0/assets/tutorial-compaction-03.png
new file mode 100644
index 000000000..d51f8f8a8
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-compaction-03.png differ
diff --git a/docs/34.0.0/assets/tutorial-compaction-04.png b/docs/34.0.0/assets/tutorial-compaction-04.png
new file mode 100644
index 000000000..46c5b1d26
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-compaction-04.png differ
diff --git a/docs/34.0.0/assets/tutorial-compaction-05.png b/docs/34.0.0/assets/tutorial-compaction-05.png
new file mode 100644
index 000000000..e692694af
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-compaction-05.png differ
diff --git a/docs/34.0.0/assets/tutorial-compaction-06.png b/docs/34.0.0/assets/tutorial-compaction-06.png
new file mode 100644
index 000000000..55c999f9d
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-compaction-06.png differ
diff --git a/docs/34.0.0/assets/tutorial-compaction-07.png b/docs/34.0.0/assets/tutorial-compaction-07.png
new file mode 100644
index 000000000..661e89784
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-compaction-07.png differ
diff --git a/docs/34.0.0/assets/tutorial-compaction-08.png b/docs/34.0.0/assets/tutorial-compaction-08.png
new file mode 100644
index 000000000..6e3f1aa03
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-compaction-08.png differ
diff --git a/docs/34.0.0/assets/tutorial-deletion-01.png b/docs/34.0.0/assets/tutorial-deletion-01.png
new file mode 100644
index 000000000..942f057d7
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-deletion-01.png differ
diff --git a/docs/34.0.0/assets/tutorial-deletion-02.png b/docs/34.0.0/assets/tutorial-deletion-02.png
new file mode 100644
index 000000000..516fdf7fe
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-deletion-02.png differ
diff --git a/docs/34.0.0/assets/tutorial-deletion-03.png b/docs/34.0.0/assets/tutorial-deletion-03.png
new file mode 100644
index 000000000..666ff7a89
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-deletion-03.png differ
diff --git a/docs/34.0.0/assets/tutorial-kafka-data-loader-01.png b/docs/34.0.0/assets/tutorial-kafka-data-loader-01.png
new file mode 100644
index 000000000..7f8d0daac
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-kafka-data-loader-01.png differ
diff --git a/docs/34.0.0/assets/tutorial-kafka-data-loader-02.png b/docs/34.0.0/assets/tutorial-kafka-data-loader-02.png
new file mode 100644
index 000000000..8475eeba2
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-kafka-data-loader-02.png differ
diff --git a/docs/34.0.0/assets/tutorial-kafka-data-loader-03.png b/docs/34.0.0/assets/tutorial-kafka-data-loader-03.png
new file mode 100644
index 000000000..dc7400404
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-kafka-data-loader-03.png differ
diff --git a/docs/34.0.0/assets/tutorial-kafka-data-loader-04.png b/docs/34.0.0/assets/tutorial-kafka-data-loader-04.png
new file mode 100644
index 000000000..570306695
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-kafka-data-loader-04.png differ
diff --git a/docs/34.0.0/assets/tutorial-kafka-data-loader-05.png b/docs/34.0.0/assets/tutorial-kafka-data-loader-05.png
new file mode 100644
index 000000000..c920f0565
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-kafka-data-loader-05.png differ
diff --git a/docs/34.0.0/assets/tutorial-kafka-data-loader-06.png b/docs/34.0.0/assets/tutorial-kafka-data-loader-06.png
new file mode 100644
index 000000000..4fb96dd47
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-kafka-data-loader-06.png differ
diff --git a/docs/34.0.0/assets/tutorial-kafka-data-loader-07.png b/docs/34.0.0/assets/tutorial-kafka-data-loader-07.png
new file mode 100644
index 000000000..b3013b735
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-kafka-data-loader-07.png differ
diff --git a/docs/34.0.0/assets/tutorial-kafka-data-loader-08.png b/docs/34.0.0/assets/tutorial-kafka-data-loader-08.png
new file mode 100644
index 000000000..b1cdd2df1
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-kafka-data-loader-08.png differ
diff --git a/docs/34.0.0/assets/tutorial-kafka-data-loader-09.png b/docs/34.0.0/assets/tutorial-kafka-data-loader-09.png
new file mode 100644
index 000000000..e2045ac89
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-kafka-data-loader-09.png differ
diff --git a/docs/34.0.0/assets/tutorial-kafka-data-loader-10.png b/docs/34.0.0/assets/tutorial-kafka-data-loader-10.png
new file mode 100644
index 000000000..39eaa3750
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-kafka-data-loader-10.png differ
diff --git a/docs/34.0.0/assets/tutorial-kafka-data-loader-11.png b/docs/34.0.0/assets/tutorial-kafka-data-loader-11.png
new file mode 100644
index 000000000..7bd3d9a25
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-kafka-data-loader-11.png differ
diff --git a/docs/34.0.0/assets/tutorial-kafka-data-loader-12.png b/docs/34.0.0/assets/tutorial-kafka-data-loader-12.png
new file mode 100644
index 000000000..ed952b135
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-kafka-data-loader-12.png differ
diff --git a/docs/34.0.0/assets/tutorial-kafka-submit-supervisor-01.png b/docs/34.0.0/assets/tutorial-kafka-submit-supervisor-01.png
new file mode 100644
index 000000000..809c0c673
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-kafka-submit-supervisor-01.png differ
diff --git a/docs/34.0.0/assets/tutorial-query-01.png b/docs/34.0.0/assets/tutorial-query-01.png
new file mode 100644
index 000000000..99354cbdf
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-query-01.png differ
diff --git a/docs/34.0.0/assets/tutorial-query-02.png b/docs/34.0.0/assets/tutorial-query-02.png
new file mode 100644
index 000000000..4d789f598
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-query-02.png differ
diff --git a/docs/34.0.0/assets/tutorial-query-03.png b/docs/34.0.0/assets/tutorial-query-03.png
new file mode 100644
index 000000000..841d36bfe
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-query-03.png differ
diff --git a/docs/34.0.0/assets/tutorial-query-04.png b/docs/34.0.0/assets/tutorial-query-04.png
new file mode 100644
index 000000000..7c713e367
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-query-04.png differ
diff --git a/docs/34.0.0/assets/tutorial-query-05.png b/docs/34.0.0/assets/tutorial-query-05.png
new file mode 100644
index 000000000..4b3d78d15
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-query-05.png differ
diff --git a/docs/34.0.0/assets/tutorial-query-06.png b/docs/34.0.0/assets/tutorial-query-06.png
new file mode 100644
index 000000000..cb35a0787
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-query-06.png differ
diff --git a/docs/34.0.0/assets/tutorial-query-07.png b/docs/34.0.0/assets/tutorial-query-07.png
new file mode 100644
index 000000000..aa94d629f
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-query-07.png differ
diff --git a/docs/34.0.0/assets/tutorial-query-deepstorage-retention-rule.png b/docs/34.0.0/assets/tutorial-query-deepstorage-retention-rule.png
new file mode 100644
index 000000000..9dee37bde
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-query-deepstorage-retention-rule.png differ
diff --git a/docs/34.0.0/assets/tutorial-quickstart-01.png b/docs/34.0.0/assets/tutorial-quickstart-01.png
new file mode 100644
index 000000000..649708b7c
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-quickstart-01.png differ
diff --git a/docs/34.0.0/assets/tutorial-quickstart-02.png b/docs/34.0.0/assets/tutorial-quickstart-02.png
new file mode 100644
index 000000000..5edec67c3
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-quickstart-02.png differ
diff --git a/docs/34.0.0/assets/tutorial-quickstart-03.png b/docs/34.0.0/assets/tutorial-quickstart-03.png
new file mode 100644
index 000000000..917f25d04
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-quickstart-03.png differ
diff --git a/docs/34.0.0/assets/tutorial-quickstart-04.png b/docs/34.0.0/assets/tutorial-quickstart-04.png
new file mode 100644
index 000000000..e847ef550
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-quickstart-04.png differ
diff --git a/docs/34.0.0/assets/tutorial-quickstart-05.png b/docs/34.0.0/assets/tutorial-quickstart-05.png
new file mode 100644
index 000000000..da3ed0dfa
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-quickstart-05.png differ
diff --git a/docs/34.0.0/assets/tutorial-retention-00.png b/docs/34.0.0/assets/tutorial-retention-00.png
new file mode 100644
index 000000000..a3f84a9fe
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-retention-00.png differ
diff --git a/docs/34.0.0/assets/tutorial-retention-01.png b/docs/34.0.0/assets/tutorial-retention-01.png
new file mode 100644
index 000000000..35a97c262
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-retention-01.png differ
diff --git a/docs/34.0.0/assets/tutorial-retention-02.png b/docs/34.0.0/assets/tutorial-retention-02.png
new file mode 100644
index 000000000..f38fad0d2
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-retention-02.png differ
diff --git a/docs/34.0.0/assets/tutorial-retention-03.png b/docs/34.0.0/assets/tutorial-retention-03.png
new file mode 100644
index 000000000..256836a2d
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-retention-03.png differ
diff --git a/docs/34.0.0/assets/tutorial-retention-04.png b/docs/34.0.0/assets/tutorial-retention-04.png
new file mode 100644
index 000000000..d39495f87
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-retention-04.png differ
diff --git a/docs/34.0.0/assets/tutorial-retention-05.png b/docs/34.0.0/assets/tutorial-retention-05.png
new file mode 100644
index 000000000..638a752fa
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-retention-05.png differ
diff --git a/docs/34.0.0/assets/tutorial-retention-06.png b/docs/34.0.0/assets/tutorial-retention-06.png
new file mode 100644
index 000000000..f47cbffbb
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-retention-06.png differ
diff --git a/docs/34.0.0/assets/tutorial-sql-aggregate-query.png b/docs/34.0.0/assets/tutorial-sql-aggregate-query.png
new file mode 100644
index 000000000..0ffbff60e
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-sql-aggregate-query.png differ
diff --git a/docs/34.0.0/assets/tutorial-sql-auto-queries.png b/docs/34.0.0/assets/tutorial-sql-auto-queries.png
new file mode 100644
index 000000000..dc04a8de6
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-sql-auto-queries.png differ
diff --git a/docs/34.0.0/assets/tutorial-sql-count-distinct-help.png b/docs/34.0.0/assets/tutorial-sql-count-distinct-help.png
new file mode 100644
index 000000000..5327972d2
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-sql-count-distinct-help.png differ
diff --git a/docs/34.0.0/assets/tutorial-sql-count-distinct.png b/docs/34.0.0/assets/tutorial-sql-count-distinct.png
new file mode 100644
index 000000000..5fb9b2ae0
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-sql-count-distinct.png differ
diff --git a/docs/34.0.0/assets/tutorial-sql-demo-queries.png b/docs/34.0.0/assets/tutorial-sql-demo-queries.png
new file mode 100644
index 000000000..16fc040a6
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-sql-demo-queries.png differ
diff --git a/docs/34.0.0/assets/tutorial-sql-query-plan.png b/docs/34.0.0/assets/tutorial-sql-query-plan.png
new file mode 100644
index 000000000..03f3c3cc6
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-sql-query-plan.png differ
diff --git a/docs/34.0.0/assets/tutorial-sql-result-column-actions.png b/docs/34.0.0/assets/tutorial-sql-result-column-actions.png
new file mode 100644
index 000000000..16518d4bf
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-sql-result-column-actions.png differ
diff --git a/docs/34.0.0/assets/tutorial-theta-01.png b/docs/34.0.0/assets/tutorial-theta-01.png
new file mode 100644
index 000000000..2411fbf19
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-theta-01.png differ
diff --git a/docs/34.0.0/assets/tutorial-theta-02.png b/docs/34.0.0/assets/tutorial-theta-02.png
new file mode 100644
index 000000000..ce849fd36
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-theta-02.png differ
diff --git a/docs/34.0.0/assets/tutorial-theta-03.png b/docs/34.0.0/assets/tutorial-theta-03.png
new file mode 100644
index 000000000..316bf7f0b
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-theta-03.png differ
diff --git a/docs/34.0.0/assets/tutorial-theta-04.png b/docs/34.0.0/assets/tutorial-theta-04.png
new file mode 100644
index 000000000..21f383af6
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-theta-04.png differ
diff --git a/docs/34.0.0/assets/tutorial-theta-05.png b/docs/34.0.0/assets/tutorial-theta-05.png
new file mode 100644
index 000000000..ec2c8df6d
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-theta-05.png differ
diff --git a/docs/34.0.0/assets/tutorial-theta-06.png b/docs/34.0.0/assets/tutorial-theta-06.png
new file mode 100644
index 000000000..4048aa238
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-theta-06.png differ
diff --git a/docs/34.0.0/assets/tutorial-theta-07.png b/docs/34.0.0/assets/tutorial-theta-07.png
new file mode 100644
index 000000000..369b5914a
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-theta-07.png differ
diff --git a/docs/34.0.0/assets/tutorial-theta-08.png b/docs/34.0.0/assets/tutorial-theta-08.png
new file mode 100644
index 000000000..59a6bc051
Binary files /dev/null and b/docs/34.0.0/assets/tutorial-theta-08.png differ
diff --git a/docs/34.0.0/assets/web-console-0.7-tasks.png b/docs/34.0.0/assets/web-console-0.7-tasks.png
new file mode 100644
index 000000000..80080ba8e
Binary files /dev/null and b/docs/34.0.0/assets/web-console-0.7-tasks.png differ
diff --git a/docs/34.0.0/assets/web-console-01-home-view.png b/docs/34.0.0/assets/web-console-01-home-view.png
new file mode 100644
index 000000000..39b6e8a1a
Binary files /dev/null and b/docs/34.0.0/assets/web-console-01-home-view.png differ
diff --git a/docs/34.0.0/assets/web-console-02-data-loader-1.png b/docs/34.0.0/assets/web-console-02-data-loader-1.png
new file mode 100644
index 000000000..ecd18c01f
Binary files /dev/null and b/docs/34.0.0/assets/web-console-02-data-loader-1.png differ
diff --git a/docs/34.0.0/assets/web-console-03-data-loader-2.png b/docs/34.0.0/assets/web-console-03-data-loader-2.png
new file mode 100644
index 000000000..bfb7be59c
Binary files /dev/null and b/docs/34.0.0/assets/web-console-03-data-loader-2.png differ
diff --git a/docs/34.0.0/assets/web-console-04-datasources.png b/docs/34.0.0/assets/web-console-04-datasources.png
new file mode 100644
index 000000000..fab3cec45
Binary files /dev/null and b/docs/34.0.0/assets/web-console-04-datasources.png differ
diff --git a/docs/34.0.0/assets/web-console-05-retention.png b/docs/34.0.0/assets/web-console-05-retention.png
new file mode 100644
index 000000000..96278525a
Binary files /dev/null and b/docs/34.0.0/assets/web-console-05-retention.png differ
diff --git a/docs/34.0.0/assets/web-console-06-segments.png b/docs/34.0.0/assets/web-console-06-segments.png
new file mode 100644
index 000000000..9e9e9ab98
Binary files /dev/null and b/docs/34.0.0/assets/web-console-06-segments.png differ
diff --git a/docs/34.0.0/assets/web-console-07-supervisors.png b/docs/34.0.0/assets/web-console-07-supervisors.png
new file mode 100644
index 000000000..70391bd64
Binary files /dev/null and b/docs/34.0.0/assets/web-console-07-supervisors.png differ
diff --git a/docs/34.0.0/assets/web-console-08-supervisor-status.png b/docs/34.0.0/assets/web-console-08-supervisor-status.png
new file mode 100644
index 000000000..1bcfccdfe
Binary files /dev/null and b/docs/34.0.0/assets/web-console-08-supervisor-status.png differ
diff --git a/docs/34.0.0/assets/web-console-09-task-status.png b/docs/34.0.0/assets/web-console-09-task-status.png
new file mode 100644
index 000000000..100e8ada0
Binary files /dev/null and b/docs/34.0.0/assets/web-console-09-task-status.png differ
diff --git a/docs/34.0.0/assets/web-console-10-servers.png b/docs/34.0.0/assets/web-console-10-servers.png
new file mode 100644
index 000000000..a3e0084e1
Binary files /dev/null and b/docs/34.0.0/assets/web-console-10-servers.png differ
diff --git a/docs/34.0.0/assets/web-console-11-query-sql.png b/docs/34.0.0/assets/web-console-11-query-sql.png
new file mode 100644
index 000000000..a144774f4
Binary files /dev/null and b/docs/34.0.0/assets/web-console-11-query-sql.png differ
diff --git a/docs/34.0.0/assets/web-console-12-query-rune.png b/docs/34.0.0/assets/web-console-12-query-rune.png
new file mode 100644
index 000000000..8c5e27056
Binary files /dev/null and b/docs/34.0.0/assets/web-console-12-query-rune.png differ
diff --git a/docs/34.0.0/assets/web-console-13-lookups.png b/docs/34.0.0/assets/web-console-13-lookups.png
new file mode 100644
index 000000000..fa0bd0b06
Binary files /dev/null and b/docs/34.0.0/assets/web-console-13-lookups.png differ
diff --git a/docs/34.0.0/comparisons/druid-vs-elasticsearch.md b/docs/34.0.0/comparisons/druid-vs-elasticsearch.md
new file mode 100644
index 000000000..82752aa7a
--- /dev/null
+++ b/docs/34.0.0/comparisons/druid-vs-elasticsearch.md
@@ -0,0 +1,39 @@
+---
+id: druid-vs-elasticsearch
+title: "Apache Druid vs Elasticsearch"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+We are not experts on search systems, if anything is incorrect about our portrayal, please let us know on the mailing list or via some other means.
+
+Elasticsearch is a search system based on Apache Lucene. It provides full text search for schema-free documents
+and provides access to raw event level data. Elasticsearch is increasingly adding more support for analytics and aggregations.
+[Some members of the community](https://groups.google.com/forum/#!msg/druid-development/nlpwTHNclj8/sOuWlKOzPpYJ) have pointed out
+the resource requirements for data ingestion and aggregation in Elasticsearch is much higher than those of Druid.
+
+Elasticsearch also does not support data summarization/roll-up at ingestion time, which can compact the data that needs to be
+stored up to 100x with real-world data sets. This leads to Elasticsearch having greater storage requirements.
+
+Druid focuses on OLAP work flows. Druid is optimized for high performance (fast aggregation and ingestion) at low cost,
+and supports a wide range of analytic operations. Druid has some basic search support for structured event data, but does not support
+full text search. Druid also does not support completely unstructured data. Measures must be defined in a Druid schema such that
+summarization/roll-up can be done.
diff --git a/docs/34.0.0/comparisons/druid-vs-key-value.md b/docs/34.0.0/comparisons/druid-vs-key-value.md
new file mode 100644
index 000000000..57f3dec66
--- /dev/null
+++ b/docs/34.0.0/comparisons/druid-vs-key-value.md
@@ -0,0 +1,46 @@
+---
+id: druid-vs-key-value
+title: "Apache Druid vs. Key/Value Stores (HBase/Cassandra/OpenTSDB)"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+Druid is highly optimized for scans and aggregations, it supports arbitrarily deep drill downs into data sets. This same functionality
+is supported in key/value stores in 2 ways:
+
+1. Pre-compute all permutations of possible user queries
+2. Range scans on event data
+
+When pre-computing results, the key is the exact parameters of the query, and the value is the result of the query.
+The queries return extremely quickly, but at the cost of flexibility, as ad-hoc exploratory queries are not possible with
+pre-computing every possible query permutation. Pre-computing all permutations of all ad-hoc queries leads to result sets
+that grow exponentially with the number of columns of a data set, and pre-computing queries for complex real-world data sets
+can require hours of pre-processing time.
+
+The other approach to using key/value stores for aggregations to use the dimensions of an event as the key and the event measures as the value.
+Aggregations are done by issuing range scans on this data. Timeseries specific databases such as OpenTSDB use this approach.
+One of the limitations here is that the key/value storage model does not have indexes for any kind of filtering other than prefix ranges,
+which can be used to filter a query down to a metric and time range, but cannot resolve complex predicates to narrow the exact data to scan.
+When the number of rows to scan gets large, this limitation can greatly reduce performance. It is also harder to achieve good
+locality with key/value stores because most don’t support pushing down aggregates to the storage layer.
+
+For arbitrary exploration of data (flexible data filtering), Druid's custom column format enables ad-hoc queries without pre-computation. The format
+also enables fast scans on columns, which is important for good aggregation performance.
diff --git a/docs/34.0.0/comparisons/druid-vs-kudu.md b/docs/34.0.0/comparisons/druid-vs-kudu.md
new file mode 100644
index 000000000..b992a1633
--- /dev/null
+++ b/docs/34.0.0/comparisons/druid-vs-kudu.md
@@ -0,0 +1,39 @@
+---
+id: druid-vs-kudu
+title: "Apache Druid vs Kudu"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+Kudu's storage format enables single row updates, whereas updates to existing Druid segments requires recreating the segment, so theoretically
+the process for updating old values should be higher latency in Druid. However, the requirements in Kudu for maintaining extra head space to store
+updates as well as organizing data by id instead of time has the potential to introduce some extra latency and accessing
+of data that is not needed to answer a query at query time.
+
+Druid summarizes/rollups up data at ingestion time, which in practice reduces the raw data that needs to be
+stored significantly (up to 40 times on average), and increases performance of scanning raw data significantly.
+Druid segments also contain bitmap indexes for fast filtering, which Kudu does not currently support.
+Druid's segment architecture is heavily geared towards fast aggregates and filters, and for OLAP workflows. Appends are very
+fast in Druid, whereas updates of older data are higher latency. This is by design as the data Druid is good for is typically event data,
+and does not need to be updated too frequently. Kudu supports arbitrary primary keys with uniqueness constraints, and
+efficient lookup by ranges of those keys. Kudu chooses not to include the execution engine, but supports sufficient
+operations so as to allow node-local processing from the execution engines. This means that Kudu can support multiple frameworks on the same data (e.g., MR, Spark, and SQL).
+Druid includes its own query layer that allows it to push down aggregations and computations directly to data processes for faster query processing.
diff --git a/docs/34.0.0/comparisons/druid-vs-redshift.md b/docs/34.0.0/comparisons/druid-vs-redshift.md
new file mode 100644
index 000000000..3e2c7b9ea
--- /dev/null
+++ b/docs/34.0.0/comparisons/druid-vs-redshift.md
@@ -0,0 +1,62 @@
+---
+id: druid-vs-redshift
+title: "Apache Druid vs Redshift"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+### How does Druid compare to Redshift?
+
+In terms of drawing a differentiation, Redshift started out as ParAccel (Actian), which Amazon is licensing and has since heavily modified.
+
+Aside from potential performance differences, there are some functional differences:
+
+### Real-time data ingestion
+
+Because Druid is optimized to provide insight against massive quantities of streaming data; it is able to load and aggregate data in real-time.
+
+Generally traditional data warehouses including column stores work only with batch ingestion and are not optimal for streaming data in regularly.
+
+### Druid is a read oriented analytical data store
+
+Druid’s write semantics are not as fluid and does not support full joins (we support large table to small table joins). Redshift provides full SQL support including joins and insert/update statements.
+
+### Data distribution model
+
+Druid’s data distribution is segment-based and leverages a highly available "deep" storage such as S3 or HDFS. Scaling up (or down) does not require massive copy actions or downtime; in fact, losing any number of Historical processes does not result in data loss because new Historical processes can always be brought up by reading data from "deep" storage.
+
+To contrast, ParAccel’s data distribution model is hash-based. Expanding the cluster requires re-hashing the data across the nodes, making it difficult to perform without taking downtime. Amazon’s Redshift works around this issue with a multi-step process:
+
+* set cluster into read-only mode
+* copy data from cluster to new cluster that exists in parallel
+* redirect traffic to new cluster
+
+### Replication strategy
+
+Druid employs segment-level data distribution meaning that more processes can be added and rebalanced without having to perform a staged swap. The replication strategy also makes all replicas available for querying. Replication is done automatically and without any impact to performance.
+
+ParAccel’s hash-based distribution generally means that replication is conducted via hot spares. This puts a numerical limit on the number of nodes you can lose without losing data, and this replication strategy often does not allow the hot spare to help share query load.
+
+### Indexing strategy
+
+Along with column oriented structures, Druid uses indexing structures to speed up query execution when a filter is provided. Indexing structures do increase storage overhead (and make it more difficult to allow for mutation), but they also significantly speed up queries.
+
+ParAccel does not appear to employ indexing strategies.
diff --git a/docs/34.0.0/comparisons/druid-vs-spark.md b/docs/34.0.0/comparisons/druid-vs-spark.md
new file mode 100644
index 000000000..4d3a6b43d
--- /dev/null
+++ b/docs/34.0.0/comparisons/druid-vs-spark.md
@@ -0,0 +1,42 @@
+---
+id: druid-vs-spark
+title: "Apache Druid vs Spark"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+Druid and Spark are complementary solutions as Druid can be used to accelerate OLAP queries in Spark.
+
+Spark is a general cluster computing framework initially designed around the concept of Resilient Distributed Datasets (RDDs).
+RDDs enable data reuse by persisting intermediate results
+in memory and enable Spark to provide fast computations for iterative algorithms.
+This is especially beneficial for certain work flows such as machine
+learning, where the same operation may be applied over and over
+again until some result is converged upon. The generality of Spark makes it very suitable as an engine to process (clean or transform) data.
+Although Spark provides the ability to query data through Spark SQL, much like Hadoop, the query latencies are not specifically targeted to be interactive (sub-second).
+
+Druid's focus is on extremely low latency queries, and is ideal for powering applications used by thousands of users, and where each query must
+return fast enough such that users can interactively explore through data. Druid fully indexes all data, and can act as a middle layer between Spark and your application.
+One typical setup seen in production is to process data in Spark, and load the processed data into Druid for faster access.
+
+For more information about using Druid and Spark together, including benchmarks of the two systems, please see:
+
+https://www.linkedin.com/pulse/combining-druid-spark-interactive-flexible-analytics-scale-butani
diff --git a/docs/34.0.0/comparisons/druid-vs-sql-on-hadoop.md b/docs/34.0.0/comparisons/druid-vs-sql-on-hadoop.md
new file mode 100644
index 000000000..00e447312
--- /dev/null
+++ b/docs/34.0.0/comparisons/druid-vs-sql-on-hadoop.md
@@ -0,0 +1,82 @@
+---
+id: druid-vs-sql-on-hadoop
+title: "Apache Druid vs SQL-on-Hadoop"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+SQL-on-Hadoop engines provide an
+execution engine for various data formats and data stores, and
+many can be made to push down computations down to Druid, while providing a SQL interface to Druid.
+
+For a direct comparison between the technologies and when to only use one or the other, things basically comes down to your
+product requirements and what the systems were designed to do.
+
+Druid was designed to
+
+1. be an always on service
+1. ingest data in real-time
+1. handle slice-n-dice style ad-hoc queries
+
+SQL-on-Hadoop engines generally sidestep Map/Reduce, instead querying data directly from HDFS or, in some cases, other storage systems.
+Some of these engines (including Impala and Presto) can be co-located with HDFS data nodes and coordinate with them to achieve data locality for queries.
+What does this mean? We can talk about it in terms of three general areas
+
+1. Queries
+1. Data Ingestion
+1. Query Flexibility
+
+### Queries
+
+Druid segments stores data in a custom column format. Segments are scanned directly as part of queries and each Druid server
+calculates a set of results that are eventually merged at the Broker level. This means the data that is transferred between servers
+are queries and results, and all computation is done internally as part of the Druid servers.
+
+Most SQL-on-Hadoop engines are responsible for query planning and execution for underlying storage layers and storage formats.
+They are processes that stay on even if there is no query running (eliminating the JVM startup costs from Hadoop MapReduce).
+Some (Impala/Presto) SQL-on-Hadoop engines have daemon processes that can be run where the data is stored, virtually eliminating network transfer costs. There is still
+some latency overhead (e.g. serialization/deserialization time) associated with pulling data from the underlying storage layer into the computation layer. We are unaware of exactly
+how much of a performance impact this makes.
+
+### Data Ingestion
+
+Druid is built to allow for real-time ingestion of data. You can ingest data and query it immediately upon ingestion,
+the latency between how quickly the event is reflected in the data is dominated by how long it takes to deliver the event to Druid.
+
+SQL-on-Hadoop, being based on data in HDFS or some other backing store, are limited in their data ingestion rates by the
+rate at which that backing store can make data available. Generally, the backing store is the biggest bottleneck for
+how quickly data can become available.
+
+### Query Flexibility
+
+Druid's query language is fairly low level and maps to how Druid operates internally. Although Druid can be combined with a high level query
+planner to support most SQL queries and analytic SQL queries (minus joins among large tables),
+base Druid is less flexible than SQL-on-Hadoop solutions for generic processing.
+
+SQL-on-Hadoop support SQL style queries with full joins.
+
+## Druid vs Parquet
+
+Parquet is a column storage format that is designed to work with SQL-on-Hadoop engines. Parquet doesn't have a query execution engine, and instead
+relies on external sources to pull data out of it.
+
+Druid's storage format is highly optimized for linear scans. Although Druid has support for nested data, Parquet's storage format is much
+more hierarchical, and is more designed for binary chunking. In theory, this should lead to faster scans in Druid.
diff --git a/docs/34.0.0/configuration/extensions.md b/docs/34.0.0/configuration/extensions.md
new file mode 100644
index 000000000..fb040188b
--- /dev/null
+++ b/docs/34.0.0/configuration/extensions.md
@@ -0,0 +1,177 @@
+---
+id: extensions
+title: "Extensions"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+Druid implements an extension system that allows for adding functionality at runtime. Extensions
+are commonly used to add support for deep storages (like HDFS and S3), metadata stores (like MySQL
+and PostgreSQL), new aggregators, new input formats, and so on.
+
+Production clusters will generally use at least two extensions; one for deep storage and one for a
+metadata store. Many clusters will also use additional extensions.
+
+## Core extensions
+
+Core extensions are maintained by Druid committers.
+
+|Name|Description|Docs|
+|----|-----------|----|
+|druid-avro-extensions|Support for data in Apache Avro data format.|[link](../development/extensions-core/avro.md)|
+|druid-azure-extensions|Microsoft Azure deep storage.|[link](../development/extensions-core/azure.md)|
+|druid-basic-security|Support for Basic HTTP authentication and role-based access control.|[link](../development/extensions-core/druid-basic-security.md)|
+|druid-bloom-filter|Support for providing Bloom filters in druid queries.|[link](../development/extensions-core/bloom-filter.md)|
+|druid-catalog|This extension allows users to configure, update, retrieve, and manage metadata stored in Druid's catalog. |[link](../development/extensions-core/catalog.md)|
+|druid-datasketches|Support for approximate counts and set operations with [Apache DataSketches](https://datasketches.apache.org/).|[link](../development/extensions-core/datasketches-extension.md)|
+|druid-google-extensions|Google Cloud Storage deep storage.|[link](../development/extensions-core/google.md)|
+|druid-hdfs-storage|HDFS deep storage.|[link](../development/extensions-core/hdfs.md)|
+|druid-histogram|Approximate histograms and quantiles aggregator. Deprecated, please use the [DataSketches quantiles aggregator](../development/extensions-core/datasketches-quantiles.md) from the `druid-datasketches` extension instead.|[link](../development/extensions-core/approximate-histograms.md)|
+|druid-kafka-extraction-namespace|Apache Kafka-based namespaced lookup. Requires namespace lookup extension.|[link](../querying/kafka-extraction-namespace.md)|
+|druid-kafka-indexing-service|Supervised exactly-once Apache Kafka ingestion for the indexing service.|[link](../ingestion/kafka-ingestion.md)|
+|druid-kinesis-indexing-service|Supervised exactly-once Kinesis ingestion for the indexing service.|[link](../ingestion/kinesis-ingestion.md)|
+|druid-kerberos|Kerberos authentication for druid processes.|[link](../development/extensions-core/druid-kerberos.md)|
+|druid-lookups-cached-global|A module for [lookups](../querying/lookups.md) providing a jvm-global eager caching for lookups. It provides JDBC and URI implementations for fetching lookup data.|[link](../querying/lookups-cached-global.md)|
+|druid-lookups-cached-single| Per lookup caching module to support the use cases where a lookup need to be isolated from the global pool of lookups |[link](../development/extensions-core/druid-lookups.md)|
+|druid-multi-stage-query| Support for the multi-stage query architecture for Apache Druid and the multi-stage query task engine.|[link](../multi-stage-query/index.md)|
+|druid-orc-extensions|Support for data in Apache ORC data format.|[link](../development/extensions-core/orc.md)|
+|druid-parquet-extensions|Support for data in Apache Parquet data format. Requires druid-avro-extensions to be loaded.|[link](../development/extensions-core/parquet.md)|
+|druid-protobuf-extensions| Support for data in Protobuf data format.|[link](../development/extensions-core/protobuf.md)|
+|druid-s3-extensions|Interfacing with data in Amazon S3, and using S3 as deep storage.|[link](../development/extensions-core/s3.md)|
+|druid-ec2-extensions|Interfacing with AWS EC2 for autoscaling middle managers|UNDOCUMENTED|
+|druid-aws-rds-extensions|Support for AWS token based access to AWS RDS DB Cluster.|[link](../development/extensions-core/druid-aws-rds.md)|
+|druid-stats|Statistics related module including variance and standard deviation.|[link](../development/extensions-core/stats.md)|
+|mysql-metadata-storage|MySQL metadata store.|[link](../development/extensions-core/mysql.md)|
+|postgresql-metadata-storage|PostgreSQL metadata store.|[link](../development/extensions-core/postgresql.md)|
+|simple-client-sslcontext|Simple SSLContext provider module to be used by Druid's internal HttpClient when talking to other Druid processes over HTTPS.|[link](../development/extensions-core/simple-client-sslcontext.md)|
+|druid-pac4j|OpenID Connect authentication for druid processes.|[link](../development/extensions-core/druid-pac4j.md)|
+|druid-kubernetes-extensions|Druid cluster deployment on Kubernetes without Zookeeper.|[link](../development/extensions-core/kubernetes.md)|
+|druid-kubernetes-overlord-extensions|Support for launching tasks in k8s without Middle Managers|[link](../development/extensions-core/k8s-jobs.md)|
+
+## Community extensions
+
+:::info
+ Community extensions are not maintained by Druid committers, although we accept patches from community members using these extensions. They may not have been as extensively tested as the core extensions.
+:::
+
+A number of community members have contributed their own extensions to Druid that are not packaged with the default Druid tarball.
+If you'd like to take on maintenance for a community extension, please post on [dev@druid.apache.org](https://lists.apache.org/list.html?dev@druid.apache.org) to let us know!
+
+All of these community extensions can be downloaded using [pull-deps](../operations/pull-deps.md) while specifying a `-c` coordinate option to pull `org.apache.druid.extensions.contrib:{EXTENSION_NAME}:{DRUID_VERSION}`.
+
+|Name|Description|Docs|
+|----|-----------|----|
+|aliyun-oss-extensions|Aliyun OSS deep storage |[link](../development/extensions-contrib/aliyun-oss-extensions.md)|
+|ambari-metrics-emitter|Ambari Metrics Emitter |[link](../development/extensions-contrib/ambari-metrics-emitter.md)|
+|druid-cassandra-storage|Apache Cassandra deep storage.|[link](../development/extensions-contrib/cassandra.md)|
+|druid-cloudfiles-extensions|Rackspace Cloudfiles deep storage.|[link](../development/extensions-contrib/cloudfiles.md)|
+|druid-compressed-bigdecimal|Compressed Big Decimal Type | [link](../development/extensions-contrib/compressed-big-decimal.md)|
+|druid-ddsketch|Support for DDSketch approximate quantiles based on [DDSketch](https://github.com/datadog/sketches-java) | [link](../development/extensions-contrib/ddsketch-quantiles.md)|
+|druid-deltalake-extensions|Support for ingesting Delta Lake tables.|[link](../development/extensions-contrib/delta-lake.md)|
+|druid-distinctcount|DistinctCount aggregator|[link](../development/extensions-contrib/distinctcount.md)|
+|druid-iceberg-extensions|Support for ingesting Iceberg tables.|[link](../development/extensions-contrib/iceberg.md)|
+|druid-redis-cache|A cache implementation for Druid based on Redis.|[link](../development/extensions-contrib/redis-cache.md)|
+|druid-time-min-max|Min/Max aggregator for timestamp.|[link](../development/extensions-contrib/time-min-max.md)|
+|sqlserver-metadata-storage|Microsoft SQLServer metadata store.|[link](../development/extensions-contrib/sqlserver.md)|
+|graphite-emitter|Graphite metrics emitter|[link](../development/extensions-contrib/graphite.md)|
+|statsd-emitter|StatsD metrics emitter|[link](../development/extensions-contrib/statsd.md)|
+|kafka-emitter|Kafka metrics emitter|[link](../development/extensions-contrib/kafka-emitter.md)|
+|druid-thrift-extensions|Support thrift ingestion |[link](../development/extensions-contrib/thrift.md)|
+|druid-opentsdb-emitter|OpenTSDB metrics emitter |[link](../development/extensions-contrib/opentsdb-emitter.md)|
+|materialized-view-selection, materialized-view-maintenance|Materialized View|[link](../development/extensions-contrib/materialized-view.md)|
+|druid-moving-average-query|Support for [Moving Average](https://en.wikipedia.org/wiki/Moving_average) and other Aggregate [Window Functions](https://en.wikibooks.org/wiki/Structured_Query_Language/Window_functions) in Druid queries.|[link](../development/extensions-contrib/moving-average-query.md)|
+|druid-influxdb-emitter|InfluxDB metrics emitter|[link](../development/extensions-contrib/influxdb-emitter.md)|
+|druid-momentsketch|Support for approximate quantile queries using the [momentsketch](https://github.com/stanford-futuredata/momentsketch) library|[link](../development/extensions-contrib/momentsketch-quantiles.md)|
+|druid-tdigestsketch|Support for approximate sketch aggregators based on [T-Digest](https://github.com/tdunning/t-digest)|[link](../development/extensions-contrib/tdigestsketch-quantiles.md)|
+|gce-extensions|GCE Extensions|[link](../development/extensions-contrib/gce-extensions.md)|
+|prometheus-emitter|Exposes [Druid metrics](../operations/metrics.md) for [Prometheus](https://prometheus.io/)|[link](../development/extensions-contrib/prometheus.md)|
+|druid-spectator-histogram|Support for efficient approximate percentile queries|[link](../development/extensions-contrib/spectator-histogram.md)|
+|druid-rabbit-indexing-service|Support for creating and managing [RabbitMQ](https://www.rabbitmq.com/) indexing tasks|[link](../development/extensions-contrib/rabbit-stream-ingestion.md)|
+|druid-ranger-security|Support for access control through Apache Ranger.|[link](../development/extensions-contrib/druid-ranger-security.md)|
+
+## Promoting community extensions to core extensions
+
+Please post on [dev@druid.apache.org](https://lists.apache.org/list.html?dev@druid.apache.org) if you'd like an extension to be promoted to core.
+If we see a community extension actively supported by the community, we can promote it to core based on community feedback.
+
+For information how to create your own extension, please see [here](../development/modules.md).
+
+## Loading extensions
+
+### Loading core extensions
+
+Apache Druid bundles all [core extensions](../configuration/extensions.md#core-extensions) out of the box.
+See the [list of extensions](../configuration/extensions.md#core-extensions) for your options. You
+can load bundled extensions by adding their names to your common.runtime.properties
+`druid.extensions.loadList` property. For example, to load the postgresql-metadata-storage and
+druid-hdfs-storage extensions, use the configuration:
+
+```properties
+druid.extensions.loadList=["postgresql-metadata-storage", "druid-hdfs-storage"]
+```
+
+These extensions are located in the `extensions` directory of the distribution.
+
+:::info
+ Druid bundles two sets of configurations: one for the [quickstart](../tutorials/index.md) and
+ one for a [clustered configuration](../tutorials/cluster.md). Make sure you are updating the correct
+ `common.runtime.properties` for your setup.
+:::
+
+:::info
+ Because of licensing, the mysql-metadata-storage extension does not include the required MySQL JDBC driver. For instructions
+ on how to install this library, see the [MySQL extension page](../development/extensions-core/mysql.md).
+:::
+
+### Loading community extensions
+
+You can also load community and third-party extensions not already bundled with Druid. To do this, first download the extension and
+then install it into your `extensions` directory. You can download extensions from their distributors directly, or
+if they are available from Maven, the included [pull-deps](../operations/pull-deps.md) can download them for you. To use *pull-deps*,
+specify the full Maven coordinate of the extension in the form `groupId:artifactId:version`. For example,
+for the (hypothetical) extension *com.example:druid-example-extension:1.0.0*, run:
+
+```shell
+java \
+ -cp "lib/*" \
+ -Ddruid.extensions.directory="extensions" \
+ -Ddruid.extensions.hadoopDependenciesDir="hadoop-dependencies" \
+ org.apache.druid.cli.Main tools pull-deps \
+ --no-default-hadoop \
+ -c "com.example:druid-example-extension:1.0.0"
+```
+
+You only have to install the extension once. Then, add `"druid-example-extension"` to
+`druid.extensions.loadList` in common.runtime.properties to instruct Druid to load the extension.
+
+:::info
+ Please make sure all the Extensions related configuration properties listed [here](../configuration/index.md#extensions) are set correctly.
+:::
+
+:::info
+ The Maven `groupId` for almost every [community extension](../configuration/extensions.md#community-extensions) is `org.apache.druid.extensions.contrib`. The `artifactId` is the name
+ of the extension, and the version is the latest Druid stable version.
+:::
+
+### Loading extensions from the classpath
+
+If you add your extension jar to the classpath at runtime, Druid will also load it into the system. This mechanism is relatively easy to reason about,
+but it also means that you have to ensure that all dependency jars on the classpath are compatible. That is, Druid makes no provisions while using
+this method to maintain class loader isolation so you must make sure that the jars on your classpath are mutually compatible.
diff --git a/docs/34.0.0/configuration/human-readable-byte.md b/docs/34.0.0/configuration/human-readable-byte.md
new file mode 100644
index 000000000..0f412b69a
--- /dev/null
+++ b/docs/34.0.0/configuration/human-readable-byte.md
@@ -0,0 +1,98 @@
+---
+id: human-readable-byte
+title: "Human-readable Byte Configuration Reference"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+This page documents configuration properties related to bytes.
+
+These properties can be configured through 2 ways:
+1. a simple number in bytes
+2. a number with a unit suffix
+
+## A number in bytes
+
+Given that cache size is 3G, there's a configuration as below
+
+```properties
+# 3G bytes = 3_000_000_000 bytes
+druid.cache.sizeInBytes=3000000000
+```
+
+
+## A number with a unit suffix
+
+When you have to put a large number for some configuration as above, it is easy to make a mistake such as extra or missing 0s. Druid supports a better way, a number with a unit suffix.
+
+Given a disk of 1T, the configuration can be
+
+```properties
+druid.segmentCache.locations=[{"path":"/segment-cache-00","maxSize":"1t"},{"path":"/segment-cache-01","maxSize":"1200g"}]
+```
+
+Note: in above example, both `1t` and `1T` are acceptable since it's case-insensitive.
+Also, only integers are valid as the number part. For example, you can't replace `1200g` with `1.2t`.
+
+### Supported Units
+In the world of computer, a unit like `K` is ambiguous. It means 1000 or 1024 in different contexts, for more information please see [Here](https://en.wikipedia.org/wiki/Binary_prefix).
+
+To make it clear, the base of units are defined in Druid as below
+
+| Unit | Description | Base |
+|---|---|---|
+| K | Kilo Decimal Byte | 1_000 |
+| M | Mega Decimal Byte | 1_000_000 |
+| G | Giga Decimal Byte | 1_000_000_000 |
+| T | Tera Decimal Byte | 1_000_000_000_000 |
+| P | Peta Decimal Byte | 1_000_000_000_000_000 |
+| Ki | Kilo Binary Byte | 1024 |
+| Mi | Mega Binary Byte | 1024 * 1024 |
+| Gi | Giga Binary Byte | 1024 * 1024 * 1024 |
+| Ti | Tera Binary Byte | 1024 * 1024 * 1024 * 1024 |
+| Pi | Peta Binary Byte | 1024 * 1024 * 1024 * 1024 * 1024 |
+| KiB | Kilo Binary Byte | 1024 |
+| MiB | Mega Binary Byte | 1024 * 1024 |
+| GiB | Giga Binary Byte | 1024 * 1024 * 1024 |
+| TiB | Tera Binary Byte | 1024 * 1024 * 1024 * 1024 |
+| PiB | Peta Binary Byte | 1024 * 1024 * 1024 * 1024 * 1024 |
+
+Unit is case-insensitive. `k`, `kib`, `ki`, `KiB`, `Ki`, `kiB` are all acceptable.
+
+Here are some examples
+
+```properties
+# 1G bytes = 1_000_000_000 bytes
+druid.cache.sizeInBytes=1g
+```
+
+```properties
+# 256MiB bytes = 256 * 1024 * 1024 bytes
+druid.cache.sizeInBytes=256MiB
+```
+
+```properties
+# 256Mi = 256MiB = 256 * 1024 * 1024 bytes
+druid.cache.sizeInBytes=256Mi
+```
+
+
+
diff --git a/docs/latest/configuration/index.md b/docs/34.0.0/configuration/index.md
similarity index 94%
copy from docs/latest/configuration/index.md
copy to docs/34.0.0/configuration/index.md
index 91168d947..0976190cf 100644
--- a/docs/latest/configuration/index.md
+++ b/docs/34.0.0/configuration/index.md
@@ -148,6 +148,7 @@ We recommend just setting the base ZK path and the ZK service host, but all ZK p
|`druid.zk.service.connectionTimeoutMs`|ZooKeeper connection timeout, in milliseconds.|`15000`|
|`druid.zk.service.compress`|Boolean flag for whether or not created Znodes should be compressed.|`true`|
|`druid.zk.service.acl`|Boolean flag for whether or not to enable ACL security for ZooKeeper. If ACL is enabled, zNode creators will have all permissions.|`false`|
+|`druid.zk.service.pathChildrenCacheStrategy`|Dictates the underlying caching strategy for service announcements. Set true to let announcers to use Apache Curator's PathChildrenCache strategy, otherwise NodeCache strategy. Consider using NodeCache strategy when you are dealing with huge number of ZooKeeper watches in your cluster.|`true`|
#### Path configuration
@@ -275,9 +276,10 @@ The `file` request logger stores daily request logs on disk.
|Property|Description|Default|
|--------|-----------|-------|
-|`druid.request.logging.dir`|Historical, Realtime, and Broker services maintain request logs of all of the requests they get (interaction is via POST, so normal request logs don’t generally capture information about the actual query), this specifies the directory to store the request logs in|none|
-|`druid.request.logging.filePattern`|[Joda datetime format](http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html) for each file|"yyyy-MM-dd'.log'"|
-| `druid.request.logging.durationToRetain`| Period to retain the request logs on disk. The period should be at least longer than `P1D`.| none|
+|`druid.request.logging.dir`| Historical, Realtime, and Broker services maintain request logs of all of the requests they get (interaction is via POST, so normal request logs don’t generally capture information about the actual query), this specifies the directory to store the request logs in. | none|
+|`druid.request.logging.filePattern`| [Joda datetime format](http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html) for each file.| "yyyy-MM-dd'.log'"|
+|`druid.request.logging.durationToRetain`| Period to retain the request logs on disk. The period should be at least as long as roll period.| none|
+|`druid.request.logging.rollPeriod`| Defines the log rotation period for request logs. The period should be at least `PT1H`. For periods smaller than 1 day, it is recommended to use `"yyyy-MM-dd-HH'.log'"` as the file pattern.| P1D|
The format of request logs is TSV, one line per requests, with five fields: timestamp, remote\_addr, native\_query, query\_context, sql\_query.
@@ -295,7 +297,7 @@ For SQL query request, the `native_query` field is empty. For example:
#### Emitter request logging
-The `emitter` request logger emits every request to the external location specified in the [emitter](#enabling-metrics) configuration.
+The `emitter` request logger emits every request to the external location specified in the [emitter](#metrics-monitors) configuration.
|Property|Description|Default|
|--------|-----------|-------|
@@ -370,144 +372,6 @@ Coordinator and Overlord log changes to lookups, segment load/drop rules, and dy
|`druid.audit.manager.maxPayloadSizeBytes`|The maximum size of audit payload to store in Druid's metadata store audit table. If the size of audit payload exceeds this value, the audit log would be stored with a message indicating that the payload was omitted instead. Setting `maxPayloadSizeBytes` to -1 (default value) disables this check, meaning Druid will always store audit payload regardless of it's size. Setting to any negative number other than `-1` is invalid. Human-readable format [...]
|`druid.audit.manager.skipNullField`|If true, the audit payload stored in metadata store will exclude any field with null value. |false|
-### Enabling metrics
-
-You can configure Druid services to emit [metrics](../operations/metrics.md) regularly from a number of [monitors](#metrics-monitors) via [emitters](#metrics-emitters).
-
-|Property|Description|Default|
-|--------|-----------|-------|
-|`druid.monitoring.emissionPeriod`| Frequency that Druid emits metrics.|`PT1M`|
-|[`druid.monitoring.monitors`](#metrics-monitors)|Sets list of Druid monitors used by a service.<br /><br />Because individual monitors sometimes only work on specific process types, it is best to set this property for each process type individually in e.g. `historical/runtime.properties` rather than `_common/common.runtime.properties`.|none (no monitors)|
-|[`druid.emitter`](#metrics-emitters)|Setting this value initializes one of the emitter modules.|`noop` (metric emission disabled by default)|
-
-#### Metrics monitors
-
-Metric monitoring is an essential part of Druid operations. The following monitors are available:
-
-|Name|Description|
-|----|-----------|
-|`org.apache.druid.client.cache.CacheMonitor`|Emits metrics (to logs) about the segment results cache for Historical and Broker services. Reports typical cache statistics include hits, misses, rates, and size (bytes and number of entries), as well as timeouts and and errors.|
-|`org.apache.druid.java.util.metrics.SysMonitor`|Reports on various system activities and statuses using the [SIGAR library](https://github.com/hyperic/sigar). Requires execute privileges on files in `java.io.tmpdir`. Do not set `java.io.tmpdir` to `noexec` when using `SysMonitor`.|
-|`org.apache.druid.java.util.metrics.JvmMonitor`|Reports various JVM-related statistics.|
-|`org.apache.druid.java.util.metrics.JvmCpuMonitor`|Reports statistics of CPU consumption by the JVM.|
-|`org.apache.druid.java.util.metrics.CpuAcctDeltaMonitor`|Reports consumed CPU as per the cpuacct cgroup.|
-|`org.apache.druid.java.util.metrics.JvmThreadsMonitor`|Reports Thread statistics in the JVM, like numbers of total, daemon, started, died threads.|
-|`org.apache.druid.java.util.metrics.CgroupCpuMonitor`|Reports CPU shares and quotas as per the `cpu` cgroup.|
-|`org.apache.druid.java.util.metrics.CgroupCpuSetMonitor`|Reports CPU core/HT and memory node allocations as per the `cpuset` cgroup.|
-|`org.apache.druid.java.util.metrics.CgroupDiskMonitor`|Reports disk statistic as per the blkio cgroup.|
-|`org.apache.druid.java.util.metrics.CgroupMemoryMonitor`|Reports memory statistic as per the memory cgroup.|
-|`org.apache.druid.java.util.metrics.CgroupV2CpuMonitor`| **EXPERIMENTAL** Reports CPU usage from `cpu.stat` file. Only applicable to `cgroupv2`.|
-|`org.apache.druid.java.util.metrics.CgroupV2DiskMonitor`| **EXPERIMENTAL** Reports disk usage from `io.stat` file. Only applicable to `cgroupv2`.|
-|`org.apache.druid.java.util.metrics.CgroupV2MemoryMonitor`| **EXPERIMENTAL** Reports memory usage from `memory.current` and `memory.max` files. Only applicable to `cgroupv2`.|
-|`org.apache.druid.server.metrics.HistoricalMetricsMonitor`|Reports statistics on Historical services. Available only on Historical services.|
-|`org.apache.druid.server.metrics.SegmentStatsMonitor` | **EXPERIMENTAL** Reports statistics about segments on Historical services. Available only on Historical services. Not to be used when lazy loading is configured.|
-|`org.apache.druid.server.metrics.QueryCountStatsMonitor`|Reports how many queries have been successful/failed/interrupted.|
-|`org.apache.druid.server.metrics.SubqueryCountStatsMonitor`|Reports how many subqueries have been materialized as rows or bytes and various other statistics related to the subquery execution|
-|`org.apache.druid.server.emitter.HttpEmittingMonitor`|Reports internal metrics of `http` or `parametrized` emitter (see below). Must not be used with another emitter type. See the description of the metrics here: https://github.com/apache/druid/pull/4973.|
-|`org.apache.druid.server.metrics.TaskCountStatsMonitor`|Reports how many ingestion tasks are currently running/pending/waiting and also the number of successful/failed tasks per emission period.|
-|`org.apache.druid.server.metrics.TaskSlotCountStatsMonitor`|Reports metrics about task slot usage per emission period.|
-|`org.apache.druid.server.metrics.WorkerTaskCountStatsMonitor`|Reports how many ingestion tasks are currently running/pending/waiting, the number of successful/failed tasks, and metrics about task slot usage for the reporting worker, per emission period. Only supported by Middle Manager node types.|
-|`org.apache.druid.server.metrics.ServiceStatusMonitor`|Reports a heartbeat for the service.|
-|`org.apache.druid.server.metrics.GroupByStatsMonitor`|Report metrics for groupBy queries like disk and merge buffer utilization. |
-
-For example, you might configure monitors on all services for system and JVM information within `common.runtime.properties` as follows:
-
-```properties
-druid.monitoring.monitors=["org.apache.druid.java.util.metrics.SysMonitor","org.apache.druid.java.util.metrics.JvmMonitor"]
-```
-
-You can override cluster-wide configuration by amending the `runtime.properties` of individual services.
-
-#### Metrics emitters
-
-There are several emitters available:
-
-* `noop` (default) disables metric emission.
-* [`logging`](#logging-emitter-module) emits logs using Log4j2.
-* [`http`](#http-emitter-module) sends `POST` requests of JSON events.
-* [`parametrized`](#parametrized-http-emitter-module) operates like the `http` emitter but fine-tunes the recipient URL based on the event feed.
-* [`composing`](#composing-emitter-module) initializes multiple emitter modules.
-* [`graphite`](#graphite-emitter) emits metrics to a [Graphite](https://graphiteapp.org/) Carbon service.
-* [`switching`](#switching-emitter) initializes and emits to multiple emitter modules based on the event feed.
-
-##### Logging emitter module
-
-The use this emitter module, set `druid.emitter=logging`. The `logging` emitter uses a Log4j2 logger named
-`druid.emitter.logging.loggerClass` to emit events. Each event is logged as a single `json` object with a
-[Marker](https://logging.apache.org/log4j/2.x/manual/markers.html) as the feed of the event. Users may wish to edit the
-log4j config to route these logs to different sources based on the feed of the event.
-
-|Property|Description| Default|
-|--------|-----------|--------|
-|`druid.emitter.logging.loggerClass`|The class used for logging.|`org.apache.druid.java.util.emitter.core.LoggingEmitter`|
-|`druid.emitter.logging.logLevel`|Choices: debug, info, warn, error. The log level at which message are logged.|info|
-
-##### HTTP emitter module
-
-|Property|Description|Default|
-|--------|-----------|-------|
-|`druid.emitter.http.flushMillis`|How often the internal message buffer is flushed (data is sent).|60000|
-|`druid.emitter.http.flushCount`|How many messages the internal message buffer can hold before flushing (sending).|500|
-|`druid.emitter.http.basicAuthentication`|[Password Provider](../operations/password-provider.md) for providing login and password for authentication in `"login:password"` form. For example, `druid.emitter.http.basicAuthentication=admin:adminpassword` uses Default Password Provider which allows plain text passwords.|not specified = no authentication|
-|`druid.emitter.http.flushTimeOut`|The timeout after which an event should be sent to the endpoint, even if internal buffers are not filled, in milliseconds.|not specified = no timeout|
-|`druid.emitter.http.batchingStrategy`|The strategy of how the batch is formatted. "ARRAY" means `[event1,event2]`, "NEWLINES" means `event1\nevent2`, ONLY_EVENTS means `event1event2`.|ARRAY|
-|`druid.emitter.http.maxBatchSize`|The maximum batch size, in bytes.|the minimum of (10% of JVM heap size divided by 2) or (5242880 (i. e. 5 MiB))|
-|`druid.emitter.http.batchQueueSizeLimit`|The maximum number of batches in emitter queue, if there are problems with emitting.|the maximum of (2) or (10% of the JVM heap size divided by 5MiB)|
-|`druid.emitter.http.minHttpTimeoutMillis`|If the speed of filling batches imposes timeout smaller than that, not even trying to send batch to endpoint, because it will likely fail, not being able to send the data that fast. Configure this depending based on emitter/successfulSending/minTimeMs metric. Reasonable values are 10ms..100ms.|0|
-|`druid.emitter.http.recipientBaseUrl`|The base URL to emit messages to. Druid will POST JSON to be consumed at the HTTP endpoint specified by this property.|none, required config|
-
-##### HTTP emitter module TLS overrides
-
-By default, when sending events to a TLS-enabled receiver, the HTTP Emitter uses an SSLContext obtained from the service described at [Druid's internal communication over TLS](../operations/tls-support.md), that is the same SSLContext that would be used for internal communications between Druid services.
-
-In some use cases it may be desirable to have the HTTP Emitter use its own separate truststore configuration. For example, there may be organizational policies that prevent the TLS-enabled metrics receiver's certificate from being added to the same truststore used by Druid's internal HTTP client.
-
-The following properties allow the HTTP Emitter to use its own truststore configuration when building its SSLContext.
-
-|Property|Description|Default|
-|--------|-----------|-------|
-|`druid.emitter.http.ssl.useDefaultJavaContext`|If set to true, the HttpEmitter will use `SSLContext.getDefault()`, the default Java SSLContext, and all other properties below are ignored.|false|
-|`druid.emitter.http.ssl.trustStorePath`|The file path or URL of the TLS/SSL Key store where trusted root certificates are stored. If this is unspecified, the HTTP Emitter will use the same SSLContext as Druid's internal HTTP client, as described in the beginning of this section, and all other properties below are ignored.|null|
-|`druid.emitter.http.ssl.trustStoreType`|The type of the key store where trusted root certificates are stored.|`java.security.KeyStore.getDefaultType()`|
-|`druid.emitter.http.ssl.trustStoreAlgorithm`|Algorithm to be used by TrustManager to validate certificate chains|`javax.net.ssl.TrustManagerFactory.getDefaultAlgorithm()`|
-|`druid.emitter.http.ssl.trustStorePassword`|The [Password Provider](../operations/password-provider.md) or String password for the Trust Store.|none|
-|`druid.emitter.http.ssl.protocol`|TLS protocol to use.|"TLSv1.2"|
-
-##### Parametrized HTTP emitter module
-
-The parametrized emitter takes the same configs as the [`http` emitter](#http-emitter-module) using the prefix `druid.emitter.parametrized.httpEmitting.`.
-For example:
-
-* `druid.emitter.parametrized.httpEmitting.flushMillis`
-* `druid.emitter.parametrized.httpEmitting.flushCount`
-* `druid.emitter.parametrized.httpEmitting.ssl.trustStorePath`
-
-Do not specify `recipientBaseUrl` with the parametrized emitter.
-Instead use `recipientBaseUrlPattern` described in the table below.
-
-|Property|Description|Default|
-|--------|-----------|-------|
-|`druid.emitter.parametrized.recipientBaseUrlPattern`|The URL pattern to send an event to, based on the event's feed. For example, `http://foo.bar/{feed}`, that will send event to `http://foo.bar/metrics` if the event's feed is "metrics".|none, required config|
-
-##### Composing emitter module
-
-|Property|Description|Default|
-|--------|-----------|-------|
-|`druid.emitter.composing.emitters`|List of emitter modules to load, such as ["logging","http"].|[]|
-
-##### Graphite emitter
-
-To use graphite as emitter set `druid.emitter=graphite`. For configuration details, see [Graphite emitter](../development/extensions-contrib/graphite.md) for the Graphite emitter Druid extension.
-
-##### Switching emitter
-
-To use switching as emitter set `druid.emitter=switching`.
-
-|Property|Description|Default|
-|--------|-----------|-------|
-|`druid.emitter.switching.emitters`|JSON map of feed to list of emitter modules that will be used for the mapped feed, such as `{"metrics":["http"], "alerts":["logging"]}`|{}|
-|`druid.emitter.switching.defaultEmitters`|JSON list of emitter modules to load that will be used if there is no emitter specifically designated for that event's feed, such as `["logging","http"]`.|[]|
-
### Metadata storage
These properties specify the JDBC connection and other configuration around the metadata storage. The only services that connect to the metadata storage with these properties are the [Coordinator](../design/coordinator.md) and [Overlord](../design/overlord.md).
@@ -585,19 +449,23 @@ This deep storage is used to interface with Cassandra. You must load the `druid-
|`druid.storage.host`|Cassandra host.|none|
|`druid.storage.keyspace`|Cassandra key space.|none|
-#### Centralized datasource schema
+#### Centralized datasource schema (Experimental)
+
+This is an [experimental feature](../development/experimental.md) to improve datasource schema management by persisting segment schemas to the metadata store and caching them on the Coordinator.
+Traditionally, Brokers issue segment metadata queries to data nodes and tasks to fetch the schemas of all available segments.
+Each Broker then individually builds the schema of a datasource by combining the schemas of all the segments of that datasource.
+This mechanism is redundant and prone to errors as there is no single source of truth for schemas.
-Centralized datasource schema is an [experimental feature](../development/experimental.md) to centralized datasource schema building within the Coordinator.
-Traditionally, the datasource schema is built in the Brokers by combining schema for all the available segments of a datasource.
-Brokers issue segment metadata query to data nodes and tasks to fetch segment schema.
-In the new arrangement, tasks publish segment schema along with segment metadata to the database and schema for realtime segments is periodically pushed to the Coordinator in the segment announcement flow.
-This enables Coordinator to cache segment schemas and build datasource schema by combining segment schema.
-Brokers query the datasource schema from the Coordinator, while retaining the ability to build table schema if the
-need arises.
+Centralized schema management improves upon this design as follows:
+- Tasks publish segment schema along with segment metadata to the database.
+- Tasks announce schema for realtime segments periodically to the Coordinator.
+- Coordinator caches segment schemas and builds a combined schema for each datasource.
+- Broker poll the datasource schema cached on the Coordinator rather than building it on their own.
+- Brokers still retain the ability to build a datasource schema if they are unable to fetch it from the Coordinator.
|Property|Description|Default|Required|
-|-----|-----------|-------|--------|
-|`druid.centralizedDatasourceSchema.enabled`|Boolean flag for enabling datasource schema building in the Coordinator, this should be specified in the common runtime properties.|false|No.|
+|--------|-----------|-------|--------|
+|`druid.centralizedDatasourceSchema.enabled`|Boolean flag for enabling datasource schema building and caching on the Coordinator. This property should be specified in the common runtime properties.|false|No.|
|`druid.indexer.fork.property.druid.centralizedDatasourceSchema.enabled`| This config should be set when CentralizedDatasourceSchema feature is enabled. This should be specified in the Middle Manager runtime properties.|false|No.|
If you enable this feature, you can query datasources that are only stored in deep storage and are not loaded on a Historical. For more information, see [Query from deep storage](../querying/query-from-deep-storage.md).
@@ -872,9 +740,19 @@ These Coordinator static configurations can be defined in the `coordinator/runti
|Property|Description|Default|
|--------|-----------|-------|
|`druid.coordinator.period`|The run period for the Coordinator. The Coordinator operates by maintaining the current state of the world in memory and periodically looking at the set of "used" segments and segments being served to make decisions about whether any changes need to be made to the data topology. This property sets the delay between each of these runs.|`PT60S`|
-|`druid.coordinator.period.indexingPeriod`|How often to send compact/merge/conversion tasks to the indexing service. It's recommended to be longer than `druid.manager.segments.pollDuration`|`PT1800S` (30 mins)|
|`druid.coordinator.startDelay`|The operation of the Coordinator works on the assumption that it has an up-to-date view of the state of the world when it runs, the current ZooKeeper interaction code, however, is written in a way that doesn’t allow the Coordinator to know for a fact that it’s done loading the current state of the world. This delay is a hack to give it enough time to believe that it has all the data.|`PT300S`|
|`druid.coordinator.load.timeout`|The timeout duration for when the Coordinator assigns a segment to a Historical service.|`PT15M`|
+|`druid.coordinator.balancer.strategy`|The [balancing strategy](../design/coordinator.md#balancing-segments-in-a-tier) used by the Coordinator to distribute segments among the Historical servers in a tier. The `cost` strategy distributes segments by minimizing a cost function, `diskNormalized` weights these costs with the disk usage ratios of the servers and `random` distributes segments randomly.|`cost`|
+|`druid.coordinator.loadqueuepeon.http.repeatDelay`|The start and repeat delay (in milliseconds) for the load queue peon, which manages the load/drop queue of segments for any server.|1 minute|
+|`druid.coordinator.loadqueuepeon.http.batchSize`|Number of segment load/drop requests to batch in one HTTP request. Note that it must be smaller than or equal to the `druid.segmentCache.numLoadingThreads` config on Historical service. If this value is not configured, the coordinator uses the value of the `numLoadingThreads` for the respective server. | `druid.segmentCache.numLoadingThreads` |
+|`druid.coordinator.asOverlord.enabled`|Boolean value for whether this Coordinator service should act like an Overlord as well. This configuration allows users to simplify a Druid cluster by not having to deploy any standalone Overlord services. If set to true, then Overlord console is available at `http://coordinator-host:port/console.html` and be sure to set `druid.coordinator.asOverlord.overlordService` also.|false|
+|`druid.coordinator.asOverlord.overlordService`| Required, if `druid.coordinator.asOverlord.enabled` is `true`. This must be same value as `druid.service` on standalone Overlord services and `druid.selectors.indexing.serviceName` on Middle Managers.|NULL|
+
+##### Data management
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.coordinator.period.indexingPeriod`|Period to run data management duties on the Coordinator including launching compact tasks and performing clean up of unused data. It is recommended to keep this value longer than `druid.manager.segments.pollDuration`.|`PT1800S` (30 mins)|
|`druid.coordinator.kill.pendingSegments.on`|Boolean flag for whether or not the Coordinator clean up old entries in the `pendingSegments` table of metadata store. If set to true, Coordinator will check the created time of most recently complete task. If it doesn't exist, it finds the created time of the earliest running/pending/waiting tasks. Once the created time is found, then for all datasources not in the `killPendingSegmentsSkipList` (see [Dynamic configuration](#dynamic-configurat [...]
|`druid.coordinator.kill.on`|Boolean flag to enable the Coordinator to submit a kill task for unused segments and delete them permanently from the metadata store and deep storage.|false|
|`druid.coordinator.kill.period`| The frequency of sending kill tasks to the indexing service. The value must be greater than or equal to `druid.coordinator.period.indexingPeriod`. Only applies if kill is turned on.|Same as `druid.coordinator.period.indexingPeriod`|
@@ -883,11 +761,6 @@ These Coordinator static configurations can be defined in the `coordinator/runti
|`druid.coordinator.kill.bufferPeriod`|The amount of time that a segment must be unused before it is able to be permanently removed from metadata and deep storage. This can serve as a buffer period to prevent data loss if data ends up being needed after being marked unused.|`P30D`|
|`druid.coordinator.kill.maxSegments`|The number of unused segments to kill per kill task. This number must be greater than 0. This only applies when `druid.coordinator.kill.on=true`.|100|
|`druid.coordinator.kill.maxInterval`|The largest interval, as an [ISO 8601 duration](https://en.wikipedia.org/wiki/ISO_8601#Durations), of segments to delete per kill task. Set to zero, e.g. `PT0S`, for unlimited. This only applies when `druid.coordinator.kill.on=true`.|`P30D`|
-|`druid.coordinator.balancer.strategy`|The [balancing strategy](../design/coordinator.md#balancing-segments-in-a-tier) used by the Coordinator to distribute segments among the Historical servers in a tier. The `cost` strategy distributes segments by minimizing a cost function, `diskNormalized` weights these costs with the disk usage ratios of the servers and `random` distributes segments randomly.|`cost`|
-|`druid.coordinator.loadqueuepeon.http.repeatDelay`|The start and repeat delay (in milliseconds) for the load queue peon, which manages the load/drop queue of segments for any server.|1 minute|
-|`druid.coordinator.loadqueuepeon.http.batchSize`|Number of segment load/drop requests to batch in one HTTP request. Note that it must be smaller than or equal to the `druid.segmentCache.numLoadingThreads` config on Historical service. If this value is not configured, the coordinator uses the value of the `numLoadingThreads` for the respective server. | `druid.segmentCache.numLoadingThreads` |
-|`druid.coordinator.asOverlord.enabled`|Boolean value for whether this Coordinator service should act like an Overlord as well. This configuration allows users to simplify a Druid cluster by not having to deploy any standalone Overlord services. If set to true, then Overlord console is available at `http://coordinator-host:port/console.html` and be sure to set `druid.coordinator.asOverlord.overlordService` also.|false|
-|`druid.coordinator.asOverlord.overlordService`| Required, if `druid.coordinator.asOverlord.enabled` is `true`. This must be same value as `druid.service` on standalone Overlord services and `druid.selectors.indexing.serviceName` on Middle Managers.|NULL|
##### Metadata management
@@ -908,7 +781,7 @@ These Coordinator static configurations can be defined in the `coordinator/runti
|`druid.coordinator.kill.datasource.on`| Boolean value for whether to enable automatic deletion of datasource metadata (Note: datasource metadata only exists for datasource created from supervisor). If set to true, Coordinator will periodically remove datasource metadata of terminated supervisor from the datasource table in metadata storage. | No | True|
|`druid.coordinator.kill.datasource.period`| How often to do automatic deletion of datasource metadata in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Value must be equal to or greater than `druid.coordinator.period.metadataStoreManagementPeriod`. Only applies if `druid.coordinator.kill.datasource.on` is set to true.| No| `P1D`|
|`druid.coordinator.kill.datasource.durationToRetain`| Duration of datasource metadata to be retained from created time in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Only applies if `druid.coordinator.kill.datasource.on` is set to true.| Yes if `druid.coordinator.kill.datasource.on` is set to true.| `P90D`|
-|`druid.coordinator.kill.segmentSchema.on`| Boolean value for whether to enable automatic deletion of unused segment schemas. If set to true, Coordinator will periodically identify segment schemas which are not referenced by any used segment and mark them as unused. At a later point, these unused schemas are deleted. Only applies if [Centralized Datasource schema](#centralized-datasource-schema) feature is enabled. | No | True|
+|`druid.coordinator.kill.segmentSchema.on`| Boolean value for whether to enable automatic deletion of unused segment schemas. If set to true, Coordinator will periodically identify segment schemas which are not referenced by any used segment and mark them as unused. At a later point, these unused schemas are deleted. Only applies if [Centralized Datasource schema](#centralized-datasource-schema-experimental) feature is enabled. | No | True|
|`druid.coordinator.kill.segmentSchema.period`| How often to do automatic deletion of segment schemas in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Value must be equal to or greater than `druid.coordinator.period.metadataStoreManagementPeriod`. Only applies if `druid.coordinator.kill.segmentSchema.on` is set to true.| No| `P1D`|
|`druid.coordinator.kill.segmentSchema.durationToRetain`| Duration of segment schemas to be retained from the time it was marked as unused in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Only applies if `druid.coordinator.kill.segmentSchema.on` is set to true.| Yes, if `druid.coordinator.kill.segmentSchema.on` is set to true.| `P90D`|
@@ -925,6 +798,7 @@ These Coordinator static configurations can be defined in the `coordinator/runti
|--------|-----------|-------|
|`druid.manager.config.pollDuration`|How often the manager polls the config table for updates.|`PT1M`|
|`druid.manager.segments.pollDuration`|The duration between polls the Coordinator does for updates to the set of active segments. Generally defines the amount of lag time it can take for the Coordinator to notice new segments.|`PT1M`|
+|`druid.manager.segments.useIncrementalCache`|(Experimental) Denotes the usage mode of the segment metadata incremental cache. This cache provides a performance improvement over the polling mechanism currently employed by the Coordinator as it retrieves payloads of only updated segments. Possible cache modes are: (a) `never`: Incremental cache is disabled. (b) `always`: Incremental cache is enabled. Service start-up will be blocked until cache has synced with the metadata store at least [...]
|`druid.manager.rules.pollDuration`|The duration between polls the Coordinator does for updates to the set of active rules. Generally defines the amount of lag time it can take for the Coordinator to notice rules.|`PT1M`|
|`druid.manager.rules.defaultRule`|The default rule for the cluster|`_default`|
|`druid.manager.rules.alertThreshold`|The duration after a failed poll upon which an alert should be emitted.|`PT10M`|
@@ -954,6 +828,7 @@ The following table shows the dynamic configuration properties for the Coordinat
|`pauseCoordination`|Boolean flag for whether or not the Coordinator should execute its various duties of coordinating the cluster. Setting this to true essentially pauses all coordination work while allowing the API to remain up. Duties that are paused include all classes that implement the `CoordinatorDuty` interface. Such duties include: segment balancing, segment compaction, submitting kill tasks for unused segments (if enabled), logging of used segments in the cluster, marking of ne [...]
|`replicateAfterLoadTimeout`|Boolean flag for whether or not additional replication is needed for segments that have failed to load due to the expiry of `druid.coordinator.load.timeout`. If this is set to true, the Coordinator will attempt to replicate the failed segment on a different historical server. This helps improve the segment availability if there are a few slow Historicals in the cluster. However, the slow Historical may still load the segment later and the Coordinator may issu [...]
|`turboLoadingNodes`| Experimental. List of Historical servers to place in turbo loading mode. These servers use a larger thread-pool to load segments faster but at the cost of query performance. For servers specified in `turboLoadingNodes`, `druid.coordinator.loadqueuepeon.http.batchSize` is ignored and the coordinator uses the value of the respective `numLoadingThreads` instead.<br/>Please use this config with caution. All servers should eventually be removed from this list once the se [...]
+|`cloneServers`| Experimental. Map from target Historical server to source Historical server which should be cloned by the target. The target Historical does not participate in regular segment assignment or balancing. Instead, the Coordinator mirrors any segment assignment made to the source Historical onto the target Historical, so that the target becomes an exact copy of the source. Segments on the target Historical do not count towards replica counts either. If the source disappears, [...]
##### Smart segment loading
@@ -1000,11 +875,11 @@ You can configure automatic compaction through the following properties:
|Property|Description|Required|
|--------|-----------|--------|
|`dataSource`|The datasource name to be compacted.|yes|
-|`taskPriority`|[Priority](../ingestion/tasks.md#priority) of compaction task.|no (default = 25)|
+|`taskPriority`|[Priority](../ingestion/tasks.md#lock-priority) of compaction task.|no (default = 25)|
|`inputSegmentSizeBytes`|Maximum number of total segment bytes processed per compaction task. Since a time chunk must be processed in its entirety, if the segments for a particular time chunk have a total size in bytes greater than this parameter, compaction will not run for that time chunk.|no (default = 100,000,000,000,000 i.e. 100TB)|
|`skipOffsetFromLatest`|The offset for searching segments to be compacted in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Strongly recommended to set for realtime datasources. See [Data handling with compaction](../data-management/compaction.md#data-handling-with-compaction).|no (default = "P1D")|
|`tuningConfig`|Tuning config for compaction tasks. See below [Automatic compaction tuningConfig](#automatic-compaction-tuningconfig).|no|
-|`taskContext`|[Task context](../ingestion/tasks.md#context) for compaction tasks.|no|
+|`taskContext`|[Task context](../ingestion/tasks.md#context-parameters) for compaction tasks.|no|
|`granularitySpec`|Custom `granularitySpec`. See [Automatic compaction granularitySpec](#automatic-compaction-granularityspec).|no|
|`dimensionsSpec`|Custom `dimensionsSpec`. See [Automatic compaction dimensionsSpec](#automatic-compaction-dimensionsspec).|no|
|`transformSpec`|Custom `transformSpec`. See [Automatic compaction transformSpec](#automatic-compaction-transformspec).|no|
@@ -1110,9 +985,10 @@ These Overlord static configurations can be defined in the `overlord/runtime.pro
|`druid.indexer.runner.type`|Indicates whether tasks should be run locally using `local` or in a distributed environment using `remote`. The recommended option is `httpRemote`, which is similar to `remote` but uses HTTP to interact with Middle Managers instead of ZooKeeper.|`httpRemote`|
|`druid.indexer.storage.type`|Indicates whether incoming tasks should be stored locally (in heap) or in metadata storage. One of `local` or `metadata`. `local` is mainly for internal testing while `metadata` is recommended in production because storing incoming tasks in metadata storage allows for tasks to be resumed if the Overlord should fail.|`local`|
|`druid.indexer.storage.recentlyFinishedThreshold`|Duration of time to store task results. Default is 24 hours. If you have hundreds of tasks running in a day, consider increasing this threshold.|`PT24H`|
-|`druid.indexer.tasklock.forceTimeChunkLock`|**Setting this to false is still experimental**<br/> If set, all tasks are enforced to use time chunk lock. If not set, each task automatically chooses a lock type to use. This configuration can be overwritten by setting `forceTimeChunkLock` in the [task context](../ingestion/tasks.md#context). See [Task Locking & Priority](../ingestion/tasks.md#context) for more details about locking in tasks.|true|
+|`druid.indexer.tasklock.forceTimeChunkLock`|**Setting this to false is still experimental**<br/> If set, all tasks are enforced to use time chunk lock. If not set, each task automatically chooses a lock type to use. This configuration can be overwritten by setting `forceTimeChunkLock` in the [task context](../ingestion/tasks.md#context-parameters). See [Task lock system](../ingestion/tasks.md#task-lock-system) for more details about locking in tasks.|true|
|`druid.indexer.tasklock.batchSegmentAllocation`| If set to true, Druid performs segment allocate actions in batches to improve throughput and reduce the average `task/action/run/time`. See [batching `segmentAllocate` actions](../ingestion/tasks.md#batching-segmentallocate-actions) for details.|true|
|`druid.indexer.tasklock.batchAllocationWaitTime`|Number of milliseconds after Druid adds the first segment allocate action to a batch, until it executes the batch. Allows the batch to add more requests and improve the average segment allocation run time. This configuration takes effect only if `batchSegmentAllocation` is enabled.|0|
+|`druid.indexer.tasklock.batchAllocationNumThreads`|Number of worker threads to use for batch segment allocation. This represents the maximum number of allocation batches that can be processed in parallel for distinct datasources. Batches for a single datasource are always processed sequentially. This configuration takes effect only if `batchSegmentAllocation` is enabled.|5|
|`druid.indexer.task.default.context`|Default task context that is applied to all tasks submitted to the Overlord. Any default in this config does not override neither the context values the user provides nor `druid.indexer.tasklock.forceTimeChunkLock`.|empty context|
|`druid.indexer.queue.maxSize`|Maximum number of active tasks at one time.|`Integer.MAX_VALUE`|
|`druid.indexer.queue.startDelay`|Sleep this long before starting Overlord queue management. This can be useful to give a cluster time to re-orient itself (for example, after a widespread network issue).|`PT1M`|
@@ -1169,14 +1045,24 @@ If autoscaling is enabled, you can set these additional configs:
The `druid.supervisor.idleConfig.*` specification in the Overlord runtime properties defines the default behavior for the entire cluster. See [Idle Configuration in Kafka Supervisor IOConfig](../ingestion/kinesis-ingestion.md#io-configuration) to override it for an individual supervisor.
-##### Segment metadata cache (EXPERIMENTAL)
+##### Segment metadata cache (Experimental)
The following properties pertain to segment metadata caching on the Overlord that may be used to speed up segment allocation and other metadata operations.
|Property|Description|Default|
|--------|-----------|-------|
-|`druid.manager.segments.useCache`|Denotes the usage mode of the segment metadata cache. Possible modes are: (a) `never`: Cache is disabled. (b) `always`: Reads are always done from the cache. Service start-up will be blocked until cache has synced with the metadata store at least once. Transactions will block until cache has synced with the metadata store at least once after becoming leader. (c) `ifSynced`: Reads are done from the cache only if it has already synced with the metadata st [...]
-|`druid.manager.segments.pollDuration`|Duration (in ISO 8601 format) between successive syncs of the cache with the metadata store. This property is used only when `druid.manager.segments.useCache` is set to `always` or `ifSynced`.|`PT1M` (1 minute)|
+|`druid.manager.segments.useIncrementalCache`|Denotes the usage mode of the segment metadata incremental cache. Possible modes are: (a) `never`: Cache is disabled. (b) `always`: Reads are always done from the cache. Service start-up will be blocked until cache has synced with the metadata store at least once. Transactions will block until cache has synced with the metadata store at least once after becoming leader. (c) `ifSynced`: Reads are done from the cache only if it has already sync [...]
+|`druid.manager.segments.pollDuration`|Duration (in ISO 8601 format) between successive syncs of the cache with the metadata store. This property is used only when `druid.manager.segments.useIncrementalCache` is set to `always` or `ifSynced`.|`PT1M` (1 minute)|
+
+##### Auto-kill unused segments (Experimental)
+
+These configs pertain to the new embedded mode of running [kill tasks on the Overlord](../data-management/delete.md#auto-kill-data-on-the-overlord-experimental).
+None of the configs that apply to [auto-kill performed by the Coordinator](../data-management/delete.md#auto-kill-data-using-coordinator-duties) are used by this feature.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.manager.segments.killUnused.enabled`|Boolean flag to enable auto-kill of eligible unused segments on the Overlord. This feature can be used only when [segment metadata caching](#segment-metadata-cache-experimental) is enabled on the Overlord and MUST NOT be enabled if `druid.coordinator.kill.on` is already set to `true` on the Coordinator.|`true`|
+|`druid.manager.segments.killUnused.bufferPeriod`|Period after which a segment marked as unused becomes eligible for auto-kill on the Overlord. This config is effective only if `druid.manager.segments.killUnused.enabled` is set to `true`.|`P30D` (30 days)|
#### Overlord dynamic configuration
@@ -1465,7 +1351,7 @@ Middle Managers pass their configurations down to their child peons. The Middle
|`druid.worker.baseTaskDirs`|List of base temporary working directories, one of which is assigned per task in a round-robin fashion. This property can be used to allow usage of multiple disks for indexing. This property is recommended in place of and takes precedence over `${druid.indexer.task.baseTaskDir}`. If this configuration is not set, `${druid.indexer.task.baseTaskDir}` is used. For example, `druid.worker.baseTaskDirs=[\"PATH1\",\"PATH2\",...]`.|null|
|`druid.worker.baseTaskDirSize`|The total amount of bytes that can be used by tasks on any single task dir. This value is treated symmetrically across all directories, that is, if this is 500 GB and there are 3 `baseTaskDirs`, then each of those task directories is assumed to allow for 500 GB to be used and a total of 1.5 TB will potentially be available across all tasks. The actual amount of memory assigned to each task is discussed in [Configuring task storage sizes](../ingestion/tasks [...]
|`druid.worker.category`|A string to name the category that the Middle Manager node belongs to.|`_default_worker_category`|
-|`druid.indexer.fork.property.druid.centralizedDatasourceSchema.enabled`| This config should be set when [Centralized Datasource Schema](#centralized-datasource-schema) feature is enabled. |false|
+|`druid.indexer.fork.property.druid.centralizedDatasourceSchema.enabled`| This config should be set when [Centralized Datasource Schema](#centralized-datasource-schema-experimental) feature is enabled. |false|
#### Peon processing
@@ -1528,6 +1414,7 @@ Additional Peon configs include:
|`druid.indexer.task.ignoreTimestampSpecForDruidInputSource`|If true, tasks using the [Druid input source](../ingestion/input-sources.md) will ignore the provided timestampSpec, and will use the `__time` column of the input datasource. This option is provided for compatibility with ingestion specs written before Druid 0.22.0.|false|
|`druid.indexer.task.storeEmptyColumns`|Boolean value for whether or not to store empty columns during ingestion. When set to true, Druid stores every column specified in the [`dimensionsSpec`](../ingestion/ingestion-spec.md#dimensionsspec). If you use the string-based schemaless ingestion and don't specify any dimensions to ingest, you must also set [`includeAllDimensions`](../ingestion/ingestion-spec.md#dimensionsspec) for Druid to store empty columns.<br/><br/>If you set `storeEmptyCo [...]
|`druid.indexer.task.tmpStorageBytesPerTask`|Maximum number of bytes per task to be used to store temporary files on disk. This config is generally intended for internal usage. Attempts to set it are very likely to be overwritten by the TaskRunner that executes the task, so be sure of what you expect to happen before directly adjusting this configuration parameter. The config is documented here primarily to provide an understanding of what it means if/when someone sees that it has been s [...]
+|`druid.indexer.task.allowHadoopTaskExecution`|Conditional dictating if the cluster allows `index_hadoop` tasks to be executed. `index_hadoop` is deprecated, and defaulting to false will force cluster operators to acknowledge the deprecation and consciously opt in to using index_hadoop with the understanding that it will be removed in the future.|false|
|`druid.indexer.server.maxChatRequests`|Maximum number of concurrent requests served by a task's chat handler. Set to 0 to disable limiting.|0|
If the Peon is running in remote mode, there must be an Overlord up and running. Peons in remote mode can set the following configurations:
@@ -1536,7 +1423,7 @@ If the Peon is running in remote mode, there must be an Overlord up and running.
|--------|-----------|-------|
|`druid.peon.taskActionClient.retry.minWait`|The minimum retry time to communicate with Overlord.|`PT5S`|
|`druid.peon.taskActionClient.retry.maxWait`|The maximum retry time to communicate with Overlord.|`PT1M`|
-|`druid.peon.taskActionClient.retry.maxRetryCount`|The maximum number of retries to communicate with Overlord.|60|
+|`druid.peon.taskActionClient.retry.maxRetryCount`|The maximum number of retries to communicate with Overlord.|13 (about 10 minutes of retrying)|
##### SegmentWriteOutMediumFactory
@@ -1589,7 +1476,7 @@ For most types of tasks, `SegmentWriteOutMediumFactory` can be configured per-ta
|`druid.indexer.task.storeEmptyColumns`|Boolean value for whether or not to store empty columns during ingestion. When set to true, Druid stores every column specified in the [`dimensionsSpec`](../ingestion/ingestion-spec.md#dimensionsspec). <br/><br/>If you set `storeEmptyColumns` to false, Druid SQL queries referencing empty columns will fail. If you intend to leave `storeEmptyColumns` disabled, you should either ingest placeholder data for empty columns or else not query on empty colu [...]
|`druid.peon.taskActionClient.retry.minWait`|The minimum retry time to communicate with Overlord.|`PT5S`|
|`druid.peon.taskActionClient.retry.maxWait`|The maximum retry time to communicate with Overlord.|`PT1M`|
-|`druid.peon.taskActionClient.retry.maxRetryCount`|The maximum number of retries to communicate with Overlord.|60|
+|`druid.peon.taskActionClient.retry.maxRetryCount`|The maximum number of retries to communicate with Overlord.|13 (about 10 minutes of retrying)|
#### Indexer concurrent requests
@@ -1781,8 +1668,10 @@ These Broker configurations can be defined in the `broker/runtime.properties` fi
|Property|Possible Values|Description|Default|
|--------|---------------|-----------|-------|
|`druid.broker.balancer.type`|`random`, `connectionCount`|Determines how the broker balances connections to Historical processes. `random` choose randomly, `connectionCount` picks the process with the fewest number of active connections to|`random`|
-|`druid.broker.select.tier`|`highestPriority`, `lowestPriority`, `custom`|If segments are cross-replicated across tiers in a cluster, you can tell the broker to prefer to select segments in a tier with a certain priority.|`highestPriority`|
+|`druid.broker.select.tier`|`highestPriority`, `lowestPriority`, `custom`, `preferred`|If segments are cross-replicated across tiers in a cluster, you can tell the broker to prefer to select segments in a tier with a certain priority.|`highestPriority`|
|`druid.broker.select.tier.custom.priorities`|An array of integer priorities, such as `[-1, 0, 1, 2]`|Select servers in tiers with a custom priority list.|The config only has effect if `druid.broker.select.tier` is set to `custom`. If `druid.broker.select.tier` is set to `custom` but this config is not specified, the effect is the same as `druid.broker.select.tier` set to `highestPriority`. Any of the integers in this config can be ignored if there's no corresponding tiers with such prio [...]
+|`druid.broker.select.tier.preferred.tier`| The preferred tier name. E.g., `_default_tier` | A non-empty value that specifies the preferred tier in which historical servers will be picked up for queries. If there are not enough historical servers from the preferred tier, servers from other tiers (if there are any) will be selected. This config only has effect if `druid.broker.select.tier` is set to `preferred` | null |
+|`druid.broker.select.tier.preferred.priority`| `highest`, `lowest` | If there are multiple candidates in a preferred tier, specifies the priority to pick up candidates. By default, the higher priority a historical, the higher chances it will be picked up. This config only has effect if `druid.broker.select.tier` is set to `preferred`| `highest` |
##### Query prioritization and laning
@@ -1974,6 +1863,7 @@ The Druid SQL server is configured through the following properties on the Broke
|`druid.sql.planner.useApproximateCountDistinct`|Whether to use an approximate cardinality algorithm for `COUNT(DISTINCT foo)`.|true|
|`druid.sql.planner.useGroupingSetForExactDistinct`|Only relevant when `useApproximateCountDistinct` is disabled. If set to true, exact distinct queries are re-written using grouping sets. Otherwise, exact distinct queries are re-written using joins. This should be set to true for group by query with multiple exact distinct aggregations. This flag can be overridden per query.|false|
|`druid.sql.planner.useApproximateTopN`|Whether to use approximate [TopN queries](../querying/topnquery.md) when a SQL query could be expressed as such. If false, exact [GroupBy queries](../querying/groupbyquery.md) will be used instead.|true|
+|`druid.sql.planner.useLexicographicTopN`|Whether to use [TopN queries](../querying/topnquery.md) with lexicographic dimension ordering. If false, [GroupBy queries](../querying/groupbyquery.md) will be used instead for lexicographic ordering. When both this and `useApproximateTopN` are false, TopN queries are never used.|false|
|`druid.sql.planner.requireTimeCondition`|Whether to require SQL to have filter conditions on `__time` column so that all generated native queries will have user specified intervals. If true, all queries without filter condition on `__time` column will fail|false|
|`druid.sql.planner.sqlTimeZone`|Sets the default time zone for the server, which will affect how time functions and timestamp literals behave. Should be a time zone name like "America/Los_Angeles" or offset like "-08:00".|UTC|
|`druid.sql.planner.metadataSegmentCacheEnable`|Whether to keep a cache of published segments in broker. If true, broker polls coordinator in background to get segments from metadata store and maintains a local cache. If false, coordinator's REST API will be invoked when broker needs published segments info.|false|
@@ -2022,6 +1912,164 @@ See [cache configuration](#cache-configuration) for how to configure cache setti
|`druid.broker.segment.watchRealtimeTasks`|Boolean|The Broker watches segment announcements from processes that serve segments to build a cache to relate each process to the segments it serves. When `watchRealtimeTasks` is true, the Broker watches for segment announcements from both Historicals and realtime processes. To configure a broker to exclude segments served by realtime processes, set `watchRealtimeTasks` to false. |true|
|`druid.broker.segment.awaitInitializationOnStart`|Boolean|Whether the Broker will wait for its view of segments to fully initialize before starting up. If set to 'true', the Broker's HTTP server will not start up, and the Broker will not announce itself as available, until the server view is initialized. See also `druid.sql.planner.awaitInitializationOnStart`, a related setting.|true|
+## Metrics monitors
+
+You can configure Druid services to emit [metrics](../operations/metrics.md) regularly from a number of [monitors](#metrics-monitors-for-each-service) via [emitters](#metrics-emitters). The following table lists general configurations for metrics:
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.monitoring.emissionPeriod`| Frequency that Druid emits metrics.|`PT1M`|
+|[`druid.monitoring.monitors`](#metrics-monitors-for-each-service)|Sets list of Druid monitors used by a service.|none (no monitors)|
+|[`druid.emitter`](#metrics-emitters)|Setting this value initializes one of the emitter modules.|`noop` (metric emission disabled by default)|
+
+### Metrics monitors for each service
+
+Metric monitoring is an essential part of Druid operations. Druid includes
+
+:::caution
+
+The `runtime.properties` file for each service overrides the common configuration file (`common.runtime.properties`). They are not additive. This means that if you add any monitors to a specific service, that service only has the monitors specified in its `runtime.properties` file even if there are additional ones listed in the common file.
+
+:::
+
+
+The following table lists the monitors that are available and the services you could configure the monitor for:
+
+|Name|Description|Service|
+|----|-----------|-------|
+|`org.apache.druid.client.cache.CacheMonitor`|Emits metrics (to logs) about the segment results cache for Historical and Broker services. Reports typical cache statistics include hits, misses, rates, and size (bytes and number of entries), as well as timeouts and and errors.|Broker, Historical, Indexer, Peon|
+|`org.apache.druid.java.util.metrics.OshiSysMonitor`|Reports on various system activities and statuses using the [OSHI](https://github.com/oshi/oshi), a JNA-based (native) Operating System and Hardware Information library for Java.|Any|
+|`org.apache.druid.java.util.metrics.JvmMonitor`|Reports various JVM-related statistics.|Any|
+|`org.apache.druid.java.util.metrics.JvmCpuMonitor`|Reports statistics of CPU consumption by the JVM.|Any|
+|`org.apache.druid.java.util.metrics.CpuAcctDeltaMonitor`|Reports consumed CPU as per the cpuacct cgroup.|Any|
+|`org.apache.druid.java.util.metrics.JvmThreadsMonitor`|Reports Thread statistics in the JVM, like numbers of total, daemon, started, died threads.|Any|
+|`org.apache.druid.java.util.metrics.CgroupCpuMonitor`|Reports CPU shares and quotas as per the `cpu` cgroup.|Any|
+|`org.apache.druid.java.util.metrics.CgroupCpuSetMonitor`|Reports CPU core/HT and memory node allocations as per the `cpuset` cgroup.|Any|
+|`org.apache.druid.java.util.metrics.CgroupDiskMonitor`|Reports disk statistic as per the blkio cgroup.|Any|
+|`org.apache.druid.java.util.metrics.CgroupMemoryMonitor`|Reports memory statistic as per the memory cgroup.|Any|
+|`org.apache.druid.java.util.metrics.CgroupV2CpuMonitor`| **EXPERIMENTAL** Reports CPU usage from `cpu.stat` file. Only applicable to `cgroupv2`.|Any|
+|`org.apache.druid.java.util.metrics.CgroupV2DiskMonitor`| **EXPERIMENTAL** Reports disk usage from `io.stat` file. Only applicable to `cgroupv2`.|Any|
+|`org.apache.druid.java.util.metrics.CgroupV2MemoryMonitor`| **EXPERIMENTAL** Reports memory usage from `memory.current` and `memory.max` files. Only applicable to `cgroupv2`.|Any|
+|`org.apache.druid.server.metrics.HistoricalMetricsMonitor`|Reports statistics on Historical services.|Historical|
+|`org.apache.druid.server.metrics.SegmentStatsMonitor` | **EXPERIMENTAL** Reports statistics about segments on Historical services. Not to be used when lazy loading is configured.|Historical|
+|`org.apache.druid.server.metrics.QueryCountStatsMonitor`|Reports how many queries have been successful/failed/interrupted.|Broker, Historical, Router, Indexer, Peon|
+|`org.apache.druid.server.metrics.SubqueryCountStatsMonitor`|Reports how many subqueries have been materialized as rows or bytes and various other statistics related to the subquery execution|Broker|
+|`org.apache.druid.server.emitter.HttpEmittingMonitor`|Reports internal metrics of `http` or `parametrized` emitter (see below). Must not be used with another emitter type. See the description of the metrics here: https://github.com/apache/druid/pull/4973.|Any|
+|`org.apache.druid.server.metrics.TaskCountStatsMonitor`|Reports how many ingestion tasks are currently running/pending/waiting and also the number of successful/failed tasks per emission period.|Overlord|
+|`org.apache.druid.server.metrics.TaskSlotCountStatsMonitor`|Reports metrics about task slot usage per emission period.|Overlord|
+|`org.apache.druid.server.metrics.WorkerTaskCountStatsMonitor`|Reports how many ingestion tasks are currently running/pending/waiting, the number of successful/failed tasks, and metrics about task slot usage for the reporting worker, per emission period. |MiddleManager, Indexer|
+|`org.apache.druid.server.metrics.ServiceStatusMonitor`|Reports a heartbeat for the service.|Any|
+|`org.apache.druid.server.metrics.GroupByStatsMonitor`|Report metrics for groupBy queries like disk and merge buffer utilization. |Broker, Historical, Indexer, Peon|
+
+For example, if you only wanted monitors on all services for system and JVM information, you'd add the following to `common.runtime.properties`:
+
+```properties
+druid.monitoring.monitors=["org.apache.druid.java.util.metrics.OshiSysMonitor","org.apache.druid.java.util.metrics.JvmMonitor"]
+```
+
+All the services in your Druid deployment would have these two monitors.
+
+If you want any service specific monitors though, you need to add all the monitors you want to run for that service to the service's `runtime.properties` file even if they are listed in the common file. The service specific properties take precedence.
+
+The following example adds the `TaskCountStatsMonitor` and `TaskSlotCountStatsMonitor` as well as the `OshiSysMonitor` and `JvmMonitor` from the previous example to the Overlord service (`coordinator-overlord/runtime.properties`):
+
+```properties
+druid.monitoring.monitors=["org.apache.druid.server.metrics.TaskCountStatsMonitor", "org.apache.druid.server.metrics.TaskSlotCountStatsMonitor", "org.apache.druid.java.util.metrics.OshiSysMonitor","org.apache.druid.java.util.metrics.JvmMonitor"]
+```
+
+If you don't include `OshiSysMonitor` and `JvmMonitor` in the Overlord's `runtime.properties` file, the monitors don't get loaded onto the Overlord despite being specified in the common file.
+
+### Metrics emitters
+
+There are several emitters available:
+
+* `noop` (default) disables metric emission.
+* [`logging`](#logging-emitter-module) emits logs using Log4j2.
+* [`http`](#http-emitter-module) sends `POST` requests of JSON events.
+* [`parametrized`](#parametrized-http-emitter-module) operates like the `http` emitter but fine-tunes the recipient URL based on the event feed.
+* [`composing`](#composing-emitter-module) initializes multiple emitter modules.
+* [`graphite`](#graphite-emitter) emits metrics to a [Graphite](https://graphiteapp.org/) Carbon service.
+* [`switching`](#switching-emitter) initializes and emits to multiple emitter modules based on the event feed.
+
+#### Logging emitter module
+
+The use this emitter module, set `druid.emitter=logging`. The `logging` emitter uses a Log4j2 logger named
+`druid.emitter.logging.loggerClass` to emit events. Each event is logged as a single `json` object with a
+[Marker](https://logging.apache.org/log4j/2.x/manual/markers.html) as the feed of the event. Users may wish to edit the
+log4j config to route these logs to different sources based on the feed of the event.
+
+|Property|Description| Default|
+|--------|-----------|--------|
+|`druid.emitter.logging.loggerClass`|The class used for logging.|`org.apache.druid.java.util.emitter.core.LoggingEmitter`|
+|`druid.emitter.logging.logLevel`|Choices: debug, info, warn, error. The log level at which message are logged.|info|
+
+#### HTTP emitter module
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.emitter.http.flushMillis`|How often the internal message buffer is flushed (data is sent).|60000|
+|`druid.emitter.http.flushCount`|How many messages the internal message buffer can hold before flushing (sending).|500|
+|`druid.emitter.http.basicAuthentication`|[Password Provider](../operations/password-provider.md) for providing login and password for authentication in `"login:password"` form. For example, `druid.emitter.http.basicAuthentication=admin:adminpassword` uses Default Password Provider which allows plain text passwords.|not specified = no authentication|
+|`druid.emitter.http.flushTimeOut`|The timeout after which an event should be sent to the endpoint, even if internal buffers are not filled, in milliseconds.|not specified = no timeout|
+|`druid.emitter.http.batchingStrategy`|The strategy of how the batch is formatted. "ARRAY" means `[event1,event2]`, "NEWLINES" means `event1\nevent2`, ONLY_EVENTS means `event1event2`.|ARRAY|
+|`druid.emitter.http.maxBatchSize`|The maximum batch size, in bytes.|the minimum of (10% of JVM heap size divided by 2) or (5242880 (i. e. 5 MiB))|
+|`druid.emitter.http.batchQueueSizeLimit`|The maximum number of batches in emitter queue, if there are problems with emitting.|the maximum of (2) or (10% of the JVM heap size divided by 5MiB)|
+|`druid.emitter.http.minHttpTimeoutMillis`|If the speed of filling batches imposes timeout smaller than that, not even trying to send batch to endpoint, because it will likely fail, not being able to send the data that fast. Configure this depending based on emitter/successfulSending/minTimeMs metric. Reasonable values are 10ms..100ms.|0|
+|`druid.emitter.http.recipientBaseUrl`|The base URL to emit messages to. Druid will POST JSON to be consumed at the HTTP endpoint specified by this property.|none, required config|
+
+#### HTTP emitter module TLS overrides
+
+By default, when sending events to a TLS-enabled receiver, the HTTP Emitter uses an SSLContext obtained from the service described at [Druid's internal communication over TLS](../operations/tls-support.md), that is the same SSLContext that would be used for internal communications between Druid services.
+
+In some use cases it may be desirable to have the HTTP Emitter use its own separate truststore configuration. For example, there may be organizational policies that prevent the TLS-enabled metrics receiver's certificate from being added to the same truststore used by Druid's internal HTTP client.
+
+The following properties allow the HTTP Emitter to use its own truststore configuration when building its SSLContext.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.emitter.http.ssl.useDefaultJavaContext`|If set to true, the HttpEmitter will use `SSLContext.getDefault()`, the default Java SSLContext, and all other properties below are ignored.|false|
+|`druid.emitter.http.ssl.trustStorePath`|The file path or URL of the TLS/SSL Key store where trusted root certificates are stored. If this is unspecified, the HTTP Emitter will use the same SSLContext as Druid's internal HTTP client, as described in the beginning of this section, and all other properties below are ignored.|null|
+|`druid.emitter.http.ssl.trustStoreType`|The type of the key store where trusted root certificates are stored.|`java.security.KeyStore.getDefaultType()`|
+|`druid.emitter.http.ssl.trustStoreAlgorithm`|Algorithm to be used by TrustManager to validate certificate chains|`javax.net.ssl.TrustManagerFactory.getDefaultAlgorithm()`|
+|`druid.emitter.http.ssl.trustStorePassword`|The [Password Provider](../operations/password-provider.md) or String password for the Trust Store.|none|
+|`druid.emitter.http.ssl.protocol`|TLS protocol to use.|"TLSv1.2"|
+
+#### Parametrized HTTP emitter module
+
+The parametrized emitter takes the same configs as the [`http` emitter](#http-emitter-module) using the prefix `druid.emitter.parametrized.httpEmitting.`.
+For example:
+
+* `druid.emitter.parametrized.httpEmitting.flushMillis`
+* `druid.emitter.parametrized.httpEmitting.flushCount`
+* `druid.emitter.parametrized.httpEmitting.ssl.trustStorePath`
+
+Do not specify `recipientBaseUrl` with the parametrized emitter.
+Instead use `recipientBaseUrlPattern` described in the table below.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.emitter.parametrized.recipientBaseUrlPattern`|The URL pattern to send an event to, based on the event's feed. For example, `http://foo.bar/{feed}`, that will send event to `http://foo.bar/metrics` if the event's feed is "metrics".|none, required config|
+
+#### Composing emitter module
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.emitter.composing.emitters`|List of emitter modules to load, such as ["logging","http"].|[]|
+
+#### Graphite emitter
+
+To use graphite as emitter set `druid.emitter=graphite`. For configuration details, see [Graphite emitter](../development/extensions-contrib/graphite.md) for the Graphite emitter Druid extension.
+
+#### Switching emitter
+
+To use switching as emitter set `druid.emitter=switching`.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.emitter.switching.emitters`|JSON map of feed to list of emitter modules that will be used for the mapped feed, such as `{"metrics":["http"], "alerts":["logging"]}`|{}|
+|`druid.emitter.switching.defaultEmitters`|JSON list of emitter modules to load that will be used if there is no emitter specifically designated for that event's feed, such as `["logging","http"]`.|[]|
+
+
## Cache configuration
This section describes caching configuration that is common to Broker, Historical, and Middle Manager/Peon processes.
diff --git a/docs/latest/configuration/logging.md b/docs/34.0.0/configuration/logging.md
similarity index 99%
copy from docs/latest/configuration/logging.md
copy to docs/34.0.0/configuration/logging.md
index 36b3492bb..d740f38b0 100644
--- a/docs/latest/configuration/logging.md
+++ b/docs/34.0.0/configuration/logging.md
@@ -24,7 +24,7 @@ title: "Logging"
Apache Druid services emit logs that to help you debug.
-The same services also emit periodic [metrics](../configuration/index.md#enabling-metrics) about their state.
+The same services also emit periodic [metrics](../configuration/index.md#metrics-monitors) about their state.
To disable metric info logs set the following runtime property: `-Ddruid.emitter.logging.logLevel=debug`.
Druid uses [log4j2](http://logging.apache.org/log4j/2.x/) for logging.
@@ -106,7 +106,7 @@ Peons always output logs to standard output. Middle Managers redirect task logs
[long-term storage](index.md#log-long-term-storage).
:::info
- NOTE:
+
Druid shares the log4j configuration file among all services, including task peon processes.
However, you must define a console appender in the logger for your peon processes.
If you don't define a console appender, Druid creates and configures a new console appender
diff --git a/docs/latest/data-management/automatic-compaction.md b/docs/34.0.0/data-management/automatic-compaction.md
similarity index 99%
copy from docs/latest/data-management/automatic-compaction.md
copy to docs/34.0.0/data-management/automatic-compaction.md
index 7d16eae26..1a0803baf 100644
--- a/docs/latest/data-management/automatic-compaction.md
+++ b/docs/34.0.0/data-management/automatic-compaction.md
@@ -52,7 +52,7 @@ The automatic compaction system uses the following syntax:
}
```
-:::info Experimental
+:::info[Experimental]
The MSQ task engine is available as a compaction engine when you run automatic compaction as a compaction supervisor. For more information, see [Auto-compaction using compaction supervisors](#auto-compaction-using-compaction-supervisors).
@@ -85,7 +85,7 @@ For more details on each of the specs in an auto-compaction configuration, see [
## Auto-compaction using Coordinator duties
-You can control how often the Coordinator checks to see if auto-compaction is needed. The Coordinator [indexing period](../configuration/index.md#coordinator-operation), `druid.coordinator.period.indexingPeriod`, controls the frequency of compaction tasks.
+You can control how often the Coordinator checks to see if auto-compaction is needed. The Coordinator [indexing period](../configuration/index.md#data-management), `druid.coordinator.period.indexingPeriod`, controls the frequency of compaction tasks.
The default indexing period is 30 minutes, meaning that the Coordinator first checks for segments to compact at most 30 minutes from when auto-compaction is enabled.
This time period also affects other Coordinator duties such as cleanup of unused segments and stale pending segments.
To configure the auto-compaction time period without interfering with `indexingPeriod`, see [Set frequency of compaction runs](#change-compaction-frequency).
@@ -230,7 +230,7 @@ The following auto-compaction configuration compacts updates the `wikipedia` seg
## Auto-compaction using compaction supervisors
-:::info Experimental
+:::info[Experimental]
Compaction supervisors are experimental. For production use, we recommend [auto-compaction using Coordinator duties](#auto-compaction-using-coordinator-duties).
:::
diff --git a/docs/34.0.0/data-management/compaction.md b/docs/34.0.0/data-management/compaction.md
new file mode 100644
index 000000000..51bf7ee86
--- /dev/null
+++ b/docs/34.0.0/data-management/compaction.md
@@ -0,0 +1,113 @@
+---
+id: compaction
+title: "Compaction"
+description: "Defines compaction and automatic compaction (auto-compaction or autocompaction) for segment optimization. Use cases and strategies for compaction. Describes compaction task configuration."
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+Query performance in Apache Druid depends on optimally sized segments. Compaction is one strategy you can use to optimize segment size for your Druid database. Compaction tasks read an existing set of segments for a given time interval and combine the data into a new "compacted" set of segments. In some cases the compacted segments are larger, but there are fewer of them. In other cases the compacted segments may be smaller. Compaction tends to increase performance because optimized segm [...]
+
+## Compaction guidelines
+
+There are several cases to consider compaction for segment optimization:
+
+- With streaming ingestion, data can arrive out of chronological order creating many small segments.
+- If you append data using `appendToExisting` for [native batch](../ingestion/native-batch.md) ingestion creating suboptimal segments.
+- When you use `index_parallel` for parallel batch indexing and the parallel ingestion tasks create many small segments.
+- When a misconfigured ingestion task creates oversized segments.
+
+By default, compaction does not modify the underlying data of the segments. However, there are cases when you may want to modify data during compaction to improve query performance:
+
+- If, after ingestion, you realize that data for the time interval is sparse, you can use compaction to increase the segment granularity.
+- If you don't need fine-grained granularity for older data, you can use compaction to change older segments to a coarser query granularity. For example, from `minute` to `hour` or `hour` to `day`. This reduces the storage space required for older data.
+- You can change the dimension order to improve sorting and reduce segment size.
+- You can remove unused columns in compaction or implement an aggregation metric for older data.
+- You can change segment rollup from dynamic partitioning with best-effort rollup to hash or range partitioning with perfect rollup. For more information on rollup, see [perfect vs best-effort rollup](../ingestion/rollup.md#perfect-rollup-vs-best-effort-rollup).
+
+Compaction does not improve performance in all situations. For example, if you rewrite your data with each ingestion task, you don't need to use compaction. See [Segment optimization](../operations/segment-optimization.md) for additional guidance to determine if compaction will help in your environment.
+
+## Ways to run compaction
+
+Automatic compaction, also called auto-compaction, works in most use cases and should be your first option.
+
+The Coordinator uses its [segment search policy](../design/coordinator.md#segment-search-policy-in-automatic-compaction) to periodically identify segments for compaction starting from newest to oldest. When the Coordinator discovers segments that have not been compacted or segments that were compacted with a different or changed spec, it submits compaction tasks for the time interval covering those segments.
+
+To learn more, see [Automatic compaction](../data-management/automatic-compaction.md).
+
+In cases where you require more control over compaction, you can manually submit compaction tasks. For example:
+
+- Automatic compaction is running into the limit of task slots available to it, so tasks are waiting for previous automatic compaction tasks to complete. Manual compaction can use all available task slots, therefore you can complete compaction more quickly by submitting more concurrent tasks for more intervals.
+- You want to force compaction for a specific time range or you want to compact data out of chronological order.
+
+See [Setting up a manual compaction task](./manual-compaction.md#setting-up-manual-compaction) for more about manual compaction tasks.
+
+## Data handling with compaction
+
+During compaction, Druid overwrites the original set of segments with the compacted set. Druid also locks the segments for the time interval being compacted to ensure data consistency. By default, compaction tasks do not modify the underlying data. You can configure the compaction task to change the query granularity or add or remove dimensions in the compaction task. This means that the only changes to query results should be the result of intentional, not automatic, changes.
+
+You can set `dropExisting` in `ioConfig` to "true" in the compaction task to configure Druid to replace all existing segments fully contained by the interval. See the suggestion for reindexing with finer granularity under [Implementation considerations](../ingestion/native-batch.md#implementation-considerations) for an example.
+:::info
+ WARNING: `dropExisting` in `ioConfig` is a beta feature.
+:::
+
+If an ingestion task needs to write data to a segment for a time interval locked for compaction, by default the ingestion task supersedes the compaction task and the compaction task fails without finishing. For manual compaction tasks, you can adjust the input spec interval to avoid conflicts between ingestion and compaction. For automatic compaction, you can set the `skipOffsetFromLatest` key to adjust the auto-compaction starting point from the current time to reduce the chance of conf [...]
+Another option is to set the compaction task to higher priority than the ingestion task.
+For more information, see [Avoid conflicts with ingestion](../data-management/automatic-compaction.md#avoid-conflicts-with-ingestion).
+
+### Segment granularity handling
+
+Unless you modify the segment granularity in [`granularitySpec`](manual-compaction.md#compaction-granularity-spec), Druid attempts to retain the granularity for the compacted segments. When segments have different segment granularities with no overlap in interval Druid creates a separate compaction task for each to retain the segment granularity in the compacted segment.
+
+If segments have different segment granularities before compaction but there is some overlap in interval, Druid attempts find start and end of the overlapping interval and uses the closest segment granularity level for the compacted segment.
+
+For example consider two overlapping segments: segment "A" for the interval 01/01/2020-01/02/2020 with day granularity and segment "B" for the interval 01/01/2020-02/01/2020. Druid attempts to combine and compact the overlapped segments. In this example, the earliest start time for the two segments is 01/01/2020 and the latest end time of the two segments is 02/01/2020. Druid compacts the segments together even though they have different segment granularity. Druid uses month segment gran [...]
+
+### Query granularity handling
+
+Unless you modify the query granularity in the [`granularitySpec`](manual-compaction.md#compaction-granularity-spec), Druid retains the query granularity for the compacted segments. If segments have different query granularities before compaction, Druid chooses the finest level of granularity for the resulting compacted segment. For example if a compaction task combines two segments, one with day query granularity and one with minute query granularity, the resulting segment uses minute q [...]
+
+:::info
+ In Apache Druid 0.21.0 and prior, Druid sets the granularity for compacted segments to the default granularity of `NONE` regardless of the query granularity of the original segments.
+:::
+
+If you configure query granularity in compaction to go from a finer granularity like month to a coarser query granularity like year, then Druid overshadows the original segment with coarser granularity. Because the new segments have a coarser granularity, running a kill task to remove the overshadowed segments for those intervals will cause you to permanently lose the finer granularity data.
+
+### Dimension handling
+
+Apache Druid supports schema changes. Therefore, dimensions can be different across segments even if they are a part of the same datasource. See [Segments with different schemas](../design/segments.md#segments-with-different-schemas). If the input segments have different dimensions, the resulting compacted segment includes all dimensions of the input segments.
+
+Even when the input segments have the same set of dimensions, the dimension order or the data type of dimensions can be different. The dimensions of recent segments precede that of old segments in terms of data types and the ordering because more recent segments are more likely to have the preferred order and data types.
+
+If you want to control dimension ordering or ensure specific values for dimension types, you can configure a custom `dimensionsSpec` in the compaction task spec.
+
+### Rollup
+
+Druid only rolls up the output segment when `rollup` is set for all input segments.
+See [Roll-up](../ingestion/rollup.md) for more details.
+You can check that your segments are rolled up or not by using [Segment Metadata Queries](../querying/segmentmetadataquery.md#analysistypes).
+
+## Learn more
+
+See the following topics for more information:
+- [Segment optimization](../operations/segment-optimization.md) for guidance to determine if compaction will help in your case.
+- [Manual compaction](./manual-compaction.md) for how to run a one-time compaction task.
+- [Automatic compaction](automatic-compaction.md) for how to enable and configure automatic compaction.
+
diff --git a/docs/latest/data-management/delete.md b/docs/34.0.0/data-management/delete.md
similarity index 74%
copy from docs/latest/data-management/delete.md
copy to docs/34.0.0/data-management/delete.md
index c444a2507..e37ba48b5 100644
--- a/docs/latest/data-management/delete.md
+++ b/docs/34.0.0/data-management/delete.md
@@ -22,7 +22,7 @@ title: "Data deletion"
~ under the License.
-->
-## By time range, manually
+## Delete data for a time range manually
Apache Druid stores data [partitioned by time chunk](../design/storage.md) and supports
deleting data for time chunks by dropping segments. This is a fast, metadata-only operation.
@@ -42,17 +42,17 @@ For documentation on disabling segments using the Coordinator API, see the
A data deletion tutorial is available at [Tutorial: Deleting data](../tutorials/tutorial-delete-data.md).
-## By time range, automatically
+## Delete data automatically using drop rules
Druid supports [load and drop rules](../operations/rule-configuration.md), which are used to define intervals of time
where data should be preserved, and intervals where data should be discarded. Data that falls under a drop rule is
-marked unused, in the same manner as if you [manually mark that time range unused](#by-time-range-manually). This is a
+marked unused, in the same manner as if you [manually mark that time range unused](#delete-data-for-a-time-range-manually). This is a
fast, metadata-only operation.
Data that is dropped in this way is marked unused, but remains in deep storage. To permanently delete it, use a
[`kill` task](#kill-task).
-## Specific records
+## Delete specific records
Druid supports deleting specific records using [reindexing](update.md#reindex) with a filter. The filter specifies which
data remains after reindexing, so it must be the inverse of the data you want to delete. Because segments must be
@@ -62,7 +62,7 @@ For example, to delete records where `userName` is `'bob'` with native batch ind
[`transformSpec`](../ingestion/ingestion-spec.md#transformspec) with filter `{"type": "not", "field": {"type":
"selector", "dimension": "userName", "value": "bob"}}`.
-To delete the same records using SQL, use [REPLACE](../multi-stage-query/concepts.md#replace) with `WHERE userName <> 'bob'`.
+To delete the same records using SQL, use [REPLACE](../multi-stage-query/concepts.md#overwrite-data-with-replace) with `WHERE userName <> 'bob'`.
To reindex using [native batch](../ingestion/native-batch.md), use the [`druid` input
source](../ingestion/input-sources.md#druid-input-source). If needed,
@@ -74,15 +74,15 @@ used to filter, modify, or enrich the data during the reindexing job.
Data that is deleted in this way is marked unused, but remains in deep storage. To permanently delete it, use a [`kill`
task](#kill-task).
-## Entire table
+## Delete an entire table
-Deleting an entire table works the same way as [deleting part of a table by time range](#by-time-range-manually). First,
+Deleting an entire table works the same way as [deleting part of a table by time range](#delete-data-for-a-time-range-manually). First,
mark all segments unused using the Coordinator API or web console. Then, optionally, delete it permanently using a
[`kill` task](#kill-task).
<a name="kill-task"></a>
-## Permanently (`kill` task)
+## Delete data permanently using `kill` tasks
Data that has been overwritten or soft-deleted still remains as segments that have been marked unused. You can use a
`kill` task to permanently delete this data.
@@ -116,3 +116,33 @@ Some of the parameters used in the task payload are further explained below:
**WARNING:** The `kill` task permanently removes all information about the affected segments from the metadata store and
deep storage. This operation cannot be undone.
+### Auto-kill data using Coordinator duties
+
+Instead of submitting `kill` tasks manually to permanently delete data for a given interval, you can enable auto-kill of unused segments on the Coordinator.
+The Coordinator runs a duty periodically to identify intervals containing unused segments that are eligible for kill. It then launches a `kill` task for each of these intervals.
+
+Refer to [Data management on the Coordinator](../configuration/index.md#data-management) to configure auto-kill of unused segments on the Coordinator.
+
+### Auto-kill data on the Overlord (Experimental)
+
+:::info
+This is an experimental feature that:
+- Can be used only if [segment metadata caching](../configuration/index.md#segment-metadata-cache-experimental) is enabled on the Overlord.
+- MUST NOT be used if auto-kill of unused segments is already enabled on the Coordinator.
+:::
+
+This is an experimental feature to run kill tasks in an "embedded" mode on the Overlord itself.
+
+These embedded tasks offer several advantages over auto-kill performed by the Coordinator as they:
+- avoid a lot of unnecessary REST API calls to the Overlord from tasks or the Coordinator.
+- kill unused segments as soon as they become eligible.
+- run on the Overlord and do not take up task slots.
+- finish faster as they save on the overhead of launching a task process.
+- kill a small number of segments per task, to ensure that locks on an interval are not held for too long.
+- skip locked intervals to avoid head-of-line blocking in kill tasks.
+- require little to no configuration.
+- can keep up with a large number of unused segments in the cluster.
+- take advantage of the segment metadata cache on the Overlord.
+
+Refer to [Auto-kill unused segments on the Overlord](../configuration/index.md#auto-kill-unused-segments-experimental) to configure auto-kill of unused segments on the Overlord.
+See [Auto-kill metrics](../operations/metrics.md#auto-kill-unused-segments) for the metrics emitted by embedded kill tasks.
diff --git a/docs/34.0.0/data-management/index.md b/docs/34.0.0/data-management/index.md
new file mode 100644
index 000000000..0e0e09ac8
--- /dev/null
+++ b/docs/34.0.0/data-management/index.md
@@ -0,0 +1,34 @@
+---
+id: index
+title: "Data management"
+sidebar_label: "Overview"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+Apache Druid stores data [partitioned by time chunk](../design/storage.md) in immutable
+files called [segments](../design/segments.md). Data management operations involving replacing, or deleting,
+these segments include:
+
+- [Updates](update.md) to existing data.
+- [Deletion](delete.md) of existing data.
+- [Schema changes](schema-changes.md) for new and existing data.
+- [Compaction](compaction.md) and [automatic compaction](automatic-compaction.md), which reindex existing data to
+ optimize storage footprint and performance.
diff --git a/docs/latest/data-management/manual-compaction.md b/docs/34.0.0/data-management/manual-compaction.md
similarity index 99%
copy from docs/latest/data-management/manual-compaction.md
copy to docs/34.0.0/data-management/manual-compaction.md
index a2cd61b36..e6e34dba8 100644
--- a/docs/latest/data-management/manual-compaction.md
+++ b/docs/34.0.0/data-management/manual-compaction.md
@@ -57,7 +57,7 @@ You can perform manual compaction where you submit a one-time compaction task fo
|`segmentGranularity`|Deprecated. Use `granularitySpec`.|No|
|`tuningConfig`|[Tuning configuration](../ingestion/native-batch.md#tuningconfig) for parallel indexing. `awaitSegmentAvailabilityTimeoutMillis` value is not supported for compaction tasks. Leave this parameter at the default value, 0.|No|
|`granularitySpec`|When set, the compaction task uses the specified `granularitySpec` rather than generating one from existing segments. See [Compaction `granularitySpec`](#compaction-granularity-spec) for details.|No|
-|`context`|[Task context](../ingestion/tasks.md#context)|No|
+|`context`|[Task context](../ingestion/tasks.md#context-parameters)|No|
:::info
Note: Use `granularitySpec` over `segmentGranularity` and only set one of these values. If you specify different values for these in the same compaction spec, the task fails.
diff --git a/docs/34.0.0/data-management/schema-changes.md b/docs/34.0.0/data-management/schema-changes.md
new file mode 100644
index 000000000..0771da3ce
--- /dev/null
+++ b/docs/34.0.0/data-management/schema-changes.md
@@ -0,0 +1,39 @@
+---
+id: schema-changes
+title: "Schema changes"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+## For new data
+
+Apache Druid allows you to provide a new schema for new data without the need to update the schema of any existing data.
+It is sufficient to update your supervisor spec, if using [streaming ingestion](../ingestion/index.md#streaming), or to
+provide the new schema the next time you do a [batch ingestion](../ingestion/index.md#batch). This is made possible by
+the fact that each [segment](../design/segments.md), at the time it is created, stores a
+copy of its own schema. Druid reconciles all of these individual segment schemas automatically at query time.
+
+## For existing data
+
+Schema changes are sometimes necessary for existing data. For example, you may want to change the type of a column in
+previously-ingested data, or drop a column entirely. Druid handles this using [reindexing](update.md), the same method
+it uses to handle updates of existing data. Reindexing involves rewriting all affected segments and can be a
+time-consuming operation.
diff --git a/docs/34.0.0/data-management/update.md b/docs/34.0.0/data-management/update.md
new file mode 100644
index 000000000..a8c75a5d3
--- /dev/null
+++ b/docs/34.0.0/data-management/update.md
@@ -0,0 +1,78 @@
+---
+id: update
+title: "Data updates"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+## Overwrite
+
+Apache Druid stores data [partitioned by time chunk](../design/storage.md) and supports
+overwriting existing data using time ranges. Data outside the replacement time range is not touched. Overwriting of
+existing data is done using the same mechanisms as [batch ingestion](../ingestion/index.md#batch).
+
+For example:
+
+- [Native batch](../ingestion/native-batch.md) with `appendToExisting: false`, and `intervals` set to a specific
+ time range, overwrites data for that time range.
+- [SQL `REPLACE <table> OVERWRITE [ALL | WHERE ...]`](../multi-stage-query/reference.md#replace) overwrites data for
+ the entire table or for a specified time range.
+
+In both cases, Druid's atomic update mechanism ensures that queries will flip seamlessly from the old data to the new
+data on a time-chunk-by-time-chunk basis.
+
+Ingestion and overwriting cannot run concurrently for the same time range of the same datasource. While an overwrite job
+is ongoing for a particular time range of a datasource, new ingestions for that time range are queued up. Ingestions for
+other time ranges proceed as normal. Read-only queries also proceed as normal, using the pre-existing version of the
+data.
+
+:::info
+ Druid does not support single-record updates by primary key.
+:::
+
+## Reindex
+
+Reindexing is an [overwrite of existing data](#overwrite) where the source of new data is the existing data itself. It
+is used to perform schema changes, repartition data, filter out unwanted data, enrich existing data, and so on. This
+behaves just like any other [overwrite](#overwrite) with regard to atomic updates and locking.
+
+With [native batch](../ingestion/native-batch.md), use the [`druid` input
+source](../ingestion/input-sources.md#druid-input-source). If needed,
+[`transformSpec`](../ingestion/ingestion-spec.md#transformspec) can be used to filter or modify data during the
+reindexing job.
+
+With SQL, use [`REPLACE <table> OVERWRITE`](../multi-stage-query/reference.md#replace) with `SELECT ... FROM <table>`.
+(Druid does not have `UPDATE` or `ALTER TABLE` statements.) Any SQL SELECT query can be used to filter,
+modify, or enrich the data during the reindexing job.
+
+## Rolled-up datasources
+
+Rolled-up datasources can be effectively updated using appends, without rewrites. When you append a row that has an
+identical set of dimensions to an existing row, queries that use aggregation operators automatically combine those two
+rows together at query time.
+
+[Compaction](compaction.md) or [automatic compaction](automatic-compaction.md) can be used to physically combine these
+matching rows together later on, by rewriting segments in the background.
+
+## Lookups
+
+If you have a dimension where values need to be updated frequently, try first using [lookups](../querying/lookups.md). A
+classic use case of lookups is when you have an ID dimension stored in a Druid segment, and want to map the ID dimension to a
+human-readable string that may need to be updated periodically.
diff --git a/docs/34.0.0/design/architecture.md b/docs/34.0.0/design/architecture.md
new file mode 100644
index 000000000..04498defb
--- /dev/null
+++ b/docs/34.0.0/design/architecture.md
@@ -0,0 +1,185 @@
+---
+id: architecture
+title: "Architecture"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+Druid has a distributed architecture that is designed to be cloud-friendly and easy to operate. You can configure and scale services independently for maximum flexibility over cluster operations. This design includes enhanced fault tolerance: an outage of one component does not immediately affect other components.
+
+The following diagram shows the services that make up the Druid architecture, their typical arrangement across servers, and how queries and data flow through this architecture.
+
+
+
+The following sections describe the components of this architecture.
+
+## Druid services
+
+Druid has several types of services:
+
+* [Coordinator](../design/coordinator.md) manages data availability on the cluster.
+* [Overlord](../design/overlord.md) controls the assignment of data ingestion workloads.
+* [Broker](../design/broker.md) handles queries from external clients.
+* [Router](../design/router.md) routes requests to Brokers, Coordinators, and Overlords.
+* [Historical](../design/historical.md) stores queryable data.
+* [Middle Manager](../design/middlemanager.md) and [Peon](../design/peons.md) ingest data.
+* [Indexer](../design/indexer.md) serves as an alternative to the Middle Manager + Peon task execution system.
+
+You can view services in the **Services** tab in the web console:
+
+
+
+## Druid servers
+
+You can deploy Druid services according to your preferences. For ease of deployment, we recommend organizing them into three server types: [Master](#master-server), [Query](#query-server), and [Data](#data-server).
+
+### Master server
+
+A Master server manages data ingestion and availability. It is responsible for starting new ingestion jobs and coordinating availability of data on the [Data server](#data-server).
+
+Master servers divide operations between Coordinator and Overlord services.
+
+#### Coordinator service
+
+[Coordinator](../design/coordinator.md) services watch over the Historical services on the Data servers. They are responsible for assigning segments to specific servers, and for ensuring segments are well-balanced across Historicals.
+
+#### Overlord service
+
+[Overlord](../design/overlord.md) services watch over the Middle Manager services on the Data servers and are the controllers of data ingestion into Druid. They are responsible for assigning ingestion tasks to Middle Managers and for coordinating segment publishing.
+
+### Query server
+
+A Query server provides the endpoints that users and client applications interact with, routing queries to Data servers or other Query servers (and optionally proxied Master server requests).
+
+Query servers divide operations between Broker and Router services.
+
+#### Broker service
+
+[Broker](../design/broker.md) services receive queries from external clients and forward those queries to Data servers. When Brokers receive results from those subqueries, they merge those results and return them to the caller. Typically, you query Brokers rather than querying Historical or Middle Manager services on Data servers directly.
+
+#### Router service
+
+[**Router**](../design/router.md) services provide a unified API gateway in front of Brokers, Overlords, and Coordinators.
+
+The Router service also runs the [web console](../operations/web-console.md), a UI for loading data, managing datasources and tasks, and viewing server status and segment information.
+
+### Data server
+
+A Data server executes ingestion jobs and stores queryable data.
+
+Data servers divide operations between Historical and Middle Manager services.
+
+#### Historical service
+
+[**Historical**](../design/historical.md) services handle storage and querying on historical data, including any streaming data that has been in the system long enough to be committed. Historical services download segments from deep storage and respond to queries about these segments. They don't accept writes.
+
+#### Middle Manager service
+
+[**Middle Manager**](../design/middlemanager.md) services handle ingestion of new data into the cluster. They are responsible
+for reading from external data sources and publishing new Druid segments.
+
+##### Peon service
+
+[**Peon**](../design/peons.md) services are task execution engines spawned by Middle Managers. Each Peon runs a separate JVM and is responsible for executing a single task. Peons always run on the same host as the Middle Manager that spawned them.
+
+#### Indexer service (optional)
+
+[**Indexer**](../design/indexer.md) services are an alternative to Middle Managers and Peons. Instead of
+forking separate JVM processes per-task, the Indexer runs tasks as individual threads within a single JVM process.
+
+The Indexer is designed to be easier to configure and deploy compared to the MiddleManager + Peon system and to better enable resource sharing across tasks, which can help streaming ingestion. The Indexer is currently designated [experimental](../development/experimental.md).
+
+Typically, you would deploy one of the following: MiddleManagers, [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md), or Indexers. You wouldn't deploy more than one of these options.
+
+## Colocation of services
+
+Colocating Druid services by server type generally results in better utilization of hardware resources for most clusters.
+For very large scale clusters, it can be desirable to split the Druid services such that they run on individual servers to avoid resource contention.
+
+This section describes guidelines and configuration parameters related to service colocation.
+
+### Coordinators and Overlords
+
+The workload on the Coordinator service tends to increase with the number of segments in the cluster. The Overlord's workload also increases based on the number of segments in the cluster, but to a lesser degree than the Coordinator.
+
+In clusters with very high segment counts, it can make sense to separate the Coordinator and Overlord services to provide more resources for the Coordinator's segment balancing workload.
+
+You can run the Coordinator and Overlord services as a single combined service by setting the `druid.coordinator.asOverlord.enabled` property.
+For more information, see [Coordinator Operation](../configuration/index.md#coordinator-operation).
+
+### Historicals and Middle Managers
+
+With higher levels of ingestion or query load, it can make sense to deploy the Historical and Middle Manager services on separate hosts to to avoid CPU and memory contention.
+
+The Historical service also benefits from having free memory for memory mapped segments, which can be another reason to deploy the Historical and Middle Manager services separately.
+
+## External dependencies
+
+In addition to its built-in service types, Druid also has three external dependencies. These are intended to be able to
+leverage existing infrastructure, where present.
+
+### Deep storage
+
+Druid uses deep storage to store any data that has been ingested into the system. Deep storage is shared file
+storage accessible by every Druid server. In a clustered deployment, this is typically a distributed object store like S3 or
+HDFS, or a network mounted filesystem. In a single-server deployment, this is typically local disk.
+
+Druid uses deep storage for the following purposes:
+
+- To store all the data you ingest. Segments that get loaded onto Historical services for low latency queries are also kept in deep storage for backup purposes. Additionally, segments that are only in deep storage can be used for [queries from deep storage](../querying/query-from-deep-storage.md).
+- As a way to transfer data in the background between Druid services. Druid stores data in files called _segments_.
+
+Historical services cache data segments on local disk and serve queries from that cache as well as from an in-memory cache.
+Segments on disk for Historical services provide the low latency querying performance Druid is known for.
+
+You can also query directly from deep storage. When you query segments that exist only in deep storage, you trade some performance for the ability to query more of your data without necessarily having to scale your Historical services.
+
+When determining sizing for your storage, keep the following in mind:
+
+- Deep storage needs to be able to hold all the data that you ingest into Druid.
+- On disk storage for Historical services need to be able to accommodate the data you want to load onto them to run queries. The data on Historical services should be data you access frequently and need to run low latency queries for.
+
+Deep storage is an important part of Druid's elastic, fault-tolerant design. Druid bootstraps from deep storage even
+if every single data server is lost and re-provisioned.
+
+For more details, please see the [Deep storage](../design/deep-storage.md) page.
+
+### Metadata storage
+
+The metadata storage holds various shared system metadata such as segment usage information and task information. In a
+clustered deployment, this is typically a traditional RDBMS like PostgreSQL or MySQL. In a single-server
+deployment, it is typically a locally-stored Apache Derby database.
+
+For more details, please see the [Metadata storage](../design/metadata-storage.md) page.
+
+### ZooKeeper
+
+Used for internal service discovery, coordination, and leader election.
+
+For more details, please see the [ZooKeeper](zookeeper.md) page.
+
+## Learn more
+
+See the following topics for more information:
+
+* [Storage components](storage.md) to learn about data storage in Druid.
+* [Segments](segments.md) to learn about segment files.
+* [Query processing](../querying/query-processing.md) for a high-level overview of how Druid processes queries.
\ No newline at end of file
diff --git a/docs/34.0.0/design/broker.md b/docs/34.0.0/design/broker.md
new file mode 100644
index 000000000..bbd6b94f2
--- /dev/null
+++ b/docs/34.0.0/design/broker.md
@@ -0,0 +1,54 @@
+---
+id: broker
+title: "Broker service"
+sidebar_label: "Broker"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+The Broker service routes queries in a distributed cluster setup. It interprets the metadata published to ZooKeeper about segment distribution across services and routes queries accordingly. Additionally, the Broker service consolidates result sets from individual services.
+
+## Configuration
+
+For Apache Druid Broker service configuration, see [Broker Configuration](../configuration/index.md#broker).
+
+For basic tuning guidance for the Broker service, see [Basic cluster tuning](../operations/basic-cluster-tuning.md#broker).
+
+## HTTP endpoints
+
+For a list of API endpoints supported by the Broker, see [Broker API](../api-reference/legacy-metadata-api.md#broker).
+
+## Running
+
+```
+org.apache.druid.cli.Main server broker
+```
+
+## Forwarding queries
+
+Most Druid queries contain an interval object that indicates a span of time for which data is requested. Similarly, Druid partitions [segments](../design/segments.md) to contain data for some interval of time and distributes the segments across a cluster. Consider a simple datasource with seven segments where each segment contains data for a given day of the week. Any query issued to the datasource for more than one day of data will hit more than one segment. These segments will likely b [...]
+
+To determine which services to forward queries to, the Broker service first builds a view of the world from information in ZooKeeper. ZooKeeper maintains information about [Historical](../design/historical.md) and streaming ingestion [Peon](../design/peons.md) services and the segments they are serving. For every datasource in ZooKeeper, the Broker service builds a timeline of segments and the services that serve them. When queries are received for a specific datasource and interval, the [...]
+
+## Caching
+
+Broker services employ a cache with an LRU cache invalidation strategy. The Broker cache stores per-segment results. The cache can be local to each Broker service or shared across multiple services using an external distributed cache such as [memcached](http://memcached.org/). Each time a Broker service receives a query, it first maps the query to a set of segments. A subset of these segment results may already exist in the cache and the results can be directly pulled from the cache. For [...]
+Historical services. Once the Historical services return their results, the Broker will store those results in the cache. Real-time segments are never cached and hence requests for real-time data will always be forwarded to real-time services. Real-time data is perpetually changing and caching the results would be unreliable.
\ No newline at end of file
diff --git a/docs/34.0.0/design/coordinator.md b/docs/34.0.0/design/coordinator.md
new file mode 100644
index 000000000..bc4c5ebc1
--- /dev/null
+++ b/docs/34.0.0/design/coordinator.md
@@ -0,0 +1,176 @@
+---
+id: coordinator
+title: "Coordinator service"
+sidebar_label: "Coordinator"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+The Coordinator service is primarily responsible for segment management and distribution. More specifically, the
+Coordinator service communicates to Historical services to load or drop segments based on configurations. The Coordinator is responsible for loading new segments, dropping outdated segments, ensuring that segments are "replicated" (that is, loaded on multiple different Historical nodes) proper (configured) number of times, and moving
+("balancing") segments between Historical nodes to keep the latter evenly loaded.
+
+The Coordinator runs its duties periodically and the time between each run is a configurable parameter. On each
+run, the Coordinator assesses the current state of the cluster before deciding on the appropriate actions to take.
+Similar to the Broker and Historical services, the Coordinator maintains a connection to a ZooKeeper cluster for
+current cluster information. The Coordinator also maintains a connection to a database containing information about
+"used" segments (that is, the segments that *should* be loaded in the cluster) and the loading rules.
+
+Before any unassigned segments are serviced by Historical services, the Historical services for each tier are first
+sorted in terms of capacity, with least capacity servers having the highest priority. Unassigned segments are always
+assigned to the services with least capacity to maintain a level of balance between services. The Coordinator does not
+directly communicate with a Historical service when assigning it a new segment; instead the Coordinator creates some
+temporary information about the new segment under load queue path of the Historical service. Once this request is seen,
+the Historical service loads the segment and begins servicing it.
+
+## Configuration
+
+For Apache Druid Coordinator service configuration, see [Coordinator configuration](../configuration/index.md#coordinator).
+
+For basic tuning guidance for the Coordinator service, see [Basic cluster tuning](../operations/basic-cluster-tuning.md#coordinator).
+
+## HTTP endpoints
+
+For a list of API endpoints supported by the Coordinator, see [Service status API reference](../api-reference/service-status-api.md#coordinator).
+
+## Running
+
+```
+org.apache.druid.cli.Main server coordinator
+```
+
+## Rules
+
+Segments can be automatically loaded and dropped from the cluster based on a set of rules. For more information on rules, see [Rule Configuration](../operations/rule-configuration.md).
+
+### Clean up overshadowed segments
+
+On each run, the Coordinator compares the set of used segments in the database with the segments served by some
+Historical nodes in the cluster. The Coordinator sends requests to Historical nodes to unload unused segments or segments
+that are removed from the database.
+
+Segments that are overshadowed (their versions are too old and their data has been replaced by newer segments) are
+marked as unused. During the next Coordinator's run, they will be unloaded from Historical nodes in the cluster.
+
+### Clean up non-overshadowed eternity tombstone segments
+
+On each run, the Coordinator determines and cleans up unneeded eternity tombstone segments for each datasource. These segments must fit all the following criteria:
+- It is a tombstone segment that starts at -INF or ends at INF (for example, a tombstone with an interval of `-146136543-09-08T08:23:32.096Z/2000-01-01` or `2020-01-01/146140482-04-24T15:36:27.903Z` or `-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z`)
+- It does not overlap with any overshadowed segment
+- It has 0 core partitions
+
+## Segment availability
+
+If a Historical service restarts or becomes unavailable for any reason, the Coordinator notices that a service has gone missing and treats all segments served by that service as being dropped. The segments are then reassigned to other Historical services in the cluster. However, each segment that is dropped is not immediately forgotten. Instead, there is a transitional data structure that stores all dropped segments with an associated lifetime. The lifetime represents a period of time in [...]
+
+## Balancing segments in a tier
+
+Druid queries perform optimally when segments are distributed evenly across Historical services. An ideal distribution would ensure that all Historicals participate equally in the query load thus avoiding hot-spots in the system. To some extent, this can be achieved by keeping multiple replicas of a segment in a cluster.
+But in a tier with several Historicals (or a low replication factor), segment replication is not sufficient to attain balance.
+Thus, the Coordinator constantly monitors the set of segments present on each Historical in a tier and employs one of the following strategies to identify segments that may be moved from one Historical to another to retain balance.
+
+- `cost` (default): For a given segment in a tier, this strategy picks the server with the minimum "cost" of placing that segment. The cost is a function of the data interval of the segment and the data intervals of all the segments already present on the candidate server. In essence, this strategy tries to avoid placing segments with adjacent or overlapping data intervals on the same server. This is based on the premise that adjacent-interval segments are more likely to be used together [...]
+- `diskNormalized`: A derivative of the `cost` strategy that weights the cost of placing a segment on a server with the disk usage ratio of the server. There are known issues with this strategy and is not recommended for a production cluster.
+- `random`: Distributes segments randomly across servers. This is an experimental strategy and is not recommended for a production cluster.
+
+All of the above strategies prioritize moving segments from the Historical with the least available disk space.
+
+## Automatic compaction
+
+The Coordinator manages the [automatic compaction system](../data-management/automatic-compaction.md).
+Each run, the Coordinator compacts segments by merging small segments or splitting a large one. This is useful when the size of your segments is not optimized which may degrade query performance.
+See [Segment size optimization](../operations/segment-optimization.md) for details.
+
+The Coordinator first finds the segments to compact based on the [segment search policy](#segment-search-policy-in-automatic-compaction).
+Once some segments are found, it issues a [compaction task](../ingestion/tasks.md#compact) to compact those segments.
+The maximum number of running compaction tasks is `min(sum of worker capacity * slotRatio, maxSlots)`.
+Note that even if `min(sum of worker capacity * slotRatio, maxSlots) = 0`, at least one compaction task is always submitted
+if the compaction is enabled for a dataSource.
+See [Automatic compaction configuration API](../api-reference/automatic-compaction-api.md#manage-automatic-compaction) and [Automatic compaction configuration](../configuration/index.md#automatic-compaction-dynamic-configuration) to enable and configure automatic compaction.
+
+Compaction tasks might fail due to the following reasons:
+
+- If the input segments of a compaction task are removed or overshadowed before it starts, that compaction task fails immediately.
+- If a task of a higher priority acquires a [time chunk lock](../ingestion/tasks.md#locking) for an interval overlapping with the interval of a compaction task, the compaction task fails.
+
+Once a compaction task fails, the Coordinator simply checks the segments in the interval of the failed task again, and issues another compaction task in the next run.
+
+Note that Compacting Segments Coordinator Duty is automatically enabled and run as part of the Indexing Service Duties group. However, Compacting Segments Coordinator Duty can be configured to run in isolation as a separate Coordinator duty group. This allows changing the period of Compacting Segments Coordinator Duty without impacting the period of other Indexing Service Duties. This can be done by setting the following properties. For more details, see [custom pluggable Coordinator Dut [...]
+```
+druid.coordinator.dutyGroups=[<SOME_GROUP_NAME>]
+druid.coordinator.<SOME_GROUP_NAME>.duties=["compactSegments"]
+druid.coordinator.<SOME_GROUP_NAME>.period=<PERIOD_TO_RUN_COMPACTING_SEGMENTS_DUTY>
+```
+
+## Segment search policy in automatic compaction
+
+At every Coordinator run, this policy looks up time chunks from newest to oldest and checks whether the segments in those time chunks
+need compaction.
+A set of segments needs compaction if all conditions below are satisfied:
+
+* Total size of segments in the time chunk is smaller than or equal to the configured `inputSegmentSizeBytes`.
+* Segments have never been compacted yet or compaction spec has been updated since the last compaction: `maxTotalRows` or `indexSpec`.
+
+Here are some details with an example. Suppose we have two dataSources (`foo`, `bar`) as seen below:
+
+- `foo`
+ - `foo_2017-11-01T00:00:00.000Z_2017-12-01T00:00:00.000Z_VERSION`
+ - `foo_2017-11-01T00:00:00.000Z_2017-12-01T00:00:00.000Z_VERSION_1`
+ - `foo_2017-09-01T00:00:00.000Z_2017-10-01T00:00:00.000Z_VERSION`
+- `bar`
+ - `bar_2017-10-01T00:00:00.000Z_2017-11-01T00:00:00.000Z_VERSION`
+ - `bar_2017-10-01T00:00:00.000Z_2017-11-01T00:00:00.000Z_VERSION_1`
+
+Assuming that each segment is 10 MB and haven't been compacted yet, this policy first returns two segments of
+`foo_2017-11-01T00:00:00.000Z_2017-12-01T00:00:00.000Z_VERSION` and `foo_2017-11-01T00:00:00.000Z_2017-12-01T00:00:00.000Z_VERSION_1` to compact together because
+`2017-11-01T00:00:00.000Z/2017-12-01T00:00:00.000Z` is the most recent time chunk.
+
+If the Coordinator has enough task slots for compaction, this policy will continue searching for the next segments and return
+`bar_2017-10-01T00:00:00.000Z_2017-11-01T00:00:00.000Z_VERSION` and `bar_2017-10-01T00:00:00.000Z_2017-11-01T00:00:00.000Z_VERSION_1`.
+Finally, `foo_2017-09-01T00:00:00.000Z_2017-10-01T00:00:00.000Z_VERSION` will be picked up even though there is only one segment in the time chunk of `2017-09-01T00:00:00.000Z/2017-10-01T00:00:00.000Z`.
+
+The search start point can be changed by setting `skipOffsetFromLatest`.
+If this is set, this policy will ignore the segments falling into the time chunk of (the end time of the most recent segment - `skipOffsetFromLatest`).
+This is to avoid conflicts between compaction tasks and realtime tasks.
+Note that realtime tasks have a higher priority than compaction tasks by default. Realtime tasks will revoke the locks of compaction tasks if their intervals overlap, resulting in the termination of the compaction task.
+For more information, see [Avoid conflicts with ingestion](../data-management/automatic-compaction.md#avoid-conflicts-with-ingestion).
+
+:::info
+ This policy currently cannot handle the situation when there are a lot of small segments which have the same interval,
+ and their total size exceeds [`inputSegmentSizeBytes`](../configuration/index.md#automatic-compaction-dynamic-configuration).
+ If it finds such segments, it simply skips them.
+:::
+
+## FAQ
+
+1. **Do clients ever contact the Coordinator service?**
+
+ The Coordinator is not involved in a query.
+
+ Historical services never directly contact the Coordinator service. The Coordinator tells the Historical services to load/drop data via ZooKeeper, but the Historical services are completely unaware of the Coordinator.
+
+ Brokers also never contact the Coordinator. Brokers base their understanding of the data topology on metadata exposed by the Historical services via ZooKeeper and are completely unaware of the Coordinator.
+
+2. **Does it matter if the Coordinator service starts up before or after other services?**
+
+ No. If the Coordinator is not started up, no new segments will be loaded in the cluster and outdated segments will not be dropped. However, the Coordinator service can be started up at any time, and after a configurable delay, will start running Coordinator tasks.
+
+ This also means that if you have a working cluster and all of your Coordinators die, the cluster will continue to function, it just won’t experience any changes to its data topology.
diff --git a/docs/34.0.0/design/deep-storage.md b/docs/34.0.0/design/deep-storage.md
new file mode 100644
index 000000000..0674f3242
--- /dev/null
+++ b/docs/34.0.0/design/deep-storage.md
@@ -0,0 +1,88 @@
+---
+id: deep-storage
+title: "Deep storage"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+
+Deep storage is where segments are stored. It is a storage mechanism that Apache Druid does not provide. This deep storage infrastructure defines the level of durability of your data. As long as Druid processes can see this storage infrastructure and get at the segments stored on it, you will not lose data no matter how many Druid nodes you lose. If segments disappear from this storage layer, then you will lose whatever data those segments represented.
+
+In addition to being the backing store for segments, you can use [query from deep storage](#querying-from-deep-storage) and run queries against segments stored primarily in deep storage. The [load rules](../operations/rule-configuration.md#load-rules) you configure determine whether segments exist primarily in deep storage or in a combination of deep storage and Historical processes.
+
+## Deep storage options
+
... 40898 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org