[PR] feat: add OpenLineage request logger extension (druid)

["mshahid6 (via GitHub)" <gi...@apache.org>]

mshahid6 opened a new pull request, #19107:
URL: https://github.com/apache/druid/pull/19107

   ### Description
   Added `extensions-contrib/openlineage-emitter` as a contrib extension that uses the `RequestLogger` to transform and send lineage information to any [OpenLineage](https://openlineage.io)-compatible API.
   For SQL queries, the SQL text is parsed with the Calcite parser to extract input datasources (FROM clauses, JOINs, CTEs) and output datasources (INSERT INTO). For native queries, table names are read from `DataSource.getTableNames()`. Native sub-queries spawned by a SQL execution are deduplicated against the SQL-level event.
   Each event includes standard OpenLineage facets (`processing_engine`, `jobType`, `sql`,`errorMessage`) and custom Druid facets (`druid_query_context` with user identity and query metadata, `druid_query_statistics` with duration and bytes).
   
   Transport is configurable: `CONSOLE` (default) logs JSON to the Druid log; `HTTP` POSTs to an OpenLineage endpoint such as [Marquez](https://marquezproject.ai). Can be combined with other loggers via the `composing` provider.
   
   
   This PR has:
   
   - [ ] been self-reviewed.
   - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] a release note entry in the PR description.
   - [x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [ ] added integration tests.
   - [ ] been tested in a test Druid cluster.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["FrankChen021 (via GitHub)" <gi...@apache.org>]

FrankChen021 commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3208936475


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,626 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlNodeList;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.RejectedExecutionHandler;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  // Matches INSERT INTO or REPLACE INTO when the standard Calcite parser fails (e.g. EXTERN,
+  // PARTITIONED BY). Captures the output table name; handles quoted and unquoted identifiers.
+  private static final Pattern WRITE_INTO_PATTERN =
+      Pattern.compile("(?i)^\\s*(?:INSERT|REPLACE)\\s+INTO\\s+\"?([^\"\\s]+)\"?");
+  // Extracts the SELECT subquery from INSERT INTO or REPLACE INTO for input lineage.
+  // Stops before PARTITIONED BY / CLUSTERED BY which are Druid-specific clauses.
+  private static final Pattern SELECT_FROM_WRITE_PATTERN =
+      Pattern.compile("(?is)\\b(SELECT\\s+.+?)(?:\\s+PARTITIONED\\s+BY|\\s+CLUSTERED\\s+BY|$)");
+
+  static final int DEFAULT_EMIT_QUEUE_CAPACITY = 1000;
+  static final int DEFAULT_EMIT_THREAD_COUNT = 1;
+  private static final int DISCARD_WARNING_INTERVAL = 1000;
+
+  static final String UNKNOWN_QUERY_ID = "unknown-query-id";
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  private final Set<String> excludedNativeQueryTypes;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+  private final AtomicLong discardedEventCount = new AtomicLong(0);
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         DEFAULT_EMIT_QUEUE_CAPACITY, DEFAULT_EMIT_THREAD_COUNT, null);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount,
+      @Nullable HttpClient httpClient
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    this.excludedNativeQueryTypes = excludedNativeQueryTypes;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = httpClient;

Review Comment:
   [P2] Direct HTTP logger construction leaves client null
   
   The public 5-argument constructor still delegates here with a null httpClient. After this update, HTTP mode stores that null instead of creating a default client, so direct construction with HTTP and a transportUrl enqueues events whose worker hits a NullPointerException at httpClient.execute and never posts lineage. Please keep default client creation when httpClient is null, or make the client-injection path non-public and route all HTTP construction through the provider.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["jtuglu1 (via GitHub)" <gi...@apache.org>]

jtuglu1 commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3091137636


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLoggerProvider.java:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.guice.annotations.Json;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.server.DruidNode;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.server.log.RequestLoggerProvider;
+
+import javax.annotation.Nullable;
+import javax.validation.constraints.NotNull;
+import java.util.Set;
+
+/**
+ *  Configure logging type, namespace, transport type (http or default console), transportUrl in {@code runtime.properties} 
+ */
+@JsonTypeName("openlineage")
+public class OpenLineageRequestLoggerProvider implements RequestLoggerProvider
+{
+  private static final Logger log = new Logger(OpenLineageRequestLoggerProvider.class);
+
+  public enum TransportType
+  {
+    CONSOLE,
+    HTTP
+  }
+
+  @JacksonInject
+  @Json
+  @NotNull
+  private ObjectMapper jsonMapper;
+
+  @JsonProperty
+  @NotNull
+  private String namespace = "druid://" + DruidNode.getDefaultHost();
+
+  @JsonProperty
+  @NotNull
+  private TransportType transportType = TransportType.CONSOLE;
+
+  @Nullable
+  @JsonProperty
+  private String transportUrl;
+
+  @JsonProperty
+  @NotNull
+  private Set<String> excludedNativeQueryTypes = Set.of(
+      "segmentMetadata",
+      "dataSourceMetadata",
+      "timeBoundary"

Review Comment:
   Why excluding timeBoundary?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["github-advanced-security[bot] (via GitHub)" <gi...@apache.org>]

github-advanced-security[bot] commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3169991966


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,656 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlNodeList;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.RejectedExecutionHandler;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  // Matches: INSERT INTO <table> ... (fallback when EXTERN prevents standard Calcite parsing)
+  // Handles both quoted ("table") and unquoted (table) identifiers.
+  private static final Pattern INSERT_INTO_PATTERN =
+      Pattern.compile("(?i)^\\s*INSERT\\s+INTO\\s+\"?([^\"\\s]+)\"?");
+  // Matches: REPLACE INTO <table> OVERWRITE ... (Druid-specific, not parseable by standard Calcite)
+  private static final Pattern REPLACE_INTO_PATTERN =
+      Pattern.compile("(?i)^\\s*REPLACE\\s+INTO\\s+\"?([^\"\\s]+)\"?");
+  // Extracts the SELECT subquery from a REPLACE INTO statement for input lineage
+  private static final Pattern REPLACE_SELECT_PATTERN =
+      Pattern.compile("(?is)\\b(SELECT\\s+.+?)(?:\\s+PARTITIONED\\s+BY|\\s+CLUSTERED\\s+BY|$)");
+
+  static final int DEFAULT_EMIT_QUEUE_CAPACITY = 1000;
+  static final int DEFAULT_EMIT_THREAD_COUNT = 1;
+  private static final int DISCARD_WARNING_INTERVAL = 1000;
+
+  static final String UNKNOWN_QUERY_ID = "unknown-query-id";
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  private final Set<String> excludedNativeQueryTypes;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+  private final AtomicLong discardedEventCount = new AtomicLong(0);
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         DEFAULT_EMIT_QUEUE_CAPACITY, DEFAULT_EMIT_THREAD_COUNT);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         emitQueueCapacity, emitThreadCount, null);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount,
+      @Nullable HttpClient httpClient
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    this.excludedNativeQueryTypes = excludedNativeQueryTypes;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = httpClient != null ? httpClient : HttpClientBuilder.create().build();
+      // Bounded queue: if the queue is full, drop the event rather than blocking the query thread.
+      // A warning is logged on the first drop and every DISCARD_WARNING_INTERVAL drops thereafter.
+      this.emitExecutor = new ThreadPoolExecutor(
+          emitThreadCount,
+          emitThreadCount,
+          60L,
+          TimeUnit.SECONDS,
+          new ArrayBlockingQueue<>(emitQueueCapacity),
+          Execs.makeThreadFactory("OpenLineageEmitter-%d"),
+          new DiscardWithWarningPolicy(discardedEventCount)
+      );
+    } else {
+      this.httpClient = null;
+      this.emitExecutor = null;
+    }
+  }
+
+  // Note: ComposingRequestLogger does not delegate @LifecycleStart to sub-loggers, so this method
+  // may not be called when used in a composing configuration. HTTP URL validation is therefore
+  // performed in the constructor instead. This method is retained for direct lifecycle use.
+  @LifecycleStart
+  @Override
+  public void start()
+  {
+    log.info(
+        "Started OpenLineage {} transport{}",
+        transportType,
+        transportUrl != null ? " to [" + transportUrl + "]" : ""
+    );
+  }
+
+  @LifecycleStop
+  @Override
+  public void stop()
+  {
+    if (emitExecutor != null) {
+      emitExecutor.shutdown();
+      try {
+        if (!emitExecutor.awaitTermination(5, TimeUnit.SECONDS)) {
+          emitExecutor.shutdownNow();
+        }
+      }
+      catch (InterruptedException e) {
+        emitExecutor.shutdownNow();
+        Thread.currentThread().interrupt();
+      }
+    }
+    if (httpClient instanceof Closeable) {
+      CloseableUtils.closeAndSuppressExceptions(
+          (Closeable) httpClient,
+          e -> log.warn(e, "Failed to close OpenLineage HTTP client")
+      );
+    }
+    log.info("Stopped OpenLineage request logger");
+  }
+
+  @Override
+  public void logNativeQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    if (requestLogLine.getQuery() == null) {
+      return;
+    }
+
+    // Skip native sub-queries of a SQL execution to avoid duplicating the SQL-level event.
+    if (requestLogLine.getQuery().getContext().get(BaseQuery.SQL_QUERY_ID) != null) {
+      return;
+    }
+
+    String queryType = requestLogLine.getQuery().getType();
+
+    if (excludedNativeQueryTypes.contains(queryType)) {
+      return;
+    }
+
+    List<String> inputs = new ArrayList<>(new LinkedHashSet<>(requestLogLine.getQuery().getDataSource().getTableNames()));
+    String queryId = requestLogLine.getQuery().getId();
+    if (queryId == null) {
+      log.debug("Native query reached OpenLineage logger without a query ID");
+      queryId = UNKNOWN_QUERY_ID;
+    }
+
+    emit(buildRunEvent(queryId, null, queryType, requestLogLine, inputs, null));
+  }
+
+  @Override
+  public void logSqlQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    String sql = requestLogLine.getSql();
+    List<String> inputs = new ArrayList<>();
+    String output = null;
+
+    if (sql != null) {
+      try {
+        SqlNode parsed = SqlParser.create(sql, SQL_PARSER_CONFIG).parseQuery();
+        inputs = extractInputs(parsed);
+        output = extractOutput(parsed);
+      }
+      catch (SqlParseException e) {
+        // Druid-specific SQL extensions (REPLACE INTO, EXTERN, etc.) may not parse with the
+        // standard Calcite parser. Attempt to extract lineage from REPLACE INTO via regex;
+        // for other unparseable statements, emit the event without table-level lineage.
+        if (sql != null) {

Review Comment:
   ## CodeQL / Useless null check
   
   This check is useless. [sql](1) cannot be null at this check, since it is guarded by [... != ...](2).
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/11168)



##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,656 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlNodeList;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.RejectedExecutionHandler;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  // Matches: INSERT INTO <table> ... (fallback when EXTERN prevents standard Calcite parsing)
+  // Handles both quoted ("table") and unquoted (table) identifiers.
+  private static final Pattern INSERT_INTO_PATTERN =
+      Pattern.compile("(?i)^\\s*INSERT\\s+INTO\\s+\"?([^\"\\s]+)\"?");
+  // Matches: REPLACE INTO <table> OVERWRITE ... (Druid-specific, not parseable by standard Calcite)
+  private static final Pattern REPLACE_INTO_PATTERN =
+      Pattern.compile("(?i)^\\s*REPLACE\\s+INTO\\s+\"?([^\"\\s]+)\"?");
+  // Extracts the SELECT subquery from a REPLACE INTO statement for input lineage
+  private static final Pattern REPLACE_SELECT_PATTERN =
+      Pattern.compile("(?is)\\b(SELECT\\s+.+?)(?:\\s+PARTITIONED\\s+BY|\\s+CLUSTERED\\s+BY|$)");
+
+  static final int DEFAULT_EMIT_QUEUE_CAPACITY = 1000;
+  static final int DEFAULT_EMIT_THREAD_COUNT = 1;
+  private static final int DISCARD_WARNING_INTERVAL = 1000;
+
+  static final String UNKNOWN_QUERY_ID = "unknown-query-id";
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  private final Set<String> excludedNativeQueryTypes;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+  private final AtomicLong discardedEventCount = new AtomicLong(0);
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         DEFAULT_EMIT_QUEUE_CAPACITY, DEFAULT_EMIT_THREAD_COUNT);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         emitQueueCapacity, emitThreadCount, null);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount,
+      @Nullable HttpClient httpClient
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    this.excludedNativeQueryTypes = excludedNativeQueryTypes;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = httpClient != null ? httpClient : HttpClientBuilder.create().build();
+      // Bounded queue: if the queue is full, drop the event rather than blocking the query thread.
+      // A warning is logged on the first drop and every DISCARD_WARNING_INTERVAL drops thereafter.
+      this.emitExecutor = new ThreadPoolExecutor(
+          emitThreadCount,
+          emitThreadCount,
+          60L,
+          TimeUnit.SECONDS,
+          new ArrayBlockingQueue<>(emitQueueCapacity),
+          Execs.makeThreadFactory("OpenLineageEmitter-%d"),
+          new DiscardWithWarningPolicy(discardedEventCount)
+      );
+    } else {
+      this.httpClient = null;
+      this.emitExecutor = null;
+    }
+  }
+
+  // Note: ComposingRequestLogger does not delegate @LifecycleStart to sub-loggers, so this method
+  // may not be called when used in a composing configuration. HTTP URL validation is therefore
+  // performed in the constructor instead. This method is retained for direct lifecycle use.
+  @LifecycleStart
+  @Override
+  public void start()
+  {
+    log.info(
+        "Started OpenLineage {} transport{}",
+        transportType,
+        transportUrl != null ? " to [" + transportUrl + "]" : ""
+    );

Review Comment:
   ## CodeQL / Unused format argument
   
   This format call refers to 0 argument(s) but supplies 2 argument(s).
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/11167)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["jtuglu1 (via GitHub)" <gi...@apache.org>]

jtuglu1 commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3132206846


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,498 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOrderBy;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  private static final int EMIT_QUEUE_CAPACITY = 1000;
+  private static final int EMIT_THREAD_COUNT = 1;
+
+  /**
+   * Internal broker query types that are automatically issued for schema discovery and metadata
+   * caching. These are not user-initiated and produce noisy, low-value lineage events.
+   */
+  private static final Set<String> INTERNAL_QUERY_TYPES = Set.of(
+      "segmentMetadata",
+      "dataSourceMetadata",
+      "timeBoundary"
+  );
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = HttpClientBuilder.create().build();
+      // Bounded queue: if the queue is full, silently drop the event rather than blocking
+      // the query thread. Uses Druid's Execs for daemon thread naming conventions.
+      this.emitExecutor = new ThreadPoolExecutor(
+          EMIT_THREAD_COUNT,
+          EMIT_THREAD_COUNT,
+          60L,
+          TimeUnit.SECONDS,
+          new ArrayBlockingQueue<>(EMIT_QUEUE_CAPACITY),
+          Execs.makeThreadFactory("OpenLineageEmitter-%d"),
+          new ThreadPoolExecutor.DiscardPolicy()
+      );
+    } else {
+      this.httpClient = null;
+      this.emitExecutor = null;
+    }
+  }
+
+  @LifecycleStart
+  @Override
+  public void start() throws Exception
+  {
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      log.info("Started OpenLineage HTTP transport to [%s]", transportUrl);
+    } else {
+      log.info("Started OpenLineage console transport");
+    }
+  }
+
+  @LifecycleStop
+  @Override
+  public void stop()
+  {
+    if (emitExecutor != null) {
+      emitExecutor.shutdown();
+      try {
+        if (!emitExecutor.awaitTermination(5, TimeUnit.SECONDS)) {
+          emitExecutor.shutdownNow();
+        }
+      }
+      catch (InterruptedException e) {
+        emitExecutor.shutdownNow();
+        Thread.currentThread().interrupt();
+      }
+    }
+    if (httpClient instanceof Closeable) {
+      CloseableUtils.closeAndSuppressExceptions(
+          (Closeable) httpClient,
+          e -> log.warn(e, "Failed to close OpenLineage HTTP client")
+      );
+    }
+    log.info("Stopped OpenLineage request logger");
+  }
+
+  @Override
+  public void logNativeQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    if (requestLogLine.getQuery() == null) {
+      return;
+    }
+
+    // Skip native sub-queries of a SQL execution to avoid duplicating the SQL-level event.
+    if (requestLogLine.getQuery().getContext().get(BaseQuery.SQL_QUERY_ID) != null) {
+      return;
+    }
+
+    String queryType = requestLogLine.getQuery().getType();
+
+    if (INTERNAL_QUERY_TYPES.contains(queryType)) {
+      return;
+    }
+
+    List<String> inputs = new ArrayList<>(new LinkedHashSet<>(requestLogLine.getQuery().getDataSource().getTableNames()));
+    String queryId = requestLogLine.getQuery().getId();
+    if (queryId == null) {
+      queryId = UUID.randomUUID().toString();
+    }
+
+    emit(buildRunEvent(queryId, null, queryType, requestLogLine, inputs, null));
+  }
+
+  @Override
+  public void logSqlQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    String sql = requestLogLine.getSql();
+    List<String> inputs = new ArrayList<>();
+    String output = null;
+
+    if (sql != null) {
+      try {
+        SqlNode parsed = SqlParser.create(sql, SQL_PARSER_CONFIG).parseQuery();
+        inputs = extractInputs(parsed);
+        output = extractOutput(parsed);
+      }
+      catch (SqlParseException e) {
+        // Druid-specific SQL extensions (REPLACE, EXTERN, etc.) may not parse with the standard
+        // Calcite parser. Emit the event without table-level lineage rather than failing.
+        log.debug(
+            "OpenLineage: could not parse SQL for lineage extraction (query will still be emitted): %s",
+            e.getMessage()
+        );
+      }
+    }
+
+    String queryId = extractSqlQueryId(requestLogLine);
+    emit(buildRunEvent(queryId, sql, "sql", requestLogLine, new ArrayList<>(new LinkedHashSet<>(inputs)), output));
+  }
+
+  private ObjectNode buildRunEvent(
+      String queryId,
+      @Nullable String sql,
+      String queryType,
+      RequestLogLine requestLogLine,
+      List<String> inputs,
+      @Nullable String output
+  )
+  {
+    Map<String, Object> stats = requestLogLine.getQueryStats().getStats();
+    boolean success = Boolean.TRUE.equals(stats.get("success"));
+
+    ObjectNode event = jsonMapper.createObjectNode();
+    event.put("eventType", success ? "COMPLETE" : "FAIL");
+    event.put("eventTime", requestLogLine.getTimestamp().toInstant().toString());
+    event.put("producer", PRODUCER);
+    event.put("schemaURL", SCHEMA_URL);
+    event.set("run", buildRun(queryId, queryType, requestLogLine, stats, success));
+    event.set("job", buildJob(queryId, sql));
+    event.set("inputs", buildDatasets(inputs));
+    event.set("outputs", buildDatasets(output != null ? List.of(output) : List.of()));
+    return event;
+  }
+
+  private ObjectNode buildRun(
+      String queryId,
+      String queryType,
+      RequestLogLine requestLogLine,
+      Map<String, Object> stats,
+      boolean success
+  )
+  {
+    ObjectNode run = jsonMapper.createObjectNode();
+    run.put("runId", UUID.nameUUIDFromBytes(queryId.getBytes(StandardCharsets.UTF_8)).toString());
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode engineFacet = createFacet(ENGINE_FACET_SCHEMA_URL);
+    engineFacet.put("name", "druid");
+    engineFacet.put("version", getDruidVersion());
+    facets.set("processing_engine", engineFacet);
+
+    ObjectNode contextFacet = createFacet(null);
+    contextFacet.put("queryType", queryType);
+    contextFacet.put("remoteAddress", requestLogLine.getRemoteAddr());
+    Object identity = stats.get("identity");
+    if (identity != null) {
+      contextFacet.put("identity", identity.toString());
+    }
+    Object nativeQueryIds = requestLogLine.getSqlQueryContext().get("nativeQueryIds");
+    if (nativeQueryIds != null) {
+      contextFacet.put("nativeQueryIds", nativeQueryIds.toString());
+    }
+    facets.set("druid_query_context", contextFacet);
+
+    ObjectNode statsFacet = createFacet(null);
+    putLongStat(statsFacet, "durationMs", stats, "sqlQuery/time", "query/time");
+    putLongStat(statsFacet, "bytes", stats, "sqlQuery/bytes", "query/bytes");
+    putLongStat(statsFacet, "planningTimeMs", stats, "sqlQuery/planningTimeMs");
+    Object statusCode = stats.get("statusCode");
+    if (statusCode != null) {
+      statsFacet.put("statusCode", statusCode.toString());
+    }
+    facets.set("druid_query_statistics", statsFacet);
+
+    if (!success) {
+      Object exception = stats.get("exception");
+      if (exception != null) {
+        ObjectNode errorFacet = createFacet(ERROR_FACET_SCHEMA_URL);
+        errorFacet.put("message", exception.toString());
+        if ("sql".equals(queryType)) {
+          errorFacet.put("programmingLanguage", "SQL");
+        }
+        facets.set("errorMessage", errorFacet);
+      }
+    }
+
+    run.set("facets", facets);
+    return run;
+  }
+
+  private ObjectNode buildJob(String queryId, @Nullable String sql)
+  {
+    ObjectNode job = jsonMapper.createObjectNode();
+    job.put("namespace", namespace);
+    job.put("name", queryId);
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode jobTypeFacet = createFacet(JOB_TYPE_FACET_SCHEMA_URL);
+    jobTypeFacet.put("processingType", "BATCH");
+    jobTypeFacet.put("integration", "DRUID");
+    jobTypeFacet.put("jobType", "QUERY");
+    facets.set("jobType", jobTypeFacet);
+
+    if (sql != null) {

Review Comment:
   We should be emitting lineage for all both types. `logNativeQuery` handles the native path.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["mshahid6 (via GitHub)" <gi...@apache.org>]

mshahid6 commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3148675400


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLoggerProvider.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.guice.annotations.Json;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.server.DruidNode;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.server.log.RequestLoggerProvider;
+
+import javax.annotation.Nullable;
+import javax.validation.constraints.NotNull;
+import java.util.Set;
+
+/**
+ *  Configure logging type, namespace, transport type (http or default console), transportUrl in {@code runtime.properties} 
+ */
+@JsonTypeName("openlineage")
+public class OpenLineageRequestLoggerProvider implements RequestLoggerProvider
+{
+  private static final Logger log = new Logger(OpenLineageRequestLoggerProvider.class);
+
+  public enum TransportType
+  {
+    CONSOLE,
+    HTTP
+  }
+
+  @JacksonInject
+  @Json
+  @NotNull
+  private ObjectMapper jsonMapper;
+
+  @JsonProperty
+  @NotNull
+  private String namespace = "druid://" + DruidNode.getDefaultHost();
+
+  @JsonProperty
+  @NotNull
+  private TransportType transportType = TransportType.CONSOLE;
+
+  @Nullable
+  @JsonProperty
+  private String transportUrl;
+
+  @JsonProperty
+  @NotNull
+  private Set<String> excludedNativeQueryTypes = Set.of(

Review Comment:
   Initially had it (https://github.com/apache/druid/pull/19107#discussion_r3091137636), can re-add timeBoundary



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["FrankChen021 (via GitHub)" <gi...@apache.org>]

FrankChen021 commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3281522301


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,449 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.RejectedExecutionHandler;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  static final int DEFAULT_EMIT_QUEUE_CAPACITY = 1000;
+  static final int DEFAULT_EMIT_THREAD_COUNT = 1;
+  private static final int DISCARD_WARNING_INTERVAL = 1000;
+
+  static final String UNKNOWN_QUERY_ID = "unknown-query-id";
+
+  // Matches the output table of MSQ DML: INSERT INTO "foo" or REPLACE INTO foo
+  private static final Pattern MSQ_DML_OUTPUT_PATTERN = Pattern.compile(
+      "(?si)^\\s*(?:INSERT|REPLACE)\\s+INTO\\s+\"?([^\"\\s(,]+)\"?"

Review Comment:
   [P2] MSQ output regex reports the wrong destination for valid INSERT targets
   
   The new SQL DML path extracts the output dataset from raw SQL with this regex, but Druid accepts targets that do not map 1:1 to that text. For example, `INSERT INTO druid.dst ...` is valid and is normalized by the ingest planner to datasource `dst`, while this emits output dataset `druid.dst`; `INSERT INTO EXTERN(...) AS CSV ...` is also valid and this emits a bogus Druid dataset named `EXTERN`. This produces incorrect OpenLineage outputs for supported MSQ statements. Use the parsed ingest target or the planner's resolved destination instead of matching the SQL string.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["github-advanced-security[bot] (via GitHub)" <gi...@apache.org>]

github-advanced-security[bot] commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r2899187067


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,414 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOrderBy;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+  }
+
+  @Override
+  public void logNativeQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    // Skip native sub-queries of a SQL execution to avoid duplicating the SQL-level event.
+    if (requestLogLine.getQuery().getContextValue(BaseQuery.SQL_QUERY_ID) != null) {

Review Comment:
   ## Deprecated method or constructor invocation
   
   Invoking [Query.getContextValue](1) should be avoided because it has been deprecated.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/10865)



##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,414 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOrderBy;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+  }
+
+  @Override
+  public void logNativeQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    // Skip native sub-queries of a SQL execution to avoid duplicating the SQL-level event.
+    if (requestLogLine.getQuery().getContextValue(BaseQuery.SQL_QUERY_ID) != null) {
+      return;
+    }
+
+    List<String> inputs = new ArrayList<>(requestLogLine.getQuery().getDataSource().getTableNames());
+    String queryId = requestLogLine.getQuery().getId();
+    if (queryId == null) {
+      queryId = UUID.randomUUID().toString();
+    }
+    String queryType = requestLogLine.getQuery().getType();
+
+    emit(buildRunEvent(queryId, null, queryType, requestLogLine, inputs, Optional.empty()));
+  }
+
+  @Override
+  public void logSqlQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    String sql = requestLogLine.getSql();
+    List<String> inputs = new ArrayList<>();
+    Optional<String> output = Optional.empty();
+
+    if (sql != null) {
+      try {
+        SqlNode parsed = SqlParser.create(sql, SqlParser.config()).parseQuery();
+        inputs = extractInputs(parsed);
+        output = extractOutput(parsed);
+      }
+      catch (SqlParseException e) {
+        // Druid-specific SQL extensions (REPLACE, EXTERN, etc.) may not parse with the standard
+        // Calcite parser. Emit the event without table-level lineage rather than failing.
+        log.debug(
+            "OpenLineage: could not parse SQL for lineage extraction (query will still be emitted): %s",
+            e.getMessage()
+        );
+      }
+    }
+
+    String queryId = extractSqlQueryId(requestLogLine);
+    emit(buildRunEvent(queryId, sql, "sql", requestLogLine, inputs, output));
+  }
+
+  private ObjectNode buildRunEvent(
+      String queryId,
+      @Nullable String sql,
+      String queryType,
+      RequestLogLine requestLogLine,
+      List<String> inputs,
+      Optional<String> output
+  )
+  {
+    Map<String, Object> stats = requestLogLine.getQueryStats().getStats();
+    boolean success = Boolean.TRUE.equals(stats.get("success"));
+
+    ObjectNode event = jsonMapper.createObjectNode();
+    event.put("eventType", success ? "COMPLETE" : "FAIL");
+    event.put("eventTime", requestLogLine.getTimestamp().toInstant().toString());
+    event.put("producer", PRODUCER);
+    event.put("schemaURL", SCHEMA_URL);
+    event.set("run", buildRun(queryId, queryType, requestLogLine, stats, success));
+    event.set("job", buildJob(queryId, sql));
+    event.set("inputs", buildDatasets(inputs));
+    event.set("outputs", buildDatasets(output.map(List::of).orElse(List.of())));
+    return event;
+  }
+
+  private ObjectNode buildRun(
+      String queryId,
+      String queryType,
+      RequestLogLine requestLogLine,
+      Map<String, Object> stats,
+      boolean success
+  )
+  {
+    ObjectNode run = jsonMapper.createObjectNode();
+    run.put("runId", UUID.nameUUIDFromBytes(queryId.getBytes(StandardCharsets.UTF_8)).toString());
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode engineFacet = jsonMapper.createObjectNode();
+    engineFacet.put("_producer", PRODUCER);
+    engineFacet.put("_schemaURL", ENGINE_FACET_SCHEMA_URL);
+    engineFacet.put("name", "druid");
+    facets.set("processing_engine", engineFacet);
+
+    ObjectNode contextFacet = jsonMapper.createObjectNode();
+    contextFacet.put("_producer", PRODUCER);
+    contextFacet.put("queryType", queryType);
+    contextFacet.put("remoteAddress", requestLogLine.getRemoteAddr());
+    Object identity = stats.get("identity");
+    if (identity != null) {
+      contextFacet.put("identity", identity.toString());
+    }
+    Object nativeQueryIds = requestLogLine.getSqlQueryContext().get("nativeQueryIds");
+    if (nativeQueryIds != null) {
+      contextFacet.put("nativeQueryIds", nativeQueryIds.toString());
+    }
+    facets.set("druid_query_context", contextFacet);
+
+    ObjectNode statsFacet = jsonMapper.createObjectNode();
+    statsFacet.put("_producer", PRODUCER);
+    putLongStat(statsFacet, "durationMs", stats, "sqlQuery/time", "query/time");
+    putLongStat(statsFacet, "bytes", stats, "sqlQuery/bytes", "query/bytes");
+    putLongStat(statsFacet, "planningTimeMs", stats, "sqlQuery/planningTimeMs");
+    Object statusCode = stats.get("statusCode");
+    if (statusCode != null) {
+      statsFacet.put("statusCode", statusCode.toString());
+    }
+    facets.set("druid_query_statistics", statsFacet);
+
+    if (!success) {
+      Object exception = stats.get("exception");
+      if (exception != null) {
+        ObjectNode errorFacet = jsonMapper.createObjectNode();
+        errorFacet.put("_producer", PRODUCER);
+        errorFacet.put("_schemaURL", ERROR_FACET_SCHEMA_URL);
+        errorFacet.put("message", exception.toString());
+        errorFacet.put("programmingLanguage", "SQL");
+        facets.set("errorMessage", errorFacet);
+      }
+    }
+
+    run.set("facets", facets);
+    return run;
+  }
+
+  private ObjectNode buildJob(String queryId, @Nullable String sql)
+  {
+    ObjectNode job = jsonMapper.createObjectNode();
+    job.put("namespace", namespace);
+    job.put("name", queryId);
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode jobTypeFacet = jsonMapper.createObjectNode();
+    jobTypeFacet.put("_producer", PRODUCER);
+    jobTypeFacet.put("_schemaURL", JOB_TYPE_FACET_SCHEMA_URL);
+    jobTypeFacet.put("processingType", "BATCH");
+    jobTypeFacet.put("integration", "DRUID");
+    jobTypeFacet.put("jobType", "QUERY");
+    facets.set("jobType", jobTypeFacet);
+
+    if (sql != null) {
+      ObjectNode sqlFacet = jsonMapper.createObjectNode();
+      sqlFacet.put("_producer", PRODUCER);
+      sqlFacet.put("_schemaURL", SQL_FACET_SCHEMA_URL);
+      sqlFacet.put("query", sql);
+      facets.set("sql", sqlFacet);
+    }
+
+    job.set("facets", facets);
+    return job;
+  }
+
+  private ArrayNode buildDatasets(List<String> tableNames)
+  {
+    ArrayNode array = jsonMapper.createArrayNode();
+    for (String name : tableNames) {
+      ObjectNode node = jsonMapper.createObjectNode();
+      node.put("namespace", namespace);
+      node.put("name", name);
+      // Schema and columnLineage facets require Druid coordinator metadata lookups and are not
+      // populated here. Column-level lineage is a future enhancement.
+      node.set("facets", jsonMapper.createObjectNode());
+      array.add(node);
+    }
+    return array;
+  }
+
+  /**
+   * Extracts the names of input datasources referenced in FROM clauses.
+   * CTEs (WITH clause names) are excluded since they are not physical datasources.
+   */
+  private List<String> extractInputs(SqlNode root)
+  {
+    List<String> tables = new ArrayList<>();
+    if (root instanceof SqlWith) {
+      SqlWith with = (SqlWith) root;
+      Set<String> cteNames = new HashSet<>();
+      for (SqlNode item : with.withList) {
+        if (item instanceof SqlWithItem) {
+          cteNames.add(((SqlWithItem) item).name.getSimple());
+          collectFromClause(((SqlWithItem) item).query, tables, cteNames);
+        }
+      }
+      collectFromClause(with.body, tables, cteNames);
+    } else if (root instanceof SqlInsert) {
+      collectFromClause(((SqlInsert) root).getSource(), tables, Set.of());
+    } else {
+      collectFromClause(root, tables, Set.of());
+    }
+    return tables;
+  }
+
+  /**
+   * Extracts the output datasource name for INSERT statements. Returns empty for SELECT queries.
+   */
+  private Optional<String> extractOutput(SqlNode root)
+  {
+    if (root instanceof SqlInsert) {
+      SqlNode target = ((SqlInsert) root).getTargetTable();
+      if (target instanceof SqlIdentifier) {
+        return Optional.of(String.join(".", ((SqlIdentifier) target).names));
+      }
+    }
+    return Optional.empty();
+  }
+
+  /**
+   * Recursively walks a FROM clause node and appends physical table names to {@code tables}.
+   * CTE names in {@code excludeNames} are skipped.
+   */
+  private void collectFromClause(SqlNode from, List<String> tables, Set<String> excludeNames)
+  {
+    if (from == null) {
+      return;
+    }
+    if (from instanceof SqlIdentifier) {

Review Comment:
   ## Chain of 'instanceof' tests
   
   This if block performs a chain of 6 type tests - consider alternatives, e.g. polymorphism or the visitor pattern.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/10866)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["FrankChen021 (via GitHub)" <gi...@apache.org>]

FrankChen021 commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3141585779


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,551 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOrderBy;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.RejectedExecutionHandler;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  static final int DEFAULT_EMIT_QUEUE_CAPACITY = 1000;
+  static final int DEFAULT_EMIT_THREAD_COUNT = 1;
+  private static final int DISCARD_WARNING_INTERVAL = 1000;
+
+  static final String UNKNOWN_QUERY_ID = "unknown-query-id";
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  private final Set<String> excludedNativeQueryTypes;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+  private final AtomicLong discardedEventCount = new AtomicLong(0);
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         DEFAULT_EMIT_QUEUE_CAPACITY, DEFAULT_EMIT_THREAD_COUNT);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         emitQueueCapacity, emitThreadCount, null);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount,
+      @Nullable HttpClient httpClient
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    this.excludedNativeQueryTypes = excludedNativeQueryTypes;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = httpClient != null ? httpClient : HttpClientBuilder.create().build();
+      // Bounded queue: if the queue is full, drop the event rather than blocking the query thread.
+      // A warning is logged on the first drop and every DISCARD_WARNING_INTERVAL drops thereafter.
+      this.emitExecutor = new ThreadPoolExecutor(
+          emitThreadCount,
+          emitThreadCount,
+          60L,
+          TimeUnit.SECONDS,
+          new ArrayBlockingQueue<>(emitQueueCapacity),
+          Execs.makeThreadFactory("OpenLineageEmitter-%d"),
+          new DiscardWithWarningPolicy(discardedEventCount)
+      );
+    } else {
+      this.httpClient = null;
+      this.emitExecutor = null;
+    }
+  }
+
+  @LifecycleStart
+  @Override
+  public void start() throws Exception
+  {
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      log.info("Started OpenLineage HTTP transport to [%s]", transportUrl);
+    } else {
+      log.info("Started OpenLineage console transport");
+    }
+  }
+
+  @LifecycleStop
+  @Override
+  public void stop()
+  {
+    if (emitExecutor != null) {
+      emitExecutor.shutdown();
+      try {
+        if (!emitExecutor.awaitTermination(5, TimeUnit.SECONDS)) {
+          emitExecutor.shutdownNow();
+        }
+      }
+      catch (InterruptedException e) {
+        emitExecutor.shutdownNow();
+        Thread.currentThread().interrupt();
+      }
+    }
+    if (httpClient instanceof Closeable) {
+      CloseableUtils.closeAndSuppressExceptions(
+          (Closeable) httpClient,
+          e -> log.warn(e, "Failed to close OpenLineage HTTP client")
+      );
+    }
+    log.info("Stopped OpenLineage request logger");
+  }
+
+  @Override
+  public void logNativeQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    if (requestLogLine.getQuery() == null) {
+      return;
+    }
+
+    // Skip native sub-queries of a SQL execution to avoid duplicating the SQL-level event.
+    if (requestLogLine.getQuery().getContext().get(BaseQuery.SQL_QUERY_ID) != null) {
+      return;
+    }
+
+    String queryType = requestLogLine.getQuery().getType();
+
+    if (excludedNativeQueryTypes.contains(queryType)) {
+      return;
+    }
+
+    List<String> inputs = new ArrayList<>(new LinkedHashSet<>(requestLogLine.getQuery().getDataSource().getTableNames()));
+    String queryId = requestLogLine.getQuery().getId();
+    if (queryId == null) {
+      log.debug("Native query reached OpenLineage logger without a query ID");
+      queryId = UNKNOWN_QUERY_ID;
+    }
+
+    emit(buildRunEvent(queryId, null, queryType, requestLogLine, inputs, null));
+  }
+
+  @Override
+  public void logSqlQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    String sql = requestLogLine.getSql();
+    List<String> inputs = new ArrayList<>();
+    String output = null;
+
+    if (sql != null) {
+      try {
+        SqlNode parsed = SqlParser.create(sql, SQL_PARSER_CONFIG).parseQuery();
+        inputs = extractInputs(parsed);
+        output = extractOutput(parsed);
+      }
+      catch (SqlParseException e) {
+        // Druid-specific SQL extensions (REPLACE, EXTERN, etc.) may not parse with the standard
+        // Calcite parser. Emit the event without table-level lineage rather than failing.
+        log.debug(
+            "OpenLineage: could not parse SQL for lineage extraction (query will still be emitted): %s",
+            e.getMessage()
+        );
+      }
+    }
+
+    String queryId = extractSqlQueryId(requestLogLine);
+    emit(buildRunEvent(queryId, sql, "sql", requestLogLine, new ArrayList<>(new LinkedHashSet<>(inputs)), output));
+  }
+
+  private ObjectNode buildRunEvent(
+      String queryId,
+      @Nullable String sql,
+      String queryType,
+      RequestLogLine requestLogLine,
+      List<String> inputs,
+      @Nullable String output
+  )
+  {
+    Map<String, Object> stats = requestLogLine.getQueryStats().getStats();
+    boolean success = Boolean.TRUE.equals(stats.get("success"));
+
+    ObjectNode event = jsonMapper.createObjectNode();
+    event.put("eventType", success ? "COMPLETE" : "FAIL");
+    event.put("eventTime", requestLogLine.getTimestamp().toInstant().toString());
+    event.put("producer", PRODUCER);
+    event.put("schemaURL", SCHEMA_URL);
+    event.set("run", buildRun(queryId, queryType, requestLogLine, stats, success));
+    event.set("job", buildJob(queryId, sql));
+    event.set("inputs", buildDatasets(inputs));
+    event.set("outputs", buildDatasets(output != null ? List.of(output) : List.of()));
+    return event;
+  }
+
+  private ObjectNode buildRun(
+      String queryId,
+      String queryType,
+      RequestLogLine requestLogLine,
+      Map<String, Object> stats,
+      boolean success
+  )
+  {
+    ObjectNode run = jsonMapper.createObjectNode();
+    run.put("runId", UUID.nameUUIDFromBytes(queryId.getBytes(StandardCharsets.UTF_8)).toString());
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode engineFacet = createFacet(ENGINE_FACET_SCHEMA_URL);
+    engineFacet.put("name", "druid");
+    engineFacet.put("version", getDruidVersion());
+    facets.set("processing_engine", engineFacet);
+
+    ObjectNode contextFacet = createFacet(null);
+    contextFacet.put("queryType", queryType);
+    contextFacet.put("remoteAddress", requestLogLine.getRemoteAddr());
+    Object identity = stats.get("identity");
+    if (identity != null) {
+      contextFacet.put("identity", identity.toString());
+    }
+    Object nativeQueryIds = requestLogLine.getSqlQueryContext().get("nativeQueryIds");
+    if (nativeQueryIds != null) {
+      contextFacet.put("nativeQueryIds", nativeQueryIds.toString());
+    }
+    facets.set("druid_query_context", contextFacet);
+
+    ObjectNode statsFacet = createFacet(null);
+    putLongStat(statsFacet, "durationMs", stats, "sqlQuery/time", "query/time");
+    putLongStat(statsFacet, "bytes", stats, "sqlQuery/bytes", "query/bytes");
+    putLongStat(statsFacet, "planningTimeMs", stats, "sqlQuery/planningTimeMs");
+    Object statusCode = stats.get("statusCode");
+    if (statusCode != null) {
+      statsFacet.put("statusCode", statusCode.toString());
+    }
+    facets.set("druid_query_statistics", statsFacet);
+
+    if (!success) {
+      Object exception = stats.get("exception");
+      if (exception != null) {
+        ObjectNode errorFacet = createFacet(ERROR_FACET_SCHEMA_URL);
+        errorFacet.put("message", exception.toString());
+        if ("sql".equals(queryType)) {
+          errorFacet.put("programmingLanguage", "SQL");
+        }
+        facets.set("errorMessage", errorFacet);
+      }
+    }
+
+    run.set("facets", facets);
+    return run;
+  }
+
+  private ObjectNode buildJob(String queryId, @Nullable String sql)
+  {
+    ObjectNode job = jsonMapper.createObjectNode();
+    job.put("namespace", namespace);
+    job.put("name", queryId);
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode jobTypeFacet = createFacet(JOB_TYPE_FACET_SCHEMA_URL);
+    jobTypeFacet.put("processingType", "BATCH");
+    jobTypeFacet.put("integration", "DRUID");
+    jobTypeFacet.put("jobType", "QUERY");
+    facets.set("jobType", jobTypeFacet);
+
+    if (sql != null) {
+      ObjectNode sqlFacet = createFacet(SQL_FACET_SCHEMA_URL);
+      sqlFacet.put("query", sql);
+      facets.set("sql", sqlFacet);
+    }
+
+    job.set("facets", facets);
+    return job;
+  }
+
+  private ObjectNode createFacet(@Nullable String schemaUrl)
+  {
+    ObjectNode facet = jsonMapper.createObjectNode();
+    facet.put("_producer", PRODUCER);
+    if (schemaUrl != null) {
+      facet.put("_schemaURL", schemaUrl);
+    }
+    return facet;
+  }
+
+  private ArrayNode buildDatasets(List<String> tableNames)
+  {
+    ArrayNode array = jsonMapper.createArrayNode();
+    for (String name : tableNames) {
+      ObjectNode node = jsonMapper.createObjectNode();
+      node.put("namespace", namespace);
+      node.put("name", name);
+      node.set("facets", jsonMapper.createObjectNode());
+      array.add(node);
+    }
+    return array;
+  }
+
+  private List<String> extractInputs(SqlNode root)
+  {
+    List<String> tables = new ArrayList<>();
+    if (root instanceof SqlWith) {
+      SqlWith with = (SqlWith) root;
+      Set<String> cteNames = new HashSet<>();
+      for (SqlNode item : with.withList) {
+        if (item instanceof SqlWithItem) {
+          cteNames.add(((SqlWithItem) item).name.getSimple());
+          collectFromClause(((SqlWithItem) item).query, tables, cteNames);
+        }
+      }
+      collectFromClause(with.body, tables, cteNames);
+    } else if (root instanceof SqlInsert) {
+      collectFromClause(((SqlInsert) root).getSource(), tables, Set.of());
+    } else {
+      collectFromClause(root, tables, Set.of());
+    }
+    return tables;
+  }
+
+  @Nullable
+  private String extractOutput(SqlNode root)
+  {
+    if (root instanceof SqlInsert) {
+      SqlNode target = ((SqlInsert) root).getTargetTable();
+      if (target instanceof SqlIdentifier) {
+        return String.join(".", ((SqlIdentifier) target).names);
+      }
+    }
+    return null;
+  }
+
+  private void collectFromClause(SqlNode from, List<String> tables, Set<String> excludeNames)
+  {
+    if (from == null) {
+      return;
+    }
+    if (from instanceof SqlIdentifier) {
+      String name = String.join(".", ((SqlIdentifier) from).names);
+      if (!excludeNames.contains(name)) {
+        tables.add(name);
+      }
+    } else if (from instanceof SqlJoin) {
+      SqlJoin join = (SqlJoin) from;
+      collectFromClause(join.getLeft(), tables, excludeNames);
+      collectFromClause(join.getRight(), tables, excludeNames);
+    } else if (from instanceof SqlSelect) {

Review Comment:
   P2 SQL lineage misses common nested inputs
   
   The SQL walker only recurses into SqlSelect.getFrom() and only handles SqlBasicCall when it is AS, so inputs referenced by subqueries in WHERE/HAVING/SELECT clauses are omitted, and set operations such as SELECT ... FROM a UNION ALL SELECT ... FROM b can emit no inputs at all. This makes the emitted OpenLineage event materially incomplete for valid Druid SQL; consider walking the full SqlNode tree with the relevant CTE alias exclusions instead of only FROM/JOIN nodes.



##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLoggerProvider.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.guice.annotations.Json;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.server.DruidNode;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.server.log.RequestLoggerProvider;
+
+import javax.annotation.Nullable;
+import javax.validation.constraints.NotNull;
+import java.util.Set;
+
+/**
+ *  Configure logging type, namespace, transport type (http or default console), transportUrl in {@code runtime.properties} 
+ */
+@JsonTypeName("openlineage")
+public class OpenLineageRequestLoggerProvider implements RequestLoggerProvider
+{
+  private static final Logger log = new Logger(OpenLineageRequestLoggerProvider.class);
+
+  public enum TransportType
+  {
+    CONSOLE,
+    HTTP
+  }
+
+  @JacksonInject
+  @Json
+  @NotNull
+  private ObjectMapper jsonMapper;
+
+  @JsonProperty
+  @NotNull
+  private String namespace = "druid://" + DruidNode.getDefaultHost();
+
+  @JsonProperty
+  @NotNull
+  private TransportType transportType = TransportType.CONSOLE;
+
+  @Nullable
+  @JsonProperty
+  private String transportUrl;
+
+  @JsonProperty
+  @NotNull
+  private Set<String> excludedNativeQueryTypes = Set.of(

Review Comment:
   P3 Default exclusions differ from docs
   
   The provider default excludes only segmentMetadata and dataSourceMetadata, but the new docs and test constant list timeBoundary as a default exclusion too. Users who rely on the documented default will still emit low-value timeBoundary native query events unless they override the property explicitly. Add timeBoundary here or update the docs/tests to match the actual default.



##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,551 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOrderBy;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.RejectedExecutionHandler;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  static final int DEFAULT_EMIT_QUEUE_CAPACITY = 1000;
+  static final int DEFAULT_EMIT_THREAD_COUNT = 1;
+  private static final int DISCARD_WARNING_INTERVAL = 1000;
+
+  static final String UNKNOWN_QUERY_ID = "unknown-query-id";
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  private final Set<String> excludedNativeQueryTypes;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+  private final AtomicLong discardedEventCount = new AtomicLong(0);
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         DEFAULT_EMIT_QUEUE_CAPACITY, DEFAULT_EMIT_THREAD_COUNT);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         emitQueueCapacity, emitThreadCount, null);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount,
+      @Nullable HttpClient httpClient
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    this.excludedNativeQueryTypes = excludedNativeQueryTypes;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = httpClient != null ? httpClient : HttpClientBuilder.create().build();
+      // Bounded queue: if the queue is full, drop the event rather than blocking the query thread.
+      // A warning is logged on the first drop and every DISCARD_WARNING_INTERVAL drops thereafter.
+      this.emitExecutor = new ThreadPoolExecutor(
+          emitThreadCount,
+          emitThreadCount,
+          60L,
+          TimeUnit.SECONDS,
+          new ArrayBlockingQueue<>(emitQueueCapacity),
+          Execs.makeThreadFactory("OpenLineageEmitter-%d"),
+          new DiscardWithWarningPolicy(discardedEventCount)
+      );
+    } else {
+      this.httpClient = null;
+      this.emitExecutor = null;
+    }
+  }
+
+  @LifecycleStart
+  @Override
+  public void start() throws Exception
+  {
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      log.info("Started OpenLineage HTTP transport to [%s]", transportUrl);
+    } else {
+      log.info("Started OpenLineage console transport");
+    }
+  }
+
+  @LifecycleStop
+  @Override
+  public void stop()
+  {
+    if (emitExecutor != null) {
+      emitExecutor.shutdown();
+      try {
+        if (!emitExecutor.awaitTermination(5, TimeUnit.SECONDS)) {
+          emitExecutor.shutdownNow();
+        }
+      }
+      catch (InterruptedException e) {
+        emitExecutor.shutdownNow();
+        Thread.currentThread().interrupt();
+      }
+    }
+    if (httpClient instanceof Closeable) {
+      CloseableUtils.closeAndSuppressExceptions(
+          (Closeable) httpClient,
+          e -> log.warn(e, "Failed to close OpenLineage HTTP client")
+      );
+    }
+    log.info("Stopped OpenLineage request logger");
+  }
+
+  @Override
+  public void logNativeQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    if (requestLogLine.getQuery() == null) {
+      return;
+    }
+
+    // Skip native sub-queries of a SQL execution to avoid duplicating the SQL-level event.
+    if (requestLogLine.getQuery().getContext().get(BaseQuery.SQL_QUERY_ID) != null) {
+      return;
+    }
+
+    String queryType = requestLogLine.getQuery().getType();
+
+    if (excludedNativeQueryTypes.contains(queryType)) {
+      return;
+    }
+
+    List<String> inputs = new ArrayList<>(new LinkedHashSet<>(requestLogLine.getQuery().getDataSource().getTableNames()));
+    String queryId = requestLogLine.getQuery().getId();
+    if (queryId == null) {
+      log.debug("Native query reached OpenLineage logger without a query ID");
+      queryId = UNKNOWN_QUERY_ID;
+    }
+
+    emit(buildRunEvent(queryId, null, queryType, requestLogLine, inputs, null));
+  }
+
+  @Override
+  public void logSqlQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    String sql = requestLogLine.getSql();
+    List<String> inputs = new ArrayList<>();
+    String output = null;
+
+    if (sql != null) {
+      try {
+        SqlNode parsed = SqlParser.create(sql, SQL_PARSER_CONFIG).parseQuery();
+        inputs = extractInputs(parsed);
+        output = extractOutput(parsed);
+      }
+      catch (SqlParseException e) {
+        // Druid-specific SQL extensions (REPLACE, EXTERN, etc.) may not parse with the standard
+        // Calcite parser. Emit the event without table-level lineage rather than failing.
+        log.debug(
+            "OpenLineage: could not parse SQL for lineage extraction (query will still be emitted): %s",
+            e.getMessage()
+        );
+      }
+    }
+
+    String queryId = extractSqlQueryId(requestLogLine);
+    emit(buildRunEvent(queryId, sql, "sql", requestLogLine, new ArrayList<>(new LinkedHashSet<>(inputs)), output));
+  }
+
+  private ObjectNode buildRunEvent(
+      String queryId,
+      @Nullable String sql,
+      String queryType,
+      RequestLogLine requestLogLine,
+      List<String> inputs,
+      @Nullable String output
+  )
+  {
+    Map<String, Object> stats = requestLogLine.getQueryStats().getStats();
+    boolean success = Boolean.TRUE.equals(stats.get("success"));
+
+    ObjectNode event = jsonMapper.createObjectNode();
+    event.put("eventType", success ? "COMPLETE" : "FAIL");
+    event.put("eventTime", requestLogLine.getTimestamp().toInstant().toString());
+    event.put("producer", PRODUCER);
+    event.put("schemaURL", SCHEMA_URL);
+    event.set("run", buildRun(queryId, queryType, requestLogLine, stats, success));
+    event.set("job", buildJob(queryId, sql));
+    event.set("inputs", buildDatasets(inputs));
+    event.set("outputs", buildDatasets(output != null ? List.of(output) : List.of()));
+    return event;
+  }
+
+  private ObjectNode buildRun(
+      String queryId,
+      String queryType,
+      RequestLogLine requestLogLine,
+      Map<String, Object> stats,
+      boolean success
+  )
+  {
+    ObjectNode run = jsonMapper.createObjectNode();
+    run.put("runId", UUID.nameUUIDFromBytes(queryId.getBytes(StandardCharsets.UTF_8)).toString());
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode engineFacet = createFacet(ENGINE_FACET_SCHEMA_URL);
+    engineFacet.put("name", "druid");
+    engineFacet.put("version", getDruidVersion());
+    facets.set("processing_engine", engineFacet);
+
+    ObjectNode contextFacet = createFacet(null);
+    contextFacet.put("queryType", queryType);
+    contextFacet.put("remoteAddress", requestLogLine.getRemoteAddr());
+    Object identity = stats.get("identity");
+    if (identity != null) {
+      contextFacet.put("identity", identity.toString());
+    }
+    Object nativeQueryIds = requestLogLine.getSqlQueryContext().get("nativeQueryIds");
+    if (nativeQueryIds != null) {
+      contextFacet.put("nativeQueryIds", nativeQueryIds.toString());
+    }
+    facets.set("druid_query_context", contextFacet);
+
+    ObjectNode statsFacet = createFacet(null);
+    putLongStat(statsFacet, "durationMs", stats, "sqlQuery/time", "query/time");
+    putLongStat(statsFacet, "bytes", stats, "sqlQuery/bytes", "query/bytes");
+    putLongStat(statsFacet, "planningTimeMs", stats, "sqlQuery/planningTimeMs");
+    Object statusCode = stats.get("statusCode");
+    if (statusCode != null) {
+      statsFacet.put("statusCode", statusCode.toString());
+    }
+    facets.set("druid_query_statistics", statsFacet);
+
+    if (!success) {
+      Object exception = stats.get("exception");
+      if (exception != null) {
+        ObjectNode errorFacet = createFacet(ERROR_FACET_SCHEMA_URL);
+        errorFacet.put("message", exception.toString());
+        if ("sql".equals(queryType)) {
+          errorFacet.put("programmingLanguage", "SQL");
+        }
+        facets.set("errorMessage", errorFacet);
+      }
+    }
+
+    run.set("facets", facets);
+    return run;
+  }
+
+  private ObjectNode buildJob(String queryId, @Nullable String sql)
+  {
+    ObjectNode job = jsonMapper.createObjectNode();
+    job.put("namespace", namespace);
+    job.put("name", queryId);
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode jobTypeFacet = createFacet(JOB_TYPE_FACET_SCHEMA_URL);
+    jobTypeFacet.put("processingType", "BATCH");
+    jobTypeFacet.put("integration", "DRUID");
+    jobTypeFacet.put("jobType", "QUERY");
+    facets.set("jobType", jobTypeFacet);
+
+    if (sql != null) {
+      ObjectNode sqlFacet = createFacet(SQL_FACET_SCHEMA_URL);
+      sqlFacet.put("query", sql);
+      facets.set("sql", sqlFacet);
+    }
+
+    job.set("facets", facets);
+    return job;
+  }
+
+  private ObjectNode createFacet(@Nullable String schemaUrl)
+  {
+    ObjectNode facet = jsonMapper.createObjectNode();
+    facet.put("_producer", PRODUCER);
+    if (schemaUrl != null) {
+      facet.put("_schemaURL", schemaUrl);
+    }
+    return facet;
+  }
+
+  private ArrayNode buildDatasets(List<String> tableNames)
+  {
+    ArrayNode array = jsonMapper.createArrayNode();
+    for (String name : tableNames) {
+      ObjectNode node = jsonMapper.createObjectNode();
+      node.put("namespace", namespace);
+      node.put("name", name);
+      node.set("facets", jsonMapper.createObjectNode());
+      array.add(node);
+    }
+    return array;
+  }
+
+  private List<String> extractInputs(SqlNode root)
+  {
+    List<String> tables = new ArrayList<>();
+    if (root instanceof SqlWith) {
+      SqlWith with = (SqlWith) root;
+      Set<String> cteNames = new HashSet<>();
+      for (SqlNode item : with.withList) {
+        if (item instanceof SqlWithItem) {
+          cteNames.add(((SqlWithItem) item).name.getSimple());
+          collectFromClause(((SqlWithItem) item).query, tables, cteNames);
+        }
+      }
+      collectFromClause(with.body, tables, cteNames);
+    } else if (root instanceof SqlInsert) {
+      collectFromClause(((SqlInsert) root).getSource(), tables, Set.of());
+    } else {
+      collectFromClause(root, tables, Set.of());
+    }
+    return tables;
+  }
+
+  @Nullable
+  private String extractOutput(SqlNode root)
+  {
+    if (root instanceof SqlInsert) {
+      SqlNode target = ((SqlInsert) root).getTargetTable();
+      if (target instanceof SqlIdentifier) {
+        return String.join(".", ((SqlIdentifier) target).names);
+      }
+    }
+    return null;
+  }
+
+  private void collectFromClause(SqlNode from, List<String> tables, Set<String> excludeNames)
+  {
+    if (from == null) {
+      return;
+    }
+    if (from instanceof SqlIdentifier) {
+      String name = String.join(".", ((SqlIdentifier) from).names);
+      if (!excludeNames.contains(name)) {
+        tables.add(name);
+      }
+    } else if (from instanceof SqlJoin) {
+      SqlJoin join = (SqlJoin) from;
+      collectFromClause(join.getLeft(), tables, excludeNames);
+      collectFromClause(join.getRight(), tables, excludeNames);
+    } else if (from instanceof SqlSelect) {
+      collectFromClause(((SqlSelect) from).getFrom(), tables, excludeNames);
+    } else if (from instanceof SqlWith) {
+      SqlWith with = (SqlWith) from;
+      Set<String> innerExcludes = new HashSet<>(excludeNames);
+      for (SqlNode item : with.withList) {
+        if (item instanceof SqlWithItem) {
+          innerExcludes.add(((SqlWithItem) item).name.getSimple());
+          collectFromClause(((SqlWithItem) item).query, tables, innerExcludes);
+        }
+      }
+      collectFromClause(with.body, tables, innerExcludes);
+    } else if (from instanceof SqlOrderBy) {
+      collectFromClause(((SqlOrderBy) from).query, tables, excludeNames);
+    } else if (from instanceof SqlBasicCall && from.getKind() == SqlKind.AS) {
+      collectFromClause(((SqlBasicCall) from).operand(0), tables, excludeNames);
+    }
+  }
+
+  protected void emit(ObjectNode event)
+  {
+    try {
+      String json = jsonMapper.writeValueAsString(event);
+      if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+        emitExecutor.submit(() -> emitHttp(json));
+      } else {
+        log.debug("OpenLineage event: %s", json);
+      }
+    }
+    catch (IOException e) {
+      log.error(e, "Failed to serialize OpenLineage event");
+    }
+  }
+
+  private void emitHttp(String json)
+  {
+    HttpPost post = new HttpPost(transportUrl);
+    post.setEntity(new StringEntity(json, ContentType.APPLICATION_JSON));
+    try {
+      org.apache.http.HttpResponse response = httpClient.execute(post);

Review Comment:
   P2 HTTP transport silently drops rejected events
   
   emitHttp treats any HTTP response as success: it executes the POST and consumes the entity without checking the status code. If Marquez/OpenLineage returns 400/401/429/500 due to schema, auth, rate limiting, or backend errors, the async logger discards the event with no warning, causing silent lineage loss. Check for non-2xx responses and log enough detail to diagnose delivery failures.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["maytasm (via GitHub)" <gi...@apache.org>]

maytasm commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3171450696


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;

Review Comment:
   Would these Calcite imports cause issue when loading this extension on nodes without Calcite on the classpath?



##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlNodeList;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.RejectedExecutionHandler;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  // Matches: INSERT INTO <table> ... (fallback when EXTERN prevents standard Calcite parsing)
+  // Handles both quoted ("table") and unquoted (table) identifiers.
+  private static final Pattern INSERT_INTO_PATTERN =
+      Pattern.compile("(?i)^\\s*INSERT\\s+INTO\\s+\"?([^\"\\s]+)\"?");
+  // Matches: REPLACE INTO <table> OVERWRITE ... (Druid-specific, not parseable by standard Calcite)
+  private static final Pattern REPLACE_INTO_PATTERN =
+      Pattern.compile("(?i)^\\s*REPLACE\\s+INTO\\s+\"?([^\"\\s]+)\"?");
+  // Extracts the SELECT subquery from a REPLACE INTO statement for input lineage
+  private static final Pattern REPLACE_SELECT_PATTERN =
+      Pattern.compile("(?is)\\b(SELECT\\s+.+?)(?:\\s+PARTITIONED\\s+BY|\\s+CLUSTERED\\s+BY|$)");
+
+  static final int DEFAULT_EMIT_QUEUE_CAPACITY = 1000;
+  static final int DEFAULT_EMIT_THREAD_COUNT = 1;
+  private static final int DISCARD_WARNING_INTERVAL = 1000;
+
+  static final String UNKNOWN_QUERY_ID = "unknown-query-id";
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  private final Set<String> excludedNativeQueryTypes;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+  private final AtomicLong discardedEventCount = new AtomicLong(0);
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         DEFAULT_EMIT_QUEUE_CAPACITY, DEFAULT_EMIT_THREAD_COUNT);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         emitQueueCapacity, emitThreadCount, null);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount,
+      @Nullable HttpClient httpClient
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    this.excludedNativeQueryTypes = excludedNativeQueryTypes;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = httpClient != null ? httpClient : HttpClientBuilder.create().build();
+      // Bounded queue: if the queue is full, drop the event rather than blocking the query thread.
+      // A warning is logged on the first drop and every DISCARD_WARNING_INTERVAL drops thereafter.
+      this.emitExecutor = new ThreadPoolExecutor(
+          emitThreadCount,
+          emitThreadCount,
+          60L,
+          TimeUnit.SECONDS,
+          new ArrayBlockingQueue<>(emitQueueCapacity),
+          Execs.makeThreadFactory("OpenLineageEmitter-%d"),
+          new DiscardWithWarningPolicy(discardedEventCount)
+      );
+    } else {
+      this.httpClient = null;
+      this.emitExecutor = null;
+    }
+  }
+
+  // Note: ComposingRequestLogger does not delegate @LifecycleStart to sub-loggers, so this method
+  // may not be called when used in a composing configuration. HTTP URL validation is therefore
+  // performed in the constructor instead. This method is retained for direct lifecycle use.
+  @LifecycleStart
+  @Override
+  public void start()
+  {
+    log.info(
+        "Started OpenLineage %s transport%s",
+        transportType,
+        transportUrl != null ? " to [" + transportUrl + "]" : ""
+    );
+  }
+
+  @LifecycleStop
+  @Override
+  public void stop()
+  {
+    if (emitExecutor != null) {
+      emitExecutor.shutdown();
+      try {
+        if (!emitExecutor.awaitTermination(5, TimeUnit.SECONDS)) {
+          emitExecutor.shutdownNow();
+        }
+      }
+      catch (InterruptedException e) {
+        emitExecutor.shutdownNow();
+        Thread.currentThread().interrupt();
+      }
+    }
+    if (httpClient instanceof Closeable) {
+      CloseableUtils.closeAndSuppressExceptions(
+          (Closeable) httpClient,
+          e -> log.warn(e, "Failed to close OpenLineage HTTP client")
+      );
+    }
+    log.info("Stopped OpenLineage request logger");
+  }
+
+  @Override
+  public void logNativeQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    if (requestLogLine.getQuery() == null) {
+      return;
+    }
+
+    // Skip native sub-queries of a SQL execution to avoid duplicating the SQL-level event.
+    if (requestLogLine.getQuery().getContext().get(BaseQuery.SQL_QUERY_ID) != null) {
+      return;
+    }
+
+    String queryType = requestLogLine.getQuery().getType();
+
+    if (excludedNativeQueryTypes.contains(queryType)) {
+      return;
+    }
+
+    List<String> inputs = new ArrayList<>(new LinkedHashSet<>(requestLogLine.getQuery().getDataSource().getTableNames()));
+    String queryId = requestLogLine.getQuery().getId();
+    if (queryId == null) {
+      log.debug("Native query reached OpenLineage logger without a query ID");
+      queryId = UNKNOWN_QUERY_ID;
+    }
+
+    emit(buildRunEvent(queryId, null, queryType, requestLogLine, inputs, null));
+  }
+
+  @Override
+  public void logSqlQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    String sql = requestLogLine.getSql();
+    List<String> inputs = new ArrayList<>();
+    String output = null;
+
+    if (sql != null) {
+      try {
+        SqlNode parsed = SqlParser.create(sql, SQL_PARSER_CONFIG).parseQuery();
+        inputs = extractInputs(parsed);
+        output = extractOutput(parsed);
+      }
+      catch (SqlParseException e) {
+        // Druid-specific SQL extensions (REPLACE INTO, EXTERN, etc.) may not parse with the
+        // standard Calcite parser. Attempt to extract lineage via regex fallback;
+        // for other unparseable statements, emit the event without table-level lineage.
+        Matcher insertMatcher = INSERT_INTO_PATTERN.matcher(sql);
+        if (insertMatcher.find()) {
+          output = insertMatcher.group(1);
+          // Also try to extract inputs from the SELECT subquery (handles PARTITIONED BY and
+          // other Druid-specific clauses that prevent full parsing). EXTERN-sourced inputs
+          // will fail to re-parse and fall through with empty inputs, which is correct.
+          Matcher selectMatcher = REPLACE_SELECT_PATTERN.matcher(sql);
+          if (selectMatcher.find()) {
+            try {
+              SqlNode selectNode = SqlParser.create(selectMatcher.group(1), SQL_PARSER_CONFIG).parseQuery();
+              inputs = extractInputs(selectNode);
+            }
+            catch (SqlParseException ignored) {
+              // EXTERN or other non-standard sources — inputs left empty
+            }
+          }
+        }
+        Matcher m = REPLACE_INTO_PATTERN.matcher(sql);

Review Comment:
   This matcher will run even when we already have a match from INSERT_INTO_PATTERN. This is unnecessary since INSERT INTO and REPLACE INTO are mutually exclusive patterns 



##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLoggerProvider.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.guice.annotations.Json;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.server.DruidNode;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.server.log.RequestLoggerProvider;
+
+import javax.annotation.Nullable;
+import javax.validation.constraints.NotNull;
+import java.util.Set;
+
+/**
+ *  Configure logging type, namespace, transport type (http or default console), transportUrl in {@code runtime.properties} 
+ */
+@JsonTypeName("openlineage")
+public class OpenLineageRequestLoggerProvider implements RequestLoggerProvider
+{
+  private static final Logger log = new Logger(OpenLineageRequestLoggerProvider.class);
+
+  public enum TransportType
+  {
+    CONSOLE,
+    HTTP
+  }
+
+  @JacksonInject
+  @Json
+  @NotNull
+  private ObjectMapper jsonMapper;
+
+  @JsonProperty
+  @NotNull
+  private String namespace = "druid://" + DruidNode.getDefaultHost();
+
+  @JsonProperty
+  @NotNull
+  private TransportType transportType = TransportType.CONSOLE;
+
+  @Nullable
+  @JsonProperty
+  private String transportUrl;
+
+  @JsonProperty
+  @NotNull
+  private Set<String> excludedNativeQueryTypes = Set.of(
+      "segmentMetadata",
+      "dataSourceMetadata",
+      "timeBoundary"
+  );
+
+  @JsonProperty
+  private int emitQueueCapacity = OpenLineageRequestLogger.DEFAULT_EMIT_QUEUE_CAPACITY;

Review Comment:
   This seems to be configurable but don't see it in the docs



##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLoggerProvider.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.guice.annotations.Json;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.server.DruidNode;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.server.log.RequestLoggerProvider;
+
+import javax.annotation.Nullable;
+import javax.validation.constraints.NotNull;
+import java.util.Set;
+
+/**
+ *  Configure logging type, namespace, transport type (http or default console), transportUrl in {@code runtime.properties} 
+ */
+@JsonTypeName("openlineage")
+public class OpenLineageRequestLoggerProvider implements RequestLoggerProvider
+{
+  private static final Logger log = new Logger(OpenLineageRequestLoggerProvider.class);
+
+  public enum TransportType
+  {
+    CONSOLE,
+    HTTP
+  }
+
+  @JacksonInject
+  @Json
+  @NotNull
+  private ObjectMapper jsonMapper;
+
+  @JsonProperty
+  @NotNull
+  private String namespace = "druid://" + DruidNode.getDefaultHost();
+
+  @JsonProperty
+  @NotNull
+  private TransportType transportType = TransportType.CONSOLE;
+
+  @Nullable
+  @JsonProperty
+  private String transportUrl;
+
+  @JsonProperty
+  @NotNull
+  private Set<String> excludedNativeQueryTypes = Set.of(
+      "segmentMetadata",
+      "dataSourceMetadata",
+      "timeBoundary"
+  );
+
+  @JsonProperty
+  private int emitQueueCapacity = OpenLineageRequestLogger.DEFAULT_EMIT_QUEUE_CAPACITY;
+
+  @JsonProperty
+  private int emitThreadCount = OpenLineageRequestLogger.DEFAULT_EMIT_THREAD_COUNT;

Review Comment:
   This seems to be configurable but don't see it in the docs



##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlNodeList;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.RejectedExecutionHandler;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  // Matches: INSERT INTO <table> ... (fallback when EXTERN prevents standard Calcite parsing)
+  // Handles both quoted ("table") and unquoted (table) identifiers.
+  private static final Pattern INSERT_INTO_PATTERN =
+      Pattern.compile("(?i)^\\s*INSERT\\s+INTO\\s+\"?([^\"\\s]+)\"?");
+  // Matches: REPLACE INTO <table> OVERWRITE ... (Druid-specific, not parseable by standard Calcite)
+  private static final Pattern REPLACE_INTO_PATTERN =
+      Pattern.compile("(?i)^\\s*REPLACE\\s+INTO\\s+\"?([^\"\\s]+)\"?");
+  // Extracts the SELECT subquery from a REPLACE INTO statement for input lineage
+  private static final Pattern REPLACE_SELECT_PATTERN =
+      Pattern.compile("(?is)\\b(SELECT\\s+.+?)(?:\\s+PARTITIONED\\s+BY|\\s+CLUSTERED\\s+BY|$)");
+
+  static final int DEFAULT_EMIT_QUEUE_CAPACITY = 1000;
+  static final int DEFAULT_EMIT_THREAD_COUNT = 1;
+  private static final int DISCARD_WARNING_INTERVAL = 1000;
+
+  static final String UNKNOWN_QUERY_ID = "unknown-query-id";
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  private final Set<String> excludedNativeQueryTypes;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+  private final AtomicLong discardedEventCount = new AtomicLong(0);
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         DEFAULT_EMIT_QUEUE_CAPACITY, DEFAULT_EMIT_THREAD_COUNT);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         emitQueueCapacity, emitThreadCount, null);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount,
+      @Nullable HttpClient httpClient
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    this.excludedNativeQueryTypes = excludedNativeQueryTypes;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = httpClient != null ? httpClient : HttpClientBuilder.create().build();
+      // Bounded queue: if the queue is full, drop the event rather than blocking the query thread.
+      // A warning is logged on the first drop and every DISCARD_WARNING_INTERVAL drops thereafter.
+      this.emitExecutor = new ThreadPoolExecutor(
+          emitThreadCount,
+          emitThreadCount,
+          60L,
+          TimeUnit.SECONDS,
+          new ArrayBlockingQueue<>(emitQueueCapacity),
+          Execs.makeThreadFactory("OpenLineageEmitter-%d"),
+          new DiscardWithWarningPolicy(discardedEventCount)
+      );
+    } else {
+      this.httpClient = null;
+      this.emitExecutor = null;
+    }
+  }
+
+  // Note: ComposingRequestLogger does not delegate @LifecycleStart to sub-loggers, so this method
+  // may not be called when used in a composing configuration. HTTP URL validation is therefore
+  // performed in the constructor instead. This method is retained for direct lifecycle use.
+  @LifecycleStart
+  @Override
+  public void start()
+  {
+    log.info(
+        "Started OpenLineage %s transport%s",
+        transportType,
+        transportUrl != null ? " to [" + transportUrl + "]" : ""
+    );
+  }
+
+  @LifecycleStop
+  @Override
+  public void stop()
+  {
+    if (emitExecutor != null) {
+      emitExecutor.shutdown();
+      try {
+        if (!emitExecutor.awaitTermination(5, TimeUnit.SECONDS)) {
+          emitExecutor.shutdownNow();
+        }
+      }
+      catch (InterruptedException e) {
+        emitExecutor.shutdownNow();
+        Thread.currentThread().interrupt();
+      }
+    }
+    if (httpClient instanceof Closeable) {
+      CloseableUtils.closeAndSuppressExceptions(
+          (Closeable) httpClient,
+          e -> log.warn(e, "Failed to close OpenLineage HTTP client")
+      );
+    }
+    log.info("Stopped OpenLineage request logger");
+  }
+
+  @Override
+  public void logNativeQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    if (requestLogLine.getQuery() == null) {
+      return;
+    }
+
+    // Skip native sub-queries of a SQL execution to avoid duplicating the SQL-level event.
+    if (requestLogLine.getQuery().getContext().get(BaseQuery.SQL_QUERY_ID) != null) {
+      return;
+    }
+
+    String queryType = requestLogLine.getQuery().getType();
+
+    if (excludedNativeQueryTypes.contains(queryType)) {
+      return;
+    }
+
+    List<String> inputs = new ArrayList<>(new LinkedHashSet<>(requestLogLine.getQuery().getDataSource().getTableNames()));
+    String queryId = requestLogLine.getQuery().getId();
+    if (queryId == null) {
+      log.debug("Native query reached OpenLineage logger without a query ID");
+      queryId = UNKNOWN_QUERY_ID;
+    }
+
+    emit(buildRunEvent(queryId, null, queryType, requestLogLine, inputs, null));
+  }
+
+  @Override
+  public void logSqlQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    String sql = requestLogLine.getSql();
+    List<String> inputs = new ArrayList<>();
+    String output = null;
+
+    if (sql != null) {
+      try {
+        SqlNode parsed = SqlParser.create(sql, SQL_PARSER_CONFIG).parseQuery();
+        inputs = extractInputs(parsed);
+        output = extractOutput(parsed);
+      }
+      catch (SqlParseException e) {
+        // Druid-specific SQL extensions (REPLACE INTO, EXTERN, etc.) may not parse with the
+        // standard Calcite parser. Attempt to extract lineage via regex fallback;
+        // for other unparseable statements, emit the event without table-level lineage.
+        Matcher insertMatcher = INSERT_INTO_PATTERN.matcher(sql);
+        if (insertMatcher.find()) {
+          output = insertMatcher.group(1);
+          // Also try to extract inputs from the SELECT subquery (handles PARTITIONED BY and
+          // other Druid-specific clauses that prevent full parsing). EXTERN-sourced inputs
+          // will fail to re-parse and fall through with empty inputs, which is correct.
+          Matcher selectMatcher = REPLACE_SELECT_PATTERN.matcher(sql);
+          if (selectMatcher.find()) {
+            try {
+              SqlNode selectNode = SqlParser.create(selectMatcher.group(1), SQL_PARSER_CONFIG).parseQuery();
+              inputs = extractInputs(selectNode);
+            }
+            catch (SqlParseException ignored) {
+              // EXTERN or other non-standard sources — inputs left empty
+            }
+          }
+        }
+        Matcher m = REPLACE_INTO_PATTERN.matcher(sql);
+        if (m.find()) {
+          output = m.group(1);
+          Matcher selectMatcher = REPLACE_SELECT_PATTERN.matcher(sql);
+          if (selectMatcher.find()) {
+            try {
+              SqlNode selectNode = SqlParser.create(selectMatcher.group(1), SQL_PARSER_CONFIG).parseQuery();
+              inputs = extractInputs(selectNode);
+            }
+            catch (SqlParseException ignored) {
+              // fall through with empty inputs
+            }
+          }
+        }
+        log.debug(
+            "OpenLineage: could not parse SQL with standard Calcite parser (query will still be emitted): %s",
+            e.getMessage()
+        );
+      }
+    }
+
+    String queryId = extractSqlQueryId(requestLogLine);
+    emit(buildRunEvent(queryId, sql, "sql", requestLogLine, new ArrayList<>(new LinkedHashSet<>(inputs)), output));
+  }
+
+  private ObjectNode buildRunEvent(
+      String queryId,
+      @Nullable String sql,
+      String queryType,
+      RequestLogLine requestLogLine,
+      List<String> inputs,
+      @Nullable String output
+  )
+  {
+    Map<String, Object> stats = requestLogLine.getQueryStats().getStats();
+    boolean success = Boolean.TRUE.equals(stats.get("success"));
+
+    ObjectNode event = jsonMapper.createObjectNode();
+    event.put("eventType", success ? "COMPLETE" : "FAIL");
+    event.put("eventTime", requestLogLine.getTimestamp().toInstant().toString());
+    event.put("producer", PRODUCER);
+    event.put("schemaURL", SCHEMA_URL);
+    event.set("run", buildRun(queryId, queryType, requestLogLine, stats, success));
+    event.set("job", buildJob(queryId, sql));
+    event.set("inputs", buildDatasets(inputs));
+    event.set("outputs", buildDatasets(output != null ? List.of(output) : List.of()));
+    return event;
+  }
+
+  private ObjectNode buildRun(
+      String queryId,
+      String queryType,
+      RequestLogLine requestLogLine,
+      Map<String, Object> stats,
+      boolean success
+  )
+  {
+    ObjectNode run = jsonMapper.createObjectNode();
+    run.put("runId", UUID.nameUUIDFromBytes(queryId.getBytes(StandardCharsets.UTF_8)).toString());
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode engineFacet = createFacet(ENGINE_FACET_SCHEMA_URL);
+    engineFacet.put("name", "druid");
+    engineFacet.put("version", getDruidVersion());
+    facets.set("processing_engine", engineFacet);
+
+    ObjectNode contextFacet = createFacet(null);
+    contextFacet.put("queryType", queryType);
+    contextFacet.put("remoteAddress", requestLogLine.getRemoteAddr());
+    Object identity = stats.get("identity");
+    if (identity != null) {
+      contextFacet.put("identity", identity.toString());
+    }
+    Object nativeQueryIds = requestLogLine.getSqlQueryContext().get("nativeQueryIds");

Review Comment:
   getSqlQueryContext is Nullable



##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlNodeList;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.RejectedExecutionHandler;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  // Matches: INSERT INTO <table> ... (fallback when EXTERN prevents standard Calcite parsing)
+  // Handles both quoted ("table") and unquoted (table) identifiers.
+  private static final Pattern INSERT_INTO_PATTERN =
+      Pattern.compile("(?i)^\\s*INSERT\\s+INTO\\s+\"?([^\"\\s]+)\"?");
+  // Matches: REPLACE INTO <table> OVERWRITE ... (Druid-specific, not parseable by standard Calcite)
+  private static final Pattern REPLACE_INTO_PATTERN =
+      Pattern.compile("(?i)^\\s*REPLACE\\s+INTO\\s+\"?([^\"\\s]+)\"?");
+  // Extracts the SELECT subquery from a REPLACE INTO statement for input lineage
+  private static final Pattern REPLACE_SELECT_PATTERN =
+      Pattern.compile("(?is)\\b(SELECT\\s+.+?)(?:\\s+PARTITIONED\\s+BY|\\s+CLUSTERED\\s+BY|$)");
+
+  static final int DEFAULT_EMIT_QUEUE_CAPACITY = 1000;
+  static final int DEFAULT_EMIT_THREAD_COUNT = 1;
+  private static final int DISCARD_WARNING_INTERVAL = 1000;
+
+  static final String UNKNOWN_QUERY_ID = "unknown-query-id";
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  private final Set<String> excludedNativeQueryTypes;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+  private final AtomicLong discardedEventCount = new AtomicLong(0);
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         DEFAULT_EMIT_QUEUE_CAPACITY, DEFAULT_EMIT_THREAD_COUNT);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         emitQueueCapacity, emitThreadCount, null);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount,
+      @Nullable HttpClient httpClient
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    this.excludedNativeQueryTypes = excludedNativeQueryTypes;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = httpClient != null ? httpClient : HttpClientBuilder.create().build();

Review Comment:
   Might also want to make properties like trustStorePath, trustStorePassword, keyStorePath, keyStorePassword configurable. Something similar to InfluxdbEmitterConfig



##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlNodeList;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.RejectedExecutionHandler;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  // Matches: INSERT INTO <table> ... (fallback when EXTERN prevents standard Calcite parsing)
+  // Handles both quoted ("table") and unquoted (table) identifiers.
+  private static final Pattern INSERT_INTO_PATTERN =
+      Pattern.compile("(?i)^\\s*INSERT\\s+INTO\\s+\"?([^\"\\s]+)\"?");
+  // Matches: REPLACE INTO <table> OVERWRITE ... (Druid-specific, not parseable by standard Calcite)
+  private static final Pattern REPLACE_INTO_PATTERN =
+      Pattern.compile("(?i)^\\s*REPLACE\\s+INTO\\s+\"?([^\"\\s]+)\"?");
+  // Extracts the SELECT subquery from a REPLACE INTO statement for input lineage
+  private static final Pattern REPLACE_SELECT_PATTERN =
+      Pattern.compile("(?is)\\b(SELECT\\s+.+?)(?:\\s+PARTITIONED\\s+BY|\\s+CLUSTERED\\s+BY|$)");
+
+  static final int DEFAULT_EMIT_QUEUE_CAPACITY = 1000;
+  static final int DEFAULT_EMIT_THREAD_COUNT = 1;
+  private static final int DISCARD_WARNING_INTERVAL = 1000;
+
+  static final String UNKNOWN_QUERY_ID = "unknown-query-id";
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  private final Set<String> excludedNativeQueryTypes;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+  private final AtomicLong discardedEventCount = new AtomicLong(0);
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         DEFAULT_EMIT_QUEUE_CAPACITY, DEFAULT_EMIT_THREAD_COUNT);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         emitQueueCapacity, emitThreadCount, null);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount,
+      @Nullable HttpClient httpClient
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    this.excludedNativeQueryTypes = excludedNativeQueryTypes;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = httpClient != null ? httpClient : HttpClientBuilder.create().build();

Review Comment:
   Generally best practice to be able to set and have reasonable defaults for socket, connection, Connection Request timeouts



##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlNodeList;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.RejectedExecutionHandler;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  // Matches: INSERT INTO <table> ... (fallback when EXTERN prevents standard Calcite parsing)
+  // Handles both quoted ("table") and unquoted (table) identifiers.
+  private static final Pattern INSERT_INTO_PATTERN =
+      Pattern.compile("(?i)^\\s*INSERT\\s+INTO\\s+\"?([^\"\\s]+)\"?");
+  // Matches: REPLACE INTO <table> OVERWRITE ... (Druid-specific, not parseable by standard Calcite)
+  private static final Pattern REPLACE_INTO_PATTERN =
+      Pattern.compile("(?i)^\\s*REPLACE\\s+INTO\\s+\"?([^\"\\s]+)\"?");
+  // Extracts the SELECT subquery from a REPLACE INTO statement for input lineage
+  private static final Pattern REPLACE_SELECT_PATTERN =
+      Pattern.compile("(?is)\\b(SELECT\\s+.+?)(?:\\s+PARTITIONED\\s+BY|\\s+CLUSTERED\\s+BY|$)");
+
+  static final int DEFAULT_EMIT_QUEUE_CAPACITY = 1000;
+  static final int DEFAULT_EMIT_THREAD_COUNT = 1;
+  private static final int DISCARD_WARNING_INTERVAL = 1000;
+
+  static final String UNKNOWN_QUERY_ID = "unknown-query-id";
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  private final Set<String> excludedNativeQueryTypes;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+  private final AtomicLong discardedEventCount = new AtomicLong(0);
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         DEFAULT_EMIT_QUEUE_CAPACITY, DEFAULT_EMIT_THREAD_COUNT);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         emitQueueCapacity, emitThreadCount, null);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount,
+      @Nullable HttpClient httpClient
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    this.excludedNativeQueryTypes = excludedNativeQueryTypes;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = httpClient != null ? httpClient : HttpClientBuilder.create().build();
+      // Bounded queue: if the queue is full, drop the event rather than blocking the query thread.
+      // A warning is logged on the first drop and every DISCARD_WARNING_INTERVAL drops thereafter.
+      this.emitExecutor = new ThreadPoolExecutor(
+          emitThreadCount,
+          emitThreadCount,
+          60L,
+          TimeUnit.SECONDS,
+          new ArrayBlockingQueue<>(emitQueueCapacity),
+          Execs.makeThreadFactory("OpenLineageEmitter-%d"),
+          new DiscardWithWarningPolicy(discardedEventCount)
+      );
+    } else {
+      this.httpClient = null;
+      this.emitExecutor = null;
+    }
+  }
+
+  // Note: ComposingRequestLogger does not delegate @LifecycleStart to sub-loggers, so this method
+  // may not be called when used in a composing configuration. HTTP URL validation is therefore
+  // performed in the constructor instead. This method is retained for direct lifecycle use.
+  @LifecycleStart
+  @Override
+  public void start()
+  {
+    log.info(
+        "Started OpenLineage %s transport%s",
+        transportType,
+        transportUrl != null ? " to [" + transportUrl + "]" : ""
+    );
+  }
+
+  @LifecycleStop
+  @Override
+  public void stop()
+  {
+    if (emitExecutor != null) {
+      emitExecutor.shutdown();
+      try {
+        if (!emitExecutor.awaitTermination(5, TimeUnit.SECONDS)) {
+          emitExecutor.shutdownNow();
+        }
+      }
+      catch (InterruptedException e) {
+        emitExecutor.shutdownNow();
+        Thread.currentThread().interrupt();
+      }
+    }
+    if (httpClient instanceof Closeable) {
+      CloseableUtils.closeAndSuppressExceptions(
+          (Closeable) httpClient,
+          e -> log.warn(e, "Failed to close OpenLineage HTTP client")
+      );
+    }
+    log.info("Stopped OpenLineage request logger");
+  }
+
+  @Override
+  public void logNativeQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    if (requestLogLine.getQuery() == null) {
+      return;
+    }
+
+    // Skip native sub-queries of a SQL execution to avoid duplicating the SQL-level event.
+    if (requestLogLine.getQuery().getContext().get(BaseQuery.SQL_QUERY_ID) != null) {
+      return;
+    }
+
+    String queryType = requestLogLine.getQuery().getType();
+
+    if (excludedNativeQueryTypes.contains(queryType)) {
+      return;
+    }
+
+    List<String> inputs = new ArrayList<>(new LinkedHashSet<>(requestLogLine.getQuery().getDataSource().getTableNames()));
+    String queryId = requestLogLine.getQuery().getId();
+    if (queryId == null) {
+      log.debug("Native query reached OpenLineage logger without a query ID");
+      queryId = UNKNOWN_QUERY_ID;
+    }
+
+    emit(buildRunEvent(queryId, null, queryType, requestLogLine, inputs, null));
+  }
+
+  @Override
+  public void logSqlQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    String sql = requestLogLine.getSql();
+    List<String> inputs = new ArrayList<>();
+    String output = null;
+
+    if (sql != null) {
+      try {
+        SqlNode parsed = SqlParser.create(sql, SQL_PARSER_CONFIG).parseQuery();
+        inputs = extractInputs(parsed);
+        output = extractOutput(parsed);
+      }
+      catch (SqlParseException e) {
+        // Druid-specific SQL extensions (REPLACE INTO, EXTERN, etc.) may not parse with the
+        // standard Calcite parser. Attempt to extract lineage via regex fallback;
+        // for other unparseable statements, emit the event without table-level lineage.
+        Matcher insertMatcher = INSERT_INTO_PATTERN.matcher(sql);
+        if (insertMatcher.find()) {
+          output = insertMatcher.group(1);
+          // Also try to extract inputs from the SELECT subquery (handles PARTITIONED BY and
+          // other Druid-specific clauses that prevent full parsing). EXTERN-sourced inputs
+          // will fail to re-parse and fall through with empty inputs, which is correct.
+          Matcher selectMatcher = REPLACE_SELECT_PATTERN.matcher(sql);
+          if (selectMatcher.find()) {
+            try {
+              SqlNode selectNode = SqlParser.create(selectMatcher.group(1), SQL_PARSER_CONFIG).parseQuery();
+              inputs = extractInputs(selectNode);
+            }
+            catch (SqlParseException ignored) {
+              // EXTERN or other non-standard sources — inputs left empty
+            }
+          }
+        }
+        Matcher m = REPLACE_INTO_PATTERN.matcher(sql);
+        if (m.find()) {
+          output = m.group(1);
+          Matcher selectMatcher = REPLACE_SELECT_PATTERN.matcher(sql);
+          if (selectMatcher.find()) {
+            try {
+              SqlNode selectNode = SqlParser.create(selectMatcher.group(1), SQL_PARSER_CONFIG).parseQuery();
+              inputs = extractInputs(selectNode);
+            }
+            catch (SqlParseException ignored) {
+              // fall through with empty inputs
+            }
+          }
+        }
+        log.debug(
+            "OpenLineage: could not parse SQL with standard Calcite parser (query will still be emitted): %s",
+            e.getMessage()
+        );
+      }
+    }
+
+    String queryId = extractSqlQueryId(requestLogLine);
+    emit(buildRunEvent(queryId, sql, "sql", requestLogLine, new ArrayList<>(new LinkedHashSet<>(inputs)), output));
+  }
+
+  private ObjectNode buildRunEvent(
+      String queryId,
+      @Nullable String sql,
+      String queryType,
+      RequestLogLine requestLogLine,
+      List<String> inputs,
+      @Nullable String output
+  )
+  {
+    Map<String, Object> stats = requestLogLine.getQueryStats().getStats();
+    boolean success = Boolean.TRUE.equals(stats.get("success"));
+
+    ObjectNode event = jsonMapper.createObjectNode();
+    event.put("eventType", success ? "COMPLETE" : "FAIL");
+    event.put("eventTime", requestLogLine.getTimestamp().toInstant().toString());
+    event.put("producer", PRODUCER);
+    event.put("schemaURL", SCHEMA_URL);
+    event.set("run", buildRun(queryId, queryType, requestLogLine, stats, success));
+    event.set("job", buildJob(queryId, sql));
+    event.set("inputs", buildDatasets(inputs));
+    event.set("outputs", buildDatasets(output != null ? List.of(output) : List.of()));
+    return event;
+  }
+
+  private ObjectNode buildRun(
+      String queryId,
+      String queryType,
+      RequestLogLine requestLogLine,
+      Map<String, Object> stats,
+      boolean success
+  )
+  {
+    ObjectNode run = jsonMapper.createObjectNode();
+    run.put("runId", UUID.nameUUIDFromBytes(queryId.getBytes(StandardCharsets.UTF_8)).toString());
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode engineFacet = createFacet(ENGINE_FACET_SCHEMA_URL);
+    engineFacet.put("name", "druid");
+    engineFacet.put("version", getDruidVersion());
+    facets.set("processing_engine", engineFacet);
+
+    ObjectNode contextFacet = createFacet(null);
+    contextFacet.put("queryType", queryType);
+    contextFacet.put("remoteAddress", requestLogLine.getRemoteAddr());
+    Object identity = stats.get("identity");
+    if (identity != null) {
+      contextFacet.put("identity", identity.toString());
+    }
+    Object nativeQueryIds = requestLogLine.getSqlQueryContext().get("nativeQueryIds");
+    if (nativeQueryIds != null) {
+      contextFacet.put("nativeQueryIds", nativeQueryIds.toString());
+    }
+    facets.set("druid_query_context", contextFacet);
+
+    ObjectNode statsFacet = createFacet(null);
+    putLongStat(statsFacet, "durationMs", stats, "sqlQuery/time", "query/time");
+    putLongStat(statsFacet, "bytes", stats, "sqlQuery/bytes", "query/bytes");
+    putLongStat(statsFacet, "planningTimeMs", stats, "sqlQuery/planningTimeMs");
+    Object statusCode = stats.get("statusCode");
+    if (statusCode != null) {
+      statsFacet.put("statusCode", statusCode.toString());
+    }
+    facets.set("druid_query_statistics", statsFacet);
+
+    if (!success) {
+      Object exception = stats.get("exception");
+      if (exception != null) {
+        ObjectNode errorFacet = createFacet(ERROR_FACET_SCHEMA_URL);
+        errorFacet.put("message", exception.toString());
+        if ("sql".equals(queryType)) {
+          errorFacet.put("programmingLanguage", "SQL");
+        }
+        facets.set("errorMessage", errorFacet);
+      }
+    }
+
+    run.set("facets", facets);
+    return run;
+  }
+
+  private ObjectNode buildJob(String queryId, @Nullable String sql)
+  {
+    ObjectNode job = jsonMapper.createObjectNode();
+    job.put("namespace", namespace);
+    job.put("name", queryId);
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode jobTypeFacet = createFacet(JOB_TYPE_FACET_SCHEMA_URL);
+    jobTypeFacet.put("processingType", "BATCH");
+    jobTypeFacet.put("integration", "DRUID");
+    jobTypeFacet.put("jobType", "QUERY");
+    facets.set("jobType", jobTypeFacet);
+
+    if (sql != null) {
+      ObjectNode sqlFacet = createFacet(SQL_FACET_SCHEMA_URL);
+      sqlFacet.put("query", sql);
+      facets.set("sql", sqlFacet);
+    }
+
+    job.set("facets", facets);
+    return job;
+  }
+
+  private ObjectNode createFacet(@Nullable String schemaUrl)
+  {
+    ObjectNode facet = jsonMapper.createObjectNode();
+    facet.put("_producer", PRODUCER);
+    if (schemaUrl != null) {
+      facet.put("_schemaURL", schemaUrl);
+    }
+    return facet;
+  }
+
+  private ArrayNode buildDatasets(List<String> tableNames)
+  {
+    ArrayNode array = jsonMapper.createArrayNode();
+    for (String name : tableNames) {
+      ObjectNode node = jsonMapper.createObjectNode();
+      node.put("namespace", namespace);
+      node.put("name", name);
+      node.set("facets", jsonMapper.createObjectNode());
+      array.add(node);
+    }
+    return array;
+  }
+
+  private List<String> extractInputs(SqlNode root)
+  {
+    List<String> tables = new ArrayList<>();
+    if (root instanceof SqlInsert) {
+      // For INSERT/REPLACE, only walk the source query — the target table is an output, not an input.
+      collectTableNames(((SqlInsert) root).getSource(), tables, Set.of());
+    } else {
+      collectTableNames(root, tables, Set.of());
+    }
+    return tables;
+  }
+
+  @Nullable
+  private String extractOutput(SqlNode root)
+  {
+    if (root instanceof SqlInsert) {
+      SqlNode target = ((SqlInsert) root).getTargetTable();
+      if (target instanceof SqlIdentifier) {
+        return String.join(".", ((SqlIdentifier) target).names);
+      }
+    }
+    return null;
+  }
+
+  /**
+   * Walks the SQL tree to collect table references from FROM clauses, JOINs, subqueries in
+   * WHERE/HAVING/SELECT, and set operations (UNION/INTERSECT/EXCEPT). CTE alias names are
+   * tracked in {@code excludeNames} to avoid reporting them as table inputs.
+   *
+   * <p>The walker distinguishes table-reference positions (FROM, JOIN) from column-reference
+   * positions (SELECT list, WHERE expressions) by collecting {@link SqlIdentifier} nodes only
+   * from known table-reference contexts, while recursing into all subqueries to find nested
+   * FROM clauses.
+   */
+  private void collectTableNames(SqlNode node, List<String> tables, Set<String> excludeNames)
+  {
+    if (node == null) {
+      return;
+    }
+    if (node instanceof SqlWith) {
+      SqlWith with = (SqlWith) node;
+      Set<String> innerExcludes = new HashSet<>(excludeNames);
+      for (SqlNode item : with.withList) {
+        if (item instanceof SqlWithItem) {
+          innerExcludes.add(((SqlWithItem) item).name.getSimple());
+          collectTableNames(((SqlWithItem) item).query, tables, innerExcludes);
+        }
+      }
+      collectTableNames(with.body, tables, innerExcludes);
+    } else if (node instanceof SqlNodeList) {
+      // Lists (e.g., SELECT list items): recurse into each element to find subqueries.
+      for (SqlNode item : (SqlNodeList) node) {
+        collectTableNames(item, tables, excludeNames);
+      }
+    } else if (node instanceof SqlSelect) {
+      SqlSelect select = (SqlSelect) node;
+      // FROM clause: identifiers here are table references.
+      collectFromPosition(select.getFrom(), tables, excludeNames);
+      // Recurse into WHERE, HAVING, and SELECT list to find nested subqueries.
+      collectTableNames(select.getWhere(), tables, excludeNames);
+      collectTableNames(select.getHaving(), tables, excludeNames);
+      collectTableNames(select.getSelectList(), tables, excludeNames);
+    } else if (node instanceof SqlCall) {
+      SqlCall call = (SqlCall) node;
+      if (call.getKind() == SqlKind.UNION || call.getKind() == SqlKind.INTERSECT || call.getKind() == SqlKind.EXCEPT) {
+        // Set operations: each operand is a SELECT whose FROM clause has table references.
+        for (SqlNode operand : call.getOperandList()) {
+          collectTableNames(operand, tables, excludeNames);
+        }
+      } else {
+        // Other expressions (AND, OR, IN, =, function calls, etc.): recurse to find subqueries.
+        for (SqlNode operand : call.getOperandList()) {
+          if (operand instanceof SqlSelect || operand instanceof SqlWith || operand instanceof SqlCall) {
+            collectTableNames(operand, tables, excludeNames);
+          }
+        }
+      }
+    }
+  }
+
+  /**
+   * Collects table names from a FROM-clause position, where {@link SqlIdentifier} nodes
+   * represent table references (not column references).
+   */
+  private void collectFromPosition(SqlNode node, List<String> tables, Set<String> excludeNames)
+  {
+    if (node == null) {
+      return;
+    }
+    if (node instanceof SqlIdentifier) {
+      String name = String.join(".", ((SqlIdentifier) node).names);
+      if (!excludeNames.contains(name)) {
+        tables.add(name);
+      }
+    } else if (node instanceof SqlJoin) {
+      SqlJoin join = (SqlJoin) node;
+      collectFromPosition(join.getLeft(), tables, excludeNames);
+      collectFromPosition(join.getRight(), tables, excludeNames);
+    } else if (node instanceof SqlSelect) {
+      // Subquery in FROM: recurse into the full select.
+      collectTableNames(node, tables, excludeNames);
+    } else if (node instanceof SqlBasicCall && node.getKind() == SqlKind.AS) {
+      // Alias: the table reference is the first operand.
+      collectFromPosition(((SqlBasicCall) node).operand(0), tables, excludeNames);
+    } else if (node instanceof SqlCall) {
+      // LATERAL, UNNEST, or other FROM-position nodes: walk operands.
+      for (SqlNode operand : ((SqlCall) node).getOperandList()) {
+        collectFromPosition(operand, tables, excludeNames);
+      }
+    }
+  }
+
+  protected void emit(ObjectNode event)
+  {
+    try {
+      String json = jsonMapper.writeValueAsString(event);
+      if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+        emitExecutor.submit(() -> emitHttp(json));
+      } else {
+        log.debug("OpenLineage event: %s", json);
+      }
+    }
+    catch (IOException e) {
+      log.error(e, "Failed to serialize OpenLineage event");
+    }
+  }
+
+  private void emitHttp(String json)
+  {
+    HttpPost post = new HttpPost(transportUrl);
+    post.setEntity(new StringEntity(json, ContentType.APPLICATION_JSON));
+    try {
+      org.apache.http.HttpResponse response = httpClient.execute(post);
+      int statusCode = response.getStatusLine().getStatusCode();
+      if (statusCode < 200 || statusCode >= 300) {
+        log.warn(
+            "OpenLineage HTTP transport received non-2xx response [%d] from [%s]; event may have been dropped",
+            statusCode,
+            transportUrl
+        );
+      }
+      EntityUtils.consumeQuietly(response.getEntity());
+    }
+    catch (IOException e) {
+      log.error(e, "Failed to POST OpenLineage event to [%s]", transportUrl);
+    }
+    finally {
+      post.releaseConnection();
+    }
+  }
+
+  private String extractSqlQueryId(RequestLogLine requestLogLine)
+  {
+    Object id = requestLogLine.getSqlQueryContext().get(BaseQuery.SQL_QUERY_ID);

Review Comment:
   getSqlQueryContext is Nullable



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["mshahid6 (via GitHub)" <gi...@apache.org>]

mshahid6 commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3299891503


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,449 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.RejectedExecutionHandler;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  static final int DEFAULT_EMIT_QUEUE_CAPACITY = 1000;
+  static final int DEFAULT_EMIT_THREAD_COUNT = 1;
+  private static final int DISCARD_WARNING_INTERVAL = 1000;
+
+  static final String UNKNOWN_QUERY_ID = "unknown-query-id";
+
+  // Matches the output table of MSQ DML: INSERT INTO "foo" or REPLACE INTO foo
+  private static final Pattern MSQ_DML_OUTPUT_PATTERN = Pattern.compile(
+      "(?si)^\\s*(?:INSERT|REPLACE)\\s+INTO\\s+\"?([^\"\\s(,]+)\"?"

Review Comment:
   updated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["jtuglu1 (via GitHub)" <gi...@apache.org>]

jtuglu1 commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3126580931


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,503 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOrderBy;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  private static final int EMIT_QUEUE_CAPACITY = 1000;
+  private static final int EMIT_THREAD_COUNT = 1;
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  private final Set<String> excludedNativeQueryTypes;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes, null);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      @Nullable HttpClient httpClient
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    this.excludedNativeQueryTypes = excludedNativeQueryTypes;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = httpClient != null ? httpClient : HttpClientBuilder.create().build();
+      // Bounded queue: if the queue is full, silently drop the event rather than blocking

Review Comment:
   let's not silently drop events without a warning – is there a callback, etc. we can use with an atomic counter or similar? Perhaps overload ThreadPoolExecutor.DiscardPolicy() to increment an atomic, and then we can periodically poll that on emit() or similar.



##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,503 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOrderBy;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  private static final int EMIT_QUEUE_CAPACITY = 1000;

Review Comment:
   we should probably make these configurable



##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLoggerProvider.java:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.guice.annotations.Json;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.server.DruidNode;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.server.log.RequestLoggerProvider;
+
+import javax.annotation.Nullable;
+import javax.validation.constraints.NotNull;
+import java.util.Set;
+
+/**
+ *  Configure logging type, namespace, transport type (http or default console), transportUrl in {@code runtime.properties} 
+ */
+@JsonTypeName("openlineage")
+public class OpenLineageRequestLoggerProvider implements RequestLoggerProvider
+{
+  private static final Logger log = new Logger(OpenLineageRequestLoggerProvider.class);
+
+  public enum TransportType
+  {
+    CONSOLE,
+    HTTP
+  }
+
+  @JacksonInject
+  @Json
+  @NotNull
+  private ObjectMapper jsonMapper;
+
+  @JsonProperty
+  @NotNull
+  private String namespace = "druid://" + DruidNode.getDefaultHost();
+
+  @JsonProperty
+  @NotNull
+  private TransportType transportType = TransportType.CONSOLE;
+
+  @Nullable
+  @JsonProperty
+  private String transportUrl;
+
+  @JsonProperty
+  @NotNull
+  private Set<String> excludedNativeQueryTypes = Set.of(
+      "segmentMetadata",
+      "dataSourceMetadata",
+      "timeBoundary"

Review Comment:
   Why excluding these?



##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,503 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOrderBy;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  private static final int EMIT_QUEUE_CAPACITY = 1000;
+  private static final int EMIT_THREAD_COUNT = 1;
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  private final Set<String> excludedNativeQueryTypes;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes, null);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      @Nullable HttpClient httpClient
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    this.excludedNativeQueryTypes = excludedNativeQueryTypes;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = httpClient != null ? httpClient : HttpClientBuilder.create().build();
+      // Bounded queue: if the queue is full, silently drop the event rather than blocking
+      // the query thread. Uses Druid's Execs for daemon thread naming conventions.
+      this.emitExecutor = new ThreadPoolExecutor(
+          EMIT_THREAD_COUNT,
+          EMIT_THREAD_COUNT,
+          60L,
+          TimeUnit.SECONDS,
+          new ArrayBlockingQueue<>(EMIT_QUEUE_CAPACITY),
+          Execs.makeThreadFactory("OpenLineageEmitter-%d"),
+          new ThreadPoolExecutor.DiscardPolicy()
+      );
+    } else {
+      this.httpClient = null;
+      this.emitExecutor = null;
+    }
+  }
+
+  @LifecycleStart
+  @Override
+  public void start() throws Exception
+  {
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      log.info("Started OpenLineage HTTP transport to [%s]", transportUrl);
+    } else {
+      log.info("Started OpenLineage console transport");
+    }
+  }
+
+  @LifecycleStop
+  @Override
+  public void stop()
+  {
+    if (emitExecutor != null) {
+      emitExecutor.shutdown();
+      try {
+        if (!emitExecutor.awaitTermination(5, TimeUnit.SECONDS)) {
+          emitExecutor.shutdownNow();
+        }
+      }
+      catch (InterruptedException e) {
+        emitExecutor.shutdownNow();
+        Thread.currentThread().interrupt();
+      }
+    }
+    if (httpClient instanceof Closeable) {
+      CloseableUtils.closeAndSuppressExceptions(
+          (Closeable) httpClient,
+          e -> log.warn(e, "Failed to close OpenLineage HTTP client")
+      );
+    }
+    log.info("Stopped OpenLineage request logger");
+  }
+
+  @Override
+  public void logNativeQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    if (requestLogLine.getQuery() == null) {
+      return;
+    }
+
+    // Skip native sub-queries of a SQL execution to avoid duplicating the SQL-level event.
+    if (requestLogLine.getQuery().getContext().get(BaseQuery.SQL_QUERY_ID) != null) {
+      return;
+    }
+
+    String queryType = requestLogLine.getQuery().getType();
+
+    if (excludedNativeQueryTypes.contains(queryType)) {
+      return;
+    }
+
+    List<String> inputs = new ArrayList<>(new LinkedHashSet<>(requestLogLine.getQuery().getDataSource().getTableNames()));
+    String queryId = requestLogLine.getQuery().getId();
+    if (queryId == null) {
+      queryId = UUID.randomUUID().toString();
+    }
+
+    emit(buildRunEvent(queryId, null, queryType, requestLogLine, inputs, null));
+  }
+
+  @Override
+  public void logSqlQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    String sql = requestLogLine.getSql();
+    List<String> inputs = new ArrayList<>();
+    String output = null;
+
+    if (sql != null) {
+      try {
+        SqlNode parsed = SqlParser.create(sql, SQL_PARSER_CONFIG).parseQuery();
+        inputs = extractInputs(parsed);
+        output = extractOutput(parsed);
+      }
+      catch (SqlParseException e) {
+        // Druid-specific SQL extensions (REPLACE, EXTERN, etc.) may not parse with the standard
+        // Calcite parser. Emit the event without table-level lineage rather than failing.
+        log.debug(
+            "OpenLineage: could not parse SQL for lineage extraction (query will still be emitted): %s",
+            e.getMessage()
+        );
+      }
+    }
+
+    String queryId = extractSqlQueryId(requestLogLine);
+    emit(buildRunEvent(queryId, sql, "sql", requestLogLine, new ArrayList<>(new LinkedHashSet<>(inputs)), output));
+  }
+
+  private ObjectNode buildRunEvent(
+      String queryId,
+      @Nullable String sql,
+      String queryType,
+      RequestLogLine requestLogLine,
+      List<String> inputs,
+      @Nullable String output
+  )
+  {
+    Map<String, Object> stats = requestLogLine.getQueryStats().getStats();
+    boolean success = Boolean.TRUE.equals(stats.get("success"));
+
+    ObjectNode event = jsonMapper.createObjectNode();
+    event.put("eventType", success ? "COMPLETE" : "FAIL");
+    event.put("eventTime", requestLogLine.getTimestamp().toInstant().toString());
+    event.put("producer", PRODUCER);
+    event.put("schemaURL", SCHEMA_URL);
+    event.set("run", buildRun(queryId, queryType, requestLogLine, stats, success));
+    event.set("job", buildJob(queryId, sql));
+    event.set("inputs", buildDatasets(inputs));
+    event.set("outputs", buildDatasets(output != null ? List.of(output) : List.of()));
+    return event;
+  }
+
+  private ObjectNode buildRun(
+      String queryId,
+      String queryType,
+      RequestLogLine requestLogLine,
+      Map<String, Object> stats,
+      boolean success
+  )
+  {
+    ObjectNode run = jsonMapper.createObjectNode();
+    run.put("runId", UUID.nameUUIDFromBytes(queryId.getBytes(StandardCharsets.UTF_8)).toString());
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode engineFacet = createFacet(ENGINE_FACET_SCHEMA_URL);
+    engineFacet.put("name", "druid");
+    engineFacet.put("version", getDruidVersion());
+    facets.set("processing_engine", engineFacet);
+
+    ObjectNode contextFacet = createFacet(null);
+    contextFacet.put("queryType", queryType);
+    contextFacet.put("remoteAddress", requestLogLine.getRemoteAddr());
+    Object identity = stats.get("identity");
+    if (identity != null) {
+      contextFacet.put("identity", identity.toString());
+    }
+    Object nativeQueryIds = requestLogLine.getSqlQueryContext().get("nativeQueryIds");
+    if (nativeQueryIds != null) {
+      contextFacet.put("nativeQueryIds", nativeQueryIds.toString());
+    }
+    facets.set("druid_query_context", contextFacet);
+
+    ObjectNode statsFacet = createFacet(null);
+    putLongStat(statsFacet, "durationMs", stats, "sqlQuery/time", "query/time");
+    putLongStat(statsFacet, "bytes", stats, "sqlQuery/bytes", "query/bytes");
+    putLongStat(statsFacet, "planningTimeMs", stats, "sqlQuery/planningTimeMs");
+    Object statusCode = stats.get("statusCode");
+    if (statusCode != null) {
+      statsFacet.put("statusCode", statusCode.toString());
+    }
+    facets.set("druid_query_statistics", statsFacet);
+
+    if (!success) {
+      Object exception = stats.get("exception");
+      if (exception != null) {
+        ObjectNode errorFacet = createFacet(ERROR_FACET_SCHEMA_URL);
+        errorFacet.put("message", exception.toString());
+        if ("sql".equals(queryType)) {
+          errorFacet.put("programmingLanguage", "SQL");
+        }
+        facets.set("errorMessage", errorFacet);
+      }
+    }
+
+    run.set("facets", facets);
+    return run;
+  }
+
+  private ObjectNode buildJob(String queryId, @Nullable String sql)
+  {
+    ObjectNode job = jsonMapper.createObjectNode();
+    job.put("namespace", namespace);
+    job.put("name", queryId);
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode jobTypeFacet = createFacet(JOB_TYPE_FACET_SCHEMA_URL);
+    jobTypeFacet.put("processingType", "BATCH");
+    jobTypeFacet.put("integration", "DRUID");
+    jobTypeFacet.put("jobType", "QUERY");
+    facets.set("jobType", jobTypeFacet);
+
+    if (sql != null) {
+      ObjectNode sqlFacet = createFacet(SQL_FACET_SCHEMA_URL);
+      sqlFacet.put("query", sql);
+      facets.set("sql", sqlFacet);
+    }
+
+    job.set("facets", facets);
+    return job;
+  }
+
+  private ObjectNode createFacet(@Nullable String schemaUrl)
+  {
+    ObjectNode facet = jsonMapper.createObjectNode();
+    facet.put("_producer", PRODUCER);
+    if (schemaUrl != null) {
+      facet.put("_schemaURL", schemaUrl);
+    }
+    return facet;
+  }
+
+  private ArrayNode buildDatasets(List<String> tableNames)
+  {
+    ArrayNode array = jsonMapper.createArrayNode();
+    for (String name : tableNames) {
+      ObjectNode node = jsonMapper.createObjectNode();
+      node.put("namespace", namespace);
+      node.put("name", name);
+      node.set("facets", jsonMapper.createObjectNode());
+      array.add(node);
+    }
+    return array;
+  }
+
+  private List<String> extractInputs(SqlNode root)
+  {
+    List<String> tables = new ArrayList<>();
+    if (root instanceof SqlWith) {
+      SqlWith with = (SqlWith) root;
+      Set<String> cteNames = new HashSet<>();
+      for (SqlNode item : with.withList) {
+        if (item instanceof SqlWithItem) {
+          cteNames.add(((SqlWithItem) item).name.getSimple());
+          collectFromClause(((SqlWithItem) item).query, tables, cteNames);
+        }
+      }
+      collectFromClause(with.body, tables, cteNames);
+    } else if (root instanceof SqlInsert) {
+      collectFromClause(((SqlInsert) root).getSource(), tables, Set.of());
+    } else {
+      collectFromClause(root, tables, Set.of());
+    }
+    return tables;
+  }
+
+  @Nullable
+  private String extractOutput(SqlNode root)
+  {
+    if (root instanceof SqlInsert) {
+      SqlNode target = ((SqlInsert) root).getTargetTable();
+      if (target instanceof SqlIdentifier) {
+        return String.join(".", ((SqlIdentifier) target).names);
+      }
+    }
+    return null;
+  }
+
+  private void collectFromClause(SqlNode from, List<String> tables, Set<String> excludeNames)
+  {
+    if (from == null) {
+      return;
+    }
+    if (from instanceof SqlIdentifier) {
+      String name = String.join(".", ((SqlIdentifier) from).names);
+      if (!excludeNames.contains(name)) {
+        tables.add(name);
+      }
+    } else if (from instanceof SqlJoin) {
+      SqlJoin join = (SqlJoin) from;
+      collectFromClause(join.getLeft(), tables, excludeNames);
+      collectFromClause(join.getRight(), tables, excludeNames);
+    } else if (from instanceof SqlSelect) {
+      collectFromClause(((SqlSelect) from).getFrom(), tables, excludeNames);
+    } else if (from instanceof SqlWith) {
+      SqlWith with = (SqlWith) from;
+      Set<String> innerExcludes = new HashSet<>(excludeNames);
+      for (SqlNode item : with.withList) {
+        if (item instanceof SqlWithItem) {
+          innerExcludes.add(((SqlWithItem) item).name.getSimple());
+          collectFromClause(((SqlWithItem) item).query, tables, innerExcludes);
+        }
+      }
+      collectFromClause(with.body, tables, innerExcludes);
+    } else if (from instanceof SqlOrderBy) {
+      collectFromClause(((SqlOrderBy) from).query, tables, excludeNames);
+    } else if (from instanceof SqlBasicCall && from.getKind() == SqlKind.AS) {
+      collectFromClause(((SqlBasicCall) from).operand(0), tables, excludeNames);
+    }
+  }
+
+  protected void emit(ObjectNode event)
+  {
+    try {
+      String json = jsonMapper.writeValueAsString(event);
+      if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+        emitExecutor.submit(() -> emitHttp(json));
+      } else {
+        log.info("OpenLineage event: %s", json);
+      }
+    }
+    catch (IOException e) {
+      log.error(e, "Failed to serialize OpenLineage event");
+    }
+  }
+
+  private void emitHttp(String json)
+  {
+    HttpPost post = new HttpPost(transportUrl);
+    post.setEntity(new StringEntity(json, ContentType.APPLICATION_JSON));
+    try {
+      org.apache.http.HttpResponse response = httpClient.execute(post);
+      EntityUtils.consumeQuietly(response.getEntity());
+    }
+    catch (IOException e) {
+      log.error(e, "Failed to POST OpenLineage event to [%s]", transportUrl);
+    }
+    finally {
+      post.releaseConnection();
+    }
+  }
+
+  private String extractSqlQueryId(RequestLogLine requestLogLine)
+  {
+    Object id = requestLogLine.getSqlQueryContext().get(BaseQuery.SQL_QUERY_ID);
+    if (id != null) {
+      return id.toString();
+    }
+    return UUID.randomUUID().toString();

Review Comment:
   Unless this uniqueness is a absolutely required by the spec I feel like we should make this some default fixed value so we can know if there are queries somehow making it through without an ID.



##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,503 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOrderBy;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  private static final int EMIT_QUEUE_CAPACITY = 1000;
+  private static final int EMIT_THREAD_COUNT = 1;
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  private final Set<String> excludedNativeQueryTypes;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes, null);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      @Nullable HttpClient httpClient
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    this.excludedNativeQueryTypes = excludedNativeQueryTypes;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = httpClient != null ? httpClient : HttpClientBuilder.create().build();
+      // Bounded queue: if the queue is full, silently drop the event rather than blocking
+      // the query thread. Uses Druid's Execs for daemon thread naming conventions.
+      this.emitExecutor = new ThreadPoolExecutor(
+          EMIT_THREAD_COUNT,
+          EMIT_THREAD_COUNT,
+          60L,
+          TimeUnit.SECONDS,
+          new ArrayBlockingQueue<>(EMIT_QUEUE_CAPACITY),
+          Execs.makeThreadFactory("OpenLineageEmitter-%d"),
+          new ThreadPoolExecutor.DiscardPolicy()
+      );
+    } else {
+      this.httpClient = null;
+      this.emitExecutor = null;
+    }
+  }
+
+  @LifecycleStart
+  @Override
+  public void start() throws Exception
+  {
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      log.info("Started OpenLineage HTTP transport to [%s]", transportUrl);
+    } else {
+      log.info("Started OpenLineage console transport");
+    }
+  }
+
+  @LifecycleStop
+  @Override
+  public void stop()
+  {
+    if (emitExecutor != null) {
+      emitExecutor.shutdown();
+      try {
+        if (!emitExecutor.awaitTermination(5, TimeUnit.SECONDS)) {
+          emitExecutor.shutdownNow();
+        }
+      }
+      catch (InterruptedException e) {
+        emitExecutor.shutdownNow();
+        Thread.currentThread().interrupt();
+      }
+    }
+    if (httpClient instanceof Closeable) {
+      CloseableUtils.closeAndSuppressExceptions(
+          (Closeable) httpClient,
+          e -> log.warn(e, "Failed to close OpenLineage HTTP client")
+      );
+    }
+    log.info("Stopped OpenLineage request logger");
+  }
+
+  @Override
+  public void logNativeQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    if (requestLogLine.getQuery() == null) {
+      return;
+    }
+
+    // Skip native sub-queries of a SQL execution to avoid duplicating the SQL-level event.
+    if (requestLogLine.getQuery().getContext().get(BaseQuery.SQL_QUERY_ID) != null) {
+      return;
+    }
+
+    String queryType = requestLogLine.getQuery().getType();
+
+    if (excludedNativeQueryTypes.contains(queryType)) {
+      return;
+    }
+
+    List<String> inputs = new ArrayList<>(new LinkedHashSet<>(requestLogLine.getQuery().getDataSource().getTableNames()));
+    String queryId = requestLogLine.getQuery().getId();
+    if (queryId == null) {
+      queryId = UUID.randomUUID().toString();
+    }
+
+    emit(buildRunEvent(queryId, null, queryType, requestLogLine, inputs, null));
+  }
+
+  @Override
+  public void logSqlQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    String sql = requestLogLine.getSql();
+    List<String> inputs = new ArrayList<>();
+    String output = null;
+
+    if (sql != null) {
+      try {
+        SqlNode parsed = SqlParser.create(sql, SQL_PARSER_CONFIG).parseQuery();
+        inputs = extractInputs(parsed);
+        output = extractOutput(parsed);
+      }
+      catch (SqlParseException e) {
+        // Druid-specific SQL extensions (REPLACE, EXTERN, etc.) may not parse with the standard
+        // Calcite parser. Emit the event without table-level lineage rather than failing.
+        log.debug(
+            "OpenLineage: could not parse SQL for lineage extraction (query will still be emitted): %s",
+            e.getMessage()
+        );
+      }
+    }
+
+    String queryId = extractSqlQueryId(requestLogLine);
+    emit(buildRunEvent(queryId, sql, "sql", requestLogLine, new ArrayList<>(new LinkedHashSet<>(inputs)), output));
+  }
+
+  private ObjectNode buildRunEvent(
+      String queryId,
+      @Nullable String sql,
+      String queryType,
+      RequestLogLine requestLogLine,
+      List<String> inputs,
+      @Nullable String output
+  )
+  {
+    Map<String, Object> stats = requestLogLine.getQueryStats().getStats();
+    boolean success = Boolean.TRUE.equals(stats.get("success"));
+
+    ObjectNode event = jsonMapper.createObjectNode();
+    event.put("eventType", success ? "COMPLETE" : "FAIL");
+    event.put("eventTime", requestLogLine.getTimestamp().toInstant().toString());
+    event.put("producer", PRODUCER);
+    event.put("schemaURL", SCHEMA_URL);
+    event.set("run", buildRun(queryId, queryType, requestLogLine, stats, success));
+    event.set("job", buildJob(queryId, sql));
+    event.set("inputs", buildDatasets(inputs));
+    event.set("outputs", buildDatasets(output != null ? List.of(output) : List.of()));
+    return event;
+  }
+
+  private ObjectNode buildRun(
+      String queryId,
+      String queryType,
+      RequestLogLine requestLogLine,
+      Map<String, Object> stats,
+      boolean success
+  )
+  {
+    ObjectNode run = jsonMapper.createObjectNode();
+    run.put("runId", UUID.nameUUIDFromBytes(queryId.getBytes(StandardCharsets.UTF_8)).toString());
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode engineFacet = createFacet(ENGINE_FACET_SCHEMA_URL);
+    engineFacet.put("name", "druid");
+    engineFacet.put("version", getDruidVersion());
+    facets.set("processing_engine", engineFacet);
+
+    ObjectNode contextFacet = createFacet(null);
+    contextFacet.put("queryType", queryType);
+    contextFacet.put("remoteAddress", requestLogLine.getRemoteAddr());
+    Object identity = stats.get("identity");
+    if (identity != null) {
+      contextFacet.put("identity", identity.toString());
+    }
+    Object nativeQueryIds = requestLogLine.getSqlQueryContext().get("nativeQueryIds");
+    if (nativeQueryIds != null) {
+      contextFacet.put("nativeQueryIds", nativeQueryIds.toString());
+    }
+    facets.set("druid_query_context", contextFacet);
+
+    ObjectNode statsFacet = createFacet(null);
+    putLongStat(statsFacet, "durationMs", stats, "sqlQuery/time", "query/time");
+    putLongStat(statsFacet, "bytes", stats, "sqlQuery/bytes", "query/bytes");
+    putLongStat(statsFacet, "planningTimeMs", stats, "sqlQuery/planningTimeMs");
+    Object statusCode = stats.get("statusCode");
+    if (statusCode != null) {
+      statsFacet.put("statusCode", statusCode.toString());
+    }
+    facets.set("druid_query_statistics", statsFacet);
+
+    if (!success) {
+      Object exception = stats.get("exception");
+      if (exception != null) {
+        ObjectNode errorFacet = createFacet(ERROR_FACET_SCHEMA_URL);
+        errorFacet.put("message", exception.toString());
+        if ("sql".equals(queryType)) {
+          errorFacet.put("programmingLanguage", "SQL");
+        }
+        facets.set("errorMessage", errorFacet);
+      }
+    }
+
+    run.set("facets", facets);
+    return run;
+  }
+
+  private ObjectNode buildJob(String queryId, @Nullable String sql)
+  {
+    ObjectNode job = jsonMapper.createObjectNode();
+    job.put("namespace", namespace);
+    job.put("name", queryId);
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode jobTypeFacet = createFacet(JOB_TYPE_FACET_SCHEMA_URL);
+    jobTypeFacet.put("processingType", "BATCH");
+    jobTypeFacet.put("integration", "DRUID");
+    jobTypeFacet.put("jobType", "QUERY");
+    facets.set("jobType", jobTypeFacet);
+
+    if (sql != null) {
+      ObjectNode sqlFacet = createFacet(SQL_FACET_SCHEMA_URL);
+      sqlFacet.put("query", sql);
+      facets.set("sql", sqlFacet);
+    }
+
+    job.set("facets", facets);
+    return job;
+  }
+
+  private ObjectNode createFacet(@Nullable String schemaUrl)
+  {
+    ObjectNode facet = jsonMapper.createObjectNode();
+    facet.put("_producer", PRODUCER);
+    if (schemaUrl != null) {
+      facet.put("_schemaURL", schemaUrl);
+    }
+    return facet;
+  }
+
+  private ArrayNode buildDatasets(List<String> tableNames)
+  {
+    ArrayNode array = jsonMapper.createArrayNode();
+    for (String name : tableNames) {
+      ObjectNode node = jsonMapper.createObjectNode();
+      node.put("namespace", namespace);
+      node.put("name", name);
+      node.set("facets", jsonMapper.createObjectNode());
+      array.add(node);
+    }
+    return array;
+  }
+
+  private List<String> extractInputs(SqlNode root)
+  {
+    List<String> tables = new ArrayList<>();
+    if (root instanceof SqlWith) {
+      SqlWith with = (SqlWith) root;
+      Set<String> cteNames = new HashSet<>();
+      for (SqlNode item : with.withList) {
+        if (item instanceof SqlWithItem) {
+          cteNames.add(((SqlWithItem) item).name.getSimple());
+          collectFromClause(((SqlWithItem) item).query, tables, cteNames);
+        }
+      }
+      collectFromClause(with.body, tables, cteNames);
+    } else if (root instanceof SqlInsert) {
+      collectFromClause(((SqlInsert) root).getSource(), tables, Set.of());
+    } else {
+      collectFromClause(root, tables, Set.of());
+    }
+    return tables;
+  }
+
+  @Nullable
+  private String extractOutput(SqlNode root)
+  {
+    if (root instanceof SqlInsert) {
+      SqlNode target = ((SqlInsert) root).getTargetTable();
+      if (target instanceof SqlIdentifier) {
+        return String.join(".", ((SqlIdentifier) target).names);
+      }
+    }
+    return null;
+  }
+
+  private void collectFromClause(SqlNode from, List<String> tables, Set<String> excludeNames)
+  {
+    if (from == null) {
+      return;
+    }
+    if (from instanceof SqlIdentifier) {
+      String name = String.join(".", ((SqlIdentifier) from).names);
+      if (!excludeNames.contains(name)) {
+        tables.add(name);
+      }
+    } else if (from instanceof SqlJoin) {
+      SqlJoin join = (SqlJoin) from;
+      collectFromClause(join.getLeft(), tables, excludeNames);
+      collectFromClause(join.getRight(), tables, excludeNames);
+    } else if (from instanceof SqlSelect) {
+      collectFromClause(((SqlSelect) from).getFrom(), tables, excludeNames);
+    } else if (from instanceof SqlWith) {
+      SqlWith with = (SqlWith) from;
+      Set<String> innerExcludes = new HashSet<>(excludeNames);
+      for (SqlNode item : with.withList) {
+        if (item instanceof SqlWithItem) {
+          innerExcludes.add(((SqlWithItem) item).name.getSimple());
+          collectFromClause(((SqlWithItem) item).query, tables, innerExcludes);
+        }
+      }
+      collectFromClause(with.body, tables, innerExcludes);
+    } else if (from instanceof SqlOrderBy) {
+      collectFromClause(((SqlOrderBy) from).query, tables, excludeNames);
+    } else if (from instanceof SqlBasicCall && from.getKind() == SqlKind.AS) {
+      collectFromClause(((SqlBasicCall) from).operand(0), tables, excludeNames);
+    }
+  }
+
+  protected void emit(ObjectNode event)
+  {
+    try {
+      String json = jsonMapper.writeValueAsString(event);
+      if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+        emitExecutor.submit(() -> emitHttp(json));
+      } else {
+        log.info("OpenLineage event: %s", json);

Review Comment:
   nit:debug



##########
docs/development/extensions-contrib/openlineage-emitter.md:
##########
@@ -0,0 +1,92 @@
+---
+id: openlineage-emitter
+title: "OpenLineage Emitter"
+---
+
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one
+  ~ or more contributor license agreements.  See the NOTICE file
+  ~ distributed with this work for additional information
+  ~ regarding copyright ownership.  The ASF licenses this file
+  ~ to you under the Apache License, Version 2.0 (the
+  ~ "License"); you may not use this file except in compliance
+  ~ with the License.  You may obtain a copy of the License at
+  ~
+  ~   http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing,
+  ~ software distributed under the License is distributed on an
+  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  ~ KIND, either express or implied.  See the License for the
+  ~ specific language governing permissions and limitations
+  ~ under the License.
+  -->
+
+To use this Apache Druid extension, [include](../../configuration/extensions.md#loading-extensions) `openlineage-emitter` in the extensions load list.
+
+## Introduction
+
+This extension emits [OpenLineage](https://openlineage.io) `RunEvent`s for each completed Druid query, enabling data lineage tracking with any OpenLineage-compatible backend such as [Marquez](https://marquezproject.ai).
+
+For SQL queries, the SQL text is parsed to extract input datasources (FROM clauses, JOINs, CTEs) and the output datasource (INSERT INTO). For native queries, table names are resolved from the datasource tree. Native sub-queries spawned by a SQL execution are deduplicated against the SQL-level event.
+
+:::note
+SQL table extraction relies on `calcite-core` being on the classpath, which is the case on Broker nodes. Native query lineage is available on all nodes.
+:::
+
+## Configuration
+
+All configuration parameters are under `druid.request.logging`.
+
+| Property | Description | Required | Default |
+|---|---|---|---|
+| `druid.request.logging.type` | Set to `openlineage` to enable this extension. | yes | — |
+| `druid.request.logging.namespace` | Namespace used for OpenLineage job and dataset URIs. Typically the Broker URL. | no | `druid://<hostname>` |
+| `druid.request.logging.transportType` | Where to send events. `CONSOLE` logs JSON to the Druid log; `HTTP` POSTs to an OpenLineage API endpoint. | no | `CONSOLE` |
+| `druid.request.logging.transportUrl` | OpenLineage API endpoint URL. Required when `transportType=HTTP`. | no | — |
+| `druid.request.logging.excludedNativeQueryTypes` | Native query types to exclude from lineage emission. Internal broker queries like segment metadata lookups produce noisy, low-value events. | no | `["segmentMetadata", "dataSourceMetadata", "timeBoundary"]` |
+
+### Examples
+
+**Console (development)**
+
+```properties
+druid.request.logging.type=openlineage
+druid.request.logging.namespace=druid://broker.prod:8082
+```
+
+**HTTP (production)**
+
+```properties
+druid.request.logging.type=openlineage
+druid.request.logging.namespace=druid://broker.prod:8082
+druid.request.logging.transportType=HTTP
+druid.request.logging.transportUrl=http://marquez:5000/api/v1/lineage
+```
+
+**Combined with another logger using the `composing` provider**
+
+```properties
+druid.request.logging.type=composing
+druid.request.logging.loggerProviders=[{"type":"slf4j"},{"type":"openlineage","namespace":"druid://broker.prod:8082","transportType":"HTTP","transportUrl":"http://marquez:5000/api/v1/lineage"}]
+```
+
+## Event structure
+
+Each emitted event follows the [OpenLineage spec](https://openlineage.io/spec/2-0-2/OpenLineage.json) and includes the following facets.
+
+### Run facets
+
+| Facet | Description |
+|---|---|
+| `processing_engine` | Engine name (`druid`). Standard OpenLineage facet. |
+| `druid_query_context` | Query metadata: `identity` (authenticated user), `remoteAddress`, `queryType`, and `nativeQueryIds` (for SQL queries). |
+| `druid_query_statistics` | Execution stats: `durationMs`, `bytes`, `planningTimeMs`, `statusCode`. |
+| `errorMessage` | Exception message for failed queries. Standard OpenLineage facet. |
+
+### Job facets
+
+| Facet | Description |
+|---|---|
+| `jobType` | `processingType=BATCH`, `integration=DRUID`, `jobType=QUERY`. Standard OpenLineage facet. |

Review Comment:
   processingType should be QUERY?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["mshahid6 (via GitHub)" <gi...@apache.org>]

mshahid6 commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3133684365


##########
docs/development/extensions-contrib/openlineage-emitter.md:
##########
@@ -0,0 +1,92 @@
+---
+id: openlineage-emitter
+title: "OpenLineage Emitter"
+---
+
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one
+  ~ or more contributor license agreements.  See the NOTICE file
+  ~ distributed with this work for additional information
+  ~ regarding copyright ownership.  The ASF licenses this file
+  ~ to you under the Apache License, Version 2.0 (the
+  ~ "License"); you may not use this file except in compliance
+  ~ with the License.  You may obtain a copy of the License at
+  ~
+  ~   http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing,
+  ~ software distributed under the License is distributed on an
+  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  ~ KIND, either express or implied.  See the License for the
+  ~ specific language governing permissions and limitations
+  ~ under the License.
+  -->
+
+To use this Apache Druid extension, [include](../../configuration/extensions.md#loading-extensions) `openlineage-emitter` in the extensions load list.
+
+## Introduction
+
+This extension emits [OpenLineage](https://openlineage.io) `RunEvent`s for each completed Druid query, enabling data lineage tracking with any OpenLineage-compatible backend such as [Marquez](https://marquezproject.ai).
+
+For SQL queries, the SQL text is parsed to extract input datasources (FROM clauses, JOINs, CTEs) and the output datasource (INSERT INTO). For native queries, table names are resolved from the datasource tree. Native sub-queries spawned by a SQL execution are deduplicated against the SQL-level event.
+
+:::note
+SQL table extraction relies on `calcite-core` being on the classpath, which is the case on Broker nodes. Native query lineage is available on all nodes.
+:::
+
+## Configuration
+
+All configuration parameters are under `druid.request.logging`.
+
+| Property | Description | Required | Default |
+|---|---|---|---|
+| `druid.request.logging.type` | Set to `openlineage` to enable this extension. | yes | — |
+| `druid.request.logging.namespace` | Namespace used for OpenLineage job and dataset URIs. Typically the Broker URL. | no | `druid://<hostname>` |
+| `druid.request.logging.transportType` | Where to send events. `CONSOLE` logs JSON to the Druid log; `HTTP` POSTs to an OpenLineage API endpoint. | no | `CONSOLE` |
+| `druid.request.logging.transportUrl` | OpenLineage API endpoint URL. Required when `transportType=HTTP`. | no | — |
+| `druid.request.logging.excludedNativeQueryTypes` | Native query types to exclude from lineage emission. Internal broker queries like segment metadata lookups produce noisy, low-value events. | no | `["segmentMetadata", "dataSourceMetadata", "timeBoundary"]` |
+
+### Examples
+
+**Console (development)**
+
+```properties
+druid.request.logging.type=openlineage
+druid.request.logging.namespace=druid://broker.prod:8082
+```
+
+**HTTP (production)**
+
+```properties
+druid.request.logging.type=openlineage
+druid.request.logging.namespace=druid://broker.prod:8082
+druid.request.logging.transportType=HTTP
+druid.request.logging.transportUrl=http://marquez:5000/api/v1/lineage
+```
+
+**Combined with another logger using the `composing` provider**
+
+```properties
+druid.request.logging.type=composing
+druid.request.logging.loggerProviders=[{"type":"slf4j"},{"type":"openlineage","namespace":"druid://broker.prod:8082","transportType":"HTTP","transportUrl":"http://marquez:5000/api/v1/lineage"}]
+```
+
+## Event structure
+
+Each emitted event follows the [OpenLineage spec](https://openlineage.io/spec/2-0-2/OpenLineage.json) and includes the following facets.
+
+### Run facets
+
+| Facet | Description |
+|---|---|
+| `processing_engine` | Engine name (`druid`). Standard OpenLineage facet. |
+| `druid_query_context` | Query metadata: `identity` (authenticated user), `remoteAddress`, `queryType`, and `nativeQueryIds` (for SQL queries). |
+| `druid_query_statistics` | Execution stats: `durationMs`, `bytes`, `planningTimeMs`, `statusCode`. |
+| `errorMessage` | Exception message for failed queries. Standard OpenLineage facet. |
+
+### Job facets
+
+| Facet | Description |
+|---|---|
+| `jobType` | `processingType=BATCH`, `integration=DRUID`, `jobType=QUERY`. Standard OpenLineage facet. |

Review Comment:
   https://openlineage.io/docs/spec/facets/job-facets/job-type `processingType` which can be `STREAMING` or `BATCH`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["jtuglu1 (via GitHub)" <gi...@apache.org>]

jtuglu1 closed pull request #19107: feat: add OpenLineage request logger extension
URL: https://github.com/apache/druid/pull/19107


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["maytasm (via GitHub)" <gi...@apache.org>]

maytasm commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3024821410


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,498 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOrderBy;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  private static final int EMIT_QUEUE_CAPACITY = 1000;
+  private static final int EMIT_THREAD_COUNT = 1;
+
+  /**
+   * Internal broker query types that are automatically issued for schema discovery and metadata
+   * caching. These are not user-initiated and produce noisy, low-value lineage events.
+   */
+  private static final Set<String> INTERNAL_QUERY_TYPES = Set.of(
+      "segmentMetadata",
+      "dataSourceMetadata",
+      "timeBoundary"
+  );
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = HttpClientBuilder.create().build();
+      // Bounded queue: if the queue is full, silently drop the event rather than blocking
+      // the query thread. Uses Druid's Execs for daemon thread naming conventions.
+      this.emitExecutor = new ThreadPoolExecutor(
+          EMIT_THREAD_COUNT,
+          EMIT_THREAD_COUNT,
+          60L,
+          TimeUnit.SECONDS,
+          new ArrayBlockingQueue<>(EMIT_QUEUE_CAPACITY),
+          Execs.makeThreadFactory("OpenLineageEmitter-%d"),
+          new ThreadPoolExecutor.DiscardPolicy()
+      );
+    } else {
+      this.httpClient = null;
+      this.emitExecutor = null;
+    }
+  }
+
+  @LifecycleStart
+  @Override
+  public void start() throws Exception
+  {
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      log.info("Started OpenLineage HTTP transport to [%s]", transportUrl);
+    } else {
+      log.info("Started OpenLineage console transport");
+    }
+  }
+
+  @LifecycleStop
+  @Override
+  public void stop()
+  {
+    if (emitExecutor != null) {
+      emitExecutor.shutdown();
+      try {
+        if (!emitExecutor.awaitTermination(5, TimeUnit.SECONDS)) {
+          emitExecutor.shutdownNow();
+        }
+      }
+      catch (InterruptedException e) {
+        emitExecutor.shutdownNow();
+        Thread.currentThread().interrupt();
+      }
+    }
+    if (httpClient instanceof Closeable) {
+      CloseableUtils.closeAndSuppressExceptions(
+          (Closeable) httpClient,
+          e -> log.warn(e, "Failed to close OpenLineage HTTP client")
+      );
+    }
+    log.info("Stopped OpenLineage request logger");
+  }
+
+  @Override
+  public void logNativeQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    if (requestLogLine.getQuery() == null) {
+      return;
+    }
+
+    // Skip native sub-queries of a SQL execution to avoid duplicating the SQL-level event.
+    if (requestLogLine.getQuery().getContext().get(BaseQuery.SQL_QUERY_ID) != null) {
+      return;
+    }
+
+    String queryType = requestLogLine.getQuery().getType();
+
+    if (INTERNAL_QUERY_TYPES.contains(queryType)) {
+      return;
+    }
+
+    List<String> inputs = new ArrayList<>(new LinkedHashSet<>(requestLogLine.getQuery().getDataSource().getTableNames()));
+    String queryId = requestLogLine.getQuery().getId();
+    if (queryId == null) {
+      queryId = UUID.randomUUID().toString();
+    }
+
+    emit(buildRunEvent(queryId, null, queryType, requestLogLine, inputs, null));
+  }
+
+  @Override
+  public void logSqlQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    String sql = requestLogLine.getSql();
+    List<String> inputs = new ArrayList<>();
+    String output = null;
+
+    if (sql != null) {
+      try {
+        SqlNode parsed = SqlParser.create(sql, SQL_PARSER_CONFIG).parseQuery();
+        inputs = extractInputs(parsed);
+        output = extractOutput(parsed);
+      }
+      catch (SqlParseException e) {
+        // Druid-specific SQL extensions (REPLACE, EXTERN, etc.) may not parse with the standard
+        // Calcite parser. Emit the event without table-level lineage rather than failing.
+        log.debug(
+            "OpenLineage: could not parse SQL for lineage extraction (query will still be emitted): %s",
+            e.getMessage()
+        );
+      }
+    }
+
+    String queryId = extractSqlQueryId(requestLogLine);
+    emit(buildRunEvent(queryId, sql, "sql", requestLogLine, new ArrayList<>(new LinkedHashSet<>(inputs)), output));
+  }
+
+  private ObjectNode buildRunEvent(
+      String queryId,
+      @Nullable String sql,
+      String queryType,
+      RequestLogLine requestLogLine,
+      List<String> inputs,
+      @Nullable String output
+  )
+  {
+    Map<String, Object> stats = requestLogLine.getQueryStats().getStats();
+    boolean success = Boolean.TRUE.equals(stats.get("success"));
+
+    ObjectNode event = jsonMapper.createObjectNode();
+    event.put("eventType", success ? "COMPLETE" : "FAIL");
+    event.put("eventTime", requestLogLine.getTimestamp().toInstant().toString());
+    event.put("producer", PRODUCER);
+    event.put("schemaURL", SCHEMA_URL);
+    event.set("run", buildRun(queryId, queryType, requestLogLine, stats, success));
+    event.set("job", buildJob(queryId, sql));
+    event.set("inputs", buildDatasets(inputs));
+    event.set("outputs", buildDatasets(output != null ? List.of(output) : List.of()));
+    return event;
+  }
+
+  private ObjectNode buildRun(
+      String queryId,
+      String queryType,
+      RequestLogLine requestLogLine,
+      Map<String, Object> stats,
+      boolean success
+  )
+  {
+    ObjectNode run = jsonMapper.createObjectNode();
+    run.put("runId", UUID.nameUUIDFromBytes(queryId.getBytes(StandardCharsets.UTF_8)).toString());
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode engineFacet = createFacet(ENGINE_FACET_SCHEMA_URL);
+    engineFacet.put("name", "druid");
+    engineFacet.put("version", getDruidVersion());
+    facets.set("processing_engine", engineFacet);
+
+    ObjectNode contextFacet = createFacet(null);
+    contextFacet.put("queryType", queryType);
+    contextFacet.put("remoteAddress", requestLogLine.getRemoteAddr());
+    Object identity = stats.get("identity");
+    if (identity != null) {
+      contextFacet.put("identity", identity.toString());
+    }
+    Object nativeQueryIds = requestLogLine.getSqlQueryContext().get("nativeQueryIds");
+    if (nativeQueryIds != null) {
+      contextFacet.put("nativeQueryIds", nativeQueryIds.toString());
+    }
+    facets.set("druid_query_context", contextFacet);
+
+    ObjectNode statsFacet = createFacet(null);
+    putLongStat(statsFacet, "durationMs", stats, "sqlQuery/time", "query/time");
+    putLongStat(statsFacet, "bytes", stats, "sqlQuery/bytes", "query/bytes");
+    putLongStat(statsFacet, "planningTimeMs", stats, "sqlQuery/planningTimeMs");
+    Object statusCode = stats.get("statusCode");
+    if (statusCode != null) {
+      statsFacet.put("statusCode", statusCode.toString());
+    }
+    facets.set("druid_query_statistics", statsFacet);
+
+    if (!success) {
+      Object exception = stats.get("exception");
+      if (exception != null) {
+        ObjectNode errorFacet = createFacet(ERROR_FACET_SCHEMA_URL);
+        errorFacet.put("message", exception.toString());
+        if ("sql".equals(queryType)) {
+          errorFacet.put("programmingLanguage", "SQL");
+        }
+        facets.set("errorMessage", errorFacet);
+      }
+    }
+
+    run.set("facets", facets);
+    return run;
+  }
+
+  private ObjectNode buildJob(String queryId, @Nullable String sql)
+  {
+    ObjectNode job = jsonMapper.createObjectNode();
+    job.put("namespace", namespace);
+    job.put("name", queryId);
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode jobTypeFacet = createFacet(JOB_TYPE_FACET_SCHEMA_URL);
+    jobTypeFacet.put("processingType", "BATCH");
+    jobTypeFacet.put("integration", "DRUID");
+    jobTypeFacet.put("jobType", "QUERY");
+    facets.set("jobType", jobTypeFacet);
+
+    if (sql != null) {

Review Comment:
   Does it make sense to include native query if this is not sql?



##########
docs/development/extensions-contrib/openlineage-emitter.md:
##########
@@ -0,0 +1,95 @@
+---
+id: openlineage-emitter
+title: "OpenLineage Emitter"
+---
+
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one
+  ~ or more contributor license agreements.  See the NOTICE file
+  ~ distributed with this work for additional information
+  ~ regarding copyright ownership.  The ASF licenses this file
+  ~ to you under the Apache License, Version 2.0 (the
+  ~ "License"); you may not use this file except in compliance
+  ~ with the License.  You may obtain a copy of the License at
+  ~
+  ~   http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing,
+  ~ software distributed under the License is distributed on an
+  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  ~ KIND, either express or implied.  See the License for the
+  ~ specific language governing permissions and limitations
+  ~ under the License.
+  -->
+
+To use this Apache Druid extension, [include](../../configuration/extensions.md#loading-extensions) `openlineage-emitter` in the extensions load list.
+
+## Introduction
+
+This extension emits [OpenLineage](https://openlineage.io) `RunEvent`s for each completed Druid query, enabling data lineage tracking with any OpenLineage-compatible backend such as [Marquez](https://marquezproject.ai).
+
+For SQL queries, the SQL text is parsed to extract input datasources (FROM clauses, JOINs, CTEs) and the output datasource (INSERT INTO). For native queries, table names are resolved from the datasource tree. Native sub-queries spawned by a SQL execution are deduplicated against the SQL-level event.
+
+:::note
+SQL table extraction relies on `calcite-core` being on the classpath, which is the case on Broker nodes. Native query lineage is available on all nodes.
+:::
+
+## Configuration
+
+All configuration parameters are under `druid.request.logging`.
+
+| Property | Description | Required | Default |
+|---|---|---|---|
+| `druid.request.logging.type` | Set to `openlineage` to enable this extension. | yes | — |
+| `druid.request.logging.namespace` | Namespace used for OpenLineage job and dataset URIs. Typically the Broker URL. | no | `druid://localhost` |
+| `druid.request.logging.transportType` | Where to send events. `CONSOLE` logs JSON to the Druid log; `HTTP` POSTs to an OpenLineage API endpoint. | no | `CONSOLE` |
+| `druid.request.logging.transportUrl` | OpenLineage API endpoint URL. Required when `transportType=HTTP`. | no | — |
+
+### Examples
+
+**Console (development)**
+
+```properties
+druid.request.logging.type=openlineage
+druid.request.logging.namespace=druid://broker.prod:8082
+```
+
+**HTTP (production)**
+
+```properties
+druid.request.logging.type=openlineage
+druid.request.logging.namespace=druid://broker.prod:8082
+druid.request.logging.transportType=HTTP
+druid.request.logging.transportUrl=http://marquez:5000/api/v1/lineage
+```
+
+**Combined with another logger using the `composing` provider**
+
+```properties
+druid.request.logging.type=composing
+druid.request.logging.loggerProviders[0].type=slf4j

Review Comment:
   Are these correct? Don't you have to set druid.request.logging.loggerProviders to a List with a map for each Logger?



##########
docs/development/extensions-contrib/openlineage-emitter.md:
##########
@@ -0,0 +1,95 @@
+---
+id: openlineage-emitter
+title: "OpenLineage Emitter"
+---
+
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one
+  ~ or more contributor license agreements.  See the NOTICE file
+  ~ distributed with this work for additional information
+  ~ regarding copyright ownership.  The ASF licenses this file
+  ~ to you under the Apache License, Version 2.0 (the
+  ~ "License"); you may not use this file except in compliance
+  ~ with the License.  You may obtain a copy of the License at
+  ~
+  ~   http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing,
+  ~ software distributed under the License is distributed on an
+  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  ~ KIND, either express or implied.  See the License for the
+  ~ specific language governing permissions and limitations
+  ~ under the License.
+  -->
+
+To use this Apache Druid extension, [include](../../configuration/extensions.md#loading-extensions) `openlineage-emitter` in the extensions load list.
+
+## Introduction
+
+This extension emits [OpenLineage](https://openlineage.io) `RunEvent`s for each completed Druid query, enabling data lineage tracking with any OpenLineage-compatible backend such as [Marquez](https://marquezproject.ai).
+
+For SQL queries, the SQL text is parsed to extract input datasources (FROM clauses, JOINs, CTEs) and the output datasource (INSERT INTO). For native queries, table names are resolved from the datasource tree. Native sub-queries spawned by a SQL execution are deduplicated against the SQL-level event.
+
+:::note
+SQL table extraction relies on `calcite-core` being on the classpath, which is the case on Broker nodes. Native query lineage is available on all nodes.
+:::
+
+## Configuration
+
+All configuration parameters are under `druid.request.logging`.
+
+| Property | Description | Required | Default |
+|---|---|---|---|
+| `druid.request.logging.type` | Set to `openlineage` to enable this extension. | yes | — |
+| `druid.request.logging.namespace` | Namespace used for OpenLineage job and dataset URIs. Typically the Broker URL. | no | `druid://localhost` |

Review Comment:
   Can this default to the hostname?



##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,498 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOrderBy;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  private static final int EMIT_QUEUE_CAPACITY = 1000;
+  private static final int EMIT_THREAD_COUNT = 1;
+
+  /**
+   * Internal broker query types that are automatically issued for schema discovery and metadata
+   * caching. These are not user-initiated and produce noisy, low-value lineage events.
+   */
+  private static final Set<String> INTERNAL_QUERY_TYPES = Set.of(
+      "segmentMetadata",
+      "dataSourceMetadata",
+      "timeBoundary"
+  );
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = HttpClientBuilder.create().build();

Review Comment:
   We may want to be able to override this client in case we require some specific http client config (keystore, truststore, etc) 



##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,498 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOrderBy;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  private static final int EMIT_QUEUE_CAPACITY = 1000;
+  private static final int EMIT_THREAD_COUNT = 1;
+
+  /**
+   * Internal broker query types that are automatically issued for schema discovery and metadata
+   * caching. These are not user-initiated and produce noisy, low-value lineage events.
+   */
+  private static final Set<String> INTERNAL_QUERY_TYPES = Set.of(
+      "segmentMetadata",
+      "dataSourceMetadata",
+      "timeBoundary"
+  );
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = HttpClientBuilder.create().build();
+      // Bounded queue: if the queue is full, silently drop the event rather than blocking
+      // the query thread. Uses Druid's Execs for daemon thread naming conventions.
+      this.emitExecutor = new ThreadPoolExecutor(
+          EMIT_THREAD_COUNT,
+          EMIT_THREAD_COUNT,
+          60L,
+          TimeUnit.SECONDS,
+          new ArrayBlockingQueue<>(EMIT_QUEUE_CAPACITY),
+          Execs.makeThreadFactory("OpenLineageEmitter-%d"),
+          new ThreadPoolExecutor.DiscardPolicy()
+      );
+    } else {
+      this.httpClient = null;
+      this.emitExecutor = null;
+    }
+  }
+
+  @LifecycleStart
+  @Override
+  public void start() throws Exception
+  {
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      log.info("Started OpenLineage HTTP transport to [%s]", transportUrl);
+    } else {
+      log.info("Started OpenLineage console transport");
+    }
+  }
+
+  @LifecycleStop
+  @Override
+  public void stop()
+  {
+    if (emitExecutor != null) {
+      emitExecutor.shutdown();
+      try {
+        if (!emitExecutor.awaitTermination(5, TimeUnit.SECONDS)) {
+          emitExecutor.shutdownNow();
+        }
+      }
+      catch (InterruptedException e) {
+        emitExecutor.shutdownNow();
+        Thread.currentThread().interrupt();
+      }
+    }
+    if (httpClient instanceof Closeable) {
+      CloseableUtils.closeAndSuppressExceptions(
+          (Closeable) httpClient,
+          e -> log.warn(e, "Failed to close OpenLineage HTTP client")
+      );
+    }
+    log.info("Stopped OpenLineage request logger");
+  }
+
+  @Override
+  public void logNativeQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    if (requestLogLine.getQuery() == null) {
+      return;
+    }
+
+    // Skip native sub-queries of a SQL execution to avoid duplicating the SQL-level event.
+    if (requestLogLine.getQuery().getContext().get(BaseQuery.SQL_QUERY_ID) != null) {
+      return;
+    }
+
+    String queryType = requestLogLine.getQuery().getType();
+
+    if (INTERNAL_QUERY_TYPES.contains(queryType)) {
+      return;
+    }
+
+    List<String> inputs = new ArrayList<>(new LinkedHashSet<>(requestLogLine.getQuery().getDataSource().getTableNames()));
+    String queryId = requestLogLine.getQuery().getId();
+    if (queryId == null) {
+      queryId = UUID.randomUUID().toString();
+    }
+
+    emit(buildRunEvent(queryId, null, queryType, requestLogLine, inputs, null));
+  }
+
+  @Override
+  public void logSqlQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    String sql = requestLogLine.getSql();
+    List<String> inputs = new ArrayList<>();
+    String output = null;
+
+    if (sql != null) {
+      try {
+        SqlNode parsed = SqlParser.create(sql, SQL_PARSER_CONFIG).parseQuery();
+        inputs = extractInputs(parsed);
+        output = extractOutput(parsed);
+      }
+      catch (SqlParseException e) {
+        // Druid-specific SQL extensions (REPLACE, EXTERN, etc.) may not parse with the standard
+        // Calcite parser. Emit the event without table-level lineage rather than failing.
+        log.debug(
+            "OpenLineage: could not parse SQL for lineage extraction (query will still be emitted): %s",
+            e.getMessage()
+        );
+      }
+    }
+
+    String queryId = extractSqlQueryId(requestLogLine);
+    emit(buildRunEvent(queryId, sql, "sql", requestLogLine, new ArrayList<>(new LinkedHashSet<>(inputs)), output));
+  }
+
+  private ObjectNode buildRunEvent(
+      String queryId,
+      @Nullable String sql,
+      String queryType,
+      RequestLogLine requestLogLine,
+      List<String> inputs,
+      @Nullable String output
+  )
+  {
+    Map<String, Object> stats = requestLogLine.getQueryStats().getStats();
+    boolean success = Boolean.TRUE.equals(stats.get("success"));
+
+    ObjectNode event = jsonMapper.createObjectNode();
+    event.put("eventType", success ? "COMPLETE" : "FAIL");
+    event.put("eventTime", requestLogLine.getTimestamp().toInstant().toString());
+    event.put("producer", PRODUCER);
+    event.put("schemaURL", SCHEMA_URL);
+    event.set("run", buildRun(queryId, queryType, requestLogLine, stats, success));
+    event.set("job", buildJob(queryId, sql));
+    event.set("inputs", buildDatasets(inputs));
+    event.set("outputs", buildDatasets(output != null ? List.of(output) : List.of()));
+    return event;
+  }
+
+  private ObjectNode buildRun(
+      String queryId,
+      String queryType,
+      RequestLogLine requestLogLine,
+      Map<String, Object> stats,
+      boolean success
+  )
+  {
+    ObjectNode run = jsonMapper.createObjectNode();
+    run.put("runId", UUID.nameUUIDFromBytes(queryId.getBytes(StandardCharsets.UTF_8)).toString());
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode engineFacet = createFacet(ENGINE_FACET_SCHEMA_URL);
+    engineFacet.put("name", "druid");
+    engineFacet.put("version", getDruidVersion());
+    facets.set("processing_engine", engineFacet);
+
+    ObjectNode contextFacet = createFacet(null);
+    contextFacet.put("queryType", queryType);
+    contextFacet.put("remoteAddress", requestLogLine.getRemoteAddr());
+    Object identity = stats.get("identity");
+    if (identity != null) {
+      contextFacet.put("identity", identity.toString());
+    }
+    Object nativeQueryIds = requestLogLine.getSqlQueryContext().get("nativeQueryIds");
+    if (nativeQueryIds != null) {
+      contextFacet.put("nativeQueryIds", nativeQueryIds.toString());
+    }
+    facets.set("druid_query_context", contextFacet);
+
+    ObjectNode statsFacet = createFacet(null);
+    putLongStat(statsFacet, "durationMs", stats, "sqlQuery/time", "query/time");
+    putLongStat(statsFacet, "bytes", stats, "sqlQuery/bytes", "query/bytes");
+    putLongStat(statsFacet, "planningTimeMs", stats, "sqlQuery/planningTimeMs");
+    Object statusCode = stats.get("statusCode");
+    if (statusCode != null) {
+      statsFacet.put("statusCode", statusCode.toString());
+    }
+    facets.set("druid_query_statistics", statsFacet);
+
+    if (!success) {
+      Object exception = stats.get("exception");
+      if (exception != null) {
+        ObjectNode errorFacet = createFacet(ERROR_FACET_SCHEMA_URL);
+        errorFacet.put("message", exception.toString());
+        if ("sql".equals(queryType)) {
+          errorFacet.put("programmingLanguage", "SQL");
+        }
+        facets.set("errorMessage", errorFacet);
+      }
+    }
+
+    run.set("facets", facets);
+    return run;
+  }
+
+  private ObjectNode buildJob(String queryId, @Nullable String sql)
+  {
+    ObjectNode job = jsonMapper.createObjectNode();
+    job.put("namespace", namespace);
+    job.put("name", queryId);
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode jobTypeFacet = createFacet(JOB_TYPE_FACET_SCHEMA_URL);
+    jobTypeFacet.put("processingType", "BATCH");
+    jobTypeFacet.put("integration", "DRUID");
+    jobTypeFacet.put("jobType", "QUERY");
+    facets.set("jobType", jobTypeFacet);
+
+    if (sql != null) {
+      ObjectNode sqlFacet = createFacet(SQL_FACET_SCHEMA_URL);
+      sqlFacet.put("query", sql);
+      facets.set("sql", sqlFacet);
+    }
+
+    job.set("facets", facets);
+    return job;
+  }
+
+  private ObjectNode createFacet(@Nullable String schemaUrl)
+  {
+    ObjectNode facet = jsonMapper.createObjectNode();
+    facet.put("_producer", PRODUCER);
+    if (schemaUrl != null) {
+      facet.put("_schemaURL", schemaUrl);
+    }
+    return facet;
+  }
+
+  private ArrayNode buildDatasets(List<String> tableNames)
+  {
+    ArrayNode array = jsonMapper.createArrayNode();
+    for (String name : tableNames) {
+      ObjectNode node = jsonMapper.createObjectNode();
+      node.put("namespace", namespace);
+      node.put("name", name);
+      node.set("facets", jsonMapper.createObjectNode());

Review Comment:
   Can we create a Github issue to track work to include the fields?



##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,498 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOrderBy;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  private static final int EMIT_QUEUE_CAPACITY = 1000;
+  private static final int EMIT_THREAD_COUNT = 1;
+
+  /**
+   * Internal broker query types that are automatically issued for schema discovery and metadata
+   * caching. These are not user-initiated and produce noisy, low-value lineage events.
+   */
+  private static final Set<String> INTERNAL_QUERY_TYPES = Set.of(

Review Comment:
   Maybe make this configurable?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["jtuglu1 (via GitHub)" <gi...@apache.org>]

jtuglu1 merged PR #19107:
URL: https://github.com/apache/druid/pull/19107


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["FrankChen021 (via GitHub)" <gi...@apache.org>]

FrankChen021 commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3161343640


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLoggerProvider.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.guice.annotations.Json;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.server.DruidNode;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.server.log.RequestLoggerProvider;
+
+import javax.annotation.Nullable;
+import javax.validation.constraints.NotNull;
+import java.util.Set;
+
+/**
+ *  Configure logging type, namespace, transport type (http or default console), transportUrl in {@code runtime.properties} 
+ */
+@JsonTypeName("openlineage")
+public class OpenLineageRequestLoggerProvider implements RequestLoggerProvider
+{
+  private static final Logger log = new Logger(OpenLineageRequestLoggerProvider.class);
+
+  public enum TransportType
+  {
+    CONSOLE,
+    HTTP
+  }
+
+  @JacksonInject
+  @Json
+  @NotNull
+  private ObjectMapper jsonMapper;
+
+  @JsonProperty
+  @NotNull
+  private String namespace = "druid://" + DruidNode.getDefaultHost();
+
+  @JsonProperty
+  @NotNull
+  private TransportType transportType = TransportType.CONSOLE;
+
+  @Nullable
+  @JsonProperty
+  private String transportUrl;
+
+  @JsonProperty
+  @NotNull
+  private Set<String> excludedNativeQueryTypes = Set.of(

Review Comment:
   Thanks, yes, re-adding `timeBoundary` to the provider default would resolve the mismatch with the docs/tests and keep the documented default behavior accurate.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["mshahid6 (via GitHub)" <gi...@apache.org>]

mshahid6 commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3133668825


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLoggerProvider.java:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.guice.annotations.Json;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.server.DruidNode;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.server.log.RequestLoggerProvider;
+
+import javax.annotation.Nullable;
+import javax.validation.constraints.NotNull;
+import java.util.Set;
+
+/**
+ *  Configure logging type, namespace, transport type (http or default console), transportUrl in {@code runtime.properties} 
+ */
+@JsonTypeName("openlineage")
+public class OpenLineageRequestLoggerProvider implements RequestLoggerProvider
+{
+  private static final Logger log = new Logger(OpenLineageRequestLoggerProvider.class);
+
+  public enum TransportType
+  {
+    CONSOLE,
+    HTTP
+  }
+
+  @JacksonInject
+  @Json
+  @NotNull
+  private ObjectMapper jsonMapper;
+
+  @JsonProperty
+  @NotNull
+  private String namespace = "druid://" + DruidNode.getDefaultHost();
+
+  @JsonProperty
+  @NotNull
+  private TransportType transportType = TransportType.CONSOLE;
+
+  @Nullable
+  @JsonProperty
+  private String transportUrl;
+
+  @JsonProperty
+  @NotNull
+  private Set<String> excludedNativeQueryTypes = Set.of(
+      "segmentMetadata",
+      "dataSourceMetadata",
+      "timeBoundary"

Review Comment:
   wasn't sure if it'd be useful for lineage and might be noisy. it's configurable but can remove it from defaults if you think it should be tracked



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["FrankChen021 (via GitHub)" <gi...@apache.org>]

FrankChen021 commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3173377492


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlNodeList;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.RejectedExecutionHandler;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  // Matches: INSERT INTO <table> ... (fallback when EXTERN prevents standard Calcite parsing)
+  // Handles both quoted ("table") and unquoted (table) identifiers.
+  private static final Pattern INSERT_INTO_PATTERN =
+      Pattern.compile("(?i)^\\s*INSERT\\s+INTO\\s+\"?([^\"\\s]+)\"?");
+  // Matches: REPLACE INTO <table> OVERWRITE ... (Druid-specific, not parseable by standard Calcite)
+  private static final Pattern REPLACE_INTO_PATTERN =
+      Pattern.compile("(?i)^\\s*REPLACE\\s+INTO\\s+\"?([^\"\\s]+)\"?");
+  // Extracts the SELECT subquery from a REPLACE INTO statement for input lineage
+  private static final Pattern REPLACE_SELECT_PATTERN =
+      Pattern.compile("(?is)\\b(SELECT\\s+.+?)(?:\\s+PARTITIONED\\s+BY|\\s+CLUSTERED\\s+BY|$)");
+
+  static final int DEFAULT_EMIT_QUEUE_CAPACITY = 1000;
+  static final int DEFAULT_EMIT_THREAD_COUNT = 1;
+  private static final int DISCARD_WARNING_INTERVAL = 1000;
+
+  static final String UNKNOWN_QUERY_ID = "unknown-query-id";
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  private final Set<String> excludedNativeQueryTypes;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+  private final AtomicLong discardedEventCount = new AtomicLong(0);
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         DEFAULT_EMIT_QUEUE_CAPACITY, DEFAULT_EMIT_THREAD_COUNT);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         emitQueueCapacity, emitThreadCount, null);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount,
+      @Nullable HttpClient httpClient
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    this.excludedNativeQueryTypes = excludedNativeQueryTypes;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = httpClient != null ? httpClient : HttpClientBuilder.create().build();
+      // Bounded queue: if the queue is full, drop the event rather than blocking the query thread.
+      // A warning is logged on the first drop and every DISCARD_WARNING_INTERVAL drops thereafter.
+      this.emitExecutor = new ThreadPoolExecutor(
+          emitThreadCount,
+          emitThreadCount,
+          60L,
+          TimeUnit.SECONDS,
+          new ArrayBlockingQueue<>(emitQueueCapacity),
+          Execs.makeThreadFactory("OpenLineageEmitter-%d"),
+          new DiscardWithWarningPolicy(discardedEventCount)
+      );
+    } else {
+      this.httpClient = null;
+      this.emitExecutor = null;
+    }
+  }
+
+  // Note: ComposingRequestLogger does not delegate @LifecycleStart to sub-loggers, so this method
+  // may not be called when used in a composing configuration. HTTP URL validation is therefore
+  // performed in the constructor instead. This method is retained for direct lifecycle use.
+  @LifecycleStart
+  @Override
+  public void start()
+  {
+    log.info(
+        "Started OpenLineage %s transport%s",
+        transportType,
+        transportUrl != null ? " to [" + transportUrl + "]" : ""
+    );
+  }
+
+  @LifecycleStop
+  @Override
+  public void stop()
+  {
+    if (emitExecutor != null) {
+      emitExecutor.shutdown();
+      try {
+        if (!emitExecutor.awaitTermination(5, TimeUnit.SECONDS)) {
+          emitExecutor.shutdownNow();
+        }
+      }
+      catch (InterruptedException e) {
+        emitExecutor.shutdownNow();
+        Thread.currentThread().interrupt();
+      }
+    }
+    if (httpClient instanceof Closeable) {
+      CloseableUtils.closeAndSuppressExceptions(
+          (Closeable) httpClient,
+          e -> log.warn(e, "Failed to close OpenLineage HTTP client")
+      );
+    }
+    log.info("Stopped OpenLineage request logger");
+  }
+
+  @Override
+  public void logNativeQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    if (requestLogLine.getQuery() == null) {
+      return;
+    }
+
+    // Skip native sub-queries of a SQL execution to avoid duplicating the SQL-level event.
+    if (requestLogLine.getQuery().getContext().get(BaseQuery.SQL_QUERY_ID) != null) {
+      return;
+    }
+
+    String queryType = requestLogLine.getQuery().getType();
+
+    if (excludedNativeQueryTypes.contains(queryType)) {
+      return;
+    }
+
+    List<String> inputs = new ArrayList<>(new LinkedHashSet<>(requestLogLine.getQuery().getDataSource().getTableNames()));
+    String queryId = requestLogLine.getQuery().getId();
+    if (queryId == null) {
+      log.debug("Native query reached OpenLineage logger without a query ID");
+      queryId = UNKNOWN_QUERY_ID;
+    }
+
+    emit(buildRunEvent(queryId, null, queryType, requestLogLine, inputs, null));
+  }
+
+  @Override
+  public void logSqlQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    String sql = requestLogLine.getSql();
+    List<String> inputs = new ArrayList<>();
+    String output = null;
+
+    if (sql != null) {
+      try {
+        SqlNode parsed = SqlParser.create(sql, SQL_PARSER_CONFIG).parseQuery();
+        inputs = extractInputs(parsed);
+        output = extractOutput(parsed);
+      }
+      catch (SqlParseException e) {
+        // Druid-specific SQL extensions (REPLACE INTO, EXTERN, etc.) may not parse with the
+        // standard Calcite parser. Attempt to extract lineage via regex fallback;
+        // for other unparseable statements, emit the event without table-level lineage.
+        Matcher insertMatcher = INSERT_INTO_PATTERN.matcher(sql);
+        if (insertMatcher.find()) {
+          output = insertMatcher.group(1);
+          // Also try to extract inputs from the SELECT subquery (handles PARTITIONED BY and
+          // other Druid-specific clauses that prevent full parsing). EXTERN-sourced inputs
+          // will fail to re-parse and fall through with empty inputs, which is correct.
+          Matcher selectMatcher = REPLACE_SELECT_PATTERN.matcher(sql);
+          if (selectMatcher.find()) {
+            try {
+              SqlNode selectNode = SqlParser.create(selectMatcher.group(1), SQL_PARSER_CONFIG).parseQuery();
+              inputs = extractInputs(selectNode);
+            }
+            catch (SqlParseException ignored) {
+              // EXTERN or other non-standard sources — inputs left empty
+            }
+          }
+        }
+        Matcher m = REPLACE_INTO_PATTERN.matcher(sql);
+        if (m.find()) {
+          output = m.group(1);
+          Matcher selectMatcher = REPLACE_SELECT_PATTERN.matcher(sql);
+          if (selectMatcher.find()) {
+            try {
+              SqlNode selectNode = SqlParser.create(selectMatcher.group(1), SQL_PARSER_CONFIG).parseQuery();
+              inputs = extractInputs(selectNode);
+            }
+            catch (SqlParseException ignored) {
+              // fall through with empty inputs
+            }
+          }
+        }
+        log.debug(
+            "OpenLineage: could not parse SQL with standard Calcite parser (query will still be emitted): %s",
+            e.getMessage()
+        );
+      }
+    }
+
+    String queryId = extractSqlQueryId(requestLogLine);
+    emit(buildRunEvent(queryId, sql, "sql", requestLogLine, new ArrayList<>(new LinkedHashSet<>(inputs)), output));
+  }
+
+  private ObjectNode buildRunEvent(
+      String queryId,
+      @Nullable String sql,
+      String queryType,
+      RequestLogLine requestLogLine,
+      List<String> inputs,
+      @Nullable String output
+  )
+  {
+    Map<String, Object> stats = requestLogLine.getQueryStats().getStats();
+    boolean success = Boolean.TRUE.equals(stats.get("success"));
+
+    ObjectNode event = jsonMapper.createObjectNode();
+    event.put("eventType", success ? "COMPLETE" : "FAIL");
+    event.put("eventTime", requestLogLine.getTimestamp().toInstant().toString());
+    event.put("producer", PRODUCER);
+    event.put("schemaURL", SCHEMA_URL);
+    event.set("run", buildRun(queryId, queryType, requestLogLine, stats, success));
+    event.set("job", buildJob(queryId, sql));
+    event.set("inputs", buildDatasets(inputs));
+    event.set("outputs", buildDatasets(output != null ? List.of(output) : List.of()));
+    return event;
+  }
+
+  private ObjectNode buildRun(
+      String queryId,
+      String queryType,
+      RequestLogLine requestLogLine,
+      Map<String, Object> stats,
+      boolean success
+  )
+  {
+    ObjectNode run = jsonMapper.createObjectNode();
+    run.put("runId", UUID.nameUUIDFromBytes(queryId.getBytes(StandardCharsets.UTF_8)).toString());
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode engineFacet = createFacet(ENGINE_FACET_SCHEMA_URL);
+    engineFacet.put("name", "druid");
+    engineFacet.put("version", getDruidVersion());
+    facets.set("processing_engine", engineFacet);
+
+    ObjectNode contextFacet = createFacet(null);
+    contextFacet.put("queryType", queryType);
+    contextFacet.put("remoteAddress", requestLogLine.getRemoteAddr());
+    Object identity = stats.get("identity");
+    if (identity != null) {
+      contextFacet.put("identity", identity.toString());
+    }
+    Object nativeQueryIds = requestLogLine.getSqlQueryContext().get("nativeQueryIds");
+    if (nativeQueryIds != null) {
+      contextFacet.put("nativeQueryIds", nativeQueryIds.toString());
+    }
+    facets.set("druid_query_context", contextFacet);
+
+    ObjectNode statsFacet = createFacet(null);
+    putLongStat(statsFacet, "durationMs", stats, "sqlQuery/time", "query/time");
+    putLongStat(statsFacet, "bytes", stats, "sqlQuery/bytes", "query/bytes");
+    putLongStat(statsFacet, "planningTimeMs", stats, "sqlQuery/planningTimeMs");
+    Object statusCode = stats.get("statusCode");
+    if (statusCode != null) {
+      statsFacet.put("statusCode", statusCode.toString());
+    }
+    facets.set("druid_query_statistics", statsFacet);
+
+    if (!success) {
+      Object exception = stats.get("exception");
+      if (exception != null) {
+        ObjectNode errorFacet = createFacet(ERROR_FACET_SCHEMA_URL);
+        errorFacet.put("message", exception.toString());
+        if ("sql".equals(queryType)) {
+          errorFacet.put("programmingLanguage", "SQL");
+        }
+        facets.set("errorMessage", errorFacet);
+      }
+    }
+
+    run.set("facets", facets);
+    return run;
+  }
+
+  private ObjectNode buildJob(String queryId, @Nullable String sql)
+  {
+    ObjectNode job = jsonMapper.createObjectNode();
+    job.put("namespace", namespace);
+    job.put("name", queryId);
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode jobTypeFacet = createFacet(JOB_TYPE_FACET_SCHEMA_URL);
+    jobTypeFacet.put("processingType", "BATCH");
+    jobTypeFacet.put("integration", "DRUID");
+    jobTypeFacet.put("jobType", "QUERY");
+    facets.set("jobType", jobTypeFacet);
+
+    if (sql != null) {
+      ObjectNode sqlFacet = createFacet(SQL_FACET_SCHEMA_URL);
+      sqlFacet.put("query", sql);
+      facets.set("sql", sqlFacet);
+    }
+
+    job.set("facets", facets);
+    return job;
+  }
+
+  private ObjectNode createFacet(@Nullable String schemaUrl)
+  {
+    ObjectNode facet = jsonMapper.createObjectNode();
+    facet.put("_producer", PRODUCER);
+    if (schemaUrl != null) {
+      facet.put("_schemaURL", schemaUrl);
+    }
+    return facet;
+  }
+
+  private ArrayNode buildDatasets(List<String> tableNames)
+  {
+    ArrayNode array = jsonMapper.createArrayNode();
+    for (String name : tableNames) {
+      ObjectNode node = jsonMapper.createObjectNode();
+      node.put("namespace", namespace);
+      node.put("name", name);
+      node.set("facets", jsonMapper.createObjectNode());
+      array.add(node);
+    }
+    return array;
+  }
+
+  private List<String> extractInputs(SqlNode root)
+  {
+    List<String> tables = new ArrayList<>();
+    if (root instanceof SqlInsert) {
+      // For INSERT/REPLACE, only walk the source query — the target table is an output, not an input.
+      collectTableNames(((SqlInsert) root).getSource(), tables, Set.of());
+    } else {
+      collectTableNames(root, tables, Set.of());
+    }
+    return tables;
+  }
+
+  @Nullable
+  private String extractOutput(SqlNode root)
+  {
+    if (root instanceof SqlInsert) {
+      SqlNode target = ((SqlInsert) root).getTargetTable();
+      if (target instanceof SqlIdentifier) {
+        return String.join(".", ((SqlIdentifier) target).names);
+      }
+    }
+    return null;
+  }
+
+  /**
+   * Walks the SQL tree to collect table references from FROM clauses, JOINs, subqueries in
+   * WHERE/HAVING/SELECT, and set operations (UNION/INTERSECT/EXCEPT). CTE alias names are
+   * tracked in {@code excludeNames} to avoid reporting them as table inputs.
+   *
+   * <p>The walker distinguishes table-reference positions (FROM, JOIN) from column-reference
+   * positions (SELECT list, WHERE expressions) by collecting {@link SqlIdentifier} nodes only
+   * from known table-reference contexts, while recursing into all subqueries to find nested
+   * FROM clauses.
+   */
+  private void collectTableNames(SqlNode node, List<String> tables, Set<String> excludeNames)
+  {
+    if (node == null) {
+      return;
+    }
+    if (node instanceof SqlWith) {
+      SqlWith with = (SqlWith) node;
+      Set<String> innerExcludes = new HashSet<>(excludeNames);
+      for (SqlNode item : with.withList) {
+        if (item instanceof SqlWithItem) {
+          innerExcludes.add(((SqlWithItem) item).name.getSimple());
+          collectTableNames(((SqlWithItem) item).query, tables, innerExcludes);
+        }
+      }
+      collectTableNames(with.body, tables, innerExcludes);
+    } else if (node instanceof SqlNodeList) {
+      // Lists (e.g., SELECT list items): recurse into each element to find subqueries.
+      for (SqlNode item : (SqlNodeList) node) {
+        collectTableNames(item, tables, excludeNames);
+      }
+    } else if (node instanceof SqlSelect) {
+      SqlSelect select = (SqlSelect) node;
+      // FROM clause: identifiers here are table references.
+      collectFromPosition(select.getFrom(), tables, excludeNames);
+      // Recurse into WHERE, HAVING, and SELECT list to find nested subqueries.
+      collectTableNames(select.getWhere(), tables, excludeNames);
+      collectTableNames(select.getHaving(), tables, excludeNames);
+      collectTableNames(select.getSelectList(), tables, excludeNames);
+    } else if (node instanceof SqlCall) {
+      SqlCall call = (SqlCall) node;
+      if (call.getKind() == SqlKind.UNION || call.getKind() == SqlKind.INTERSECT || call.getKind() == SqlKind.EXCEPT) {
+        // Set operations: each operand is a SELECT whose FROM clause has table references.
+        for (SqlNode operand : call.getOperandList()) {
+          collectTableNames(operand, tables, excludeNames);
+        }
+      } else {
+        // Other expressions (AND, OR, IN, =, function calls, etc.): recurse to find subqueries.
+        for (SqlNode operand : call.getOperandList()) {
+          if (operand instanceof SqlSelect || operand instanceof SqlWith || operand instanceof SqlCall) {
+            collectTableNames(operand, tables, excludeNames);
+          }
+        }
+      }
+    }
+  }
+
+  /**
+   * Collects table names from a FROM-clause position, where {@link SqlIdentifier} nodes
+   * represent table references (not column references).
+   */
+  private void collectFromPosition(SqlNode node, List<String> tables, Set<String> excludeNames)
+  {
+    if (node == null) {
+      return;
+    }
+    if (node instanceof SqlIdentifier) {
+      String name = String.join(".", ((SqlIdentifier) node).names);
+      if (!excludeNames.contains(name)) {
+        tables.add(name);
+      }
+    } else if (node instanceof SqlJoin) {
+      SqlJoin join = (SqlJoin) node;
+      collectFromPosition(join.getLeft(), tables, excludeNames);
+      collectFromPosition(join.getRight(), tables, excludeNames);
+    } else if (node instanceof SqlSelect) {
+      // Subquery in FROM: recurse into the full select.
+      collectTableNames(node, tables, excludeNames);
+    } else if (node instanceof SqlBasicCall && node.getKind() == SqlKind.AS) {
+      // Alias: the table reference is the first operand.
+      collectFromPosition(((SqlBasicCall) node).operand(0), tables, excludeNames);
+    } else if (node instanceof SqlCall) {

Review Comment:
   P2 UNNEST column operands are reported as input datasets
   
   The generic SqlCall branch in collectFromPosition recurses into every operand and then treats any SqlIdentifier as a table reference. For Druid SQL such as SELECT a FROM druid.arrays, UNNEST(arrayString) AS u(a), the UNNEST operand arrayString is a column, not a datasource, but this walker will add it to the OpenLineage inputs alongside druid.arrays. This produces incorrect lineage for supported UNNEST queries; table-function or UNNEST calls should avoid collecting column operands as datasets, or only recurse into operands that are known relation-producing subqueries.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["jtuglu1 (via GitHub)" <gi...@apache.org>]

jtuglu1 commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3132193799


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,503 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOrderBy;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  private static final int EMIT_QUEUE_CAPACITY = 1000;
+  private static final int EMIT_THREAD_COUNT = 1;
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  private final Set<String> excludedNativeQueryTypes;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes, null);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      @Nullable HttpClient httpClient
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    this.excludedNativeQueryTypes = excludedNativeQueryTypes;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = httpClient != null ? httpClient : HttpClientBuilder.create().build();
+      // Bounded queue: if the queue is full, silently drop the event rather than blocking
+      // the query thread. Uses Druid's Execs for daemon thread naming conventions.
+      this.emitExecutor = new ThreadPoolExecutor(
+          EMIT_THREAD_COUNT,
+          EMIT_THREAD_COUNT,
+          60L,
+          TimeUnit.SECONDS,
+          new ArrayBlockingQueue<>(EMIT_QUEUE_CAPACITY),
+          Execs.makeThreadFactory("OpenLineageEmitter-%d"),
+          new ThreadPoolExecutor.DiscardPolicy()
+      );
+    } else {
+      this.httpClient = null;
+      this.emitExecutor = null;
+    }
+  }
+
+  @LifecycleStart
+  @Override
+  public void start() throws Exception
+  {
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      log.info("Started OpenLineage HTTP transport to [%s]", transportUrl);
+    } else {
+      log.info("Started OpenLineage console transport");
+    }
+  }
+
+  @LifecycleStop
+  @Override
+  public void stop()
+  {
+    if (emitExecutor != null) {
+      emitExecutor.shutdown();
+      try {
+        if (!emitExecutor.awaitTermination(5, TimeUnit.SECONDS)) {
+          emitExecutor.shutdownNow();
+        }
+      }
+      catch (InterruptedException e) {
+        emitExecutor.shutdownNow();
+        Thread.currentThread().interrupt();
+      }
+    }
+    if (httpClient instanceof Closeable) {
+      CloseableUtils.closeAndSuppressExceptions(
+          (Closeable) httpClient,
+          e -> log.warn(e, "Failed to close OpenLineage HTTP client")
+      );
+    }
+    log.info("Stopped OpenLineage request logger");
+  }
+
+  @Override
+  public void logNativeQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    if (requestLogLine.getQuery() == null) {
+      return;
+    }
+
+    // Skip native sub-queries of a SQL execution to avoid duplicating the SQL-level event.
+    if (requestLogLine.getQuery().getContext().get(BaseQuery.SQL_QUERY_ID) != null) {
+      return;
+    }
+
+    String queryType = requestLogLine.getQuery().getType();
+
+    if (excludedNativeQueryTypes.contains(queryType)) {
+      return;
+    }
+
+    List<String> inputs = new ArrayList<>(new LinkedHashSet<>(requestLogLine.getQuery().getDataSource().getTableNames()));
+    String queryId = requestLogLine.getQuery().getId();
+    if (queryId == null) {
+      queryId = UUID.randomUUID().toString();
+    }
+
+    emit(buildRunEvent(queryId, null, queryType, requestLogLine, inputs, null));
+  }
+
+  @Override
+  public void logSqlQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    String sql = requestLogLine.getSql();
+    List<String> inputs = new ArrayList<>();
+    String output = null;
+
+    if (sql != null) {
+      try {
+        SqlNode parsed = SqlParser.create(sql, SQL_PARSER_CONFIG).parseQuery();
+        inputs = extractInputs(parsed);
+        output = extractOutput(parsed);
+      }
+      catch (SqlParseException e) {
+        // Druid-specific SQL extensions (REPLACE, EXTERN, etc.) may not parse with the standard
+        // Calcite parser. Emit the event without table-level lineage rather than failing.
+        log.debug(
+            "OpenLineage: could not parse SQL for lineage extraction (query will still be emitted): %s",
+            e.getMessage()
+        );
+      }
+    }
+
+    String queryId = extractSqlQueryId(requestLogLine);
+    emit(buildRunEvent(queryId, sql, "sql", requestLogLine, new ArrayList<>(new LinkedHashSet<>(inputs)), output));
+  }
+
+  private ObjectNode buildRunEvent(
+      String queryId,
+      @Nullable String sql,
+      String queryType,
+      RequestLogLine requestLogLine,
+      List<String> inputs,
+      @Nullable String output
+  )
+  {
+    Map<String, Object> stats = requestLogLine.getQueryStats().getStats();
+    boolean success = Boolean.TRUE.equals(stats.get("success"));
+
+    ObjectNode event = jsonMapper.createObjectNode();
+    event.put("eventType", success ? "COMPLETE" : "FAIL");
+    event.put("eventTime", requestLogLine.getTimestamp().toInstant().toString());
+    event.put("producer", PRODUCER);
+    event.put("schemaURL", SCHEMA_URL);
+    event.set("run", buildRun(queryId, queryType, requestLogLine, stats, success));
+    event.set("job", buildJob(queryId, sql));
+    event.set("inputs", buildDatasets(inputs));
+    event.set("outputs", buildDatasets(output != null ? List.of(output) : List.of()));
+    return event;
+  }
+
+  private ObjectNode buildRun(
+      String queryId,
+      String queryType,
+      RequestLogLine requestLogLine,
+      Map<String, Object> stats,
+      boolean success
+  )
+  {
+    ObjectNode run = jsonMapper.createObjectNode();
+    run.put("runId", UUID.nameUUIDFromBytes(queryId.getBytes(StandardCharsets.UTF_8)).toString());
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode engineFacet = createFacet(ENGINE_FACET_SCHEMA_URL);
+    engineFacet.put("name", "druid");
+    engineFacet.put("version", getDruidVersion());
+    facets.set("processing_engine", engineFacet);
+
+    ObjectNode contextFacet = createFacet(null);
+    contextFacet.put("queryType", queryType);
+    contextFacet.put("remoteAddress", requestLogLine.getRemoteAddr());
+    Object identity = stats.get("identity");
+    if (identity != null) {
+      contextFacet.put("identity", identity.toString());
+    }
+    Object nativeQueryIds = requestLogLine.getSqlQueryContext().get("nativeQueryIds");
+    if (nativeQueryIds != null) {
+      contextFacet.put("nativeQueryIds", nativeQueryIds.toString());
+    }
+    facets.set("druid_query_context", contextFacet);
+
+    ObjectNode statsFacet = createFacet(null);
+    putLongStat(statsFacet, "durationMs", stats, "sqlQuery/time", "query/time");
+    putLongStat(statsFacet, "bytes", stats, "sqlQuery/bytes", "query/bytes");
+    putLongStat(statsFacet, "planningTimeMs", stats, "sqlQuery/planningTimeMs");
+    Object statusCode = stats.get("statusCode");
+    if (statusCode != null) {
+      statsFacet.put("statusCode", statusCode.toString());
+    }
+    facets.set("druid_query_statistics", statsFacet);
+
+    if (!success) {
+      Object exception = stats.get("exception");
+      if (exception != null) {
+        ObjectNode errorFacet = createFacet(ERROR_FACET_SCHEMA_URL);
+        errorFacet.put("message", exception.toString());
+        if ("sql".equals(queryType)) {
+          errorFacet.put("programmingLanguage", "SQL");
+        }
+        facets.set("errorMessage", errorFacet);
+      }
+    }
+
+    run.set("facets", facets);
+    return run;
+  }
+
+  private ObjectNode buildJob(String queryId, @Nullable String sql)
+  {
+    ObjectNode job = jsonMapper.createObjectNode();
+    job.put("namespace", namespace);
+    job.put("name", queryId);
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode jobTypeFacet = createFacet(JOB_TYPE_FACET_SCHEMA_URL);
+    jobTypeFacet.put("processingType", "BATCH");
+    jobTypeFacet.put("integration", "DRUID");
+    jobTypeFacet.put("jobType", "QUERY");
+    facets.set("jobType", jobTypeFacet);
+
+    if (sql != null) {
+      ObjectNode sqlFacet = createFacet(SQL_FACET_SCHEMA_URL);
+      sqlFacet.put("query", sql);
+      facets.set("sql", sqlFacet);
+    }
+
+    job.set("facets", facets);
+    return job;
+  }
+
+  private ObjectNode createFacet(@Nullable String schemaUrl)
+  {
+    ObjectNode facet = jsonMapper.createObjectNode();
+    facet.put("_producer", PRODUCER);
+    if (schemaUrl != null) {
+      facet.put("_schemaURL", schemaUrl);
+    }
+    return facet;
+  }
+
+  private ArrayNode buildDatasets(List<String> tableNames)
+  {
+    ArrayNode array = jsonMapper.createArrayNode();
+    for (String name : tableNames) {
+      ObjectNode node = jsonMapper.createObjectNode();
+      node.put("namespace", namespace);
+      node.put("name", name);
+      node.set("facets", jsonMapper.createObjectNode());
+      array.add(node);
+    }
+    return array;
+  }
+
+  private List<String> extractInputs(SqlNode root)
+  {
+    List<String> tables = new ArrayList<>();
+    if (root instanceof SqlWith) {
+      SqlWith with = (SqlWith) root;
+      Set<String> cteNames = new HashSet<>();
+      for (SqlNode item : with.withList) {
+        if (item instanceof SqlWithItem) {
+          cteNames.add(((SqlWithItem) item).name.getSimple());
+          collectFromClause(((SqlWithItem) item).query, tables, cteNames);
+        }
+      }
+      collectFromClause(with.body, tables, cteNames);
+    } else if (root instanceof SqlInsert) {
+      collectFromClause(((SqlInsert) root).getSource(), tables, Set.of());
+    } else {
+      collectFromClause(root, tables, Set.of());
+    }
+    return tables;
+  }
+
+  @Nullable
+  private String extractOutput(SqlNode root)
+  {
+    if (root instanceof SqlInsert) {
+      SqlNode target = ((SqlInsert) root).getTargetTable();
+      if (target instanceof SqlIdentifier) {
+        return String.join(".", ((SqlIdentifier) target).names);
+      }
+    }
+    return null;
+  }
+
+  private void collectFromClause(SqlNode from, List<String> tables, Set<String> excludeNames)
+  {
+    if (from == null) {
+      return;
+    }
+    if (from instanceof SqlIdentifier) {
+      String name = String.join(".", ((SqlIdentifier) from).names);
+      if (!excludeNames.contains(name)) {
+        tables.add(name);
+      }
+    } else if (from instanceof SqlJoin) {
+      SqlJoin join = (SqlJoin) from;
+      collectFromClause(join.getLeft(), tables, excludeNames);
+      collectFromClause(join.getRight(), tables, excludeNames);
+    } else if (from instanceof SqlSelect) {
+      collectFromClause(((SqlSelect) from).getFrom(), tables, excludeNames);
+    } else if (from instanceof SqlWith) {
+      SqlWith with = (SqlWith) from;
+      Set<String> innerExcludes = new HashSet<>(excludeNames);
+      for (SqlNode item : with.withList) {
+        if (item instanceof SqlWithItem) {
+          innerExcludes.add(((SqlWithItem) item).name.getSimple());
+          collectFromClause(((SqlWithItem) item).query, tables, innerExcludes);
+        }
+      }
+      collectFromClause(with.body, tables, innerExcludes);
+    } else if (from instanceof SqlOrderBy) {
+      collectFromClause(((SqlOrderBy) from).query, tables, excludeNames);
+    } else if (from instanceof SqlBasicCall && from.getKind() == SqlKind.AS) {
+      collectFromClause(((SqlBasicCall) from).operand(0), tables, excludeNames);
+    }
+  }
+
+  protected void emit(ObjectNode event)
+  {
+    try {
+      String json = jsonMapper.writeValueAsString(event);
+      if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+        emitExecutor.submit(() -> emitHttp(json));
+      } else {
+        log.info("OpenLineage event: %s", json);
+      }
+    }
+    catch (IOException e) {
+      log.error(e, "Failed to serialize OpenLineage event");
+    }
+  }
+
+  private void emitHttp(String json)
+  {
+    HttpPost post = new HttpPost(transportUrl);
+    post.setEntity(new StringEntity(json, ContentType.APPLICATION_JSON));
+    try {
+      org.apache.http.HttpResponse response = httpClient.execute(post);
+      EntityUtils.consumeQuietly(response.getEntity());
+    }
+    catch (IOException e) {
+      log.error(e, "Failed to POST OpenLineage event to [%s]", transportUrl);
+    }
+    finally {
+      post.releaseConnection();
+    }
+  }
+
+  private String extractSqlQueryId(RequestLogLine requestLogLine)
+  {
+    Object id = requestLogLine.getSqlQueryContext().get(BaseQuery.SQL_QUERY_ID);
+    if (id != null) {
+      return id.toString();
+    }
+    return UUID.randomUUID().toString();

Review Comment:
   Unless this uniqueness is absolutely required by the spec I feel like we should make this some default fixed value so we can know if there are queries somehow making it through without an ID.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["mshahid6 (via GitHub)" <gi...@apache.org>]

mshahid6 commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3054189925


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,498 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOrderBy;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  private static final int EMIT_QUEUE_CAPACITY = 1000;
+  private static final int EMIT_THREAD_COUNT = 1;
+
+  /**
+   * Internal broker query types that are automatically issued for schema discovery and metadata
+   * caching. These are not user-initiated and produce noisy, low-value lineage events.
+   */
+  private static final Set<String> INTERNAL_QUERY_TYPES = Set.of(
+      "segmentMetadata",
+      "dataSourceMetadata",
+      "timeBoundary"
+  );
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = HttpClientBuilder.create().build();
+      // Bounded queue: if the queue is full, silently drop the event rather than blocking
+      // the query thread. Uses Druid's Execs for daemon thread naming conventions.
+      this.emitExecutor = new ThreadPoolExecutor(
+          EMIT_THREAD_COUNT,
+          EMIT_THREAD_COUNT,
+          60L,
+          TimeUnit.SECONDS,
+          new ArrayBlockingQueue<>(EMIT_QUEUE_CAPACITY),
+          Execs.makeThreadFactory("OpenLineageEmitter-%d"),
+          new ThreadPoolExecutor.DiscardPolicy()
+      );
+    } else {
+      this.httpClient = null;
+      this.emitExecutor = null;
+    }
+  }
+
+  @LifecycleStart
+  @Override
+  public void start() throws Exception
+  {
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      log.info("Started OpenLineage HTTP transport to [%s]", transportUrl);
+    } else {
+      log.info("Started OpenLineage console transport");
+    }
+  }
+
+  @LifecycleStop
+  @Override
+  public void stop()
+  {
+    if (emitExecutor != null) {
+      emitExecutor.shutdown();
+      try {
+        if (!emitExecutor.awaitTermination(5, TimeUnit.SECONDS)) {
+          emitExecutor.shutdownNow();
+        }
+      }
+      catch (InterruptedException e) {
+        emitExecutor.shutdownNow();
+        Thread.currentThread().interrupt();
+      }
+    }
+    if (httpClient instanceof Closeable) {
+      CloseableUtils.closeAndSuppressExceptions(
+          (Closeable) httpClient,
+          e -> log.warn(e, "Failed to close OpenLineage HTTP client")
+      );
+    }
+    log.info("Stopped OpenLineage request logger");
+  }
+
+  @Override
+  public void logNativeQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    if (requestLogLine.getQuery() == null) {
+      return;
+    }
+
+    // Skip native sub-queries of a SQL execution to avoid duplicating the SQL-level event.
+    if (requestLogLine.getQuery().getContext().get(BaseQuery.SQL_QUERY_ID) != null) {
+      return;
+    }
+
+    String queryType = requestLogLine.getQuery().getType();
+
+    if (INTERNAL_QUERY_TYPES.contains(queryType)) {
+      return;
+    }
+
+    List<String> inputs = new ArrayList<>(new LinkedHashSet<>(requestLogLine.getQuery().getDataSource().getTableNames()));
+    String queryId = requestLogLine.getQuery().getId();
+    if (queryId == null) {
+      queryId = UUID.randomUUID().toString();
+    }
+
+    emit(buildRunEvent(queryId, null, queryType, requestLogLine, inputs, null));
+  }
+
+  @Override
+  public void logSqlQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    String sql = requestLogLine.getSql();
+    List<String> inputs = new ArrayList<>();
+    String output = null;
+
+    if (sql != null) {
+      try {
+        SqlNode parsed = SqlParser.create(sql, SQL_PARSER_CONFIG).parseQuery();
+        inputs = extractInputs(parsed);
+        output = extractOutput(parsed);
+      }
+      catch (SqlParseException e) {
+        // Druid-specific SQL extensions (REPLACE, EXTERN, etc.) may not parse with the standard
+        // Calcite parser. Emit the event without table-level lineage rather than failing.
+        log.debug(
+            "OpenLineage: could not parse SQL for lineage extraction (query will still be emitted): %s",
+            e.getMessage()
+        );
+      }
+    }
+
+    String queryId = extractSqlQueryId(requestLogLine);
+    emit(buildRunEvent(queryId, sql, "sql", requestLogLine, new ArrayList<>(new LinkedHashSet<>(inputs)), output));
+  }
+
+  private ObjectNode buildRunEvent(
+      String queryId,
+      @Nullable String sql,
+      String queryType,
+      RequestLogLine requestLogLine,
+      List<String> inputs,
+      @Nullable String output
+  )
+  {
+    Map<String, Object> stats = requestLogLine.getQueryStats().getStats();
+    boolean success = Boolean.TRUE.equals(stats.get("success"));
+
+    ObjectNode event = jsonMapper.createObjectNode();
+    event.put("eventType", success ? "COMPLETE" : "FAIL");
+    event.put("eventTime", requestLogLine.getTimestamp().toInstant().toString());
+    event.put("producer", PRODUCER);
+    event.put("schemaURL", SCHEMA_URL);
+    event.set("run", buildRun(queryId, queryType, requestLogLine, stats, success));
+    event.set("job", buildJob(queryId, sql));
+    event.set("inputs", buildDatasets(inputs));
+    event.set("outputs", buildDatasets(output != null ? List.of(output) : List.of()));
+    return event;
+  }
+
+  private ObjectNode buildRun(
+      String queryId,
+      String queryType,
+      RequestLogLine requestLogLine,
+      Map<String, Object> stats,
+      boolean success
+  )
+  {
+    ObjectNode run = jsonMapper.createObjectNode();
+    run.put("runId", UUID.nameUUIDFromBytes(queryId.getBytes(StandardCharsets.UTF_8)).toString());
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode engineFacet = createFacet(ENGINE_FACET_SCHEMA_URL);
+    engineFacet.put("name", "druid");
+    engineFacet.put("version", getDruidVersion());
+    facets.set("processing_engine", engineFacet);
+
+    ObjectNode contextFacet = createFacet(null);
+    contextFacet.put("queryType", queryType);
+    contextFacet.put("remoteAddress", requestLogLine.getRemoteAddr());
+    Object identity = stats.get("identity");
+    if (identity != null) {
+      contextFacet.put("identity", identity.toString());
+    }
+    Object nativeQueryIds = requestLogLine.getSqlQueryContext().get("nativeQueryIds");
+    if (nativeQueryIds != null) {
+      contextFacet.put("nativeQueryIds", nativeQueryIds.toString());
+    }
+    facets.set("druid_query_context", contextFacet);
+
+    ObjectNode statsFacet = createFacet(null);
+    putLongStat(statsFacet, "durationMs", stats, "sqlQuery/time", "query/time");
+    putLongStat(statsFacet, "bytes", stats, "sqlQuery/bytes", "query/bytes");
+    putLongStat(statsFacet, "planningTimeMs", stats, "sqlQuery/planningTimeMs");
+    Object statusCode = stats.get("statusCode");
+    if (statusCode != null) {
+      statsFacet.put("statusCode", statusCode.toString());
+    }
+    facets.set("druid_query_statistics", statsFacet);
+
+    if (!success) {
+      Object exception = stats.get("exception");
+      if (exception != null) {
+        ObjectNode errorFacet = createFacet(ERROR_FACET_SCHEMA_URL);
+        errorFacet.put("message", exception.toString());
+        if ("sql".equals(queryType)) {
+          errorFacet.put("programmingLanguage", "SQL");
+        }
+        facets.set("errorMessage", errorFacet);
+      }
+    }
+
+    run.set("facets", facets);
+    return run;
+  }
+
+  private ObjectNode buildJob(String queryId, @Nullable String sql)
+  {
+    ObjectNode job = jsonMapper.createObjectNode();
+    job.put("namespace", namespace);
+    job.put("name", queryId);
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode jobTypeFacet = createFacet(JOB_TYPE_FACET_SCHEMA_URL);
+    jobTypeFacet.put("processingType", "BATCH");
+    jobTypeFacet.put("integration", "DRUID");
+    jobTypeFacet.put("jobType", "QUERY");
+    facets.set("jobType", jobTypeFacet);
+
+    if (sql != null) {

Review Comment:
   do you mean if we should only emit lineage if sql?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["jtuglu1 (via GitHub)" <gi...@apache.org>]

jtuglu1 commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3132206846


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,498 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOrderBy;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  private static final int EMIT_QUEUE_CAPACITY = 1000;
+  private static final int EMIT_THREAD_COUNT = 1;
+
+  /**
+   * Internal broker query types that are automatically issued for schema discovery and metadata
+   * caching. These are not user-initiated and produce noisy, low-value lineage events.
+   */
+  private static final Set<String> INTERNAL_QUERY_TYPES = Set.of(
+      "segmentMetadata",
+      "dataSourceMetadata",
+      "timeBoundary"
+  );
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = HttpClientBuilder.create().build();
+      // Bounded queue: if the queue is full, silently drop the event rather than blocking
+      // the query thread. Uses Druid's Execs for daemon thread naming conventions.
+      this.emitExecutor = new ThreadPoolExecutor(
+          EMIT_THREAD_COUNT,
+          EMIT_THREAD_COUNT,
+          60L,
+          TimeUnit.SECONDS,
+          new ArrayBlockingQueue<>(EMIT_QUEUE_CAPACITY),
+          Execs.makeThreadFactory("OpenLineageEmitter-%d"),
+          new ThreadPoolExecutor.DiscardPolicy()
+      );
+    } else {
+      this.httpClient = null;
+      this.emitExecutor = null;
+    }
+  }
+
+  @LifecycleStart
+  @Override
+  public void start() throws Exception
+  {
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      log.info("Started OpenLineage HTTP transport to [%s]", transportUrl);
+    } else {
+      log.info("Started OpenLineage console transport");
+    }
+  }
+
+  @LifecycleStop
+  @Override
+  public void stop()
+  {
+    if (emitExecutor != null) {
+      emitExecutor.shutdown();
+      try {
+        if (!emitExecutor.awaitTermination(5, TimeUnit.SECONDS)) {
+          emitExecutor.shutdownNow();
+        }
+      }
+      catch (InterruptedException e) {
+        emitExecutor.shutdownNow();
+        Thread.currentThread().interrupt();
+      }
+    }
+    if (httpClient instanceof Closeable) {
+      CloseableUtils.closeAndSuppressExceptions(
+          (Closeable) httpClient,
+          e -> log.warn(e, "Failed to close OpenLineage HTTP client")
+      );
+    }
+    log.info("Stopped OpenLineage request logger");
+  }
+
+  @Override
+  public void logNativeQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    if (requestLogLine.getQuery() == null) {
+      return;
+    }
+
+    // Skip native sub-queries of a SQL execution to avoid duplicating the SQL-level event.
+    if (requestLogLine.getQuery().getContext().get(BaseQuery.SQL_QUERY_ID) != null) {
+      return;
+    }
+
+    String queryType = requestLogLine.getQuery().getType();
+
+    if (INTERNAL_QUERY_TYPES.contains(queryType)) {
+      return;
+    }
+
+    List<String> inputs = new ArrayList<>(new LinkedHashSet<>(requestLogLine.getQuery().getDataSource().getTableNames()));
+    String queryId = requestLogLine.getQuery().getId();
+    if (queryId == null) {
+      queryId = UUID.randomUUID().toString();
+    }
+
+    emit(buildRunEvent(queryId, null, queryType, requestLogLine, inputs, null));
+  }
+
+  @Override
+  public void logSqlQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    String sql = requestLogLine.getSql();
+    List<String> inputs = new ArrayList<>();
+    String output = null;
+
+    if (sql != null) {
+      try {
+        SqlNode parsed = SqlParser.create(sql, SQL_PARSER_CONFIG).parseQuery();
+        inputs = extractInputs(parsed);
+        output = extractOutput(parsed);
+      }
+      catch (SqlParseException e) {
+        // Druid-specific SQL extensions (REPLACE, EXTERN, etc.) may not parse with the standard
+        // Calcite parser. Emit the event without table-level lineage rather than failing.
+        log.debug(
+            "OpenLineage: could not parse SQL for lineage extraction (query will still be emitted): %s",
+            e.getMessage()
+        );
+      }
+    }
+
+    String queryId = extractSqlQueryId(requestLogLine);
+    emit(buildRunEvent(queryId, sql, "sql", requestLogLine, new ArrayList<>(new LinkedHashSet<>(inputs)), output));
+  }
+
+  private ObjectNode buildRunEvent(
+      String queryId,
+      @Nullable String sql,
+      String queryType,
+      RequestLogLine requestLogLine,
+      List<String> inputs,
+      @Nullable String output
+  )
+  {
+    Map<String, Object> stats = requestLogLine.getQueryStats().getStats();
+    boolean success = Boolean.TRUE.equals(stats.get("success"));
+
+    ObjectNode event = jsonMapper.createObjectNode();
+    event.put("eventType", success ? "COMPLETE" : "FAIL");
+    event.put("eventTime", requestLogLine.getTimestamp().toInstant().toString());
+    event.put("producer", PRODUCER);
+    event.put("schemaURL", SCHEMA_URL);
+    event.set("run", buildRun(queryId, queryType, requestLogLine, stats, success));
+    event.set("job", buildJob(queryId, sql));
+    event.set("inputs", buildDatasets(inputs));
+    event.set("outputs", buildDatasets(output != null ? List.of(output) : List.of()));
+    return event;
+  }
+
+  private ObjectNode buildRun(
+      String queryId,
+      String queryType,
+      RequestLogLine requestLogLine,
+      Map<String, Object> stats,
+      boolean success
+  )
+  {
+    ObjectNode run = jsonMapper.createObjectNode();
+    run.put("runId", UUID.nameUUIDFromBytes(queryId.getBytes(StandardCharsets.UTF_8)).toString());
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode engineFacet = createFacet(ENGINE_FACET_SCHEMA_URL);
+    engineFacet.put("name", "druid");
+    engineFacet.put("version", getDruidVersion());
+    facets.set("processing_engine", engineFacet);
+
+    ObjectNode contextFacet = createFacet(null);
+    contextFacet.put("queryType", queryType);
+    contextFacet.put("remoteAddress", requestLogLine.getRemoteAddr());
+    Object identity = stats.get("identity");
+    if (identity != null) {
+      contextFacet.put("identity", identity.toString());
+    }
+    Object nativeQueryIds = requestLogLine.getSqlQueryContext().get("nativeQueryIds");
+    if (nativeQueryIds != null) {
+      contextFacet.put("nativeQueryIds", nativeQueryIds.toString());
+    }
+    facets.set("druid_query_context", contextFacet);
+
+    ObjectNode statsFacet = createFacet(null);
+    putLongStat(statsFacet, "durationMs", stats, "sqlQuery/time", "query/time");
+    putLongStat(statsFacet, "bytes", stats, "sqlQuery/bytes", "query/bytes");
+    putLongStat(statsFacet, "planningTimeMs", stats, "sqlQuery/planningTimeMs");
+    Object statusCode = stats.get("statusCode");
+    if (statusCode != null) {
+      statsFacet.put("statusCode", statusCode.toString());
+    }
+    facets.set("druid_query_statistics", statsFacet);
+
+    if (!success) {
+      Object exception = stats.get("exception");
+      if (exception != null) {
+        ObjectNode errorFacet = createFacet(ERROR_FACET_SCHEMA_URL);
+        errorFacet.put("message", exception.toString());
+        if ("sql".equals(queryType)) {
+          errorFacet.put("programmingLanguage", "SQL");
+        }
+        facets.set("errorMessage", errorFacet);
+      }
+    }
+
+    run.set("facets", facets);
+    return run;
+  }
+
+  private ObjectNode buildJob(String queryId, @Nullable String sql)
+  {
+    ObjectNode job = jsonMapper.createObjectNode();
+    job.put("namespace", namespace);
+    job.put("name", queryId);
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode jobTypeFacet = createFacet(JOB_TYPE_FACET_SCHEMA_URL);
+    jobTypeFacet.put("processingType", "BATCH");
+    jobTypeFacet.put("integration", "DRUID");
+    jobTypeFacet.put("jobType", "QUERY");
+    facets.set("jobType", jobTypeFacet);
+
+    if (sql != null) {

Review Comment:
   We should be emitting lineage for both types. `logNativeQuery` handles the native path.
   
   One thing to call out is to check whether issuing an MSQ/Dart query triggers a lineage emission (I suspect it will, but just to be safe).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["mshahid6 (via GitHub)" <gi...@apache.org>]

mshahid6 commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3134211800


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,498 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOrderBy;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  private static final int EMIT_QUEUE_CAPACITY = 1000;
+  private static final int EMIT_THREAD_COUNT = 1;
+
+  /**
+   * Internal broker query types that are automatically issued for schema discovery and metadata
+   * caching. These are not user-initiated and produce noisy, low-value lineage events.
+   */
+  private static final Set<String> INTERNAL_QUERY_TYPES = Set.of(
+      "segmentMetadata",
+      "dataSourceMetadata",
+      "timeBoundary"
+  );
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = HttpClientBuilder.create().build();
+      // Bounded queue: if the queue is full, silently drop the event rather than blocking
+      // the query thread. Uses Druid's Execs for daemon thread naming conventions.
+      this.emitExecutor = new ThreadPoolExecutor(
+          EMIT_THREAD_COUNT,
+          EMIT_THREAD_COUNT,
+          60L,
+          TimeUnit.SECONDS,
+          new ArrayBlockingQueue<>(EMIT_QUEUE_CAPACITY),
+          Execs.makeThreadFactory("OpenLineageEmitter-%d"),
+          new ThreadPoolExecutor.DiscardPolicy()
+      );
+    } else {
+      this.httpClient = null;
+      this.emitExecutor = null;
+    }
+  }
+
+  @LifecycleStart
+  @Override
+  public void start() throws Exception
+  {
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      log.info("Started OpenLineage HTTP transport to [%s]", transportUrl);
+    } else {
+      log.info("Started OpenLineage console transport");
+    }
+  }
+
+  @LifecycleStop
+  @Override
+  public void stop()
+  {
+    if (emitExecutor != null) {
+      emitExecutor.shutdown();
+      try {
+        if (!emitExecutor.awaitTermination(5, TimeUnit.SECONDS)) {
+          emitExecutor.shutdownNow();
+        }
+      }
+      catch (InterruptedException e) {
+        emitExecutor.shutdownNow();
+        Thread.currentThread().interrupt();
+      }
+    }
+    if (httpClient instanceof Closeable) {
+      CloseableUtils.closeAndSuppressExceptions(
+          (Closeable) httpClient,
+          e -> log.warn(e, "Failed to close OpenLineage HTTP client")
+      );
+    }
+    log.info("Stopped OpenLineage request logger");
+  }
+
+  @Override
+  public void logNativeQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    if (requestLogLine.getQuery() == null) {
+      return;
+    }
+
+    // Skip native sub-queries of a SQL execution to avoid duplicating the SQL-level event.
+    if (requestLogLine.getQuery().getContext().get(BaseQuery.SQL_QUERY_ID) != null) {
+      return;
+    }
+
+    String queryType = requestLogLine.getQuery().getType();
+
+    if (INTERNAL_QUERY_TYPES.contains(queryType)) {
+      return;
+    }
+
+    List<String> inputs = new ArrayList<>(new LinkedHashSet<>(requestLogLine.getQuery().getDataSource().getTableNames()));
+    String queryId = requestLogLine.getQuery().getId();
+    if (queryId == null) {
+      queryId = UUID.randomUUID().toString();
+    }
+
+    emit(buildRunEvent(queryId, null, queryType, requestLogLine, inputs, null));
+  }
+
+  @Override
+  public void logSqlQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    String sql = requestLogLine.getSql();
+    List<String> inputs = new ArrayList<>();
+    String output = null;
+
+    if (sql != null) {
+      try {
+        SqlNode parsed = SqlParser.create(sql, SQL_PARSER_CONFIG).parseQuery();
+        inputs = extractInputs(parsed);
+        output = extractOutput(parsed);
+      }
+      catch (SqlParseException e) {
+        // Druid-specific SQL extensions (REPLACE, EXTERN, etc.) may not parse with the standard
+        // Calcite parser. Emit the event without table-level lineage rather than failing.
+        log.debug(
+            "OpenLineage: could not parse SQL for lineage extraction (query will still be emitted): %s",
+            e.getMessage()
+        );
+      }
+    }
+
+    String queryId = extractSqlQueryId(requestLogLine);
+    emit(buildRunEvent(queryId, sql, "sql", requestLogLine, new ArrayList<>(new LinkedHashSet<>(inputs)), output));
+  }
+
+  private ObjectNode buildRunEvent(
+      String queryId,
+      @Nullable String sql,
+      String queryType,
+      RequestLogLine requestLogLine,
+      List<String> inputs,
+      @Nullable String output
+  )
+  {
+    Map<String, Object> stats = requestLogLine.getQueryStats().getStats();
+    boolean success = Boolean.TRUE.equals(stats.get("success"));
+
+    ObjectNode event = jsonMapper.createObjectNode();
+    event.put("eventType", success ? "COMPLETE" : "FAIL");
+    event.put("eventTime", requestLogLine.getTimestamp().toInstant().toString());
+    event.put("producer", PRODUCER);
+    event.put("schemaURL", SCHEMA_URL);
+    event.set("run", buildRun(queryId, queryType, requestLogLine, stats, success));
+    event.set("job", buildJob(queryId, sql));
+    event.set("inputs", buildDatasets(inputs));
+    event.set("outputs", buildDatasets(output != null ? List.of(output) : List.of()));
+    return event;
+  }
+
+  private ObjectNode buildRun(
+      String queryId,
+      String queryType,
+      RequestLogLine requestLogLine,
+      Map<String, Object> stats,
+      boolean success
+  )
+  {
+    ObjectNode run = jsonMapper.createObjectNode();
+    run.put("runId", UUID.nameUUIDFromBytes(queryId.getBytes(StandardCharsets.UTF_8)).toString());
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode engineFacet = createFacet(ENGINE_FACET_SCHEMA_URL);
+    engineFacet.put("name", "druid");
+    engineFacet.put("version", getDruidVersion());
+    facets.set("processing_engine", engineFacet);
+
+    ObjectNode contextFacet = createFacet(null);
+    contextFacet.put("queryType", queryType);
+    contextFacet.put("remoteAddress", requestLogLine.getRemoteAddr());
+    Object identity = stats.get("identity");
+    if (identity != null) {
+      contextFacet.put("identity", identity.toString());
+    }
+    Object nativeQueryIds = requestLogLine.getSqlQueryContext().get("nativeQueryIds");
+    if (nativeQueryIds != null) {
+      contextFacet.put("nativeQueryIds", nativeQueryIds.toString());
+    }
+    facets.set("druid_query_context", contextFacet);
+
+    ObjectNode statsFacet = createFacet(null);
+    putLongStat(statsFacet, "durationMs", stats, "sqlQuery/time", "query/time");
+    putLongStat(statsFacet, "bytes", stats, "sqlQuery/bytes", "query/bytes");
+    putLongStat(statsFacet, "planningTimeMs", stats, "sqlQuery/planningTimeMs");
+    Object statusCode = stats.get("statusCode");
+    if (statusCode != null) {
+      statsFacet.put("statusCode", statusCode.toString());
+    }
+    facets.set("druid_query_statistics", statsFacet);
+
+    if (!success) {
+      Object exception = stats.get("exception");
+      if (exception != null) {
+        ObjectNode errorFacet = createFacet(ERROR_FACET_SCHEMA_URL);
+        errorFacet.put("message", exception.toString());
+        if ("sql".equals(queryType)) {
+          errorFacet.put("programmingLanguage", "SQL");
+        }
+        facets.set("errorMessage", errorFacet);
+      }
+    }
+
+    run.set("facets", facets);
+    return run;
+  }
+
+  private ObjectNode buildJob(String queryId, @Nullable String sql)
+  {
+    ObjectNode job = jsonMapper.createObjectNode();
+    job.put("namespace", namespace);
+    job.put("name", queryId);
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode jobTypeFacet = createFacet(JOB_TYPE_FACET_SCHEMA_URL);
+    jobTypeFacet.put("processingType", "BATCH");
+    jobTypeFacet.put("integration", "DRUID");
+    jobTypeFacet.put("jobType", "QUERY");
+    facets.set("jobType", jobTypeFacet);
+
+    if (sql != null) {

Review Comment:
   msq-dart query emits a lineage event



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["mshahid6 (via GitHub)" <gi...@apache.org>]

mshahid6 commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3148675400


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLoggerProvider.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.guice.annotations.Json;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.server.DruidNode;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.server.log.RequestLoggerProvider;
+
+import javax.annotation.Nullable;
+import javax.validation.constraints.NotNull;
+import java.util.Set;
+
+/**
+ *  Configure logging type, namespace, transport type (http or default console), transportUrl in {@code runtime.properties} 
+ */
+@JsonTypeName("openlineage")
+public class OpenLineageRequestLoggerProvider implements RequestLoggerProvider
+{
+  private static final Logger log = new Logger(OpenLineageRequestLoggerProvider.class);
+
+  public enum TransportType
+  {
+    CONSOLE,
+    HTTP
+  }
+
+  @JacksonInject
+  @Json
+  @NotNull
+  private ObjectMapper jsonMapper;
+
+  @JsonProperty
+  @NotNull
+  private String namespace = "druid://" + DruidNode.getDefaultHost();
+
+  @JsonProperty
+  @NotNull
+  private TransportType transportType = TransportType.CONSOLE;
+
+  @Nullable
+  @JsonProperty
+  private String transportUrl;
+
+  @JsonProperty
+  @NotNull
+  private Set<String> excludedNativeQueryTypes = Set.of(

Review Comment:
   discussion for reference (https://github.com/apache/druid/pull/19107#discussion_r3091137636), can re-add timeBoundary



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["mshahid6 (via GitHub)" <gi...@apache.org>]

mshahid6 commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3082523827


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,498 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOrderBy;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  private static final int EMIT_QUEUE_CAPACITY = 1000;
+  private static final int EMIT_THREAD_COUNT = 1;
+
+  /**
+   * Internal broker query types that are automatically issued for schema discovery and metadata
+   * caching. These are not user-initiated and produce noisy, low-value lineage events.
+   */
+  private static final Set<String> INTERNAL_QUERY_TYPES = Set.of(
+      "segmentMetadata",
+      "dataSourceMetadata",
+      "timeBoundary"
+  );
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = HttpClientBuilder.create().build();
+      // Bounded queue: if the queue is full, silently drop the event rather than blocking
+      // the query thread. Uses Druid's Execs for daemon thread naming conventions.
+      this.emitExecutor = new ThreadPoolExecutor(
+          EMIT_THREAD_COUNT,
+          EMIT_THREAD_COUNT,
+          60L,
+          TimeUnit.SECONDS,
+          new ArrayBlockingQueue<>(EMIT_QUEUE_CAPACITY),
+          Execs.makeThreadFactory("OpenLineageEmitter-%d"),
+          new ThreadPoolExecutor.DiscardPolicy()
+      );
+    } else {
+      this.httpClient = null;
+      this.emitExecutor = null;
+    }
+  }
+
+  @LifecycleStart
+  @Override
+  public void start() throws Exception
+  {
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      log.info("Started OpenLineage HTTP transport to [%s]", transportUrl);
+    } else {
+      log.info("Started OpenLineage console transport");
+    }
+  }
+
+  @LifecycleStop
+  @Override
+  public void stop()
+  {
+    if (emitExecutor != null) {
+      emitExecutor.shutdown();
+      try {
+        if (!emitExecutor.awaitTermination(5, TimeUnit.SECONDS)) {
+          emitExecutor.shutdownNow();
+        }
+      }
+      catch (InterruptedException e) {
+        emitExecutor.shutdownNow();
+        Thread.currentThread().interrupt();
+      }
+    }
+    if (httpClient instanceof Closeable) {
+      CloseableUtils.closeAndSuppressExceptions(
+          (Closeable) httpClient,
+          e -> log.warn(e, "Failed to close OpenLineage HTTP client")
+      );
+    }
+    log.info("Stopped OpenLineage request logger");
+  }
+
+  @Override
+  public void logNativeQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    if (requestLogLine.getQuery() == null) {
+      return;
+    }
+
+    // Skip native sub-queries of a SQL execution to avoid duplicating the SQL-level event.
+    if (requestLogLine.getQuery().getContext().get(BaseQuery.SQL_QUERY_ID) != null) {
+      return;
+    }
+
+    String queryType = requestLogLine.getQuery().getType();
+
+    if (INTERNAL_QUERY_TYPES.contains(queryType)) {
+      return;
+    }
+
+    List<String> inputs = new ArrayList<>(new LinkedHashSet<>(requestLogLine.getQuery().getDataSource().getTableNames()));
+    String queryId = requestLogLine.getQuery().getId();
+    if (queryId == null) {
+      queryId = UUID.randomUUID().toString();
+    }
+
+    emit(buildRunEvent(queryId, null, queryType, requestLogLine, inputs, null));
+  }
+
+  @Override
+  public void logSqlQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    String sql = requestLogLine.getSql();
+    List<String> inputs = new ArrayList<>();
+    String output = null;
+
+    if (sql != null) {
+      try {
+        SqlNode parsed = SqlParser.create(sql, SQL_PARSER_CONFIG).parseQuery();
+        inputs = extractInputs(parsed);
+        output = extractOutput(parsed);
+      }
+      catch (SqlParseException e) {
+        // Druid-specific SQL extensions (REPLACE, EXTERN, etc.) may not parse with the standard
+        // Calcite parser. Emit the event without table-level lineage rather than failing.
+        log.debug(
+            "OpenLineage: could not parse SQL for lineage extraction (query will still be emitted): %s",
+            e.getMessage()
+        );
+      }
+    }
+
+    String queryId = extractSqlQueryId(requestLogLine);
+    emit(buildRunEvent(queryId, sql, "sql", requestLogLine, new ArrayList<>(new LinkedHashSet<>(inputs)), output));
+  }
+
+  private ObjectNode buildRunEvent(
+      String queryId,
+      @Nullable String sql,
+      String queryType,
+      RequestLogLine requestLogLine,
+      List<String> inputs,
+      @Nullable String output
+  )
+  {
+    Map<String, Object> stats = requestLogLine.getQueryStats().getStats();
+    boolean success = Boolean.TRUE.equals(stats.get("success"));
+
+    ObjectNode event = jsonMapper.createObjectNode();
+    event.put("eventType", success ? "COMPLETE" : "FAIL");
+    event.put("eventTime", requestLogLine.getTimestamp().toInstant().toString());
+    event.put("producer", PRODUCER);
+    event.put("schemaURL", SCHEMA_URL);
+    event.set("run", buildRun(queryId, queryType, requestLogLine, stats, success));
+    event.set("job", buildJob(queryId, sql));
+    event.set("inputs", buildDatasets(inputs));
+    event.set("outputs", buildDatasets(output != null ? List.of(output) : List.of()));
+    return event;
+  }
+
+  private ObjectNode buildRun(
+      String queryId,
+      String queryType,
+      RequestLogLine requestLogLine,
+      Map<String, Object> stats,
+      boolean success
+  )
+  {
+    ObjectNode run = jsonMapper.createObjectNode();
+    run.put("runId", UUID.nameUUIDFromBytes(queryId.getBytes(StandardCharsets.UTF_8)).toString());
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode engineFacet = createFacet(ENGINE_FACET_SCHEMA_URL);
+    engineFacet.put("name", "druid");
+    engineFacet.put("version", getDruidVersion());
+    facets.set("processing_engine", engineFacet);
+
+    ObjectNode contextFacet = createFacet(null);
+    contextFacet.put("queryType", queryType);
+    contextFacet.put("remoteAddress", requestLogLine.getRemoteAddr());
+    Object identity = stats.get("identity");
+    if (identity != null) {
+      contextFacet.put("identity", identity.toString());
+    }
+    Object nativeQueryIds = requestLogLine.getSqlQueryContext().get("nativeQueryIds");
+    if (nativeQueryIds != null) {
+      contextFacet.put("nativeQueryIds", nativeQueryIds.toString());
+    }
+    facets.set("druid_query_context", contextFacet);
+
+    ObjectNode statsFacet = createFacet(null);
+    putLongStat(statsFacet, "durationMs", stats, "sqlQuery/time", "query/time");
+    putLongStat(statsFacet, "bytes", stats, "sqlQuery/bytes", "query/bytes");
+    putLongStat(statsFacet, "planningTimeMs", stats, "sqlQuery/planningTimeMs");
+    Object statusCode = stats.get("statusCode");
+    if (statusCode != null) {
+      statsFacet.put("statusCode", statusCode.toString());
+    }
+    facets.set("druid_query_statistics", statsFacet);
+
+    if (!success) {
+      Object exception = stats.get("exception");
+      if (exception != null) {
+        ObjectNode errorFacet = createFacet(ERROR_FACET_SCHEMA_URL);
+        errorFacet.put("message", exception.toString());
+        if ("sql".equals(queryType)) {
+          errorFacet.put("programmingLanguage", "SQL");
+        }
+        facets.set("errorMessage", errorFacet);
+      }
+    }
+
+    run.set("facets", facets);
+    return run;
+  }
+
+  private ObjectNode buildJob(String queryId, @Nullable String sql)
+  {
+    ObjectNode job = jsonMapper.createObjectNode();
+    job.put("namespace", namespace);
+    job.put("name", queryId);
+
+    ObjectNode facets = jsonMapper.createObjectNode();
+
+    ObjectNode jobTypeFacet = createFacet(JOB_TYPE_FACET_SCHEMA_URL);
+    jobTypeFacet.put("processingType", "BATCH");
+    jobTypeFacet.put("integration", "DRUID");
+    jobTypeFacet.put("jobType", "QUERY");
+    facets.set("jobType", jobTypeFacet);
+
+    if (sql != null) {
+      ObjectNode sqlFacet = createFacet(SQL_FACET_SCHEMA_URL);
+      sqlFacet.put("query", sql);
+      facets.set("sql", sqlFacet);
+    }
+
+    job.set("facets", facets);
+    return job;
+  }
+
+  private ObjectNode createFacet(@Nullable String schemaUrl)
+  {
+    ObjectNode facet = jsonMapper.createObjectNode();
+    facet.put("_producer", PRODUCER);
+    if (schemaUrl != null) {
+      facet.put("_schemaURL", schemaUrl);
+    }
+    return facet;
+  }
+
+  private ArrayNode buildDatasets(List<String> tableNames)
+  {
+    ArrayNode array = jsonMapper.createArrayNode();
+    for (String name : tableNames) {
+      ObjectNode node = jsonMapper.createObjectNode();
+      node.put("namespace", namespace);
+      node.put("name", name);
+      node.set("facets", jsonMapper.createObjectNode());

Review Comment:
   https://github.com/apache/druid/issues/19314



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["maytasm (via GitHub)" <gi...@apache.org>]

maytasm commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3172183206


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.avatica.util.Quoting;
+import org.apache.calcite.sql.SqlBasicCall;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlInsert;
+import org.apache.calcite.sql.SqlJoin;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlNodeList;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.SqlWith;
+import org.apache.calcite.sql.SqlWithItem;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.RejectedExecutionHandler;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  private static final String SQL_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-1/SQLJobFacet.json";
+
+  // Standard Calcite parser (not Druid's custom parser): sufficient for table name extraction.
+  // Druid-specific syntax (REPLACE, EXTERN, etc.) falls through to the SqlParseException handler.
+  private static final SqlParser.Config SQL_PARSER_CONFIG = SqlParser
+      .config()
+      .withCaseSensitive(true)
+      .withUnquotedCasing(Casing.UNCHANGED)
+      .withQuotedCasing(Casing.UNCHANGED)
+      .withQuoting(Quoting.DOUBLE_QUOTE);
+
+  // Matches: INSERT INTO <table> ... (fallback when EXTERN prevents standard Calcite parsing)
+  // Handles both quoted ("table") and unquoted (table) identifiers.
+  private static final Pattern INSERT_INTO_PATTERN =
+      Pattern.compile("(?i)^\\s*INSERT\\s+INTO\\s+\"?([^\"\\s]+)\"?");
+  // Matches: REPLACE INTO <table> OVERWRITE ... (Druid-specific, not parseable by standard Calcite)
+  private static final Pattern REPLACE_INTO_PATTERN =
+      Pattern.compile("(?i)^\\s*REPLACE\\s+INTO\\s+\"?([^\"\\s]+)\"?");
+  // Extracts the SELECT subquery from a REPLACE INTO statement for input lineage

Review Comment:
   nit: the commend and variable name here indicates the pattern only applies to REPLACE INTO, however, this applies to both INSERT INTO and REPLACE INTO



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: add OpenLineage request logger extension (druid)

["FrankChen021 (via GitHub)" <gi...@apache.org>]

FrankChen021 commented on code in PR #19107:
URL: https://github.com/apache/druid/pull/19107#discussion_r3252855697


##########
extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java:
##########
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.extensions.openlineage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.BaseQuery;
+import org.apache.druid.server.RequestLogLine;
+import org.apache.druid.server.log.RequestLogger;
+import org.apache.druid.utils.CloseableUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.util.EntityUtils;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.RejectedExecutionHandler;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * OpenLineage RunEvents for completed Druid queries.
+ */
+public class OpenLineageRequestLogger implements RequestLogger
+{
+  private static final Logger log = new Logger(OpenLineageRequestLogger.class);
+
+  private static final String PRODUCER =
+      "https://github.com/apache/druid/extensions-contrib/openlineage-emitter";
+  private static final String SCHEMA_URL =
+      "https://openlineage.io/spec/2-0-2/OpenLineage.json";
+  private static final String ENGINE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json";
+  private static final String ERROR_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/1-0-0/ErrorMessageRunFacet.json";
+  private static final String JOB_TYPE_FACET_SCHEMA_URL =
+      "https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json";
+  static final int DEFAULT_EMIT_QUEUE_CAPACITY = 1000;
+  static final int DEFAULT_EMIT_THREAD_COUNT = 1;
+  private static final int DISCARD_WARNING_INTERVAL = 1000;
+
+  static final String UNKNOWN_QUERY_ID = "unknown-query-id";
+
+  private final ObjectMapper jsonMapper;
+  private final String namespace;
+  private final OpenLineageRequestLoggerProvider.TransportType transportType;
+  @Nullable
+  private final String transportUrl;
+  private final Set<String> excludedNativeQueryTypes;
+  @Nullable
+  private final HttpClient httpClient;
+  @Nullable
+  private final ExecutorService emitExecutor;
+  private final AtomicLong discardedEventCount = new AtomicLong(0);
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes
+  )
+  {
+    this(jsonMapper, namespace, transportType, transportUrl, excludedNativeQueryTypes,
+         DEFAULT_EMIT_QUEUE_CAPACITY, DEFAULT_EMIT_THREAD_COUNT, null);
+  }
+
+  public OpenLineageRequestLogger(
+      ObjectMapper jsonMapper,
+      String namespace,
+      OpenLineageRequestLoggerProvider.TransportType transportType,
+      @Nullable String transportUrl,
+      Set<String> excludedNativeQueryTypes,
+      int emitQueueCapacity,
+      int emitThreadCount,
+      @Nullable HttpClient httpClient
+  )
+  {
+    this.jsonMapper = jsonMapper;
+    this.namespace = namespace;
+    this.transportType = transportType;
+    this.transportUrl = transportUrl;
+    this.excludedNativeQueryTypes = excludedNativeQueryTypes;
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP && transportUrl == null) {
+      throw new IllegalStateException(
+          "druid.request.logging.transportUrl must be set when transportType=HTTP"
+      );
+    }
+    if (transportType == OpenLineageRequestLoggerProvider.TransportType.HTTP) {
+      this.httpClient = httpClient != null ? httpClient : HttpClientBuilder.create().build();
+      // Bounded queue: if the queue is full, drop the event rather than blocking the query thread.
+      // A warning is logged on the first drop and every DISCARD_WARNING_INTERVAL drops thereafter.
+      this.emitExecutor = new ThreadPoolExecutor(
+          emitThreadCount,
+          emitThreadCount,
+          60L,
+          TimeUnit.SECONDS,
+          new ArrayBlockingQueue<>(emitQueueCapacity),
+          Execs.makeThreadFactory("OpenLineageEmitter-%d"),
+          new DiscardWithWarningPolicy(discardedEventCount)
+      );
+    } else {
+      this.httpClient = null;
+      this.emitExecutor = null;
+    }
+  }
+
+  // Note: ComposingRequestLogger does not delegate @LifecycleStart to sub-loggers, so this method
+  // may not be called when used in a composing configuration. HTTP URL validation is therefore
+  // performed in the constructor instead. This method is retained for direct lifecycle use.
+  @LifecycleStart
+  @Override
+  public void start()
+  {
+    log.info(
+        "Started OpenLineage %s transport%s",
+        transportType,
+        transportUrl != null ? " to [" + transportUrl + "]" : ""
+    );
+  }
+
+  @LifecycleStop
+  @Override
+  public void stop()
+  {
+    if (emitExecutor != null) {
+      emitExecutor.shutdown();
+      try {
+        if (!emitExecutor.awaitTermination(5, TimeUnit.SECONDS)) {
+          emitExecutor.shutdownNow();
+        }
+      }
+      catch (InterruptedException e) {
+        emitExecutor.shutdownNow();
+        Thread.currentThread().interrupt();
+      }
+    }
+    if (httpClient instanceof Closeable) {
+      CloseableUtils.closeAndSuppressExceptions(
+          (Closeable) httpClient,
+          e -> log.warn(e, "Failed to close OpenLineage HTTP client")
+      );
+    }
+    log.info("Stopped OpenLineage request logger");
+  }
+
+  @Override
+  public void logNativeQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    if (requestLogLine.getQuery() == null) {
+      return;
+    }
+
+    String queryType = requestLogLine.getQuery().getType();
+
+    if (excludedNativeQueryTypes.contains(queryType)) {
+      return;
+    }
+
+    List<String> inputs = new ArrayList<>(new LinkedHashSet<>(requestLogLine.getQuery().getDataSource().getTableNames()));
+    String queryId = requestLogLine.getQuery().getId();
+    if (queryId == null) {
+      log.debug("Native query reached OpenLineage logger without a query ID");
+      queryId = UNKNOWN_QUERY_ID;
+    }
+
+    emit(buildRunEvent(queryId, queryType, requestLogLine, inputs, null));
+  }
+
+  /**
+   * SQL-level logging is a no-op. Lineage is emitted from the native query plan via
+   * {@link #logNativeQuery}, which has structured access to datasource references
+   * (JoinDataSource, UnionDataSource, etc.) without requiring SQL parsing.
+   *
+   * <p>For SQL queries, the broker converts SQL to native queries before execution.
+   * Each native sub-query calls {@link #logNativeQuery}, which extracts inputs from
+   * {@code query.getDataSource().getTableNames()}. The native events carry the parent
+   * SQL query ID in their context for correlation.
+   */
+  @Override
+  public void logSqlQuery(RequestLogLine requestLogLine) throws IOException
+  {
+    // No-op: lineage is emitted from native sub-queries.

Review Comment:
   [P1] SQL no-op drops MSQ lineage events
   
   Making `logSqlQuery` a no-op only preserves SQL lineage for native-engine SQL queries that pass through `QueryLifecycle.logNativeQuery`. MSQ-backed SQL requests, including INSERT/REPLACE, go through `MSQTaskQueryMaker` to submit an `MSQControllerTask`; the request-log callback for the SQL request is emitted by `SqlExecutionReporter` via `logSqlQuery`, with no corresponding native request-log event on this path. With this method empty, those SQL executions emit no OpenLineage event at all, losing the ingestion/output lineage this extension is meant to report. Please either keep SQL-level emission for non-native SQL engines or add an MSQ path that emits equivalent lineage before making `logSqlQuery` a no-op.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org