You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "exceptionfactory (via GitHub)" <gi...@apache.org> on 2023/10/02 17:07:27 UTC

[PR] NIFI-12115 Add ListenOTLP to collect OpenTelemetry [nifi]

exceptionfactory opened a new pull request, #7830:
URL: https://github.com/apache/nifi/pull/7830

   # Summary
   
   [NIFI-12115](https://issues.apache.org/jira/browse/NIFI-12115) Adds a `ListenOTLP` Processor to collect OpenTelemetry information, supporting [OTLP Specification 1.0.0](https://opentelemetry.io/docs/specs/otlp/).
   
   OTLP defines HTTP as the basic communication protocol and outlines the following transport encoding options:
   
   - Protobuf over gRPC
   - Protobuf over HTTP
   - JSON over HTTP
   
   OTLP also includes support for [gzip](https://en.wikipedia.org/wiki/Gzip) compression over gRPC or HTTP.
   
   The `ListenOTLP` Processor supports the standard transport encoding options and uses content negotiation based on the HTTP `Content-Type` header to select the appropriate parsing implementation.
   
   The Processor converts incoming requests to standard OpenTelemetry objects and writes objects using the standard JSON encoding, which includes special handling for hexadecimal-encoding identifiers. Using JSON as the standard output format allows subsequent Processors to select or filter as necessary. Output FlowFiles can contain a batch of one or more resource elements, depending on data volumes and batch size configuration. `ListenOTLP` adds the `client.socket.address` and `client.socket.port` attributes to the `Resource` definition of received requests to provide an additional level of source tracking.
   
   The Processor requires an `SSL Context Service` property to enforce communication over HTTPS. The gRPC protocol requires HTTP/2, which uses Application Protocol Negotiation in the TLS handshake to select the HTTP version. With the potentially sensitive nature of telemetry information, the initial version of `ListenOTLP` requires HTTPS, and can be configured for mTLS with `Client Authentication` required.
   
   Additional changes include updates to the `nifi-event-transport` library supporting configurable TLS Cipher Suites through optional `SSLParameters` settings. HTTP/2 requires a restricted set of TLS Cipher Suites, so this adjustment supports that approach.
   
   The [opentelemetry-proto](https://github.com/open-telemetry/opentelemetry-proto) and [jackson-datatype-protobuf](https://github.com/HubSpot/jackson-datatype-protobuf) libraries are both licensed under Apache Software License Version 2 with no additional notice files.
   
   OTLP 1.0.0 JSON examples are available in the [opentelemetry-proto](https://github.com/open-telemetry/opentelemetry-proto/tree/v1.0.0/examples/) examples directory.
   
   For testing with an instrumented system, the [OpenTelemetry Java Agent](https://opentelemetry.io/docs/instrumentation/java/automatic/agent-config/) can be configured using Java System properties in `bootstrap.conf` for basic instrumentation of JVM memory and Jetty calls using the following properties:
   
   ```
   java.arg.otelAgent=-javaagent:./opentelemetry-javaagent.jar
   java.arg.otelCertificate=-Dotel.exporter.otlp.certificate=./conf/trusted.pem
   java.arg.otelEndpoint=-Dotel.exporter.otlp.endpoint=https://localhost:4317
   ```
   
   The `trusted.pem` file must contain the issuer of the server certificate configured for `ListenOTLP`. The OpenTelemetry Java Agent defaults to using gRPC for transport encoding without compression, but additional options can be configured based on the [OTLP exporter properties](https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md#otlp-exporter-span-metric-and-log-exporters).
   
   This pull request targets the main branch, with the intent of backporting the addition following minor adjustments.
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
   
   ### Pull Request Tracking
   
   - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
   - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
   
   ### Pull Request Formatting
   
   - [X] Pull Request based on current revision of the `main` branch
   - [X] Pull Request refers to a feature branch with one commit containing changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request creation.
   
   ### Build
   
   - [X] Build completed using `mvn clean install -P contrib-check`
     - [X] JDK 21
   
   ### Licensing
   
   - [X] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
   - [X] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
   
   ### Documentation
   
   - [X] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-12115 Add ListenOTLP to collect OpenTelemetry [nifi]

Posted by "joewitt (via GitHub)" <gi...@apache.org>.
joewitt commented on code in PR #7830:
URL: https://github.com/apache/nifi/pull/7830#discussion_r1344258519


##########
nifi-nar-bundles/nifi-opentelemetry-bundle/nifi-opentelemetry-processors/src/main/java/org/apache/nifi/processors/opentelemetry/encoding/TelemetryMessageSerializer.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.opentelemetry.encoding;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.google.protobuf.ByteString;
+import com.google.protobuf.Descriptors;
+import com.hubspot.jackson.datatype.protobuf.ProtobufJacksonConfig;
+import com.hubspot.jackson.datatype.protobuf.builtin.serializers.MessageSerializer;
+import org.apache.commons.codec.binary.Hex;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * OpenTelemetry extension of Protobuf Message Serializer supporting OTLP 1.0.0 customization of selected fields
+ */
+public class TelemetryMessageSerializer extends MessageSerializer {
+    private static final Set<String> HEXADECIMAL_BYTE_STRING_FIELDS = Arrays.stream(HexadecimalByteStringField.values())
+            .map(HexadecimalByteStringField::getField)
+            .collect(Collectors.toSet());
+
+    protected TelemetryMessageSerializer(final ProtobufJacksonConfig config) {
+        super(config);
+    }
+
+    /**
+     * Write value as JSON with hexadecimal encoding for selected ByteString fields
+     *
+     * @param field Message Field Decriptor

Review Comment:
   There is a typo in 'Decriptor'



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-12115 Add ListenOTLP to collect OpenTelemetry [nifi]

Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #7830:
URL: https://github.com/apache/nifi/pull/7830#issuecomment-1746013182

   Thanks for the testing and feedback @joewitt! After reviewing the request content buffering and the available Netty Handlers, I consolidated the HTTP processing to work with the Netty [FullHttpRequest](https://netty.io/4.1/api/io/netty/handler/codec/http/FullHttpRequest.html) for both HTTP/1.1 and HTTP/2. This streamlined the implementation in `HttpRequestHandler`, avoiding the previous buffer handling issue that resulted in the OutOfMemoryError.
   
   Following these changes, running a continuous stream of JSON files through `InvokeHTTP` did not produce any errors. Heap and non-heap memory usage also followed a consistent pattern.
   
   As part of streamlining the handler pipeline, I added a 10 MB maximum content length limitation. Various service providers set request limits for OpenTelemetry payloads to 4 MB or 5 MB. This could be changed to a configurable property, but larger sizes also introduce potential memory usage concerns. Along similar lines, the default Queue Capacity is now 1000 and the default Batch Size is now 100, also limiting potential memory issues in the default configuration.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-12115 Add ListenOTLP to collect OpenTelemetry [nifi]

Posted by "joewitt (via GitHub)" <gi...@apache.org>.
joewitt commented on code in PR #7830:
URL: https://github.com/apache/nifi/pull/7830#discussion_r1343491367


##########
nifi-nar-bundles/nifi-opentelemetry-bundle/nifi-opentelemetry-processors/pom.xml:
##########
@@ -0,0 +1,105 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements. See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License. You may obtain a copy of the License at
+  http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+
+    <parent>
+        <groupId>org.apache.nifi</groupId>
+        <artifactId>nifi-opentelemetry-bundle</artifactId>
+        <version>2.0.0-SNAPSHOT</version>
+    </parent>
+
+    <artifactId>nifi-opentelemetry-processors</artifactId>
+    <packaging>jar</packaging>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.nifi</groupId>
+            <artifactId>nifi-ssl-context-service-api</artifactId>
+            <version>2.0.0-SNAPSHOT</version>
+            <scope>provided</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.nifi</groupId>
+            <artifactId>nifi-event-transport</artifactId>
+            <version>2.0.0-SNAPSHOT</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.nifi</groupId>
+            <artifactId>nifi-utils</artifactId>
+            <version>2.0.0-SNAPSHOT</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.nifi</groupId>
+            <artifactId>nifi-api</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-codec-http</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-codec-http2</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>commons-codec</groupId>
+            <artifactId>commons-codec</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>com.fasterxml.jackson.core</groupId>
+            <artifactId>jackson-databind</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>com.hubspot.jackson</groupId>
+            <artifactId>jackson-datatype-protobuf</artifactId>
+            <version>0.9.14</version>
+            <exclusions>
+                <exclusion>
+                    <groupId>com.google.protobuf</groupId>
+                    <artifactId>protobuf-java-util</artifactId>
+                </exclusion>
+            </exclusions>
+        </dependency>
+        <dependency>
+            <groupId>io.opentelemetry.proto</groupId>
+            <artifactId>opentelemetry-proto</artifactId>
+            <version>1.0.0-alpha</version>

Review Comment:
   This alpha designation is super strange.  Though some googling/maven repo searching suggests this is the name of the game now.  What I find odd is there seems to be a 1.0.0 spec published so why do they call this alpha?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-12115 Add ListenOTLP to collect OpenTelemetry [nifi]

Posted by "joewitt (via GitHub)" <gi...@apache.org>.
joewitt commented on code in PR #7830:
URL: https://github.com/apache/nifi/pull/7830#discussion_r1343466382


##########
nifi-nar-bundles/nifi-opentelemetry-bundle/nifi-opentelemetry-processors/src/main/java/org/apache/nifi/processors/opentelemetry/encoding/TelemetryMessageSerializer.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.opentelemetry.encoding;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.google.protobuf.ByteString;
+import com.google.protobuf.Descriptors;
+import com.hubspot.jackson.datatype.protobuf.ProtobufJacksonConfig;
+import com.hubspot.jackson.datatype.protobuf.builtin.serializers.MessageSerializer;
+import org.apache.commons.codec.binary.Hex;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * OpenTelemetry extension of Protobuf Message Serializer supporting OTLP 1.0.0 customization of selected fields
+ */
+public class TelemetryMessageSerializer extends MessageSerializer {
+    private static final Set<String> HEXADECIMAL_BYTE_STRING_FIELDS = Arrays.stream(HexadecimalByteStringField.values())
+            .map(HexadecimalByteStringField::getField)
+            .collect(Collectors.toSet());
+
+    protected TelemetryMessageSerializer(final ProtobufJacksonConfig config) {
+        super(config);
+    }
+
+    /**
+     * Write value as JSON with hexadecimal encoding for selected ByteString fields
+     *
+     * @param field Message Field Decriptor

Review Comment:
   add an 's'/type in Descriptor



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-12115 Add ListenOTLP to collect OpenTelemetry [nifi]

Posted by "joewitt (via GitHub)" <gi...@apache.org>.
joewitt commented on code in PR #7830:
URL: https://github.com/apache/nifi/pull/7830#discussion_r1343459564


##########
nifi-nar-bundles/nifi-opentelemetry-bundle/nifi-opentelemetry-processors/src/main/java/org/apache/nifi/processors/opentelemetry/ListenOTLP.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.opentelemetry;
+
+import com.google.protobuf.Message;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.configuration.DefaultSchedule;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.event.transport.EventServer;
+import org.apache.nifi.event.transport.EventServerFactory;
+import org.apache.nifi.event.transport.netty.NettyEventServerFactory;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processors.opentelemetry.protocol.TelemetryAttributeName;
+import org.apache.nifi.processors.opentelemetry.io.RequestCallback;
+import org.apache.nifi.processors.opentelemetry.io.RequestCallbackProvider;
+import org.apache.nifi.processors.opentelemetry.server.HttpServerFactory;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.security.util.ClientAuth;
+import org.apache.nifi.ssl.SSLContextService;
+
+import javax.net.ssl.SSLContext;
+import java.net.InetAddress;
+import java.net.URI;
+import java.net.UnknownHostException;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@DefaultSchedule(period = "25 ms")
+@Tags({"OpenTelemetry", "OTel", "OTLP", "telemetry", "metrics", "traces", "logs"})
+@CapabilityDescription(
+        "Collect OpenTelemetry messages over HTTP or gRPC. " +
+        "Supports standard Export Service Request messages for logs, metrics, and traces. " +
+        "Implements OpenTelemetry OTLP Specification 1.0.0 with OTLP/gRPC and OTLP/HTTP. " +
+        "Provides protocol detection using the HTTP Content-Type header."
+)
+@WritesAttributes({
+        @WritesAttribute(attribute = TelemetryAttributeName.MIME_TYPE, description = "Content-Type set to application/json"),
+        @WritesAttribute(attribute = TelemetryAttributeName.RESOURCE_TYPE, description = "OpenTelemetry Resource Type: LOGS, METRICS, or TRACES"),
+        @WritesAttribute(attribute = TelemetryAttributeName.RESOURCE_COUNT, description = "Count of resource elements included in messages"),
+})
+public class ListenOTLP extends AbstractProcessor {
+
+    static final PropertyDescriptor ADDRESS = new PropertyDescriptor.Builder()
+            .name("Address")
+            .displayName("Address")
+            .description("Internet Protocol Address on which to listen for OTLP Export Service Requests. The default value enables listening on all addresses.")
+            .required(true)
+            .defaultValue("0.0.0.0")
+            .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+            .name("Port")
+            .displayName("Port")
+            .description("TCP port number on which to listen for OTLP Export Service Requests over HTTP and gRPC")
+            .required(true)
+            .defaultValue("4317")
+            .addValidator(StandardValidators.PORT_VALIDATOR)
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final PropertyDescriptor SSL_CONTEXT_SERVICE = new PropertyDescriptor.Builder()
+            .name("SSL Context Service")
+            .displayName("SSL Context Service")
+            .description("SSL Context Service enables TLS communication for HTTPS")
+            .required(true)
+            .identifiesControllerService(SSLContextService.class)
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final PropertyDescriptor CLIENT_AUTHENTICATION = new PropertyDescriptor.Builder()
+            .name("Client Authentication")
+            .displayName("Client Authentication")
+            .description("Client authentication policy for TLS communication with HTTPS")
+            .required(true)
+            .allowableValues(ClientAuth.values())
+            .defaultValue(ClientAuth.WANT.name())
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final PropertyDescriptor WORKER_THREADS = new PropertyDescriptor.Builder()
+            .name("Worker Threads")
+            .displayName("Worker Threads")
+            .description("Number of threads responsible for decoding and queuing incoming OTLP Export Service Requests")
+            .required(true)
+            .defaultValue("2")
+            .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final PropertyDescriptor QUEUE_CAPACITY = new PropertyDescriptor.Builder()
+            .name("Queue Capacity")
+            .displayName("Queue Capacity")
+            .description("Maximum number of OTLP request resource elements that can be received and queued")
+            .required(true)
+            .defaultValue("10000")
+            .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+            .name("Batch Size")
+            .displayName("Batch Size")
+            .description("Maximum number of OTLP request resource elements included in each FlowFile produced")
+            .required(true)
+            .defaultValue("1000")
+            .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final Relationship SUCCESS = new Relationship.Builder()
+            .name("success")
+            .description("Export Service Requests containing OTLP Telemetry")
+            .build();
+
+    private static final Set<Relationship> RELATIONSHIPS = Collections.singleton(SUCCESS);
+
+    private static final List<PropertyDescriptor> DESCRIPTORS = List.of(
+            ADDRESS,
+            PORT,
+            SSL_CONTEXT_SERVICE,
+            CLIENT_AUTHENTICATION,
+            WORKER_THREADS,
+            QUEUE_CAPACITY,
+            BATCH_SIZE
+    );
+
+    private static final String TRANSIT_URI_FORMAT = "https://%s:%d";
+
+    private Iterator<RequestCallback> requestCallbackProvider;
+
+    private EventServer server;
+
+    @Override
+    public final Set<Relationship> getRelationships() {
+        return RELATIONSHIPS;
+    }
+
+    @Override
+    public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+        return DESCRIPTORS;
+    }
+
+    @OnScheduled
+    public void onScheduled(final ProcessContext context) throws UnknownHostException {
+        final EventServerFactory eventServerFactory = createEventServerFactory(context);
+        server = eventServerFactory.getEventServer();
+    }
+
+    @OnStopped
+    public void onStopped() {
+        if (server == null) {
+            getLogger().info("Server not running");

Review Comment:
   This line seems a little strange.  It seems the only time the server could be null is if this processor was stopped so if that happens this method is already done/called.  And even if this could happen the output doesn't seem useful to the user as there is no meaning they can really derive or action they can take.  Was this from earlier testing and left in or am I not considering some case?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-12115 Add ListenOTLP to collect OpenTelemetry [nifi]

Posted by "joewitt (via GitHub)" <gi...@apache.org>.
joewitt commented on code in PR #7830:
URL: https://github.com/apache/nifi/pull/7830#discussion_r1343465307


##########
nifi-nar-bundles/nifi-opentelemetry-bundle/nifi-opentelemetry-processors/src/main/java/org/apache/nifi/processors/opentelemetry/encoding/JsonServiceRequestReader.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.opentelemetry.encoding;
+
+import com.google.protobuf.Message;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.Objects;
+
+/**
+ * Service Request Reader implementation based on JSON Parser supporting standard OTLP Request Types
+ */
+public class JsonServiceRequestReader implements ServiceRequestReader {
+    private static final RequestMapper REQUEST_MAPPER = new StandardRequestMapper();
+
+    /**
+     * Read Service Request parsed from Buffer
+     *
+     * @param buffer Byte Buffer
+     * @param requestType Request Message Type
+     * @return Service Request read
+     */
+    @Override
+    public <T extends Message> T read(final ByteBuffer buffer, final Class<T> requestType) {
+        Objects.requireNonNull(buffer, "Buffer required");
+        final byte[] bytes = getBytes(buffer);
+
+        try {
+            return REQUEST_MAPPER.readValue(bytes, requestType);

Review Comment:
   This request mapper seems to be our code and if so would we not be better off passing the byte buffer itself rather than an intermediate copy of the contents of the buffer?  I followed the call chain and it seems like we're free to use the buffer as is but I might not know the total ownership/lifecycle of the byte buffer.  If true though saving on the byte buffer extra copy at processing rate would add up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-12115 Add ListenOTLP to collect OpenTelemetry [nifi]

Posted by "joewitt (via GitHub)" <gi...@apache.org>.
joewitt commented on code in PR #7830:
URL: https://github.com/apache/nifi/pull/7830#discussion_r1343468081


##########
nifi-nar-bundles/nifi-opentelemetry-bundle/nifi-opentelemetry-processors/src/main/java/org/apache/nifi/processors/opentelemetry/encoding/TelemetryMessageSerializer.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.opentelemetry.encoding;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.google.protobuf.ByteString;
+import com.google.protobuf.Descriptors;
+import com.hubspot.jackson.datatype.protobuf.ProtobufJacksonConfig;
+import com.hubspot.jackson.datatype.protobuf.builtin.serializers.MessageSerializer;
+import org.apache.commons.codec.binary.Hex;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * OpenTelemetry extension of Protobuf Message Serializer supporting OTLP 1.0.0 customization of selected fields
+ */
+public class TelemetryMessageSerializer extends MessageSerializer {
+    private static final Set<String> HEXADECIMAL_BYTE_STRING_FIELDS = Arrays.stream(HexadecimalByteStringField.values())
+            .map(HexadecimalByteStringField::getField)
+            .collect(Collectors.toSet());
+
+    protected TelemetryMessageSerializer(final ProtobufJacksonConfig config) {
+        super(config);
+    }
+
+    /**
+     * Write value as JSON with hexadecimal encoding for selected ByteString fields
+     *
+     * @param field Message Field Decriptor
+     * @param value Value to be serialized
+     * @param generator JSON Generator
+     * @param serializerProvider JSON Serializer Provider
+     * @throws IOException Thrown on failures serializing values
+     */
+    @Override
+    protected void writeValue(
+            Descriptors.FieldDescriptor field,
+            Object value,
+            JsonGenerator generator,
+            SerializerProvider serializerProvider
+    ) throws IOException {
+        final String jsonName = field.getJsonName();
+        if (HEXADECIMAL_BYTE_STRING_FIELDS.contains(jsonName)) {
+            final ByteString byteString = (ByteString) value;
+            final String encoded = getEncodedByteString(byteString);
+            generator.writeString(encoded);
+        } else {
+            super.writeValue(field, value, generator, serializerProvider);
+        }
+    }
+
+    private String getEncodedByteString(final ByteString byteString) {
+        final byte[] bytes = byteString.toByteArray();

Review Comment:
   Would using byteString.asReadOnlyByteBuffer then passing that into encodeHexString let us skip the intermediate copy here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-12115 Add ListenOTLP to collect OpenTelemetry [nifi]

Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #7830:
URL: https://github.com/apache/nifi/pull/7830#discussion_r1344048364


##########
nifi-nar-bundles/nifi-opentelemetry-bundle/nifi-opentelemetry-processors/src/main/java/org/apache/nifi/processors/opentelemetry/encoding/TelemetryMessageSerializer.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.opentelemetry.encoding;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.google.protobuf.ByteString;
+import com.google.protobuf.Descriptors;
+import com.hubspot.jackson.datatype.protobuf.ProtobufJacksonConfig;
+import com.hubspot.jackson.datatype.protobuf.builtin.serializers.MessageSerializer;
+import org.apache.commons.codec.binary.Hex;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * OpenTelemetry extension of Protobuf Message Serializer supporting OTLP 1.0.0 customization of selected fields
+ */
+public class TelemetryMessageSerializer extends MessageSerializer {
+    private static final Set<String> HEXADECIMAL_BYTE_STRING_FIELDS = Arrays.stream(HexadecimalByteStringField.values())
+            .map(HexadecimalByteStringField::getField)
+            .collect(Collectors.toSet());
+
+    protected TelemetryMessageSerializer(final ProtobufJacksonConfig config) {
+        super(config);
+    }
+
+    /**
+     * Write value as JSON with hexadecimal encoding for selected ByteString fields
+     *
+     * @param field Message Field Decriptor

Review Comment:
   This is a singular `Field` object, even `Descriptors` is the enclosing class, are you saying the description should be `Message Field Descriptors`?



##########
nifi-nar-bundles/nifi-opentelemetry-bundle/nifi-opentelemetry-processors/src/main/java/org/apache/nifi/processors/opentelemetry/encoding/TelemetryMessageSerializer.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.opentelemetry.encoding;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.google.protobuf.ByteString;
+import com.google.protobuf.Descriptors;
+import com.hubspot.jackson.datatype.protobuf.ProtobufJacksonConfig;
+import com.hubspot.jackson.datatype.protobuf.builtin.serializers.MessageSerializer;
+import org.apache.commons.codec.binary.Hex;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * OpenTelemetry extension of Protobuf Message Serializer supporting OTLP 1.0.0 customization of selected fields
+ */
+public class TelemetryMessageSerializer extends MessageSerializer {
+    private static final Set<String> HEXADECIMAL_BYTE_STRING_FIELDS = Arrays.stream(HexadecimalByteStringField.values())
+            .map(HexadecimalByteStringField::getField)
+            .collect(Collectors.toSet());
+
+    protected TelemetryMessageSerializer(final ProtobufJacksonConfig config) {
+        super(config);
+    }
+
+    /**
+     * Write value as JSON with hexadecimal encoding for selected ByteString fields
+     *
+     * @param field Message Field Decriptor
+     * @param value Value to be serialized
+     * @param generator JSON Generator
+     * @param serializerProvider JSON Serializer Provider
+     * @throws IOException Thrown on failures serializing values
+     */
+    @Override
+    protected void writeValue(
+            Descriptors.FieldDescriptor field,
+            Object value,
+            JsonGenerator generator,
+            SerializerProvider serializerProvider
+    ) throws IOException {
+        final String jsonName = field.getJsonName();
+        if (HEXADECIMAL_BYTE_STRING_FIELDS.contains(jsonName)) {
+            final ByteString byteString = (ByteString) value;
+            final String encoded = getEncodedByteString(byteString);
+            generator.writeString(encoded);
+        } else {
+            super.writeValue(field, value, generator, serializerProvider);
+        }
+    }
+
+    private String getEncodedByteString(final ByteString byteString) {
+        final byte[] bytes = byteString.toByteArray();

Review Comment:
   Good suggestion! These values are short, usually 16 bytes or less, and the actual efficiency depends on the concrete type of the `ByteString`, but it is worth making the adjustment.



##########
nifi-nar-bundles/nifi-opentelemetry-bundle/nifi-opentelemetry-processors/src/main/java/org/apache/nifi/processors/opentelemetry/io/StandardRequestContentListener.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.opentelemetry.io;
+
+import com.google.protobuf.Message;
+import io.opentelemetry.proto.collector.logs.v1.ExportLogsServiceRequest;
+import io.opentelemetry.proto.collector.metrics.v1.ExportMetricsServiceRequest;
+import io.opentelemetry.proto.collector.trace.v1.ExportTraceServiceRequest;
+import io.opentelemetry.proto.common.v1.AnyValue;
+import io.opentelemetry.proto.common.v1.KeyValue;
+import io.opentelemetry.proto.logs.v1.ResourceLogs;
+import io.opentelemetry.proto.metrics.v1.ResourceMetrics;
+import io.opentelemetry.proto.resource.v1.Resource;
+import io.opentelemetry.proto.trace.v1.ResourceSpans;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processors.opentelemetry.encoding.JsonServiceRequestReader;
+import org.apache.nifi.processors.opentelemetry.encoding.ProtobufServiceRequestReader;
+import org.apache.nifi.processors.opentelemetry.encoding.ServiceRequestReader;
+import org.apache.nifi.processors.opentelemetry.protocol.ServiceRequestDescription;
+import org.apache.nifi.processors.opentelemetry.protocol.ServiceResponse;
+import org.apache.nifi.processors.opentelemetry.protocol.ServiceResponseStatus;
+import org.apache.nifi.processors.opentelemetry.protocol.TelemetryContentEncoding;
+import org.apache.nifi.processors.opentelemetry.protocol.TelemetryContentType;
+import org.apache.nifi.processors.opentelemetry.protocol.TelemetryRequestType;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.BlockingQueue;
+import java.util.zip.GZIPInputStream;
+
+/**
+ * Standard implementation of OTLP Request Content Listener supporting available Request and Content Types
+ */
+public class StandardRequestContentListener implements RequestContentListener {
+    private static final int ZERO_MESSAGES = 0;
+
+    private static final byte COMPRESSED = 1;
+
+    private static final String CLIENT_SOCKET_ADDRESS = "client.socket.address";
+
+    private static final String CLIENT_SOCKET_PORT = "client.socket.port";
+
+    private final ServiceRequestReader protobufReader = new ProtobufServiceRequestReader();
+
+    private final ServiceRequestReader jsonReader = new JsonServiceRequestReader();
+
+    private final ComponentLog log;
+
+    private final BlockingQueue<Message> messages;
+
+    public StandardRequestContentListener(final ComponentLog log, final BlockingQueue<Message> messages) {
+        this.log = Objects.requireNonNull(log, "Log required");
+        this.messages = Objects.requireNonNull(messages, "Messages required");
+    }
+
+    /**
+     * Process Request Buffer supporting serialized Protobuf or JSON
+     *
+     * @param buffer Request Content Buffer to be processed
+     * @param serviceRequestDescription Service Request Description describes reader attributes
+     * @return Service Response with processing status
+     */
+    @Override
+    public ServiceResponse onRequest(final ByteBuffer buffer, final ServiceRequestDescription serviceRequestDescription) {
+        Objects.requireNonNull(buffer, "Buffer required");
+        Objects.requireNonNull(serviceRequestDescription, "Description required");
+
+        ServiceResponse serviceResponse;
+
+        final InetSocketAddress remoteAddress = serviceRequestDescription.getRemoteAddress();
+        final TelemetryContentType contentType = serviceRequestDescription.getContentType();
+        if (TelemetryContentType.APPLICATION_GRPC == contentType) {
+            try {
+                final byte compression = buffer.get();
+                final int messageSize = buffer.getInt();
+                log.debug("Client Address [{}] Content-Type [{}] Message Size [{}] Compression [{}]", remoteAddress, contentType, messageSize, compression);
+
+                final ByteBuffer messageBuffer;
+                if (COMPRESSED == compression) {
+                    messageBuffer = getDecompressedBuffer(buffer);

Review Comment:
   Good question, although messages should generally be of a reasonable size, the decompression process could expand the input size in a notable way. The default behavior of the Java Agent does not enable Gzip compression, but that can be tested by setting `-Dotel.exporter.otlp.compression=gzip`. In light of other notes, I have refactored the approach to pass the wrapping `GZIPInputStream`, instead of reading everything to a new buffer, which should help.



##########
nifi-nar-bundles/nifi-opentelemetry-bundle/nifi-opentelemetry-processors/src/main/java/org/apache/nifi/processors/opentelemetry/ListenOTLP.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.opentelemetry;
+
+import com.google.protobuf.Message;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.configuration.DefaultSchedule;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.event.transport.EventServer;
+import org.apache.nifi.event.transport.EventServerFactory;
+import org.apache.nifi.event.transport.netty.NettyEventServerFactory;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processors.opentelemetry.protocol.TelemetryAttributeName;
+import org.apache.nifi.processors.opentelemetry.io.RequestCallback;
+import org.apache.nifi.processors.opentelemetry.io.RequestCallbackProvider;
+import org.apache.nifi.processors.opentelemetry.server.HttpServerFactory;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.security.util.ClientAuth;
+import org.apache.nifi.ssl.SSLContextService;
+
+import javax.net.ssl.SSLContext;
+import java.net.InetAddress;
+import java.net.URI;
+import java.net.UnknownHostException;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@DefaultSchedule(period = "25 ms")
+@Tags({"OpenTelemetry", "OTel", "OTLP", "telemetry", "metrics", "traces", "logs"})
+@CapabilityDescription(
+        "Collect OpenTelemetry messages over HTTP or gRPC. " +
+        "Supports standard Export Service Request messages for logs, metrics, and traces. " +
+        "Implements OpenTelemetry OTLP Specification 1.0.0 with OTLP/gRPC and OTLP/HTTP. " +
+        "Provides protocol detection using the HTTP Content-Type header."
+)
+@WritesAttributes({
+        @WritesAttribute(attribute = TelemetryAttributeName.MIME_TYPE, description = "Content-Type set to application/json"),
+        @WritesAttribute(attribute = TelemetryAttributeName.RESOURCE_TYPE, description = "OpenTelemetry Resource Type: LOGS, METRICS, or TRACES"),
+        @WritesAttribute(attribute = TelemetryAttributeName.RESOURCE_COUNT, description = "Count of resource elements included in messages"),
+})
+public class ListenOTLP extends AbstractProcessor {
+
+    static final PropertyDescriptor ADDRESS = new PropertyDescriptor.Builder()
+            .name("Address")
+            .displayName("Address")
+            .description("Internet Protocol Address on which to listen for OTLP Export Service Requests. The default value enables listening on all addresses.")
+            .required(true)
+            .defaultValue("0.0.0.0")
+            .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+            .name("Port")
+            .displayName("Port")
+            .description("TCP port number on which to listen for OTLP Export Service Requests over HTTP and gRPC")
+            .required(true)
+            .defaultValue("4317")
+            .addValidator(StandardValidators.PORT_VALIDATOR)
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final PropertyDescriptor SSL_CONTEXT_SERVICE = new PropertyDescriptor.Builder()
+            .name("SSL Context Service")
+            .displayName("SSL Context Service")
+            .description("SSL Context Service enables TLS communication for HTTPS")
+            .required(true)
+            .identifiesControllerService(SSLContextService.class)
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final PropertyDescriptor CLIENT_AUTHENTICATION = new PropertyDescriptor.Builder()
+            .name("Client Authentication")
+            .displayName("Client Authentication")
+            .description("Client authentication policy for TLS communication with HTTPS")
+            .required(true)
+            .allowableValues(ClientAuth.values())
+            .defaultValue(ClientAuth.WANT.name())
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final PropertyDescriptor WORKER_THREADS = new PropertyDescriptor.Builder()
+            .name("Worker Threads")
+            .displayName("Worker Threads")
+            .description("Number of threads responsible for decoding and queuing incoming OTLP Export Service Requests")
+            .required(true)
+            .defaultValue("2")
+            .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final PropertyDescriptor QUEUE_CAPACITY = new PropertyDescriptor.Builder()
+            .name("Queue Capacity")
+            .displayName("Queue Capacity")
+            .description("Maximum number of OTLP request resource elements that can be received and queued")
+            .required(true)
+            .defaultValue("10000")
+            .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+            .name("Batch Size")
+            .displayName("Batch Size")
+            .description("Maximum number of OTLP request resource elements included in each FlowFile produced")
+            .required(true)
+            .defaultValue("1000")
+            .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final Relationship SUCCESS = new Relationship.Builder()
+            .name("success")
+            .description("Export Service Requests containing OTLP Telemetry")
+            .build();
+
+    private static final Set<Relationship> RELATIONSHIPS = Collections.singleton(SUCCESS);
+
+    private static final List<PropertyDescriptor> DESCRIPTORS = List.of(
+            ADDRESS,
+            PORT,
+            SSL_CONTEXT_SERVICE,
+            CLIENT_AUTHENTICATION,
+            WORKER_THREADS,
+            QUEUE_CAPACITY,
+            BATCH_SIZE
+    );
+
+    private static final String TRANSIT_URI_FORMAT = "https://%s:%d";
+
+    private Iterator<RequestCallback> requestCallbackProvider;
+
+    private EventServer server;
+
+    @Override
+    public final Set<Relationship> getRelationships() {
+        return RELATIONSHIPS;
+    }
+
+    @Override
+    public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+        return DESCRIPTORS;
+    }
+
+    @OnScheduled
+    public void onScheduled(final ProcessContext context) throws UnknownHostException {
+        final EventServerFactory eventServerFactory = createEventServerFactory(context);
+        server = eventServerFactory.getEventServer();
+    }
+
+    @OnStopped
+    public void onStopped() {
+        if (server == null) {
+            getLogger().info("Server not running");

Review Comment:
   Thanks for the note and question. Yes, it was from some early iterations, as it highlights the server status in some edge cases when using the mock TestRunner with different arguments. However, I agree that in normal operation, it should not happen, it there is nothing to do anyway, so I will simplify the logic and avoid the log.



##########
nifi-nar-bundles/nifi-opentelemetry-bundle/nifi-opentelemetry-processors/src/main/java/org/apache/nifi/processors/opentelemetry/encoding/JsonServiceRequestReader.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.opentelemetry.encoding;
+
+import com.google.protobuf.Message;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.Objects;
+
+/**
+ * Service Request Reader implementation based on JSON Parser supporting standard OTLP Request Types
+ */
+public class JsonServiceRequestReader implements ServiceRequestReader {
+    private static final RequestMapper REQUEST_MAPPER = new StandardRequestMapper();
+
+    /**
+     * Read Service Request parsed from Buffer
+     *
+     * @param buffer Byte Buffer
+     * @param requestType Request Message Type
+     * @return Service Request read
+     */
+    @Override
+    public <T extends Message> T read(final ByteBuffer buffer, final Class<T> requestType) {
+        Objects.requireNonNull(buffer, "Buffer required");
+        final byte[] bytes = getBytes(buffer);
+
+        try {
+            return REQUEST_MAPPER.readValue(bytes, requestType);

Review Comment:
   Thanks for the question and recommendation! The request processing elements went through a few iterations, and this is a good opportunity for optimization.  The supporting Jackson ObjectMapper supports streams, which will avoid creating the intermediate copy.



##########
nifi-nar-bundles/nifi-opentelemetry-bundle/nifi-opentelemetry-processors/pom.xml:
##########
@@ -0,0 +1,105 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements. See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License. You may obtain a copy of the License at
+  http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+
+    <parent>
+        <groupId>org.apache.nifi</groupId>
+        <artifactId>nifi-opentelemetry-bundle</artifactId>
+        <version>2.0.0-SNAPSHOT</version>
+    </parent>
+
+    <artifactId>nifi-opentelemetry-processors</artifactId>
+    <packaging>jar</packaging>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.nifi</groupId>
+            <artifactId>nifi-ssl-context-service-api</artifactId>
+            <version>2.0.0-SNAPSHOT</version>
+            <scope>provided</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.nifi</groupId>
+            <artifactId>nifi-event-transport</artifactId>
+            <version>2.0.0-SNAPSHOT</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.nifi</groupId>
+            <artifactId>nifi-utils</artifactId>
+            <version>2.0.0-SNAPSHOT</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.nifi</groupId>
+            <artifactId>nifi-api</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-codec-http</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>io.netty</groupId>
+            <artifactId>netty-codec-http2</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>commons-codec</groupId>
+            <artifactId>commons-codec</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>com.fasterxml.jackson.core</groupId>
+            <artifactId>jackson-databind</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>com.hubspot.jackson</groupId>
+            <artifactId>jackson-datatype-protobuf</artifactId>
+            <version>0.9.14</version>
+            <exclusions>
+                <exclusion>
+                    <groupId>com.google.protobuf</groupId>
+                    <artifactId>protobuf-java-util</artifactId>
+                </exclusion>
+            </exclusions>
+        </dependency>
+        <dependency>
+            <groupId>io.opentelemetry.proto</groupId>
+            <artifactId>opentelemetry-proto</artifactId>
+            <version>1.0.0-alpha</version>

Review Comment:
   Good question, I also wondered about this during initial implementation.
   
   The `alpha` designation applies to the Java build of the OpenTelemetry Proto classes, not to the specification itself. The [OTLP 1.0.0 specification](https://github.com/open-telemetry/opentelemetry-proto#maturity-level) indicates the maturity level of both Protobuf and JSON representations as `Stable`, but the current [opentelemetry-proto-java](https://github.com/open-telemetry/opentelemetry-proto-java/blob/main/buildSrc/src/main/kotlin/otel.publish-conventions.gradle.kts#L16) build still appends `alpha` to the version of the published JAR files. It looks like this should be corrected in the future, but in terms of field naming and serialization formatting, this version is stable. Another alternative is to include direct compilation of Java classes from the Proto definitions as part of this module build, but that does not seem necessary.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-12115 Add ListenOTLP to collect OpenTelemetry [nifi]

Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #7830:
URL: https://github.com/apache/nifi/pull/7830#discussion_r1344209484


##########
nifi-nar-bundles/nifi-opentelemetry-bundle/nifi-opentelemetry-processors/src/main/java/org/apache/nifi/processors/opentelemetry/ListenOTLP.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.opentelemetry;
+
+import com.google.protobuf.Message;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.configuration.DefaultSchedule;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.event.transport.EventServer;
+import org.apache.nifi.event.transport.EventServerFactory;
+import org.apache.nifi.event.transport.netty.NettyEventServerFactory;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processors.opentelemetry.protocol.TelemetryAttributeName;
+import org.apache.nifi.processors.opentelemetry.io.RequestCallback;
+import org.apache.nifi.processors.opentelemetry.io.RequestCallbackProvider;
+import org.apache.nifi.processors.opentelemetry.server.HttpServerFactory;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.security.util.ClientAuth;
+import org.apache.nifi.ssl.SSLContextService;
+
+import javax.net.ssl.SSLContext;
+import java.net.InetAddress;
+import java.net.URI;
+import java.net.UnknownHostException;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@DefaultSchedule(period = "25 ms")
+@Tags({"OpenTelemetry", "OTel", "OTLP", "telemetry", "metrics", "traces", "logs"})
+@CapabilityDescription(
+        "Collect OpenTelemetry messages over HTTP or gRPC. " +
+        "Supports standard Export Service Request messages for logs, metrics, and traces. " +
+        "Implements OpenTelemetry OTLP Specification 1.0.0 with OTLP/gRPC and OTLP/HTTP. " +
+        "Provides protocol detection using the HTTP Content-Type header."
+)
+@WritesAttributes({
+        @WritesAttribute(attribute = TelemetryAttributeName.MIME_TYPE, description = "Content-Type set to application/json"),
+        @WritesAttribute(attribute = TelemetryAttributeName.RESOURCE_TYPE, description = "OpenTelemetry Resource Type: LOGS, METRICS, or TRACES"),
+        @WritesAttribute(attribute = TelemetryAttributeName.RESOURCE_COUNT, description = "Count of resource elements included in messages"),
+})
+public class ListenOTLP extends AbstractProcessor {
+
+    static final PropertyDescriptor ADDRESS = new PropertyDescriptor.Builder()
+            .name("Address")
+            .displayName("Address")
+            .description("Internet Protocol Address on which to listen for OTLP Export Service Requests. The default value enables listening on all addresses.")
+            .required(true)
+            .defaultValue("0.0.0.0")
+            .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final PropertyDescriptor PORT = new PropertyDescriptor.Builder()
+            .name("Port")
+            .displayName("Port")
+            .description("TCP port number on which to listen for OTLP Export Service Requests over HTTP and gRPC")
+            .required(true)
+            .defaultValue("4317")
+            .addValidator(StandardValidators.PORT_VALIDATOR)
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final PropertyDescriptor SSL_CONTEXT_SERVICE = new PropertyDescriptor.Builder()
+            .name("SSL Context Service")
+            .displayName("SSL Context Service")
+            .description("SSL Context Service enables TLS communication for HTTPS")
+            .required(true)
+            .identifiesControllerService(SSLContextService.class)
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final PropertyDescriptor CLIENT_AUTHENTICATION = new PropertyDescriptor.Builder()
+            .name("Client Authentication")
+            .displayName("Client Authentication")
+            .description("Client authentication policy for TLS communication with HTTPS")
+            .required(true)
+            .allowableValues(ClientAuth.values())
+            .defaultValue(ClientAuth.WANT.name())
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final PropertyDescriptor WORKER_THREADS = new PropertyDescriptor.Builder()
+            .name("Worker Threads")
+            .displayName("Worker Threads")
+            .description("Number of threads responsible for decoding and queuing incoming OTLP Export Service Requests")
+            .required(true)
+            .defaultValue("2")
+            .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final PropertyDescriptor QUEUE_CAPACITY = new PropertyDescriptor.Builder()
+            .name("Queue Capacity")
+            .displayName("Queue Capacity")
+            .description("Maximum number of OTLP request resource elements that can be received and queued")
+            .required(true)
+            .defaultValue("10000")
+            .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+            .name("Batch Size")
+            .displayName("Batch Size")
+            .description("Maximum number of OTLP request resource elements included in each FlowFile produced")
+            .required(true)
+            .defaultValue("1000")
+            .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+            .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+            .build();
+
+    static final Relationship SUCCESS = new Relationship.Builder()
+            .name("success")
+            .description("Export Service Requests containing OTLP Telemetry")
+            .build();
+
+    private static final Set<Relationship> RELATIONSHIPS = Collections.singleton(SUCCESS);
+
+    private static final List<PropertyDescriptor> DESCRIPTORS = List.of(
+            ADDRESS,
+            PORT,
+            SSL_CONTEXT_SERVICE,
+            CLIENT_AUTHENTICATION,
+            WORKER_THREADS,
+            QUEUE_CAPACITY,
+            BATCH_SIZE
+    );
+
+    private static final String TRANSIT_URI_FORMAT = "https://%s:%d";
+
+    private Iterator<RequestCallback> requestCallbackProvider;
+
+    private EventServer server;
+
+    @Override
+    public final Set<Relationship> getRelationships() {
+        return RELATIONSHIPS;
+    }
+
+    @Override
+    public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+        return DESCRIPTORS;
+    }
+
+    @OnScheduled
+    public void onScheduled(final ProcessContext context) throws UnknownHostException {
+        final EventServerFactory eventServerFactory = createEventServerFactory(context);
+        server = eventServerFactory.getEventServer();
+    }
+
+    @OnStopped
+    public void onStopped() {
+        if (server == null) {
+            getLogger().info("Server not running");

Review Comment:
   Thanks for the note and question. Yes, it was from some early iterations, as it highlights the server status in some edge cases when using the mock TestRunner with different arguments. However, I agree that in normal operation, it should not happen, there is nothing to do anyway, so I will simplify the logic and avoid the log.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-12115 Add ListenOTLP to collect OpenTelemetry [nifi]

Posted by "joewitt (via GitHub)" <gi...@apache.org>.
joewitt commented on PR #7830:
URL: https://github.com/apache/nifi/pull/7830#issuecomment-1745530248

   thanks - running now at rate with the latest commits and getting a series of buffer size failures and connection resets. 
   
   2023-10-03 11:45:16,798 WARN [ListenOTLP[f3e6ea42-018a-1000-d8d4-3a6b40e25474]-1-2] o.a.n.p.opentelemetry.ListenOTLP ListenOTLP[id=f3e6ea42-018a-1000-d8d4-3a6b40e25474] Communication Failed with Remote Address [/127.0.0.1:60346]
   java.lang.OutOfMemoryError: Cannot reserve 4194304 bytes of direct buffer memory (allocated: 1072513273, limit: 1073741824)
   	at java.base/java.nio.Bits.reserveMemory(Bits.java:178)
   	at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127)
   	at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:360)
   	at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:721)
   	at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:696)
   	at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:215)
   	at io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:197)
   	at io.netty.buffer.PoolArena.allocate(PoolArena.java:139)
   	at io.netty.buffer.PoolArena.allocate(PoolArena.java:129)
   	at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:396)
   	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
   	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
   	at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:140)
   	at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-12115 Add ListenOTLP to collect OpenTelemetry [nifi]

Posted by "asfgit (via GitHub)" <gi...@apache.org>.
asfgit closed pull request #7830: NIFI-12115 Add ListenOTLP to collect OpenTelemetry
URL: https://github.com/apache/nifi/pull/7830


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-12115 Add ListenOTLP to collect OpenTelemetry [nifi]

Posted by "joewitt (via GitHub)" <gi...@apache.org>.
joewitt commented on code in PR #7830:
URL: https://github.com/apache/nifi/pull/7830#discussion_r1344259101


##########
nifi-nar-bundles/nifi-opentelemetry-bundle/nifi-opentelemetry-processors/src/main/java/org/apache/nifi/processors/opentelemetry/encoding/TelemetryMessageSerializer.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.opentelemetry.encoding;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.google.protobuf.ByteString;
+import com.google.protobuf.Descriptors;
+import com.hubspot.jackson.datatype.protobuf.ProtobufJacksonConfig;
+import com.hubspot.jackson.datatype.protobuf.builtin.serializers.MessageSerializer;
+import org.apache.commons.codec.binary.Hex;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * OpenTelemetry extension of Protobuf Message Serializer supporting OTLP 1.0.0 customization of selected fields
+ */
+public class TelemetryMessageSerializer extends MessageSerializer {
+    private static final Set<String> HEXADECIMAL_BYTE_STRING_FIELDS = Arrays.stream(HexadecimalByteStringField.values())
+            .map(HexadecimalByteStringField::getField)
+            .collect(Collectors.toSet());
+
+    protected TelemetryMessageSerializer(final ProtobufJacksonConfig config) {
+        super(config);
+    }
+
+    /**
+     * Write value as JSON with hexadecimal encoding for selected ByteString fields
+     *
+     * @param field Message Field Decriptor

Review Comment:
   But I had a typo telling you there was a typo which made it confusing :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-12115 Add ListenOTLP to collect OpenTelemetry [nifi]

Posted by "joewitt (via GitHub)" <gi...@apache.org>.
joewitt commented on code in PR #7830:
URL: https://github.com/apache/nifi/pull/7830#discussion_r1343470578


##########
nifi-nar-bundles/nifi-opentelemetry-bundle/nifi-opentelemetry-processors/src/main/java/org/apache/nifi/processors/opentelemetry/io/StandardRequestContentListener.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.opentelemetry.io;
+
+import com.google.protobuf.Message;
+import io.opentelemetry.proto.collector.logs.v1.ExportLogsServiceRequest;
+import io.opentelemetry.proto.collector.metrics.v1.ExportMetricsServiceRequest;
+import io.opentelemetry.proto.collector.trace.v1.ExportTraceServiceRequest;
+import io.opentelemetry.proto.common.v1.AnyValue;
+import io.opentelemetry.proto.common.v1.KeyValue;
+import io.opentelemetry.proto.logs.v1.ResourceLogs;
+import io.opentelemetry.proto.metrics.v1.ResourceMetrics;
+import io.opentelemetry.proto.resource.v1.Resource;
+import io.opentelemetry.proto.trace.v1.ResourceSpans;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processors.opentelemetry.encoding.JsonServiceRequestReader;
+import org.apache.nifi.processors.opentelemetry.encoding.ProtobufServiceRequestReader;
+import org.apache.nifi.processors.opentelemetry.encoding.ServiceRequestReader;
+import org.apache.nifi.processors.opentelemetry.protocol.ServiceRequestDescription;
+import org.apache.nifi.processors.opentelemetry.protocol.ServiceResponse;
+import org.apache.nifi.processors.opentelemetry.protocol.ServiceResponseStatus;
+import org.apache.nifi.processors.opentelemetry.protocol.TelemetryContentEncoding;
+import org.apache.nifi.processors.opentelemetry.protocol.TelemetryContentType;
+import org.apache.nifi.processors.opentelemetry.protocol.TelemetryRequestType;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.BlockingQueue;
+import java.util.zip.GZIPInputStream;
+
+/**
+ * Standard implementation of OTLP Request Content Listener supporting available Request and Content Types
+ */
+public class StandardRequestContentListener implements RequestContentListener {
+    private static final int ZERO_MESSAGES = 0;
+
+    private static final byte COMPRESSED = 1;
+
+    private static final String CLIENT_SOCKET_ADDRESS = "client.socket.address";
+
+    private static final String CLIENT_SOCKET_PORT = "client.socket.port";
+
+    private final ServiceRequestReader protobufReader = new ProtobufServiceRequestReader();
+
+    private final ServiceRequestReader jsonReader = new JsonServiceRequestReader();
+
+    private final ComponentLog log;
+
+    private final BlockingQueue<Message> messages;
+
+    public StandardRequestContentListener(final ComponentLog log, final BlockingQueue<Message> messages) {
+        this.log = Objects.requireNonNull(log, "Log required");
+        this.messages = Objects.requireNonNull(messages, "Messages required");
+    }
+
+    /**
+     * Process Request Buffer supporting serialized Protobuf or JSON
+     *
+     * @param buffer Request Content Buffer to be processed
+     * @param serviceRequestDescription Service Request Description describes reader attributes
+     * @return Service Response with processing status
+     */
+    @Override
+    public ServiceResponse onRequest(final ByteBuffer buffer, final ServiceRequestDescription serviceRequestDescription) {
+        Objects.requireNonNull(buffer, "Buffer required");
+        Objects.requireNonNull(serviceRequestDescription, "Description required");
+
+        ServiceResponse serviceResponse;
+
+        final InetSocketAddress remoteAddress = serviceRequestDescription.getRemoteAddress();
+        final TelemetryContentType contentType = serviceRequestDescription.getContentType();
+        if (TelemetryContentType.APPLICATION_GRPC == contentType) {
+            try {
+                final byte compression = buffer.get();
+                final int messageSize = buffer.getInt();
+                log.debug("Client Address [{}] Content-Type [{}] Message Size [{}] Compression [{}]", remoteAddress, contentType, messageSize, compression);
+
+                final ByteBuffer messageBuffer;
+                if (COMPRESSED == compression) {
+                    messageBuffer = getDecompressedBuffer(buffer);

Review Comment:
   are we protected from this being potentially quite large?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-12115 Add ListenOTLP to collect OpenTelemetry [nifi]

Posted by "joewitt (via GitHub)" <gi...@apache.org>.
joewitt commented on code in PR #7830:
URL: https://github.com/apache/nifi/pull/7830#discussion_r1343460122


##########
nifi-nar-bundles/nifi-opentelemetry-bundle/nifi-opentelemetry-processors/src/main/java/org/apache/nifi/processors/opentelemetry/encoding/ByteStringFieldDeserializer.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.opentelemetry.encoding;
+
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.databind.DeserializationContext;
+import com.google.protobuf.ByteString;
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.Message;
+import com.hubspot.jackson.datatype.protobuf.ProtobufJacksonConfig;
+import com.hubspot.jackson.datatype.protobuf.builtin.deserializers.MessageDeserializer;
+import org.apache.commons.codec.DecoderException;
+import org.apache.commons.codec.binary.Hex;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * ByteString Field Deserializer supporting conversion from hexadecimal to ByteString for selected fields
+ *
+ * @param <T> Message Type
+ * @param <V> Message Builder Type
+ */
+public class ByteStringFieldDeserializer<T extends Message, V extends Message.Builder> extends MessageDeserializer<T, V> {
+    private static final Set<String> HEXADECIMAL_BYTE_STRING_FIELDS = Arrays.stream(HexadecimalByteStringField.values())
+            .map(HexadecimalByteStringField::getField)
+            .collect(Collectors.toSet());
+
+    /**
+     * Deserializer constructor with Message Type class to be deserialized
+     *
+     * @param messageType Message Type class to be deserialized
+     * @param config Jackson Configuration for Protobuf
+     */
+    public ByteStringFieldDeserializer(final Class<T> messageType, final ProtobufJacksonConfig config) {
+        super(messageType, config);
+    }
+
+    /**
+     * Read value from JSON Parser and decode hexadecimal ByteString fields when found
+     *
+     * @param builder Protobuf Message Builder
+     * @param field Message Field Deescriptor

Review Comment:
   little typo 'dee'



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-12115 Add ListenOTLP to collect OpenTelemetry [nifi]

Posted by "joewitt (via GitHub)" <gi...@apache.org>.
joewitt commented on PR #7830:
URL: https://github.com/apache/nifi/pull/7830#issuecomment-1744241664

   It is alive and working - very cool.  Need to learn more about what this java agent exporter really does but the json output is interesting.  I did note after running for a minute or two I received a couple of these outputs.  I was making minor edits to the flow when it happened but not this processor so dont have the underlying data it choked on.
   
   `2023-10-02 22:32:53,888 WARN [ListenOTLP[f3e6ea42-018a-1000-d8d4-3a6b40e25474]-1-1] o.a.n.p.opentelemetry.ListenOTLP ListenOTLP[id=f3e6ea42-018a-1000-d8d4-3a6b40e25474] Client Address [/127.0.0.1:55410] Content-Type [APPLICATION_GRPC] processing failed
   java.io.UncheckedIOException: Request parsing failed
   	at org.apache.nifi.processors.opentelemetry.encoding.ProtobufServiceRequestReader.read(ProtobufServiceRequestReader.java:64)
   	at org.apache.nifi.processors.opentelemetry.io.StandardRequestContentListener.readMessages(StandardRequestContentListener.java:165)
   	at org.apache.nifi.processors.opentelemetry.io.StandardRequestContentListener.onSupportedRequest(StandardRequestContentListener.java:145)
   	at org.apache.nifi.processors.opentelemetry.io.StandardRequestContentListener.onRequest(StandardRequestContentListener.java:109)
   	at org.apache.nifi.processors.opentelemetry.server.Http2RequestFrameListener.onDataRead(Http2RequestFrameListener.java:177)
   	at io.netty.handler.codec.http2.Http2FrameListenerDecorator.onDataRead(Http2FrameListenerDecorator.java:36)
   	at io.netty.handler.codec.http2.Http2EmptyDataFrameListener.onDataRead(Http2EmptyDataFrameListener.java:49)
   	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:307)
   	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:415)
   	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:250)
   	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159)
   	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173)
   	at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:63)
   	at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393)
   	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
   	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
   	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
   	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
   	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
   	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1471)
   	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1334)
   	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1383)
   	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
   	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
   	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
   	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
   	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
   	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
   	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
   	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
   	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
   	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
   	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
   	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
   	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
   	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
   	at java.base/java.lang.Thread.run(Thread.java:1583)
   Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message had invalid UTF-8.
   	at com.google.protobuf.InvalidProtocolBufferException.invalidUtf8(InvalidProtocolBufferException.java:172)
   	at com.google.protobuf.Utf8$UnsafeProcessor.decodeUtf8Direct(Utf8.java:1434)
   	at com.google.protobuf.Utf8$Processor.decodeUtf8(Utf8.java:631)
   	at com.google.protobuf.Utf8.decodeUtf8(Utf8.java:331)
   	at com.google.protobuf.CodedInputStream$UnsafeDirectNioDecoder.readStringRequireUtf8(CodedInputStream.java:1528)
   	at io.opentelemetry.proto.metrics.v1.Metric$Builder.mergeFrom(Metric.java:1061)
   	at io.opentelemetry.proto.metrics.v1.Metric$1.parsePartialFrom(Metric.java:2167)
   	at io.opentelemetry.proto.metrics.v1.Metric$1.parsePartialFrom(Metric.java:2159)
   	at com.google.protobuf.CodedInputStream$UnsafeDirectNioDecoder.readMessage(CodedInputStream.java:1600)
   	at io.opentelemetry.proto.metrics.v1.ScopeMetrics$Builder.mergeFrom(ScopeMetrics.java:604)
   	at io.opentelemetry.proto.metrics.v1.ScopeMetrics$1.parsePartialFrom(ScopeMetrics.java:1248)
   	at io.opentelemetry.proto.metrics.v1.ScopeMetrics$1.parsePartialFrom(ScopeMetrics.java:1240)
   	at com.google.protobuf.CodedInputStream$UnsafeDirectNioDecoder.readMessage(CodedInputStream.java:1600)
   	at io.opentelemetry.proto.metrics.v1.ResourceMetrics$Builder.mergeFrom(ResourceMetrics.java:603)
   	at io.opentelemetry.proto.metrics.v1.ResourceMetrics$1.parsePartialFrom(ResourceMetrics.java:1243)
   	at io.opentelemetry.proto.metrics.v1.ResourceMetrics$1.parsePartialFrom(ResourceMetrics.java:1235)
   	at com.google.protobuf.CodedInputStream$UnsafeDirectNioDecoder.readMessage(CodedInputStream.java:1600)
   	at io.opentelemetry.proto.collector.metrics.v1.ExportMetricsServiceRequest$Builder.mergeFrom(ExportMetricsServiceRequest.java:469)
   	at io.opentelemetry.proto.collector.metrics.v1.ExportMetricsServiceRequest$1.parsePartialFrom(ExportMetricsServiceRequest.java:915)
   	at io.opentelemetry.proto.collector.metrics.v1.ExportMetricsServiceRequest$1.parsePartialFrom(ExportMetricsServiceRequest.java:907)
   	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:134)
   	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:149)
   	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48)
   	at org.apache.nifi.processors.opentelemetry.encoding.ProtobufServiceRequestReader.read(ProtobufServiceRequestReader.java:56)
   	... 42 common frames omitted`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-12115 Add ListenOTLP to collect OpenTelemetry [nifi]

Posted by "joewitt (via GitHub)" <gi...@apache.org>.
joewitt commented on PR #7830:
URL: https://github.com/apache/nifi/pull/7830#issuecomment-1746025606

   All looks good.  And cruising around 9500 events/sec on default settings.  No more errors of any kind and smooth sailing. Thanks!  I'll merge to main.  Can you please adjust and merge to support as desired.  +1 on that


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-12115 Add ListenOTLP to collect OpenTelemetry [nifi]

Posted by "joewitt (via GitHub)" <gi...@apache.org>.
joewitt commented on PR #7830:
URL: https://github.com/apache/nifi/pull/7830#issuecomment-1745124928

   @exceptionfactory I should add I am not particularly convicted on the byte[]/buffer stuff.  Just seems like an opportunity to avoid intermediate copies.  My mental model is this stuff could be in some cases fairly high rate and thus we want to be as efficient as we can.  I didn't comment on the one related to decompression but it too seemed to be subject to some improvements.  I looked at what we're doing in other areas such as this and we have byte[] usage but less so are we jumping between byte buffers to then new byte[] etc..  Ultimately it probably isn't worth looking into now but if we see high utilization and profiling suggests we can do better then we deal with it then.
   
   The error I noted and stack trace above I could not get to happen again during testing last night.  It suggests some funky state can be entered but I'm not sure of the nature of it.  Might well be a problem with the agent itself.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org