You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "derrickaw (via GitHub)" <gi...@apache.org> on 2026/02/05 14:19:18 UTC
[PR] add datadog it support [beam]
derrickaw opened a new pull request, #37517:
URL: https://github.com/apache/beam/pull/37517
Add some datadog it support
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add datadog it support [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #37517:
URL: https://github.com/apache/beam/pull/37517#issuecomment-3878454706
/gemini review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add datadog it support [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #37517:
URL: https://github.com/apache/beam/pull/37517#discussion_r2788950453
##########
it/datadog/src/test/java/org/apache/beam/it/datadog/DatadogResourceManagerTest.java:
##########
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.it.datadog;
+
+import static com.google.common.truth.Truth.assertThat;
+import static org.junit.Assert.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.io.IOException;
+import java.util.Collections;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.mockito.Answers;
+import org.mockito.Mock;
+import org.mockito.junit.MockitoJUnit;
+import org.mockito.junit.MockitoRule;
+import org.mockserver.client.ForwardChainExpectation;
+import org.mockserver.client.MockServerClient;
+import org.mockserver.model.HttpRequest;
+import org.testcontainers.containers.MockServerContainer;
+
+/** Unit tests for {@link com.google.cloud.teleport.it.datadog.DatadogResourceManager}. */
Review Comment:
fixed
##########
it/datadog/src/main/java/org/apache/beam/it/datadog/conditions/package-info.java:
##########
@@ -0,0 +1,18 @@
+/*
+ * Copyright (C) 2023 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not
+ * use this file except in compliance with the License. You may obtain a copy of
+ * the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add datadog it support [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm merged PR #37517:
URL: https://github.com/apache/beam/pull/37517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add datadog it support [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #37517:
URL: https://github.com/apache/beam/pull/37517#discussion_r2788949235
##########
it/datadog/src/test/java/org/apache/beam/it/datadog/DatadogResourceManagerUtilsTest.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.it.datadog;
+
+import static com.google.common.truth.Truth.assertThat;
+import static org.apache.beam.it.datadog.DatadogResourceManagerUtils.datadogEntryToMap;
+import static org.apache.beam.it.datadog.DatadogResourceManagerUtils.generateApiKey;
+
+import java.util.HashMap;
+import java.util.Map;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Unit tests for {@link com.google.cloud.teleport.it.datadog.DatadogResourceManager}. */
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add datadog it support [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #37517:
URL: https://github.com/apache/beam/pull/37517#discussion_r2788961692
##########
it/datadog/src/main/java/org/apache/beam/it/datadog/DatadogResourceManager.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.it.datadog;
+
+import static org.apache.beam.it.datadog.DatadogResourceManagerUtils.datadogEntryToMap;
+import static org.apache.beam.it.datadog.DatadogResourceManagerUtils.generateApiKey;
+import static org.mockserver.model.HttpRequest.request;
+import static org.mockserver.model.HttpResponse.response;
+import static org.mockserver.model.JsonSchemaBody.jsonSchema;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.gson.Gson;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import java.io.IOException;
+import java.io.UnsupportedEncodingException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+import org.apache.beam.it.common.ResourceManager;
+import org.apache.beam.it.testcontainers.TestContainerResourceManager;
+import org.apache.beam.it.testcontainers.TestContainerResourceManagerException;
+import org.apache.http.client.entity.GzipCompressingEntity;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.json.JSONObject;
+import org.mockserver.client.MockServerClient;
+import org.mockserver.model.HttpRequest;
+import org.mockserver.model.MediaType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.MockServerContainer;
+import org.testcontainers.utility.DockerImageName;
+
+/**
+ * Client for managing Datadog resources.
+ *
+ * <p>The class supports one mock Datadog server instance.
+ *
+ * <p>The class is thread-safe.
+ *
+ * <p>Note: The MockServer TestContainer will only run on M1 Mac's if the Docker version is >=
+ * 4.16.0 and the "Use Rosetta for x86/amd64 emulation on Apple Silicon" setting is enabled.
+ */
+public class DatadogResourceManager extends TestContainerResourceManager<MockServerContainer>
+ implements ResourceManager {
+
+ private static final Logger LOG = LoggerFactory.getLogger(DatadogResourceManager.class);
+ private static final String DEFAULT_MOCKSERVER_CONTAINER_NAME = "mockserver/mockserver";
+
+ // A list of available Mockserver Docker image tags can be found at
+ // https://hub.docker.com/r/mockserver/mockserver/tags
+ private static final String DEFAULT_MOCKSERVER_CONTAINER_TAG;
+
+ static {
+ DEFAULT_MOCKSERVER_CONTAINER_TAG =
+ Optional.ofNullable(MockServerClient.class.getPackage())
+ .map(Package::getImplementationVersion)
+ .orElseThrow(
+ () ->
+ new TestContainerResourceManagerException(
+ "Could not determine Mockserver container version."));
+ }
+
+ // See: https://docs.datadoghq.com/api/latest/logs/#send-logs
+ private static final String SEND_LOGS_JSON_SCHEMA =
+ "{"
+ + " \"$schema\": \"http://json-schema.org/draft-04/schema#\","
+ + " \"type\": \"array\","
+ + " \"items\": ["
+ + " {"
+ + " \"type\": \"object\","
+ + " \"properties\": {"
+ + " \"ddsource\": {"
+ + " \"type\": \"string\""
+ + " },"
+ + " \"ddtags\": {"
+ + " \"type\": \"string\""
+ + " },"
+ + " \"hostname\": {"
+ + " \"type\": \"string\""
+ + " },"
+ + " \"message\": {"
+ + " \"type\": \"string\""
+ + " },"
+ + " \"service\": {"
+ + " \"type\": \"string\""
+ + " }"
+ + " },"
+ + " \"required\": ["
+ + " \"message\""
+ + " ]"
+ + " }"
+ + " ]"
+ + "}";
+
+ private static final Gson GSON = new Gson();
+
+ private final String apiKey;
+ private final DatadogClientFactory clientFactory;
+
+ private DatadogResourceManager(DatadogResourceManager.Builder builder) {
+ this(
+ new DatadogClientFactory(),
+ new MockServerContainer(
+ DockerImageName.parse(builder.containerImageName).withTag(builder.containerImageTag)),
+ builder);
+ }
+
+ @VisibleForTesting
+ DatadogResourceManager(
+ DatadogClientFactory clientFactory,
+ MockServerContainer container,
+ DatadogResourceManager.Builder builder) {
+ super(container, builder);
+
+ this.apiKey = builder.apiKey;
+ this.clientFactory = clientFactory;
+ }
+
+ public static DatadogResourceManager.Builder builder(String testId) {
+ return new DatadogResourceManager.Builder(testId);
+ }
+
+ /** Returns the port to connect to the mock Datadog server. */
+ public int getPort() {
+ return super.getPort(MockServerContainer.PORT);
+ }
+
+ /**
+ * Returns the HTTP endpoint that this mock Datadog server is configured to listen on.
+ *
+ * @return the HTTP endpoint.
+ */
+ public String getHttpEndpoint() {
+ return String.format("http://%s:%d", getHost(), getPort());
+ }
+
+ /**
+ * Returns the API endpoint that this mock Datadog server is configured to receive events at.
+ *
+ * <p>This will be the HTTP endpoint concatenated with <code>'/api/v2/logs'</code>.
+ *
+ * @return the API endpoint.
+ */
+ public String getApiEndpoint() {
+ return getHttpEndpoint() + "/api/v2/logs";
+ }
+
+ /**
+ * Returns the Datadog API key used to connect to this mock Datadog server.
+ *
+ * @return the API key.
+ */
+ public String getApiKey() {
+ return apiKey;
+ }
+
+ /**
+ * Helper method for converting the given DatadogLogEntry into JSON format.
+ *
+ * @param event The DatadogLogEntry to parse.
+ * @return JSON String.
+ */
+ private static String datadogEventToJson(DatadogLogEntry event) {
+ return new JSONObject(datadogEntryToMap(event)).toString();
+ }
Review Comment:
removed, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add datadog it support [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37517:
URL: https://github.com/apache/beam/pull/37517#issuecomment-3855060391
Assigning reviewers:
R: @damccorm added as fallback since no labels match configuration
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add datadog it support [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #37517:
URL: https://github.com/apache/beam/pull/37517#discussion_r2788618176
##########
it/datadog/build.gradle:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins { id 'org.apache.beam.module' }
+applyJavaNature(
+ automaticModuleName: 'org.apache.beam.it.datadog',
+)
+
+description = "Apache Beam :: IT :: Datadog"
+ext.summary = "Integration test utilities for Datadog."
+
+dependencies {
+ implementation project(path: ":it:common")
+ implementation project(path: ":it:testcontainers")
+ implementation project(path: ":it:truthmatchers")
+ implementation project(path: ":it:conditions")
+ implementation "org.testcontainers:mockserver:1.19.7"
+ implementation "org.testcontainers:testcontainers:1.19.7"
+ implementation "org.mock-server:mockserver-client-java:5.10.0"
+ implementation "org.mock-server:mockserver-core:5.10.0"
+ implementation library.java.json_org
+ implementation "org.apache.httpcomponents:httpclient:4.5.13"
+ implementation "org.apache.httpcomponents:httpcore:4.4.14"
+ implementation library.java.google_code_gson
+ implementation library.java.vendored_guava_32_1_2_jre
+ implementation "com.google.guava:guava:33.1.0-jre"
Review Comment:
This direct Guava dependency might conflict with the vendored Guava dependency on line 40 (`library.java.vendored_guava_32_1_2_jre`). Including two different versions of Guava can lead to classpath issues and runtime errors like `NoSuchMethodError`. It's recommended to use the project's managed dependencies. Please remove this line and rely on the vendored Guava.
##########
it/datadog/src/main/java/org/apache/beam/it/datadog/conditions/package-info.java:
##########
@@ -0,0 +1,18 @@
+/*
+ * Copyright (C) 2023 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not
+ * use this file except in compliance with the License. You may obtain a copy of
+ * the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
Review Comment:
The copyright header here is for "Google LLC". To maintain consistency with the rest of the project, please update this to the standard Apache Software Foundation (ASF) copyright header used in other files. This also applies to the other new `package-info.java` files in this PR.
##########
it/datadog/src/main/java/org/apache/beam/it/datadog/DatadogResourceManager.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.it.datadog;
+
+import static org.apache.beam.it.datadog.DatadogResourceManagerUtils.datadogEntryToMap;
+import static org.apache.beam.it.datadog.DatadogResourceManagerUtils.generateApiKey;
+import static org.mockserver.model.HttpRequest.request;
+import static org.mockserver.model.HttpResponse.response;
+import static org.mockserver.model.JsonSchemaBody.jsonSchema;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.gson.Gson;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import java.io.IOException;
+import java.io.UnsupportedEncodingException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+import org.apache.beam.it.common.ResourceManager;
+import org.apache.beam.it.testcontainers.TestContainerResourceManager;
+import org.apache.beam.it.testcontainers.TestContainerResourceManagerException;
+import org.apache.http.client.entity.GzipCompressingEntity;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.json.JSONObject;
+import org.mockserver.client.MockServerClient;
+import org.mockserver.model.HttpRequest;
+import org.mockserver.model.MediaType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.MockServerContainer;
+import org.testcontainers.utility.DockerImageName;
+
+/**
+ * Client for managing Datadog resources.
+ *
+ * <p>The class supports one mock Datadog server instance.
+ *
+ * <p>The class is thread-safe.
+ *
+ * <p>Note: The MockServer TestContainer will only run on M1 Mac's if the Docker version is >=
+ * 4.16.0 and the "Use Rosetta for x86/amd64 emulation on Apple Silicon" setting is enabled.
+ */
+public class DatadogResourceManager extends TestContainerResourceManager<MockServerContainer>
+ implements ResourceManager {
+
+ private static final Logger LOG = LoggerFactory.getLogger(DatadogResourceManager.class);
+ private static final String DEFAULT_MOCKSERVER_CONTAINER_NAME = "mockserver/mockserver";
+
+ // A list of available Mockserver Docker image tags can be found at
+ // https://hub.docker.com/r/mockserver/mockserver/tags
+ private static final String DEFAULT_MOCKSERVER_CONTAINER_TAG;
+
+ static {
+ DEFAULT_MOCKSERVER_CONTAINER_TAG =
+ Optional.ofNullable(MockServerClient.class.getPackage())
+ .map(Package::getImplementationVersion)
+ .orElseThrow(
+ () ->
+ new TestContainerResourceManagerException(
+ "Could not determine Mockserver container version."));
+ }
+
+ // See: https://docs.datadoghq.com/api/latest/logs/#send-logs
+ private static final String SEND_LOGS_JSON_SCHEMA =
+ "{"
+ + " \"$schema\": \"http://json-schema.org/draft-04/schema#\","
+ + " \"type\": \"array\","
+ + " \"items\": ["
+ + " {"
+ + " \"type\": \"object\","
+ + " \"properties\": {"
+ + " \"ddsource\": {"
+ + " \"type\": \"string\""
+ + " },"
+ + " \"ddtags\": {"
+ + " \"type\": \"string\""
+ + " },"
+ + " \"hostname\": {"
+ + " \"type\": \"string\""
+ + " },"
+ + " \"message\": {"
+ + " \"type\": \"string\""
+ + " },"
+ + " \"service\": {"
+ + " \"type\": \"string\""
+ + " }"
+ + " },"
+ + " \"required\": ["
+ + " \"message\""
+ + " ]"
+ + " }"
+ + " ]"
+ + "}";
+
+ private static final Gson GSON = new Gson();
+
+ private final String apiKey;
+ private final DatadogClientFactory clientFactory;
+
+ private DatadogResourceManager(DatadogResourceManager.Builder builder) {
+ this(
+ new DatadogClientFactory(),
+ new MockServerContainer(
+ DockerImageName.parse(builder.containerImageName).withTag(builder.containerImageTag)),
+ builder);
+ }
+
+ @VisibleForTesting
+ DatadogResourceManager(
+ DatadogClientFactory clientFactory,
+ MockServerContainer container,
+ DatadogResourceManager.Builder builder) {
+ super(container, builder);
+
+ this.apiKey = builder.apiKey;
+ this.clientFactory = clientFactory;
+ }
+
+ public static DatadogResourceManager.Builder builder(String testId) {
+ return new DatadogResourceManager.Builder(testId);
+ }
+
+ /** Returns the port to connect to the mock Datadog server. */
+ public int getPort() {
+ return super.getPort(MockServerContainer.PORT);
+ }
+
+ /**
+ * Returns the HTTP endpoint that this mock Datadog server is configured to listen on.
+ *
+ * @return the HTTP endpoint.
+ */
+ public String getHttpEndpoint() {
+ return String.format("http://%s:%d", getHost(), getPort());
+ }
+
+ /**
+ * Returns the API endpoint that this mock Datadog server is configured to receive events at.
+ *
+ * <p>This will be the HTTP endpoint concatenated with <code>'/api/v2/logs'</code>.
+ *
+ * @return the API endpoint.
+ */
+ public String getApiEndpoint() {
+ return getHttpEndpoint() + "/api/v2/logs";
+ }
+
+ /**
+ * Returns the Datadog API key used to connect to this mock Datadog server.
+ *
+ * @return the API key.
+ */
+ public String getApiKey() {
+ return apiKey;
+ }
+
+ /**
+ * Helper method for converting the given DatadogLogEntry into JSON format.
+ *
+ * @param event The DatadogLogEntry to parse.
+ * @return JSON String.
+ */
+ private static String datadogEventToJson(DatadogLogEntry event) {
+ return new JSONObject(datadogEntryToMap(event)).toString();
+ }
Review Comment:
This helper method `datadogEventToJson` appears to be unused within the class. To keep the code clean and maintainable, it's best to remove unused methods.
##########
it/datadog/src/main/java/org/apache/beam/it/datadog/DatadogResourceManager.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.it.datadog;
+
+import static org.apache.beam.it.datadog.DatadogResourceManagerUtils.datadogEntryToMap;
+import static org.apache.beam.it.datadog.DatadogResourceManagerUtils.generateApiKey;
+import static org.mockserver.model.HttpRequest.request;
+import static org.mockserver.model.HttpResponse.response;
+import static org.mockserver.model.JsonSchemaBody.jsonSchema;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.gson.Gson;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import java.io.IOException;
+import java.io.UnsupportedEncodingException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+import org.apache.beam.it.common.ResourceManager;
+import org.apache.beam.it.testcontainers.TestContainerResourceManager;
+import org.apache.beam.it.testcontainers.TestContainerResourceManagerException;
+import org.apache.http.client.entity.GzipCompressingEntity;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.json.JSONObject;
+import org.mockserver.client.MockServerClient;
+import org.mockserver.model.HttpRequest;
+import org.mockserver.model.MediaType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.MockServerContainer;
+import org.testcontainers.utility.DockerImageName;
+
+/**
+ * Client for managing Datadog resources.
+ *
+ * <p>The class supports one mock Datadog server instance.
+ *
+ * <p>The class is thread-safe.
+ *
+ * <p>Note: The MockServer TestContainer will only run on M1 Mac's if the Docker version is >=
+ * 4.16.0 and the "Use Rosetta for x86/amd64 emulation on Apple Silicon" setting is enabled.
+ */
+public class DatadogResourceManager extends TestContainerResourceManager<MockServerContainer>
+ implements ResourceManager {
+
+ private static final Logger LOG = LoggerFactory.getLogger(DatadogResourceManager.class);
+ private static final String DEFAULT_MOCKSERVER_CONTAINER_NAME = "mockserver/mockserver";
+
+ // A list of available Mockserver Docker image tags can be found at
+ // https://hub.docker.com/r/mockserver/mockserver/tags
+ private static final String DEFAULT_MOCKSERVER_CONTAINER_TAG;
+
+ static {
+ DEFAULT_MOCKSERVER_CONTAINER_TAG =
+ Optional.ofNullable(MockServerClient.class.getPackage())
+ .map(Package::getImplementationVersion)
+ .orElseThrow(
+ () ->
+ new TestContainerResourceManagerException(
+ "Could not determine Mockserver container version."));
+ }
+
+ // See: https://docs.datadoghq.com/api/latest/logs/#send-logs
+ private static final String SEND_LOGS_JSON_SCHEMA =
+ "{"
+ + " \"$schema\": \"http://json-schema.org/draft-04/schema#\","
+ + " \"type\": \"array\","
+ + " \"items\": ["
+ + " {"
+ + " \"type\": \"object\","
+ + " \"properties\": {"
+ + " \"ddsource\": {"
+ + " \"type\": \"string\""
+ + " },"
+ + " \"ddtags\": {"
+ + " \"type\": \"string\""
+ + " },"
+ + " \"hostname\": {"
+ + " \"type\": \"string\""
+ + " },"
+ + " \"message\": {"
+ + " \"type\": \"string\""
+ + " },"
+ + " \"service\": {"
+ + " \"type\": \"string\""
+ + " }"
+ + " },"
+ + " \"required\": ["
+ + " \"message\""
+ + " ]"
+ + " }"
+ + " ]"
+ + "}";
+
+ private static final Gson GSON = new Gson();
+
+ private final String apiKey;
+ private final DatadogClientFactory clientFactory;
+
+ private DatadogResourceManager(DatadogResourceManager.Builder builder) {
+ this(
+ new DatadogClientFactory(),
+ new MockServerContainer(
+ DockerImageName.parse(builder.containerImageName).withTag(builder.containerImageTag)),
+ builder);
+ }
+
+ @VisibleForTesting
+ DatadogResourceManager(
+ DatadogClientFactory clientFactory,
+ MockServerContainer container,
+ DatadogResourceManager.Builder builder) {
+ super(container, builder);
+
+ this.apiKey = builder.apiKey;
+ this.clientFactory = clientFactory;
+ }
+
+ public static DatadogResourceManager.Builder builder(String testId) {
+ return new DatadogResourceManager.Builder(testId);
+ }
+
+ /** Returns the port to connect to the mock Datadog server. */
+ public int getPort() {
+ return super.getPort(MockServerContainer.PORT);
+ }
+
+ /**
+ * Returns the HTTP endpoint that this mock Datadog server is configured to listen on.
+ *
+ * @return the HTTP endpoint.
+ */
+ public String getHttpEndpoint() {
+ return String.format("http://%s:%d", getHost(), getPort());
+ }
+
+ /**
+ * Returns the API endpoint that this mock Datadog server is configured to receive events at.
+ *
+ * <p>This will be the HTTP endpoint concatenated with <code>'/api/v2/logs'</code>.
+ *
+ * @return the API endpoint.
+ */
+ public String getApiEndpoint() {
+ return getHttpEndpoint() + "/api/v2/logs";
+ }
+
+ /**
+ * Returns the Datadog API key used to connect to this mock Datadog server.
+ *
+ * @return the API key.
+ */
+ public String getApiKey() {
+ return apiKey;
+ }
+
+ /**
+ * Helper method for converting the given DatadogLogEntry into JSON format.
+ *
+ * @param event The DatadogLogEntry to parse.
+ * @return JSON String.
+ */
+ private static String datadogEventToJson(DatadogLogEntry event) {
+ return new JSONObject(datadogEntryToMap(event)).toString();
+ }
+
+ /**
+ * Sends the given HTTP event to the mock Datadog server.
+ *
+ * @param event The {@link DatadogLogEntry} to send to the API.
+ * @return True, if the request was successful.
+ */
+ public synchronized boolean sendHttpEvent(DatadogLogEntry event) {
+ return sendHttpEvents(Collections.singletonList(event));
+ }
+
+ /**
+ * Sends the given HTTP events to the mock Datadog server.
+ *
+ * @param events The {@link DatadogLogEntry}s to send to the API.
+ * @return True, if the request was successful.
+ */
+ public synchronized boolean sendHttpEvents(Collection<DatadogLogEntry> events) {
+
+ LOG.info("Attempting to send {} events to {}.", events.size(), getApiEndpoint());
+
+ // Construct base API request
+ HttpPost httppost = new HttpPost(getApiEndpoint());
+ httppost.addHeader("Content-Encoding", "gzip");
+ httppost.addHeader("Content-Type", "application/json");
+ httppost.addHeader("dd-api-key", apiKey);
+
+ String eventsData = GSON.toJson(events);
+
+ try (CloseableHttpClient httpClient = clientFactory.getHttpClient()) {
+ // Set request data
+ try {
+ httppost.setEntity(new GzipCompressingEntity(new StringEntity(eventsData)));
+ } catch (UnsupportedEncodingException e) {
+ throw new DatadogResourceManagerException(
+ "Error setting HTTP message data to " + eventsData, e);
+ }
+
+ // Send request
+ try (CloseableHttpResponse response = httpClient.execute(httppost)) {
+ // Check error code
+ int code = response.getStatusLine().getStatusCode();
+ if (code != 202) {
+ throw new DatadogResourceManagerException(
+ "Received http error code " + code + " sending event.");
+ }
+ } catch (Exception e) {
+ throw new DatadogResourceManagerException("Error sending event.", e);
+ }
+ } catch (IOException e) {
+ throw new DatadogResourceManagerException("Error with HTTP client.", e);
+ }
+
+ LOG.info("Successfully sent {} events.", events.size());
+
+ return true;
+ }
+
+ /**
+ * Return a list of all Datadog entries retrieved from the mock Datadog server.
+ *
+ * @return All Datadog entries on the server.
+ */
+ public synchronized List<DatadogLogEntry> getEntries() {
+ MockServerClient serviceClient = clientFactory.getServiceClient(getHost(), getPort());
+ LOG.info("Reading events from Datadog");
+
+ List<DatadogLogEntry> results = new ArrayList<>();
+ for (HttpRequest request : serviceClient.retrieveRecordedRequests(request())) {
+ String requestBody = request.getBodyAsString();
+
+ List<DatadogLogEntry> events = new ArrayList<>();
+ try {
+ // Parse as a json array
+ JsonArray jsonArray = GSON.fromJson(requestBody, JsonArray.class);
+
+ // For each element, create a DatadogLogEntry
+ for (JsonElement jsonElement : jsonArray) {
+ JsonObject jsonObject = jsonElement.getAsJsonObject();
+ DatadogLogEntry.Builder builder = DatadogLogEntry.newBuilder();
+ if (jsonObject.has("ddsource")) {
+ builder.withSource(jsonObject.get("ddsource").getAsString());
+ }
+ if (jsonObject.has("ddtags")) {
+ builder.withTags(jsonObject.get("ddtags").getAsString());
+ }
+ if (jsonObject.has("hostname")) {
+ builder.withHostname(jsonObject.get("hostname").getAsString());
+ }
+ if (jsonObject.has("service")) {
+ builder.withService(jsonObject.get("service").getAsString());
+ }
+ if (jsonObject.has("message")) {
+ builder.withMessage(jsonObject.get("message").getAsString());
+ }
+ events.add(builder.build());
+ }
+ } catch (Exception e) { // Catch broader exception
Review Comment:
Catching a generic `Exception` is generally discouraged as it can hide specific issues. It would be better to catch more specific exceptions that can be thrown from the `try` block, such as `com.google.gson.JsonParseException` or `IllegalStateException`. This will make error handling more robust and easier to debug.
```suggestion
} catch (com.google.gson.JsonParseException | IllegalStateException e) { // Catch specific exceptions
```
##########
it/datadog/src/test/java/org/apache/beam/it/datadog/DatadogResourceManagerTest.java:
##########
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.it.datadog;
+
+import static com.google.common.truth.Truth.assertThat;
+import static org.junit.Assert.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.io.IOException;
+import java.util.Collections;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.mockito.Answers;
+import org.mockito.Mock;
+import org.mockito.junit.MockitoJUnit;
+import org.mockito.junit.MockitoRule;
+import org.mockserver.client.ForwardChainExpectation;
+import org.mockserver.client.MockServerClient;
+import org.mockserver.model.HttpRequest;
+import org.testcontainers.containers.MockServerContainer;
+
+/** Unit tests for {@link com.google.cloud.teleport.it.datadog.DatadogResourceManager}. */
Review Comment:
The Javadoc link points to a class in a different package (`com.google.cloud.teleport.it.datadog.DatadogResourceManager`). Please update it to point to the correct class: `org.apache.beam.it.datadog.DatadogResourceManager`.
```suggestion
/** Unit tests for {@link org.apache.beam.it.datadog.DatadogResourceManager}. */
```
##########
it/datadog/src/test/java/org/apache/beam/it/datadog/DatadogResourceManagerUtilsTest.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.it.datadog;
+
+import static com.google.common.truth.Truth.assertThat;
+import static org.apache.beam.it.datadog.DatadogResourceManagerUtils.datadogEntryToMap;
+import static org.apache.beam.it.datadog.DatadogResourceManagerUtils.generateApiKey;
+
+import java.util.HashMap;
+import java.util.Map;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Unit tests for {@link com.google.cloud.teleport.it.datadog.DatadogResourceManager}. */
Review Comment:
The Javadoc here seems to be a copy-paste error. It refers to `DatadogResourceManager` from a different package. Please update it to describe this test class, which is for `DatadogResourceManagerUtils`.
```suggestion
/** Unit tests for {@link DatadogResourceManagerUtils}. */
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add datadog it support [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on PR #37517:
URL: https://github.com/apache/beam/pull/37517#issuecomment-3853986762
## Summary of Changes
Hello @derrickaw, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request introduces a dedicated integration test module for Datadog within the Apache Beam project. It provides a robust framework for simulating Datadog interactions, allowing developers to effectively test Beam pipelines that produce or consume data from Datadog. The new module includes a resource manager to control a mock Datadog server, data structures for log entries, and assertion utilities to validate test outcomes.
### Highlights
* **New Datadog Integration Test Module**: A new Gradle module `it:datadog` has been added to provide integration test support for Datadog, enabling testing of Beam pipelines interacting with Datadog services.
* **DatadogResourceManager**: Introduced `DatadogResourceManager` which leverages Testcontainers and MockServer to simulate a Datadog API endpoint, allowing for sending and retrieving log entries in a controlled test environment.
* **DatadogLogEntry and Utilities**: Added `DatadogLogEntry` as an AutoValue class to represent Datadog log data, along with `DatadogClientFactory` for HTTP and MockServer client creation, and `DatadogResourceManagerUtils` for helper functions like API key generation and log entry mapping.
* **Testing Utilities**: Included `DatadogLogEntriesCheck` for asserting the number of log entries received by the mock server and `DatadogAsserts` for Truth-based assertions on Datadog log entries in tests.
* **Comprehensive Test Coverage**: Both unit tests (`DatadogResourceManagerTest`, `DatadogResourceManagerUtilsTest`) and an integration test (`DatadogResourceManagerIT`) have been added to ensure the reliability and functionality of the new Datadog IT support.
🧠**New Feature in Public Preview:** You can now enable **Memory** to help **Gemini Code Assist** learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. **Click [here](https://codeassist.google/code-review/login) to enable Memory in your admin console.**
<details>
<summary><b>Changelog</b></summary>
* **it/datadog/build.gradle**
* Added new Gradle build file for the `it:datadog` module.
* Configured dependencies including `it:common`, `it:testcontainers`, `it:truthmatchers`, `it:conditions`, `org.testcontainers:mockserver`, `org.mock-server:mockserver-client-java`, `org.apache.httpcomponents:httpclient`, and `com.google.code.gson`.
* **it/datadog/src/main/java/org/apache/beam/it/datadog/DatadogClientFactory.java**
* Added `DatadogClientFactory` class to provide `CloseableHttpClient` for sending HTTP messages and `MockServerClient` for interacting with the mock server.
* **it/datadog/src/main/java/org/apache/beam/it/datadog/DatadogLogEntry.java**
* Added `DatadogLogEntry` as an AutoValue class to model Datadog log entries with fields like `ddsource`, `ddtags`, `hostname`, `service`, and `message`.
* **it/datadog/src/main/java/org/apache/beam/it/datadog/DatadogResourceManager.java**
* Added `DatadogResourceManager` to manage a mock Datadog server using Testcontainers.
* Implemented methods to get HTTP and API endpoints, retrieve the API key, send single or multiple `DatadogLogEntry` events, and retrieve recorded log entries.
* Configured the mock server to accept POST requests to `/api/v2/logs` with specific headers and JSON schema.
* **it/datadog/src/main/java/org/apache/beam/it/datadog/DatadogResourceManagerException.java**
* Added `DatadogResourceManagerException` as a custom runtime exception for errors within the Datadog resource manager.
* **it/datadog/src/main/java/org/apache/beam/it/datadog/DatadogResourceManagerUtils.java**
* Added `DatadogResourceManagerUtils` with static methods to convert `DatadogLogEntry` objects to a `Map<String, Object>` and to generate a secure Datadog API key.
* **it/datadog/src/main/java/org/apache/beam/it/datadog/conditions/DatadogLogEntriesCheck.java**
* Added `DatadogLogEntriesCheck` as a `ConditionCheck` implementation to verify if the mock Datadog server has received a specified minimum (and optional maximum) number of log entries.
* **it/datadog/src/main/java/org/apache/beam/it/datadog/conditions/package-info.java**
* Added `package-info.java` for the `conditions` subpackage.
* **it/datadog/src/main/java/org/apache/beam/it/datadog/matchers/DatadogAsserts.java**
* Added `DatadogAsserts` utility class with static methods to convert `DatadogLogEntry` collections to a list of maps and to create `RecordsSubject` for Truth assertions.
* **it/datadog/src/main/java/org/apache/beam/it/datadog/matchers/package-info.java**
* Added `package-info.java` for the `matchers` subpackage.
* **it/datadog/src/main/java/org/apache/beam/it/datadog/package-info.java**
* Added `package-info.java` for the main `datadog` package.
* **it/datadog/src/test/java/org/apache/beam/it/datadog/DatadogResourceManagerIT.java**
* Added integration tests for `DatadogResourceManager` to verify end-to-end functionality of sending and retrieving log entries.
* **it/datadog/src/test/java/org/apache/beam/it/datadog/DatadogResourceManagerTest.java**
* Added unit tests for `DatadogResourceManager`, covering client factory interactions, HTTP endpoint generation, API key handling, and error scenarios during event sending and retrieval.
* **it/datadog/src/test/java/org/apache/beam/it/datadog/DatadogResourceManagerUtilsTest.java**
* Added unit tests for `DatadogResourceManagerUtils`, specifically for `datadogEntryToMap` conversion and `generateApiKey` functionality.
* **settings.gradle.kts**
* Added `include(":it:datadog")` to the `settings.gradle.kts` file to include the new Datadog integration test module in the project build.
</details>
<details>
<summary><b>Activity</b></summary>
* The pull request was opened by derrickaw with the title "add datadog it support" and a description outlining the purpose of adding Datadog integration test support.
* The description includes a standard checklist for contributions, such as mentioning issues, updating `CHANGES.md`, and filing an ICLA if applicable.
* Links to build health status and CI information are provided in the PR description.
</details>
<details>
<summary><b>Using Gemini Code Assist</b></summary>
<br>
The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips.
<b>Invoking Gemini</b>
You can request assistance from Gemini at any point by creating a comment using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands on the current page.
Feature | Command | Description
--- | --- | ---
Code Review | `/gemini review` | Performs a code review for the current pull request in its current state.
Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state.
Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help | `/gemini help` | Displays a list of available commands.
<b>Customization</b>
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github).
<b>Limitations & Feedback</b>
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up [here](https://google.qualtrics.com/jfe/form/SV_2cyuGuTWsEw84yG).
<b>You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the [Gemini Code Assist IDE Extension](https://cloud.google.com/products/gemini/code-assist).</b>
</details>
[^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add datadog it support [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on code in PR #37517:
URL: https://github.com/apache/beam/pull/37517#discussion_r2788953373
##########
it/datadog/build.gradle:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins { id 'org.apache.beam.module' }
+applyJavaNature(
+ automaticModuleName: 'org.apache.beam.it.datadog',
+)
+
+description = "Apache Beam :: IT :: Datadog"
+ext.summary = "Integration test utilities for Datadog."
+
+dependencies {
+ implementation project(path: ":it:common")
+ implementation project(path: ":it:testcontainers")
+ implementation project(path: ":it:truthmatchers")
+ implementation project(path: ":it:conditions")
+ implementation "org.testcontainers:mockserver:1.19.7"
+ implementation "org.testcontainers:testcontainers:1.19.7"
+ implementation "org.mock-server:mockserver-client-java:5.10.0"
+ implementation "org.mock-server:mockserver-core:5.10.0"
+ implementation library.java.json_org
+ implementation "org.apache.httpcomponents:httpclient:4.5.13"
+ implementation "org.apache.httpcomponents:httpcore:4.4.14"
+ implementation library.java.google_code_gson
+ implementation library.java.vendored_guava_32_1_2_jre
+ implementation "com.google.guava:guava:33.1.0-jre"
Review Comment:
removed, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add datadog it support [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #37517:
URL: https://github.com/apache/beam/pull/37517#discussion_r2788638997
##########
it/datadog/src/main/java/org/apache/beam/it/datadog/DatadogResourceManager.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.it.datadog;
+
+import static org.apache.beam.it.datadog.DatadogResourceManagerUtils.datadogEntryToMap;
+import static org.apache.beam.it.datadog.DatadogResourceManagerUtils.generateApiKey;
+import static org.mockserver.model.HttpRequest.request;
+import static org.mockserver.model.HttpResponse.response;
+import static org.mockserver.model.JsonSchemaBody.jsonSchema;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.gson.Gson;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import java.io.IOException;
+import java.io.UnsupportedEncodingException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+import org.apache.beam.it.common.ResourceManager;
+import org.apache.beam.it.testcontainers.TestContainerResourceManager;
+import org.apache.beam.it.testcontainers.TestContainerResourceManagerException;
+import org.apache.http.client.entity.GzipCompressingEntity;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.json.JSONObject;
+import org.mockserver.client.MockServerClient;
+import org.mockserver.model.HttpRequest;
+import org.mockserver.model.MediaType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.MockServerContainer;
+import org.testcontainers.utility.DockerImageName;
+
+/**
+ * Client for managing Datadog resources.
+ *
+ * <p>The class supports one mock Datadog server instance.
+ *
+ * <p>The class is thread-safe.
+ *
+ * <p>Note: The MockServer TestContainer will only run on M1 Mac's if the Docker version is >=
+ * 4.16.0 and the "Use Rosetta for x86/amd64 emulation on Apple Silicon" setting is enabled.
+ */
+public class DatadogResourceManager extends TestContainerResourceManager<MockServerContainer>
+ implements ResourceManager {
+
+ private static final Logger LOG = LoggerFactory.getLogger(DatadogResourceManager.class);
+ private static final String DEFAULT_MOCKSERVER_CONTAINER_NAME = "mockserver/mockserver";
+
+ // A list of available Mockserver Docker image tags can be found at
+ // https://hub.docker.com/r/mockserver/mockserver/tags
+ private static final String DEFAULT_MOCKSERVER_CONTAINER_TAG;
+
+ static {
+ DEFAULT_MOCKSERVER_CONTAINER_TAG =
+ Optional.ofNullable(MockServerClient.class.getPackage())
+ .map(Package::getImplementationVersion)
+ .orElseThrow(
+ () ->
+ new TestContainerResourceManagerException(
+ "Could not determine Mockserver container version."));
+ }
+
+ // See: https://docs.datadoghq.com/api/latest/logs/#send-logs
+ private static final String SEND_LOGS_JSON_SCHEMA =
+ "{"
+ + " \"$schema\": \"http://json-schema.org/draft-04/schema#\","
+ + " \"type\": \"array\","
+ + " \"items\": ["
+ + " {"
+ + " \"type\": \"object\","
+ + " \"properties\": {"
+ + " \"ddsource\": {"
+ + " \"type\": \"string\""
+ + " },"
+ + " \"ddtags\": {"
+ + " \"type\": \"string\""
+ + " },"
+ + " \"hostname\": {"
+ + " \"type\": \"string\""
+ + " },"
+ + " \"message\": {"
+ + " \"type\": \"string\""
+ + " },"
+ + " \"service\": {"
+ + " \"type\": \"string\""
+ + " }"
+ + " },"
+ + " \"required\": ["
+ + " \"message\""
+ + " ]"
+ + " }"
+ + " ]"
+ + "}";
+
+ private static final Gson GSON = new Gson();
+
+ private final String apiKey;
+ private final DatadogClientFactory clientFactory;
+
+ private DatadogResourceManager(DatadogResourceManager.Builder builder) {
+ this(
+ new DatadogClientFactory(),
+ new MockServerContainer(
+ DockerImageName.parse(builder.containerImageName).withTag(builder.containerImageTag)),
+ builder);
+ }
+
+ @VisibleForTesting
+ DatadogResourceManager(
+ DatadogClientFactory clientFactory,
+ MockServerContainer container,
+ DatadogResourceManager.Builder builder) {
+ super(container, builder);
+
+ this.apiKey = builder.apiKey;
+ this.clientFactory = clientFactory;
+ }
+
+ public static DatadogResourceManager.Builder builder(String testId) {
+ return new DatadogResourceManager.Builder(testId);
+ }
+
+ /** Returns the port to connect to the mock Datadog server. */
+ public int getPort() {
+ return super.getPort(MockServerContainer.PORT);
+ }
+
+ /**
+ * Returns the HTTP endpoint that this mock Datadog server is configured to listen on.
+ *
+ * @return the HTTP endpoint.
+ */
+ public String getHttpEndpoint() {
+ return String.format("http://%s:%d", getHost(), getPort());
+ }
+
+ /**
+ * Returns the API endpoint that this mock Datadog server is configured to receive events at.
+ *
+ * <p>This will be the HTTP endpoint concatenated with <code>'/api/v2/logs'</code>.
+ *
+ * @return the API endpoint.
+ */
+ public String getApiEndpoint() {
+ return getHttpEndpoint() + "/api/v2/logs";
+ }
+
+ /**
+ * Returns the Datadog API key used to connect to this mock Datadog server.
+ *
+ * @return the API key.
+ */
+ public String getApiKey() {
+ return apiKey;
+ }
+
+ /**
+ * Helper method for converting the given DatadogLogEntry into JSON format.
+ *
+ * @param event The DatadogLogEntry to parse.
+ * @return JSON String.
+ */
+ private static String datadogEventToJson(DatadogLogEntry event) {
+ return new JSONObject(datadogEntryToMap(event)).toString();
+ }
+
+ /**
+ * Sends the given HTTP event to the mock Datadog server.
+ *
+ * @param event The {@link DatadogLogEntry} to send to the API.
+ * @return True, if the request was successful.
+ */
+ public synchronized boolean sendHttpEvent(DatadogLogEntry event) {
+ return sendHttpEvents(Collections.singletonList(event));
+ }
+
+ /**
+ * Sends the given HTTP events to the mock Datadog server.
+ *
+ * @param events The {@link DatadogLogEntry}s to send to the API.
+ * @return True, if the request was successful.
+ */
+ public synchronized boolean sendHttpEvents(Collection<DatadogLogEntry> events) {
+
+ LOG.info("Attempting to send {} events to {}.", events.size(), getApiEndpoint());
+
+ // Construct base API request
+ HttpPost httppost = new HttpPost(getApiEndpoint());
+ httppost.addHeader("Content-Encoding", "gzip");
+ httppost.addHeader("Content-Type", "application/json");
+ httppost.addHeader("dd-api-key", apiKey);
+
+ String eventsData = GSON.toJson(events);
+
+ try (CloseableHttpClient httpClient = clientFactory.getHttpClient()) {
+ // Set request data
+ try {
+ httppost.setEntity(new GzipCompressingEntity(new StringEntity(eventsData)));
+ } catch (UnsupportedEncodingException e) {
+ throw new DatadogResourceManagerException(
+ "Error setting HTTP message data to " + eventsData, e);
+ }
+
+ // Send request
+ try (CloseableHttpResponse response = httpClient.execute(httppost)) {
+ // Check error code
+ int code = response.getStatusLine().getStatusCode();
+ if (code != 202) {
+ throw new DatadogResourceManagerException(
+ "Received http error code " + code + " sending event.");
+ }
+ } catch (Exception e) {
+ throw new DatadogResourceManagerException("Error sending event.", e);
+ }
+ } catch (IOException e) {
+ throw new DatadogResourceManagerException("Error with HTTP client.", e);
+ }
+
+ LOG.info("Successfully sent {} events.", events.size());
+
+ return true;
+ }
+
+ /**
+ * Return a list of all Datadog entries retrieved from the mock Datadog server.
+ *
+ * @return All Datadog entries on the server.
+ */
+ public synchronized List<DatadogLogEntry> getEntries() {
+ MockServerClient serviceClient = clientFactory.getServiceClient(getHost(), getPort());
+ LOG.info("Reading events from Datadog");
+
+ List<DatadogLogEntry> results = new ArrayList<>();
+ for (HttpRequest request : serviceClient.retrieveRecordedRequests(request())) {
+ String requestBody = request.getBodyAsString();
+
+ List<DatadogLogEntry> events = new ArrayList<>();
+ try {
+ // Parse as a json array
+ JsonArray jsonArray = GSON.fromJson(requestBody, JsonArray.class);
+
+ // For each element, create a DatadogLogEntry
+ for (JsonElement jsonElement : jsonArray) {
+ JsonObject jsonObject = jsonElement.getAsJsonObject();
+ DatadogLogEntry.Builder builder = DatadogLogEntry.newBuilder();
+ if (jsonObject.has("ddsource")) {
+ builder.withSource(jsonObject.get("ddsource").getAsString());
+ }
+ if (jsonObject.has("ddtags")) {
+ builder.withTags(jsonObject.get("ddtags").getAsString());
+ }
+ if (jsonObject.has("hostname")) {
+ builder.withHostname(jsonObject.get("hostname").getAsString());
+ }
+ if (jsonObject.has("service")) {
+ builder.withService(jsonObject.get("service").getAsString());
+ }
+ if (jsonObject.has("message")) {
+ builder.withMessage(jsonObject.get("message").getAsString());
+ }
+ events.add(builder.build());
+ }
+ } catch (Exception e) { // Catch broader exception
Review Comment:
This is probably fine since we're just rethrowing in a test method anyways
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org