You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "JiriOndrusek (via GitHub)" <gi...@apache.org> on 2023/08/15 07:15:17 UTC
[GitHub] [camel-quarkus] JiriOndrusek opened a new pull request, #5184: Expand Splunk test coverage
JiriOndrusek opened a new pull request, #5184:
URL: https://github.com/apache/camel-quarkus/pull/5184
fixes https://github.com/apache/camel-quarkus/issues/5154
Added test for raw data, fixed some issues in tests (to nt skip some scenarios) and added missing reflection registration (required with the full coverage)
<!-- Uncomment and fill this section if your PR is not trivial
[ ] An issue should be filed for the change unless this is a trivial change (fixing a typo or similar). One issue should ideally be fixed by not more than one commit and the other way round, each commit should fix just one issue, without pulling in other changes.
[ ] Each commit in the pull request should have a meaningful and properly spelled subject line and body. Copying the title of the associated issue is typically enough. Please include the issue number in the commit message prefixed by #.
[ ] The pull request description should explain what the pull request does, how, and why. If the info is available in the associated issue or some other external document, a link is enough.
[ ] Phrases like Fix #<issueNumber> or Fixes #<issueNumber> will auto-close the named issue upon merging the pull request. Using them is typically a good idea.
[ ] Please run mvn process-resources -Pformat (and amend the changes if necessary) before sending the pull request.
[ ] Contributor guide is your good friend: https://camel.apache.org/camel-quarkus/latest/contributor-guide.html
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] zhfeng commented on pull request #5184: Expand Splunk test coverage
Posted by "zhfeng (via GitHub)" <gi...@apache.org>.
zhfeng commented on PR #5184:
URL: https://github.com/apache/camel-quarkus/pull/5184#issuecomment-1681499747
Hi @JiriOndrusek
The ibmmq client issue should be fixed and can you rebase the PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on pull request #5184: Expand Splunk test coverage
Posted by "JiriOndrusek (via GitHub)" <gi...@apache.org>.
JiriOndrusek commented on PR #5184:
URL: https://github.com/apache/camel-quarkus/pull/5184#issuecomment-1678831910
Failure is not related `Camel Quarkus :: Integration Tests :: JMS IBM MQ Client FAILURE [02:27 min]`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on pull request #5184: Expand Splunk test coverage
Posted by "JiriOndrusek (via GitHub)" <gi...@apache.org>.
JiriOndrusek commented on PR #5184:
URL: https://github.com/apache/camel-quarkus/pull/5184#issuecomment-1678590476
@jamesnetherton Issues are fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] jamesnetherton commented on a diff in pull request #5184: Expand Splunk test coverage
Posted by "jamesnetherton (via GitHub)" <gi...@apache.org>.
jamesnetherton commented on code in PR #5184:
URL: https://github.com/apache/camel-quarkus/pull/5184#discussion_r1294306893
##########
integration-tests/splunk/src/test/java/org/apache/camel/quarkus/component/splunk/it/SplunkTest.java:
##########
@@ -16,138 +16,153 @@
*/
package org.apache.camel.quarkus.component.splunk.it;
-import java.util.List;
import java.util.Map;
import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
-import java.util.stream.Collectors;
+import java.util.function.Consumer;
import io.quarkus.test.common.QuarkusTestResource;
import io.quarkus.test.junit.QuarkusTest;
import io.restassured.RestAssured;
-import io.restassured.common.mapper.TypeRef;
import io.restassured.http.ContentType;
+import org.apache.camel.component.splunk.ProducerType;
import org.apache.camel.util.CollectionHelper;
import org.eclipse.microprofile.config.ConfigProvider;
-import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import org.testcontainers.shaded.org.apache.commons.lang3.RandomStringUtils;
+import org.testcontainers.shaded.org.awaitility.Awaitility;
import static org.hamcrest.Matchers.anyOf;
-import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.is;
@QuarkusTest
@QuarkusTestResource(SplunkTestResource.class)
class SplunkTest {
@Test
- public void testWriteTcpAndReadNormal() {
- write("_normal", SplunkTestResource.TEST_INDEX, "tcp");
-
- List<Map<String, String>> result = RestAssured.given()
- .contentType(ContentType.TEXT)
- .body(String.format(
- "search index=%s sourcetype=%s | rex field=_raw \"Name: (?<name>.*) From: (?<from>.*)\"",
- SplunkTestResource.TEST_INDEX, SplunkResource.SOURCE_TYPE))
- .post("/splunk/normal")
- .then()
- .statusCode(200)
- .extract().as(new TypeRef<>() {
- });
+ public void testNormalSearchWithSubmitWithRawData() {
+ String suffix = "_normalSearchOfSubmit";
- Assertions.assertEquals(3, result.size());
- Assertions.assertEquals("Irma_normal", result.get(0).get("name"));
- Assertions.assertEquals("Earth\"", result.get(0).get("from"));
- Assertions.assertEquals("Leonard_normal", result.get(1).get("name"));
- Assertions.assertEquals("Earth 2.0\"", result.get(1).get("from"));
- Assertions.assertEquals("Sheldon_normal", result.get(2).get("name"));
- Assertions.assertEquals("Alpha Centauri\"", result.get(2).get("from"));
- }
+ write(suffix, ProducerType.SUBMIT, 0, true);
- @Test
- public void testWriteSubmitAndReadRealtime() throws InterruptedException, ExecutionException {
+ Awaitility.await().pollInterval(1000, TimeUnit.MILLISECONDS).atMost(60, TimeUnit.SECONDS).until(
+ () -> {
- RestAssured.given()
- .body(String.format(
- "search index=%s sourcetype=%s | rex field=_raw \"Name: (?<name>.*) From: (?<from>.*)\"",
- SplunkTestResource.TEST_INDEX, SplunkResource.SOURCE_TYPE))
- .post("/splunk/startRealtimePolling");
-
- //wait some time to start polling
- TimeUnit.SECONDS.sleep(3);
- write("_realtime1", SplunkTestResource.TEST_INDEX, "submit");
- TimeUnit.SECONDS.sleep(1);
- write("_realtime2", SplunkTestResource.TEST_INDEX, "submit");
- TimeUnit.SECONDS.sleep(1);
- write("_realtime3", SplunkTestResource.TEST_INDEX, "submit");
- //wait some time to gather the pulls from splunk server
- TimeUnit.SECONDS.sleep(3);
- //there should be some data from realtime search in direct (concrete values depends on the speed of writing into index)
- //test is asserting that there are some
- RestAssured.get("/splunk/directRealtimePolling")
- .then()
- .statusCode(200)
- .body(containsString("_realtime"));
+ String result = RestAssured.given()
+ .contentType(ContentType.TEXT)
+ .post("/splunk/results/normalSearch")
+ .then()
+ .statusCode(200)
+ .extract().asString();
+
+ return result.contains("Name: Sheldon" + suffix)
+ && result.contains("Name: Leonard" + suffix)
+ && result.contains("Name: Irma" + suffix);
+ });
}
@Test
- public void testWriteStreamAndReadSaved() throws InterruptedException {
- int defaultPort = RestAssured.port;
- String defaultUri = RestAssured.baseURI;
-
+ public void testSavedSearchWithTcp() throws InterruptedException {
+ String suffix = "_SavedSearchOfTcp";
//create saved search
RestAssured.given()
.baseUri("http://localhost")
.port(ConfigProvider.getConfig().getValue(SplunkResource.PARAM_REMOTE_PORT, Integer.class))
.contentType(ContentType.JSON)
- .param("name", SplunkTestResource.SAVED_SEARCH_NAME)
+ .param("name", SplunkResource.SAVED_SEARCH_NAME)
.param("disabled", "0")
- .param("description", "descritionText")
+ .param("description", "descriptionText")
.param("search",
- "index=" + SplunkTestResource.TEST_INDEX + " sourcetype=" + SplunkResource.SOURCE_TYPE)
+ "sourcetype=\"TCP\" | rex field=_raw \"Name: (?<name>.*) From: (?<from>.*)\"")
.post("/services/saved/searches")
.then()
.statusCode(anyOf(is(201), is(409)));
- write("_s", SplunkTestResource.TEST_INDEX, "stream");
- RestAssured.given()
- .contentType(ContentType.TEXT)
- .body(SplunkTestResource.SAVED_SEARCH_NAME)
- .post("/splunk/savedSearch")
- .then()
- .statusCode(200)
- .body(containsString("Name: Sheldon_s"))
- .body(containsString("Name: Leonard_s"))
- .body(containsString("Name: Irma_s"));
- }
-
- private void write(String suffix, String index, String endpoint) {
- write(CollectionHelper.mapOf("entity", "Name: Sheldon" + suffix + " From: Alpha Centauri"), "submit",
- index);
- write(CollectionHelper.mapOf("entity", "Name: Leonard" + suffix + " From: Earth 2.0"), "submit",
- index);
- write(CollectionHelper.mapOf("entity", "Name: Irma" + suffix + " From: Earth"), "submit", index);
+ //write data via tcp
+ write(suffix, ProducerType.TCP, 0, false);
+
+ //there might by delay in receiving the data
+ Awaitility.await().pollInterval(1000, TimeUnit.MILLISECONDS).atMost(60, TimeUnit.SECONDS).until(
+ () -> {
+ String result = RestAssured.given()
+ .contentType(ContentType.TEXT)
+ .post("/splunk/results/savedSearch")
+ .then()
+ .statusCode(200)
+ .extract().asString();
+
+ return result.contains("Name: Sheldon" + suffix)
+ && result.contains("Name: Leonard" + suffix)
+ && result.contains("Name: Irma" + suffix);
+ });
}
- private void write(Map<String, String> data, String endpoint, String index) {
+ @Test
+ public void testStreamForRealtime() throws InterruptedException, ExecutionException {
+ String suffix = "_RealtimeSearchOfStream";
+ //there is a buffer for stream writing, therefore about 1MB of data has to be written into Splunk
+
+ //data are written in separated thread
+ ExecutorService executor = Executors.newSingleThreadExecutor();
+ //execute component server to wait for the result
+ Future futureResult = executor.submit(
+ () -> {
+ for (int i = 0; i < 5000; i++) {
+ write(suffix + i, ProducerType.STREAM, 100, false);
+ }
+ });
- String expectedResult = expectedResult(data);
+ try {
+ Awaitility.await().pollInterval(1000, TimeUnit.MILLISECONDS).atMost(60, TimeUnit.SECONDS).until(
+ () -> {
+
+ String result = RestAssured.given()
+ .contentType(ContentType.TEXT)
+ .post("/splunk/results/realtimeSearch")
+ .then()
+ .statusCode(200)
+ .extract().asString();
+
+ return result.contains("Name: Sheldon" + suffix)
+ && result.contains("Name: Leonard" + suffix)
+ && result.contains("Name: Irma" + suffix);
+ });
+ } finally {
+ futureResult.cancel(true);
+ }
+ }
- RestAssured.given()
+ private void write(String suffix, ProducerType producerType, int lengthOfRandomString, boolean raw) {
+ Consumer<Map> write = data -> RestAssured.given()
.contentType(ContentType.JSON)
- .queryParam("index", index)
+ .queryParam("index", SplunkTestResource.TEST_INDEX)
.body(data)
- .post("/splunk/" + endpoint)
+ .post("/splunk/write/" + producerType.name())
.then()
- .statusCode(201)
- .body(containsString(expectedResult));
+ .statusCode(201);
+
+ Map data[] = new Map[3];
Review Comment:
Seems GitHub is not highlighting the bit of code I was referring to....
I mean the usage of `Map data[] = new Map[3];`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] jamesnetherton commented on a diff in pull request #5184: Expand Splunk test coverage
Posted by "jamesnetherton (via GitHub)" <gi...@apache.org>.
jamesnetherton commented on code in PR #5184:
URL: https://github.com/apache/camel-quarkus/pull/5184#discussion_r1294270620
##########
extensions/splunk/deployment/src/main/java/org/apache/camel/quarkus/component/splunk/deployment/SplunkProcessor.java:
##########
@@ -67,9 +67,10 @@ ReflectiveClassBuildItem registerForReflection(CombinedIndexBuildItem combinedIn
@BuildStep
List<ReflectiveClassBuildItem> reflectiveClasses() {
- return Arrays.asList(ReflectiveClassBuildItem.builder("com.splunk.Index").build(),
- ReflectiveClassBuildItem.builder("com.splunk.SavedSearch").build(),
- ReflectiveClassBuildItem.builder("com.splunk.Service").build());
+ return Arrays.asList(ReflectiveClassBuildItem.builder("com.splunk.Index").constructors().build(),
Review Comment:
If these are public classes, it's better to use `Class.getName()` instead of hard coding it.
##########
integration-tests/splunk/src/test/java/org/apache/camel/quarkus/component/splunk/it/SplunkTest.java:
##########
@@ -16,138 +16,153 @@
*/
package org.apache.camel.quarkus.component.splunk.it;
-import java.util.List;
import java.util.Map;
import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
-import java.util.stream.Collectors;
+import java.util.function.Consumer;
import io.quarkus.test.common.QuarkusTestResource;
import io.quarkus.test.junit.QuarkusTest;
import io.restassured.RestAssured;
-import io.restassured.common.mapper.TypeRef;
import io.restassured.http.ContentType;
+import org.apache.camel.component.splunk.ProducerType;
import org.apache.camel.util.CollectionHelper;
import org.eclipse.microprofile.config.ConfigProvider;
-import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import org.testcontainers.shaded.org.apache.commons.lang3.RandomStringUtils;
+import org.testcontainers.shaded.org.awaitility.Awaitility;
import static org.hamcrest.Matchers.anyOf;
-import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.is;
@QuarkusTest
@QuarkusTestResource(SplunkTestResource.class)
class SplunkTest {
@Test
- public void testWriteTcpAndReadNormal() {
- write("_normal", SplunkTestResource.TEST_INDEX, "tcp");
-
- List<Map<String, String>> result = RestAssured.given()
- .contentType(ContentType.TEXT)
- .body(String.format(
- "search index=%s sourcetype=%s | rex field=_raw \"Name: (?<name>.*) From: (?<from>.*)\"",
- SplunkTestResource.TEST_INDEX, SplunkResource.SOURCE_TYPE))
- .post("/splunk/normal")
- .then()
- .statusCode(200)
- .extract().as(new TypeRef<>() {
- });
+ public void testNormalSearchWithSubmitWithRawData() {
+ String suffix = "_normalSearchOfSubmit";
- Assertions.assertEquals(3, result.size());
- Assertions.assertEquals("Irma_normal", result.get(0).get("name"));
- Assertions.assertEquals("Earth\"", result.get(0).get("from"));
- Assertions.assertEquals("Leonard_normal", result.get(1).get("name"));
- Assertions.assertEquals("Earth 2.0\"", result.get(1).get("from"));
- Assertions.assertEquals("Sheldon_normal", result.get(2).get("name"));
- Assertions.assertEquals("Alpha Centauri\"", result.get(2).get("from"));
- }
+ write(suffix, ProducerType.SUBMIT, 0, true);
- @Test
- public void testWriteSubmitAndReadRealtime() throws InterruptedException, ExecutionException {
+ Awaitility.await().pollInterval(1000, TimeUnit.MILLISECONDS).atMost(60, TimeUnit.SECONDS).until(
+ () -> {
- RestAssured.given()
- .body(String.format(
- "search index=%s sourcetype=%s | rex field=_raw \"Name: (?<name>.*) From: (?<from>.*)\"",
- SplunkTestResource.TEST_INDEX, SplunkResource.SOURCE_TYPE))
- .post("/splunk/startRealtimePolling");
-
- //wait some time to start polling
- TimeUnit.SECONDS.sleep(3);
- write("_realtime1", SplunkTestResource.TEST_INDEX, "submit");
- TimeUnit.SECONDS.sleep(1);
- write("_realtime2", SplunkTestResource.TEST_INDEX, "submit");
- TimeUnit.SECONDS.sleep(1);
- write("_realtime3", SplunkTestResource.TEST_INDEX, "submit");
- //wait some time to gather the pulls from splunk server
- TimeUnit.SECONDS.sleep(3);
- //there should be some data from realtime search in direct (concrete values depends on the speed of writing into index)
- //test is asserting that there are some
- RestAssured.get("/splunk/directRealtimePolling")
- .then()
- .statusCode(200)
- .body(containsString("_realtime"));
+ String result = RestAssured.given()
+ .contentType(ContentType.TEXT)
+ .post("/splunk/results/normalSearch")
+ .then()
+ .statusCode(200)
+ .extract().asString();
+
+ return result.contains("Name: Sheldon" + suffix)
+ && result.contains("Name: Leonard" + suffix)
+ && result.contains("Name: Irma" + suffix);
+ });
}
@Test
- public void testWriteStreamAndReadSaved() throws InterruptedException {
- int defaultPort = RestAssured.port;
- String defaultUri = RestAssured.baseURI;
-
+ public void testSavedSearchWithTcp() throws InterruptedException {
+ String suffix = "_SavedSearchOfTcp";
//create saved search
RestAssured.given()
.baseUri("http://localhost")
.port(ConfigProvider.getConfig().getValue(SplunkResource.PARAM_REMOTE_PORT, Integer.class))
.contentType(ContentType.JSON)
- .param("name", SplunkTestResource.SAVED_SEARCH_NAME)
+ .param("name", SplunkResource.SAVED_SEARCH_NAME)
.param("disabled", "0")
- .param("description", "descritionText")
+ .param("description", "descriptionText")
.param("search",
- "index=" + SplunkTestResource.TEST_INDEX + " sourcetype=" + SplunkResource.SOURCE_TYPE)
+ "sourcetype=\"TCP\" | rex field=_raw \"Name: (?<name>.*) From: (?<from>.*)\"")
.post("/services/saved/searches")
.then()
.statusCode(anyOf(is(201), is(409)));
- write("_s", SplunkTestResource.TEST_INDEX, "stream");
- RestAssured.given()
- .contentType(ContentType.TEXT)
- .body(SplunkTestResource.SAVED_SEARCH_NAME)
- .post("/splunk/savedSearch")
- .then()
- .statusCode(200)
- .body(containsString("Name: Sheldon_s"))
- .body(containsString("Name: Leonard_s"))
- .body(containsString("Name: Irma_s"));
- }
-
- private void write(String suffix, String index, String endpoint) {
- write(CollectionHelper.mapOf("entity", "Name: Sheldon" + suffix + " From: Alpha Centauri"), "submit",
- index);
- write(CollectionHelper.mapOf("entity", "Name: Leonard" + suffix + " From: Earth 2.0"), "submit",
- index);
- write(CollectionHelper.mapOf("entity", "Name: Irma" + suffix + " From: Earth"), "submit", index);
+ //write data via tcp
+ write(suffix, ProducerType.TCP, 0, false);
+
+ //there might by delay in receiving the data
+ Awaitility.await().pollInterval(1000, TimeUnit.MILLISECONDS).atMost(60, TimeUnit.SECONDS).until(
+ () -> {
+ String result = RestAssured.given()
+ .contentType(ContentType.TEXT)
+ .post("/splunk/results/savedSearch")
+ .then()
+ .statusCode(200)
+ .extract().asString();
+
+ return result.contains("Name: Sheldon" + suffix)
+ && result.contains("Name: Leonard" + suffix)
+ && result.contains("Name: Irma" + suffix);
+ });
}
- private void write(Map<String, String> data, String endpoint, String index) {
+ @Test
+ public void testStreamForRealtime() throws InterruptedException, ExecutionException {
+ String suffix = "_RealtimeSearchOfStream";
+ //there is a buffer for stream writing, therefore about 1MB of data has to be written into Splunk
+
+ //data are written in separated thread
+ ExecutorService executor = Executors.newSingleThreadExecutor();
+ //execute component server to wait for the result
+ Future futureResult = executor.submit(
+ () -> {
+ for (int i = 0; i < 5000; i++) {
+ write(suffix + i, ProducerType.STREAM, 100, false);
+ }
+ });
- String expectedResult = expectedResult(data);
+ try {
+ Awaitility.await().pollInterval(1000, TimeUnit.MILLISECONDS).atMost(60, TimeUnit.SECONDS).until(
+ () -> {
+
+ String result = RestAssured.given()
+ .contentType(ContentType.TEXT)
+ .post("/splunk/results/realtimeSearch")
+ .then()
+ .statusCode(200)
+ .extract().asString();
+
+ return result.contains("Name: Sheldon" + suffix)
+ && result.contains("Name: Leonard" + suffix)
+ && result.contains("Name: Irma" + suffix);
+ });
+ } finally {
+ futureResult.cancel(true);
+ }
+ }
- RestAssured.given()
+ private void write(String suffix, ProducerType producerType, int lengthOfRandomString, boolean raw) {
+ Consumer<Map> write = data -> RestAssured.given()
.contentType(ContentType.JSON)
- .queryParam("index", index)
+ .queryParam("index", SplunkTestResource.TEST_INDEX)
.body(data)
- .post("/splunk/" + endpoint)
+ .post("/splunk/write/" + producerType.name())
.then()
- .statusCode(201)
- .body(containsString(expectedResult));
+ .statusCode(201);
+
+ Map data[] = new Map[3];
Review Comment:
Nitpick - I assume this produces compiler warnings about usage of raw types? You can probably build the same data structure with `List.of` / `Map.of`.
##########
integration-tests/splunk/README.adoc:
##########
@@ -0,0 +1,3 @@
+Splunk UI is helpful for testing.
Review Comment:
Can we expand on this a bit. Where do I get the UI from? Docker container? What's the command to get it running.
##########
integration-tests/splunk/src/main/java/org/apache/camel/quarkus/component/splunk/it/SplunkResource.java:
##########
@@ -47,10 +46,10 @@
@ApplicationScoped
public class SplunkResource {
+ public static String SAVED_SEARCH_NAME = "savedSearchForTest";
Review Comment:
```suggestion
public static final String SAVED_SEARCH_NAME = "savedSearchForTest";
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on a diff in pull request #5184: Expand Splunk test coverage
Posted by "JiriOndrusek (via GitHub)" <gi...@apache.org>.
JiriOndrusek commented on code in PR #5184:
URL: https://github.com/apache/camel-quarkus/pull/5184#discussion_r1294289707
##########
integration-tests/splunk/src/main/java/org/apache/camel/quarkus/component/splunk/it/SplunkResource.java:
##########
@@ -47,10 +46,10 @@
@ApplicationScoped
public class SplunkResource {
+ public static String SAVED_SEARCH_NAME = "savedSearchForTest";
Review Comment:
It should be there, I missed it/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on a diff in pull request #5184: Expand Splunk test coverage
Posted by "JiriOndrusek (via GitHub)" <gi...@apache.org>.
JiriOndrusek commented on code in PR #5184:
URL: https://github.com/apache/camel-quarkus/pull/5184#discussion_r1294304159
##########
integration-tests/splunk/src/test/java/org/apache/camel/quarkus/component/splunk/it/SplunkTest.java:
##########
@@ -16,138 +16,153 @@
*/
package org.apache.camel.quarkus.component.splunk.it;
-import java.util.List;
import java.util.Map;
import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
-import java.util.stream.Collectors;
+import java.util.function.Consumer;
import io.quarkus.test.common.QuarkusTestResource;
import io.quarkus.test.junit.QuarkusTest;
import io.restassured.RestAssured;
-import io.restassured.common.mapper.TypeRef;
import io.restassured.http.ContentType;
+import org.apache.camel.component.splunk.ProducerType;
import org.apache.camel.util.CollectionHelper;
import org.eclipse.microprofile.config.ConfigProvider;
-import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import org.testcontainers.shaded.org.apache.commons.lang3.RandomStringUtils;
+import org.testcontainers.shaded.org.awaitility.Awaitility;
import static org.hamcrest.Matchers.anyOf;
-import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.is;
@QuarkusTest
@QuarkusTestResource(SplunkTestResource.class)
class SplunkTest {
@Test
- public void testWriteTcpAndReadNormal() {
- write("_normal", SplunkTestResource.TEST_INDEX, "tcp");
-
- List<Map<String, String>> result = RestAssured.given()
- .contentType(ContentType.TEXT)
- .body(String.format(
- "search index=%s sourcetype=%s | rex field=_raw \"Name: (?<name>.*) From: (?<from>.*)\"",
- SplunkTestResource.TEST_INDEX, SplunkResource.SOURCE_TYPE))
- .post("/splunk/normal")
- .then()
- .statusCode(200)
- .extract().as(new TypeRef<>() {
- });
+ public void testNormalSearchWithSubmitWithRawData() {
+ String suffix = "_normalSearchOfSubmit";
- Assertions.assertEquals(3, result.size());
- Assertions.assertEquals("Irma_normal", result.get(0).get("name"));
- Assertions.assertEquals("Earth\"", result.get(0).get("from"));
- Assertions.assertEquals("Leonard_normal", result.get(1).get("name"));
- Assertions.assertEquals("Earth 2.0\"", result.get(1).get("from"));
- Assertions.assertEquals("Sheldon_normal", result.get(2).get("name"));
- Assertions.assertEquals("Alpha Centauri\"", result.get(2).get("from"));
- }
+ write(suffix, ProducerType.SUBMIT, 0, true);
- @Test
- public void testWriteSubmitAndReadRealtime() throws InterruptedException, ExecutionException {
+ Awaitility.await().pollInterval(1000, TimeUnit.MILLISECONDS).atMost(60, TimeUnit.SECONDS).until(
+ () -> {
- RestAssured.given()
- .body(String.format(
- "search index=%s sourcetype=%s | rex field=_raw \"Name: (?<name>.*) From: (?<from>.*)\"",
- SplunkTestResource.TEST_INDEX, SplunkResource.SOURCE_TYPE))
- .post("/splunk/startRealtimePolling");
-
- //wait some time to start polling
- TimeUnit.SECONDS.sleep(3);
- write("_realtime1", SplunkTestResource.TEST_INDEX, "submit");
- TimeUnit.SECONDS.sleep(1);
- write("_realtime2", SplunkTestResource.TEST_INDEX, "submit");
- TimeUnit.SECONDS.sleep(1);
- write("_realtime3", SplunkTestResource.TEST_INDEX, "submit");
- //wait some time to gather the pulls from splunk server
- TimeUnit.SECONDS.sleep(3);
- //there should be some data from realtime search in direct (concrete values depends on the speed of writing into index)
- //test is asserting that there are some
- RestAssured.get("/splunk/directRealtimePolling")
- .then()
- .statusCode(200)
- .body(containsString("_realtime"));
+ String result = RestAssured.given()
+ .contentType(ContentType.TEXT)
+ .post("/splunk/results/normalSearch")
+ .then()
+ .statusCode(200)
+ .extract().asString();
+
+ return result.contains("Name: Sheldon" + suffix)
+ && result.contains("Name: Leonard" + suffix)
+ && result.contains("Name: Irma" + suffix);
+ });
}
@Test
- public void testWriteStreamAndReadSaved() throws InterruptedException {
- int defaultPort = RestAssured.port;
- String defaultUri = RestAssured.baseURI;
-
+ public void testSavedSearchWithTcp() throws InterruptedException {
+ String suffix = "_SavedSearchOfTcp";
//create saved search
RestAssured.given()
.baseUri("http://localhost")
.port(ConfigProvider.getConfig().getValue(SplunkResource.PARAM_REMOTE_PORT, Integer.class))
.contentType(ContentType.JSON)
- .param("name", SplunkTestResource.SAVED_SEARCH_NAME)
+ .param("name", SplunkResource.SAVED_SEARCH_NAME)
.param("disabled", "0")
- .param("description", "descritionText")
+ .param("description", "descriptionText")
.param("search",
- "index=" + SplunkTestResource.TEST_INDEX + " sourcetype=" + SplunkResource.SOURCE_TYPE)
+ "sourcetype=\"TCP\" | rex field=_raw \"Name: (?<name>.*) From: (?<from>.*)\"")
.post("/services/saved/searches")
.then()
.statusCode(anyOf(is(201), is(409)));
- write("_s", SplunkTestResource.TEST_INDEX, "stream");
- RestAssured.given()
- .contentType(ContentType.TEXT)
- .body(SplunkTestResource.SAVED_SEARCH_NAME)
- .post("/splunk/savedSearch")
- .then()
- .statusCode(200)
- .body(containsString("Name: Sheldon_s"))
- .body(containsString("Name: Leonard_s"))
- .body(containsString("Name: Irma_s"));
- }
-
- private void write(String suffix, String index, String endpoint) {
- write(CollectionHelper.mapOf("entity", "Name: Sheldon" + suffix + " From: Alpha Centauri"), "submit",
- index);
- write(CollectionHelper.mapOf("entity", "Name: Leonard" + suffix + " From: Earth 2.0"), "submit",
- index);
- write(CollectionHelper.mapOf("entity", "Name: Irma" + suffix + " From: Earth"), "submit", index);
+ //write data via tcp
+ write(suffix, ProducerType.TCP, 0, false);
+
+ //there might by delay in receiving the data
+ Awaitility.await().pollInterval(1000, TimeUnit.MILLISECONDS).atMost(60, TimeUnit.SECONDS).until(
+ () -> {
+ String result = RestAssured.given()
+ .contentType(ContentType.TEXT)
+ .post("/splunk/results/savedSearch")
+ .then()
+ .statusCode(200)
+ .extract().asString();
+
+ return result.contains("Name: Sheldon" + suffix)
+ && result.contains("Name: Leonard" + suffix)
+ && result.contains("Name: Irma" + suffix);
+ });
}
- private void write(Map<String, String> data, String endpoint, String index) {
+ @Test
+ public void testStreamForRealtime() throws InterruptedException, ExecutionException {
+ String suffix = "_RealtimeSearchOfStream";
+ //there is a buffer for stream writing, therefore about 1MB of data has to be written into Splunk
+
+ //data are written in separated thread
+ ExecutorService executor = Executors.newSingleThreadExecutor();
+ //execute component server to wait for the result
+ Future futureResult = executor.submit(
+ () -> {
+ for (int i = 0; i < 5000; i++) {
+ write(suffix + i, ProducerType.STREAM, 100, false);
+ }
+ });
- String expectedResult = expectedResult(data);
+ try {
+ Awaitility.await().pollInterval(1000, TimeUnit.MILLISECONDS).atMost(60, TimeUnit.SECONDS).until(
+ () -> {
+
+ String result = RestAssured.given()
+ .contentType(ContentType.TEXT)
+ .post("/splunk/results/realtimeSearch")
+ .then()
+ .statusCode(200)
+ .extract().asString();
+
+ return result.contains("Name: Sheldon" + suffix)
+ && result.contains("Name: Leonard" + suffix)
+ && result.contains("Name: Irma" + suffix);
+ });
+ } finally {
+ futureResult.cancel(true);
+ }
+ }
- RestAssured.given()
+ private void write(String suffix, ProducerType producerType, int lengthOfRandomString, boolean raw) {
+ Consumer<Map> write = data -> RestAssured.given()
.contentType(ContentType.JSON)
- .queryParam("index", index)
+ .queryParam("index", SplunkTestResource.TEST_INDEX)
.body(data)
- .post("/splunk/" + endpoint)
+ .post("/splunk/write/" + producerType.name())
.then()
- .statusCode(201)
- .body(containsString(expectedResult));
+ .statusCode(201);
+
+ Map data[] = new Map[3];
Review Comment:
TBH I just thought, that it is not necessary to verify results of the insert, as the status code 201 means, that insert was done (the data is then verified in the tests, but I can add it back)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on a diff in pull request #5184: Expand Splunk test coverage
Posted by "JiriOndrusek (via GitHub)" <gi...@apache.org>.
JiriOndrusek commented on code in PR #5184:
URL: https://github.com/apache/camel-quarkus/pull/5184#discussion_r1294289241
##########
integration-tests/splunk/README.adoc:
##########
@@ -0,0 +1,3 @@
+Splunk UI is helpful for testing.
Review Comment:
I was probably a little bit plain about it. I tried to explain in the following sentence: `To access UI, expose port 8000 and access http://127.0.0.1:${exposed_8000)/ in a browser.` I agree that more information would be helpful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on a diff in pull request #5184: Expand Splunk test coverage
Posted by "JiriOndrusek (via GitHub)" <gi...@apache.org>.
JiriOndrusek commented on code in PR #5184:
URL: https://github.com/apache/camel-quarkus/pull/5184#discussion_r1294326267
##########
extensions/splunk/deployment/src/main/java/org/apache/camel/quarkus/component/splunk/deployment/SplunkProcessor.java:
##########
@@ -67,9 +67,10 @@ ReflectiveClassBuildItem registerForReflection(CombinedIndexBuildItem combinedIn
@BuildStep
List<ReflectiveClassBuildItem> reflectiveClasses() {
- return Arrays.asList(ReflectiveClassBuildItem.builder("com.splunk.Index").build(),
- ReflectiveClassBuildItem.builder("com.splunk.SavedSearch").build(),
- ReflectiveClassBuildItem.builder("com.splunk.Service").build());
+ return Arrays.asList(ReflectiveClassBuildItem.builder("com.splunk.Index").constructors().build(),
Review Comment:
fixed
##########
integration-tests/splunk/src/main/java/org/apache/camel/quarkus/component/splunk/it/SplunkResource.java:
##########
@@ -47,10 +46,10 @@
@ApplicationScoped
public class SplunkResource {
+ public static String SAVED_SEARCH_NAME = "savedSearchForTest";
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] jamesnetherton commented on pull request #5184: Expand Splunk test coverage
Posted by "jamesnetherton (via GitHub)" <gi...@apache.org>.
jamesnetherton commented on PR #5184:
URL: https://github.com/apache/camel-quarkus/pull/5184#issuecomment-1679018595
@JiriOndrusek Sorry, just thinking about `integration-tests/splunk/README.adoc` again. We can simplify it a bit and just have the UI accessible by default.
We can modify `SplunkTestResource` and add port `8000` to the list of exposed ports, then print a log message at the end of `SplunkTestResource.start()`. E.g something like:
```
LOG.infof("Splunk UI running on: http://localhost:%d", container.getMappedPort(WEB_PORT));
```
Then the `README.adoc` content can just be like:
```
The Splunk UI is helpful for the testing as you can see (and search) for data inside Splunk. The URL to the Splunk UI is output to the console.
You may want to delay the Splunk container shutdown either by adding a `Thread.sleep` into `SplunkTestResource.stop()` or commenting out `container.stop()`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on a diff in pull request #5184: Expand Splunk test coverage
Posted by "JiriOndrusek (via GitHub)" <gi...@apache.org>.
JiriOndrusek commented on code in PR #5184:
URL: https://github.com/apache/camel-quarkus/pull/5184#discussion_r1294326477
##########
integration-tests/splunk/src/test/java/org/apache/camel/quarkus/component/splunk/it/SplunkTest.java:
##########
@@ -16,138 +16,153 @@
*/
package org.apache.camel.quarkus.component.splunk.it;
-import java.util.List;
import java.util.Map;
import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
-import java.util.stream.Collectors;
+import java.util.function.Consumer;
import io.quarkus.test.common.QuarkusTestResource;
import io.quarkus.test.junit.QuarkusTest;
import io.restassured.RestAssured;
-import io.restassured.common.mapper.TypeRef;
import io.restassured.http.ContentType;
+import org.apache.camel.component.splunk.ProducerType;
import org.apache.camel.util.CollectionHelper;
import org.eclipse.microprofile.config.ConfigProvider;
-import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import org.testcontainers.shaded.org.apache.commons.lang3.RandomStringUtils;
+import org.testcontainers.shaded.org.awaitility.Awaitility;
import static org.hamcrest.Matchers.anyOf;
-import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.is;
@QuarkusTest
@QuarkusTestResource(SplunkTestResource.class)
class SplunkTest {
@Test
- public void testWriteTcpAndReadNormal() {
- write("_normal", SplunkTestResource.TEST_INDEX, "tcp");
-
- List<Map<String, String>> result = RestAssured.given()
- .contentType(ContentType.TEXT)
- .body(String.format(
- "search index=%s sourcetype=%s | rex field=_raw \"Name: (?<name>.*) From: (?<from>.*)\"",
- SplunkTestResource.TEST_INDEX, SplunkResource.SOURCE_TYPE))
- .post("/splunk/normal")
- .then()
- .statusCode(200)
- .extract().as(new TypeRef<>() {
- });
+ public void testNormalSearchWithSubmitWithRawData() {
+ String suffix = "_normalSearchOfSubmit";
- Assertions.assertEquals(3, result.size());
- Assertions.assertEquals("Irma_normal", result.get(0).get("name"));
- Assertions.assertEquals("Earth\"", result.get(0).get("from"));
- Assertions.assertEquals("Leonard_normal", result.get(1).get("name"));
- Assertions.assertEquals("Earth 2.0\"", result.get(1).get("from"));
- Assertions.assertEquals("Sheldon_normal", result.get(2).get("name"));
- Assertions.assertEquals("Alpha Centauri\"", result.get(2).get("from"));
- }
+ write(suffix, ProducerType.SUBMIT, 0, true);
- @Test
- public void testWriteSubmitAndReadRealtime() throws InterruptedException, ExecutionException {
+ Awaitility.await().pollInterval(1000, TimeUnit.MILLISECONDS).atMost(60, TimeUnit.SECONDS).until(
+ () -> {
- RestAssured.given()
- .body(String.format(
- "search index=%s sourcetype=%s | rex field=_raw \"Name: (?<name>.*) From: (?<from>.*)\"",
- SplunkTestResource.TEST_INDEX, SplunkResource.SOURCE_TYPE))
- .post("/splunk/startRealtimePolling");
-
- //wait some time to start polling
- TimeUnit.SECONDS.sleep(3);
- write("_realtime1", SplunkTestResource.TEST_INDEX, "submit");
- TimeUnit.SECONDS.sleep(1);
- write("_realtime2", SplunkTestResource.TEST_INDEX, "submit");
- TimeUnit.SECONDS.sleep(1);
- write("_realtime3", SplunkTestResource.TEST_INDEX, "submit");
- //wait some time to gather the pulls from splunk server
- TimeUnit.SECONDS.sleep(3);
- //there should be some data from realtime search in direct (concrete values depends on the speed of writing into index)
- //test is asserting that there are some
- RestAssured.get("/splunk/directRealtimePolling")
- .then()
- .statusCode(200)
- .body(containsString("_realtime"));
+ String result = RestAssured.given()
+ .contentType(ContentType.TEXT)
+ .post("/splunk/results/normalSearch")
+ .then()
+ .statusCode(200)
+ .extract().asString();
+
+ return result.contains("Name: Sheldon" + suffix)
+ && result.contains("Name: Leonard" + suffix)
+ && result.contains("Name: Irma" + suffix);
+ });
}
@Test
- public void testWriteStreamAndReadSaved() throws InterruptedException {
- int defaultPort = RestAssured.port;
- String defaultUri = RestAssured.baseURI;
-
+ public void testSavedSearchWithTcp() throws InterruptedException {
+ String suffix = "_SavedSearchOfTcp";
//create saved search
RestAssured.given()
.baseUri("http://localhost")
.port(ConfigProvider.getConfig().getValue(SplunkResource.PARAM_REMOTE_PORT, Integer.class))
.contentType(ContentType.JSON)
- .param("name", SplunkTestResource.SAVED_SEARCH_NAME)
+ .param("name", SplunkResource.SAVED_SEARCH_NAME)
.param("disabled", "0")
- .param("description", "descritionText")
+ .param("description", "descriptionText")
.param("search",
- "index=" + SplunkTestResource.TEST_INDEX + " sourcetype=" + SplunkResource.SOURCE_TYPE)
+ "sourcetype=\"TCP\" | rex field=_raw \"Name: (?<name>.*) From: (?<from>.*)\"")
.post("/services/saved/searches")
.then()
.statusCode(anyOf(is(201), is(409)));
- write("_s", SplunkTestResource.TEST_INDEX, "stream");
- RestAssured.given()
- .contentType(ContentType.TEXT)
- .body(SplunkTestResource.SAVED_SEARCH_NAME)
- .post("/splunk/savedSearch")
- .then()
- .statusCode(200)
- .body(containsString("Name: Sheldon_s"))
- .body(containsString("Name: Leonard_s"))
- .body(containsString("Name: Irma_s"));
- }
-
- private void write(String suffix, String index, String endpoint) {
- write(CollectionHelper.mapOf("entity", "Name: Sheldon" + suffix + " From: Alpha Centauri"), "submit",
- index);
- write(CollectionHelper.mapOf("entity", "Name: Leonard" + suffix + " From: Earth 2.0"), "submit",
- index);
- write(CollectionHelper.mapOf("entity", "Name: Irma" + suffix + " From: Earth"), "submit", index);
+ //write data via tcp
+ write(suffix, ProducerType.TCP, 0, false);
+
+ //there might by delay in receiving the data
+ Awaitility.await().pollInterval(1000, TimeUnit.MILLISECONDS).atMost(60, TimeUnit.SECONDS).until(
+ () -> {
+ String result = RestAssured.given()
+ .contentType(ContentType.TEXT)
+ .post("/splunk/results/savedSearch")
+ .then()
+ .statusCode(200)
+ .extract().asString();
+
+ return result.contains("Name: Sheldon" + suffix)
+ && result.contains("Name: Leonard" + suffix)
+ && result.contains("Name: Irma" + suffix);
+ });
}
- private void write(Map<String, String> data, String endpoint, String index) {
+ @Test
+ public void testStreamForRealtime() throws InterruptedException, ExecutionException {
+ String suffix = "_RealtimeSearchOfStream";
+ //there is a buffer for stream writing, therefore about 1MB of data has to be written into Splunk
+
+ //data are written in separated thread
+ ExecutorService executor = Executors.newSingleThreadExecutor();
+ //execute component server to wait for the result
+ Future futureResult = executor.submit(
+ () -> {
+ for (int i = 0; i < 5000; i++) {
+ write(suffix + i, ProducerType.STREAM, 100, false);
+ }
+ });
- String expectedResult = expectedResult(data);
+ try {
+ Awaitility.await().pollInterval(1000, TimeUnit.MILLISECONDS).atMost(60, TimeUnit.SECONDS).until(
+ () -> {
+
+ String result = RestAssured.given()
+ .contentType(ContentType.TEXT)
+ .post("/splunk/results/realtimeSearch")
+ .then()
+ .statusCode(200)
+ .extract().asString();
+
+ return result.contains("Name: Sheldon" + suffix)
+ && result.contains("Name: Leonard" + suffix)
+ && result.contains("Name: Irma" + suffix);
+ });
+ } finally {
+ futureResult.cancel(true);
+ }
+ }
- RestAssured.given()
+ private void write(String suffix, ProducerType producerType, int lengthOfRandomString, boolean raw) {
+ Consumer<Map> write = data -> RestAssured.given()
.contentType(ContentType.JSON)
- .queryParam("index", index)
+ .queryParam("index", SplunkTestResource.TEST_INDEX)
.body(data)
- .post("/splunk/" + endpoint)
+ .post("/splunk/write/" + producerType.name())
.then()
- .statusCode(201)
- .body(containsString(expectedResult));
+ .statusCode(201);
+
+ Map data[] = new Map[3];
Review Comment:
fixed
##########
integration-tests/splunk/README.adoc:
##########
@@ -0,0 +1,3 @@
+Splunk UI is helpful for testing.
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] jamesnetherton commented on pull request #5184: Expand Splunk test coverage
Posted by "jamesnetherton (via GitHub)" <gi...@apache.org>.
jamesnetherton commented on PR #5184:
URL: https://github.com/apache/camel-quarkus/pull/5184#issuecomment-1681773790
Superseded by #5192.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on a diff in pull request #5184: Expand Splunk test coverage
Posted by "JiriOndrusek (via GitHub)" <gi...@apache.org>.
JiriOndrusek commented on code in PR #5184:
URL: https://github.com/apache/camel-quarkus/pull/5184#discussion_r1294304159
##########
integration-tests/splunk/src/test/java/org/apache/camel/quarkus/component/splunk/it/SplunkTest.java:
##########
@@ -16,138 +16,153 @@
*/
package org.apache.camel.quarkus.component.splunk.it;
-import java.util.List;
import java.util.Map;
import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
-import java.util.stream.Collectors;
+import java.util.function.Consumer;
import io.quarkus.test.common.QuarkusTestResource;
import io.quarkus.test.junit.QuarkusTest;
import io.restassured.RestAssured;
-import io.restassured.common.mapper.TypeRef;
import io.restassured.http.ContentType;
+import org.apache.camel.component.splunk.ProducerType;
import org.apache.camel.util.CollectionHelper;
import org.eclipse.microprofile.config.ConfigProvider;
-import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import org.testcontainers.shaded.org.apache.commons.lang3.RandomStringUtils;
+import org.testcontainers.shaded.org.awaitility.Awaitility;
import static org.hamcrest.Matchers.anyOf;
-import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.is;
@QuarkusTest
@QuarkusTestResource(SplunkTestResource.class)
class SplunkTest {
@Test
- public void testWriteTcpAndReadNormal() {
- write("_normal", SplunkTestResource.TEST_INDEX, "tcp");
-
- List<Map<String, String>> result = RestAssured.given()
- .contentType(ContentType.TEXT)
- .body(String.format(
- "search index=%s sourcetype=%s | rex field=_raw \"Name: (?<name>.*) From: (?<from>.*)\"",
- SplunkTestResource.TEST_INDEX, SplunkResource.SOURCE_TYPE))
- .post("/splunk/normal")
- .then()
- .statusCode(200)
- .extract().as(new TypeRef<>() {
- });
+ public void testNormalSearchWithSubmitWithRawData() {
+ String suffix = "_normalSearchOfSubmit";
- Assertions.assertEquals(3, result.size());
- Assertions.assertEquals("Irma_normal", result.get(0).get("name"));
- Assertions.assertEquals("Earth\"", result.get(0).get("from"));
- Assertions.assertEquals("Leonard_normal", result.get(1).get("name"));
- Assertions.assertEquals("Earth 2.0\"", result.get(1).get("from"));
- Assertions.assertEquals("Sheldon_normal", result.get(2).get("name"));
- Assertions.assertEquals("Alpha Centauri\"", result.get(2).get("from"));
- }
+ write(suffix, ProducerType.SUBMIT, 0, true);
- @Test
- public void testWriteSubmitAndReadRealtime() throws InterruptedException, ExecutionException {
+ Awaitility.await().pollInterval(1000, TimeUnit.MILLISECONDS).atMost(60, TimeUnit.SECONDS).until(
+ () -> {
- RestAssured.given()
- .body(String.format(
- "search index=%s sourcetype=%s | rex field=_raw \"Name: (?<name>.*) From: (?<from>.*)\"",
- SplunkTestResource.TEST_INDEX, SplunkResource.SOURCE_TYPE))
- .post("/splunk/startRealtimePolling");
-
- //wait some time to start polling
- TimeUnit.SECONDS.sleep(3);
- write("_realtime1", SplunkTestResource.TEST_INDEX, "submit");
- TimeUnit.SECONDS.sleep(1);
- write("_realtime2", SplunkTestResource.TEST_INDEX, "submit");
- TimeUnit.SECONDS.sleep(1);
- write("_realtime3", SplunkTestResource.TEST_INDEX, "submit");
- //wait some time to gather the pulls from splunk server
- TimeUnit.SECONDS.sleep(3);
- //there should be some data from realtime search in direct (concrete values depends on the speed of writing into index)
- //test is asserting that there are some
- RestAssured.get("/splunk/directRealtimePolling")
- .then()
- .statusCode(200)
- .body(containsString("_realtime"));
+ String result = RestAssured.given()
+ .contentType(ContentType.TEXT)
+ .post("/splunk/results/normalSearch")
+ .then()
+ .statusCode(200)
+ .extract().asString();
+
+ return result.contains("Name: Sheldon" + suffix)
+ && result.contains("Name: Leonard" + suffix)
+ && result.contains("Name: Irma" + suffix);
+ });
}
@Test
- public void testWriteStreamAndReadSaved() throws InterruptedException {
- int defaultPort = RestAssured.port;
- String defaultUri = RestAssured.baseURI;
-
+ public void testSavedSearchWithTcp() throws InterruptedException {
+ String suffix = "_SavedSearchOfTcp";
//create saved search
RestAssured.given()
.baseUri("http://localhost")
.port(ConfigProvider.getConfig().getValue(SplunkResource.PARAM_REMOTE_PORT, Integer.class))
.contentType(ContentType.JSON)
- .param("name", SplunkTestResource.SAVED_SEARCH_NAME)
+ .param("name", SplunkResource.SAVED_SEARCH_NAME)
.param("disabled", "0")
- .param("description", "descritionText")
+ .param("description", "descriptionText")
.param("search",
- "index=" + SplunkTestResource.TEST_INDEX + " sourcetype=" + SplunkResource.SOURCE_TYPE)
+ "sourcetype=\"TCP\" | rex field=_raw \"Name: (?<name>.*) From: (?<from>.*)\"")
.post("/services/saved/searches")
.then()
.statusCode(anyOf(is(201), is(409)));
- write("_s", SplunkTestResource.TEST_INDEX, "stream");
- RestAssured.given()
- .contentType(ContentType.TEXT)
- .body(SplunkTestResource.SAVED_SEARCH_NAME)
- .post("/splunk/savedSearch")
- .then()
- .statusCode(200)
- .body(containsString("Name: Sheldon_s"))
- .body(containsString("Name: Leonard_s"))
- .body(containsString("Name: Irma_s"));
- }
-
- private void write(String suffix, String index, String endpoint) {
- write(CollectionHelper.mapOf("entity", "Name: Sheldon" + suffix + " From: Alpha Centauri"), "submit",
- index);
- write(CollectionHelper.mapOf("entity", "Name: Leonard" + suffix + " From: Earth 2.0"), "submit",
- index);
- write(CollectionHelper.mapOf("entity", "Name: Irma" + suffix + " From: Earth"), "submit", index);
+ //write data via tcp
+ write(suffix, ProducerType.TCP, 0, false);
+
+ //there might by delay in receiving the data
+ Awaitility.await().pollInterval(1000, TimeUnit.MILLISECONDS).atMost(60, TimeUnit.SECONDS).until(
+ () -> {
+ String result = RestAssured.given()
+ .contentType(ContentType.TEXT)
+ .post("/splunk/results/savedSearch")
+ .then()
+ .statusCode(200)
+ .extract().asString();
+
+ return result.contains("Name: Sheldon" + suffix)
+ && result.contains("Name: Leonard" + suffix)
+ && result.contains("Name: Irma" + suffix);
+ });
}
- private void write(Map<String, String> data, String endpoint, String index) {
+ @Test
+ public void testStreamForRealtime() throws InterruptedException, ExecutionException {
+ String suffix = "_RealtimeSearchOfStream";
+ //there is a buffer for stream writing, therefore about 1MB of data has to be written into Splunk
+
+ //data are written in separated thread
+ ExecutorService executor = Executors.newSingleThreadExecutor();
+ //execute component server to wait for the result
+ Future futureResult = executor.submit(
+ () -> {
+ for (int i = 0; i < 5000; i++) {
+ write(suffix + i, ProducerType.STREAM, 100, false);
+ }
+ });
- String expectedResult = expectedResult(data);
+ try {
+ Awaitility.await().pollInterval(1000, TimeUnit.MILLISECONDS).atMost(60, TimeUnit.SECONDS).until(
+ () -> {
+
+ String result = RestAssured.given()
+ .contentType(ContentType.TEXT)
+ .post("/splunk/results/realtimeSearch")
+ .then()
+ .statusCode(200)
+ .extract().asString();
+
+ return result.contains("Name: Sheldon" + suffix)
+ && result.contains("Name: Leonard" + suffix)
+ && result.contains("Name: Irma" + suffix);
+ });
+ } finally {
+ futureResult.cancel(true);
+ }
+ }
- RestAssured.given()
+ private void write(String suffix, ProducerType producerType, int lengthOfRandomString, boolean raw) {
+ Consumer<Map> write = data -> RestAssured.given()
.contentType(ContentType.JSON)
- .queryParam("index", index)
+ .queryParam("index", SplunkTestResource.TEST_INDEX)
.body(data)
- .post("/splunk/" + endpoint)
+ .post("/splunk/write/" + producerType.name())
.then()
- .statusCode(201)
- .body(containsString(expectedResult));
+ .statusCode(201);
+
+ Map data[] = new Map[3];
Review Comment:
TBH I just thought, that it is not necessary to verify it, as the status code 201 means, that insert was done (the data is then verified in the tests, but I can add it back)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on a diff in pull request #5184: Expand Splunk test coverage
Posted by "JiriOndrusek (via GitHub)" <gi...@apache.org>.
JiriOndrusek commented on code in PR #5184:
URL: https://github.com/apache/camel-quarkus/pull/5184#discussion_r1294304159
##########
integration-tests/splunk/src/test/java/org/apache/camel/quarkus/component/splunk/it/SplunkTest.java:
##########
@@ -16,138 +16,153 @@
*/
package org.apache.camel.quarkus.component.splunk.it;
-import java.util.List;
import java.util.Map;
import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
-import java.util.stream.Collectors;
+import java.util.function.Consumer;
import io.quarkus.test.common.QuarkusTestResource;
import io.quarkus.test.junit.QuarkusTest;
import io.restassured.RestAssured;
-import io.restassured.common.mapper.TypeRef;
import io.restassured.http.ContentType;
+import org.apache.camel.component.splunk.ProducerType;
import org.apache.camel.util.CollectionHelper;
import org.eclipse.microprofile.config.ConfigProvider;
-import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import org.testcontainers.shaded.org.apache.commons.lang3.RandomStringUtils;
+import org.testcontainers.shaded.org.awaitility.Awaitility;
import static org.hamcrest.Matchers.anyOf;
-import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.is;
@QuarkusTest
@QuarkusTestResource(SplunkTestResource.class)
class SplunkTest {
@Test
- public void testWriteTcpAndReadNormal() {
- write("_normal", SplunkTestResource.TEST_INDEX, "tcp");
-
- List<Map<String, String>> result = RestAssured.given()
- .contentType(ContentType.TEXT)
- .body(String.format(
- "search index=%s sourcetype=%s | rex field=_raw \"Name: (?<name>.*) From: (?<from>.*)\"",
- SplunkTestResource.TEST_INDEX, SplunkResource.SOURCE_TYPE))
- .post("/splunk/normal")
- .then()
- .statusCode(200)
- .extract().as(new TypeRef<>() {
- });
+ public void testNormalSearchWithSubmitWithRawData() {
+ String suffix = "_normalSearchOfSubmit";
- Assertions.assertEquals(3, result.size());
- Assertions.assertEquals("Irma_normal", result.get(0).get("name"));
- Assertions.assertEquals("Earth\"", result.get(0).get("from"));
- Assertions.assertEquals("Leonard_normal", result.get(1).get("name"));
- Assertions.assertEquals("Earth 2.0\"", result.get(1).get("from"));
- Assertions.assertEquals("Sheldon_normal", result.get(2).get("name"));
- Assertions.assertEquals("Alpha Centauri\"", result.get(2).get("from"));
- }
+ write(suffix, ProducerType.SUBMIT, 0, true);
- @Test
- public void testWriteSubmitAndReadRealtime() throws InterruptedException, ExecutionException {
+ Awaitility.await().pollInterval(1000, TimeUnit.MILLISECONDS).atMost(60, TimeUnit.SECONDS).until(
+ () -> {
- RestAssured.given()
- .body(String.format(
- "search index=%s sourcetype=%s | rex field=_raw \"Name: (?<name>.*) From: (?<from>.*)\"",
- SplunkTestResource.TEST_INDEX, SplunkResource.SOURCE_TYPE))
- .post("/splunk/startRealtimePolling");
-
- //wait some time to start polling
- TimeUnit.SECONDS.sleep(3);
- write("_realtime1", SplunkTestResource.TEST_INDEX, "submit");
- TimeUnit.SECONDS.sleep(1);
- write("_realtime2", SplunkTestResource.TEST_INDEX, "submit");
- TimeUnit.SECONDS.sleep(1);
- write("_realtime3", SplunkTestResource.TEST_INDEX, "submit");
- //wait some time to gather the pulls from splunk server
- TimeUnit.SECONDS.sleep(3);
- //there should be some data from realtime search in direct (concrete values depends on the speed of writing into index)
- //test is asserting that there are some
- RestAssured.get("/splunk/directRealtimePolling")
- .then()
- .statusCode(200)
- .body(containsString("_realtime"));
+ String result = RestAssured.given()
+ .contentType(ContentType.TEXT)
+ .post("/splunk/results/normalSearch")
+ .then()
+ .statusCode(200)
+ .extract().asString();
+
+ return result.contains("Name: Sheldon" + suffix)
+ && result.contains("Name: Leonard" + suffix)
+ && result.contains("Name: Irma" + suffix);
+ });
}
@Test
- public void testWriteStreamAndReadSaved() throws InterruptedException {
- int defaultPort = RestAssured.port;
- String defaultUri = RestAssured.baseURI;
-
+ public void testSavedSearchWithTcp() throws InterruptedException {
+ String suffix = "_SavedSearchOfTcp";
//create saved search
RestAssured.given()
.baseUri("http://localhost")
.port(ConfigProvider.getConfig().getValue(SplunkResource.PARAM_REMOTE_PORT, Integer.class))
.contentType(ContentType.JSON)
- .param("name", SplunkTestResource.SAVED_SEARCH_NAME)
+ .param("name", SplunkResource.SAVED_SEARCH_NAME)
.param("disabled", "0")
- .param("description", "descritionText")
+ .param("description", "descriptionText")
.param("search",
- "index=" + SplunkTestResource.TEST_INDEX + " sourcetype=" + SplunkResource.SOURCE_TYPE)
+ "sourcetype=\"TCP\" | rex field=_raw \"Name: (?<name>.*) From: (?<from>.*)\"")
.post("/services/saved/searches")
.then()
.statusCode(anyOf(is(201), is(409)));
- write("_s", SplunkTestResource.TEST_INDEX, "stream");
- RestAssured.given()
- .contentType(ContentType.TEXT)
- .body(SplunkTestResource.SAVED_SEARCH_NAME)
- .post("/splunk/savedSearch")
- .then()
- .statusCode(200)
- .body(containsString("Name: Sheldon_s"))
- .body(containsString("Name: Leonard_s"))
- .body(containsString("Name: Irma_s"));
- }
-
- private void write(String suffix, String index, String endpoint) {
- write(CollectionHelper.mapOf("entity", "Name: Sheldon" + suffix + " From: Alpha Centauri"), "submit",
- index);
- write(CollectionHelper.mapOf("entity", "Name: Leonard" + suffix + " From: Earth 2.0"), "submit",
- index);
- write(CollectionHelper.mapOf("entity", "Name: Irma" + suffix + " From: Earth"), "submit", index);
+ //write data via tcp
+ write(suffix, ProducerType.TCP, 0, false);
+
+ //there might by delay in receiving the data
+ Awaitility.await().pollInterval(1000, TimeUnit.MILLISECONDS).atMost(60, TimeUnit.SECONDS).until(
+ () -> {
+ String result = RestAssured.given()
+ .contentType(ContentType.TEXT)
+ .post("/splunk/results/savedSearch")
+ .then()
+ .statusCode(200)
+ .extract().asString();
+
+ return result.contains("Name: Sheldon" + suffix)
+ && result.contains("Name: Leonard" + suffix)
+ && result.contains("Name: Irma" + suffix);
+ });
}
- private void write(Map<String, String> data, String endpoint, String index) {
+ @Test
+ public void testStreamForRealtime() throws InterruptedException, ExecutionException {
+ String suffix = "_RealtimeSearchOfStream";
+ //there is a buffer for stream writing, therefore about 1MB of data has to be written into Splunk
+
+ //data are written in separated thread
+ ExecutorService executor = Executors.newSingleThreadExecutor();
+ //execute component server to wait for the result
+ Future futureResult = executor.submit(
+ () -> {
+ for (int i = 0; i < 5000; i++) {
+ write(suffix + i, ProducerType.STREAM, 100, false);
+ }
+ });
- String expectedResult = expectedResult(data);
+ try {
+ Awaitility.await().pollInterval(1000, TimeUnit.MILLISECONDS).atMost(60, TimeUnit.SECONDS).until(
+ () -> {
+
+ String result = RestAssured.given()
+ .contentType(ContentType.TEXT)
+ .post("/splunk/results/realtimeSearch")
+ .then()
+ .statusCode(200)
+ .extract().asString();
+
+ return result.contains("Name: Sheldon" + suffix)
+ && result.contains("Name: Leonard" + suffix)
+ && result.contains("Name: Irma" + suffix);
+ });
+ } finally {
+ futureResult.cancel(true);
+ }
+ }
- RestAssured.given()
+ private void write(String suffix, ProducerType producerType, int lengthOfRandomString, boolean raw) {
+ Consumer<Map> write = data -> RestAssured.given()
.contentType(ContentType.JSON)
- .queryParam("index", index)
+ .queryParam("index", SplunkTestResource.TEST_INDEX)
.body(data)
- .post("/splunk/" + endpoint)
+ .post("/splunk/write/" + producerType.name())
.then()
- .statusCode(201)
- .body(containsString(expectedResult));
+ .statusCode(201);
+
+ Map data[] = new Map[3];
Review Comment:
TBH I just thought, that it is not necessary to verify results of the insert, as the status code 201 means, that insert was done (the data is then verified in the tests, but I can add it back)
---
Asserting the result of insert might have meaning, because it would verify `Producer` and fail fast, if the result is not correct. I'll return the assertion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] jamesnetherton closed pull request #5184: Expand Splunk test coverage
Posted by "jamesnetherton (via GitHub)" <gi...@apache.org>.
jamesnetherton closed pull request #5184: Expand Splunk test coverage
URL: https://github.com/apache/camel-quarkus/pull/5184
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] jamesnetherton commented on pull request #5184: Expand Splunk test coverage
Posted by "jamesnetherton (via GitHub)" <gi...@apache.org>.
jamesnetherton commented on PR #5184:
URL: https://github.com/apache/camel-quarkus/pull/5184#issuecomment-1681678170
> Hi @JiriOndrusek
>
> The ibmmq client issue should be fixed and can you rebase the PR?
I think he's on PTO. To speed things along, I'll create another PR from this and with the doc suggestion I mentioned above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org