You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "david-streamlio (via GitHub)" <gi...@apache.org> on 2026/07/17 22:21:56 UTC
[PR] [improve][test] Add integration tests for the Kinesis source connector [pulsar-connectors]
david-streamlio opened a new pull request, #123:
URL: https://github.com/apache/pulsar-connectors/pull/123
## What
Adds `KinesisSourceIntegrationTest`, the first genuine integration test for the **Kinesis source** connector. Until now the source had only unit/mock coverage (`KinesisRecordProcessorTest`, `KinesisSourceConfigTest`), while the sink had `KinesisSinkTest` against LocalStack — this closes that sink/source asymmetry.
## How
The test starts a LocalStack container with **Kinesis + DynamoDB + CloudWatch** (the KCL uses DynamoDB for its lease table and CloudWatch for metrics), creates a stream, and drives `KinesisSource` end-to-end — asserting records flow through the KCL worker into the source's queue with the expected value, partition key, and `kinesis.sequence.number` property.
It mirrors the existing `DynamoDBSourceIntegrationTest` harness (both are KCL sources against LocalStack):
- seed records before `open` and read from `TRIM_HORIZON` so they replay;
- a background writer keeps a steady supply regardless of when KCL lease/shard bootstrap completes;
- a unique `applicationName` per run → a fresh KCL lease table (no stale leases);
- reads happen on a bounded daemon executor with a timeout, because `KinesisSource.read()` blocks forever on an empty queue and must never be called on the test thread;
- `useEnhancedFanOut=false` so the source polls — enhanced fan-out (`SubscribeToShard`) is unreliable on LocalStack.
No production code and no build dependencies change — `testcontainers-localstack`, `awaitility`, and the v2 Kinesis client are already on the module's test classpath.
## Testing
Ran locally against Docker/LocalStack:
```
./gradlew :kinesis:test --tests "*KinesisSourceIntegrationTest"
KinesisSourceIntegrationTest > testReadFromKinesisStream PASSED
BUILD SUCCESSFUL in 2m 40s
```
Fixes #122
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][test] Add integration tests for the Kinesis source connector [pulsar-connectors]
Posted by "david-streamlio (via GitHub)" <gi...@apache.org>.
david-streamlio commented on code in PR #123:
URL: https://github.com/apache/pulsar-connectors/pull/123#discussion_r3606845689
##########
kinesis/src/test/java/org/apache/pulsar/io/kinesis/KinesisSourceIntegrationTest.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.io.kinesis;
+
+import static org.mockito.Mockito.mock;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertTrue;
+import java.net.URI;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.io.core.SourceContext;
+import org.awaitility.Awaitility;
+import org.testcontainers.containers.localstack.LocalStackContainer;
+import org.testcontainers.utility.DockerImageName;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisAsyncClient;
+import software.amazon.awssdk.services.kinesis.model.CreateStreamRequest;
+import software.amazon.awssdk.services.kinesis.model.DescribeStreamRequest;
+import software.amazon.awssdk.services.kinesis.model.PutRecordRequest;
+import software.amazon.awssdk.services.kinesis.model.StreamStatus;
+
+/**
+ * Integration test for {@link KinesisSource} that exercises the full path from a real Kinesis
+ * stream through the KCL (Kinesis Client Library) worker and into the source's record queue. A
+ * LocalStack container provides Kinesis, DynamoDB and CloudWatch (the KCL uses DynamoDB for its
+ * lease table and CloudWatch for metrics).
+ */
+@Slf4j
+public class KinesisSourceIntegrationTest {
+
+ private static final String REGION = "us-east-1";
+ private static final String ACCESS_KEY = "access";
+ private static final String SECRET_KEY = "secret";
+ private static final String STREAM_NAME = "pulsar-kinesis-source-it";
+
+ // KCL against LocalStack is slow to bootstrap (lease table creation + shard discovery), so the
+ // first record can take well over a minute to surface. Give each read a generous deadline while
+ // keeping the overall test bounded so an under-delivering source fails promptly rather than hangs.
+ private static final int READ_TIMEOUT_SECONDS = 240;
+ private static final int EXPECTED_RECORDS = 3;
+
+ public static final LocalStackContainer LOCAL_STACK_CONTAINER =
+ new LocalStackContainer(DockerImageName.parse("localstack/localstack:4.0.3"))
+ .withServices(
+ LocalStackContainer.Service.KINESIS,
+ LocalStackContainer.Service.DYNAMODB,
+ LocalStackContainer.Service.CLOUDWATCH)
+ .withEnv("KINESIS_PROVIDER", "kinesalite")
+ .withStartupTimeout(Duration.ofMinutes(5));
+
+ private KinesisAsyncClient client;
+ private KinesisSource source;
+ private ExecutorService readerExecutor;
+ private Thread writerThread;
+ private final AtomicBoolean keepWriting = new AtomicBoolean(false);
+
+ @BeforeClass(alwaysRun = true)
+ public void beforeClass() {
+ LOCAL_STACK_CONTAINER.start();
+ }
+
+ @AfterClass(alwaysRun = true)
+ public void afterClass() {
+ LOCAL_STACK_CONTAINER.stop();
+ }
+
+ @BeforeMethod
+ public void setup() throws Exception {
+ readerExecutor = Executors.newSingleThreadExecutor(r -> {
+ Thread t = new Thread(r, "kinesis-it-reader");
+ t.setDaemon(true);
+ return t;
+ });
+
+ client = createClient();
+ client.createStream(CreateStreamRequest.builder()
+ .streamName(STREAM_NAME)
+ .shardCount(1)
+ .build()).get();
+
+ Awaitility.await().atMost(60, TimeUnit.SECONDS).until(() ->
+ client.describeStream(DescribeStreamRequest.builder().streamName(STREAM_NAME).build())
+ .get().streamDescription().streamStatus() == StreamStatus.ACTIVE);
Review Comment:
Fixed in 88f6eef. createStream/describeStream (and deleteStream in cleanup) now use bounded get(timeout) calls, so a LocalStack stall fails @BeforeMethod promptly instead of hanging the suite.
##########
kinesis/src/test/java/org/apache/pulsar/io/kinesis/KinesisSourceIntegrationTest.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.io.kinesis;
+
+import static org.mockito.Mockito.mock;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertTrue;
+import java.net.URI;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.io.core.SourceContext;
+import org.awaitility.Awaitility;
+import org.testcontainers.containers.localstack.LocalStackContainer;
+import org.testcontainers.utility.DockerImageName;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisAsyncClient;
+import software.amazon.awssdk.services.kinesis.model.CreateStreamRequest;
+import software.amazon.awssdk.services.kinesis.model.DescribeStreamRequest;
+import software.amazon.awssdk.services.kinesis.model.PutRecordRequest;
+import software.amazon.awssdk.services.kinesis.model.StreamStatus;
+
+/**
+ * Integration test for {@link KinesisSource} that exercises the full path from a real Kinesis
+ * stream through the KCL (Kinesis Client Library) worker and into the source's record queue. A
+ * LocalStack container provides Kinesis, DynamoDB and CloudWatch (the KCL uses DynamoDB for its
+ * lease table and CloudWatch for metrics).
+ */
+@Slf4j
+public class KinesisSourceIntegrationTest {
+
+ private static final String REGION = "us-east-1";
+ private static final String ACCESS_KEY = "access";
+ private static final String SECRET_KEY = "secret";
+ private static final String STREAM_NAME = "pulsar-kinesis-source-it";
+
+ // KCL against LocalStack is slow to bootstrap (lease table creation + shard discovery), so the
+ // first record can take well over a minute to surface. Give each read a generous deadline while
+ // keeping the overall test bounded so an under-delivering source fails promptly rather than hangs.
+ private static final int READ_TIMEOUT_SECONDS = 240;
+ private static final int EXPECTED_RECORDS = 3;
+
+ public static final LocalStackContainer LOCAL_STACK_CONTAINER =
+ new LocalStackContainer(DockerImageName.parse("localstack/localstack:4.0.3"))
+ .withServices(
+ LocalStackContainer.Service.KINESIS,
+ LocalStackContainer.Service.DYNAMODB,
+ LocalStackContainer.Service.CLOUDWATCH)
+ .withEnv("KINESIS_PROVIDER", "kinesalite")
+ .withStartupTimeout(Duration.ofMinutes(5));
+
+ private KinesisAsyncClient client;
+ private KinesisSource source;
+ private ExecutorService readerExecutor;
+ private Thread writerThread;
+ private final AtomicBoolean keepWriting = new AtomicBoolean(false);
+
+ @BeforeClass(alwaysRun = true)
+ public void beforeClass() {
+ LOCAL_STACK_CONTAINER.start();
+ }
+
+ @AfterClass(alwaysRun = true)
+ public void afterClass() {
+ LOCAL_STACK_CONTAINER.stop();
+ }
+
+ @BeforeMethod
+ public void setup() throws Exception {
+ readerExecutor = Executors.newSingleThreadExecutor(r -> {
+ Thread t = new Thread(r, "kinesis-it-reader");
+ t.setDaemon(true);
+ return t;
+ });
+
+ client = createClient();
+ client.createStream(CreateStreamRequest.builder()
+ .streamName(STREAM_NAME)
+ .shardCount(1)
+ .build()).get();
+
+ Awaitility.await().atMost(60, TimeUnit.SECONDS).until(() ->
+ client.describeStream(DescribeStreamRequest.builder().streamName(STREAM_NAME).build())
+ .get().streamDescription().streamStatus() == StreamStatus.ACTIVE);
+ log.info("Created Kinesis stream {}", STREAM_NAME);
+
+ source = new KinesisSource();
+ }
+
+ @AfterMethod(alwaysRun = true)
+ public void cleanup() {
+ keepWriting.set(false);
+ if (writerThread != null) {
+ writerThread.interrupt();
+ }
+ if (readerExecutor != null) {
+ // A reader may still be blocked in read(), which never returns null.
+ readerExecutor.shutdownNow();
+ }
+ if (source != null) {
+ try {
+ source.close();
+ } catch (Exception e) {
+ log.warn("Failed to close source", e);
+ }
+ }
+ if (client != null) {
+ try {
+ client.deleteStream(builder -> builder.streamName(STREAM_NAME)).get();
+ } catch (Exception e) {
+ log.warn("Failed to delete stream", e);
+ }
+ client.close();
+ }
+ }
+
+ @Test(timeOut = 600_000)
+ public void testReadFromKinesisStream() throws Exception {
+ // Seed some records before the source starts; TRIM_HORIZON makes the worker replay them.
+ for (int i = 0; i < EXPECTED_RECORDS; i++) {
+ putRecord("seed-" + i);
+ }
+
+ // Keep writing after open as well so the worker has a steady supply regardless of exactly
+ // when its lease/shard bootstrap completes.
+ startBackgroundWriter();
+
+ SourceContext sourceContext = mock(SourceContext.class);
+ source.open(buildConfig(), sourceContext);
+
+ int received = 0;
+ for (int i = 0; i < EXPECTED_RECORDS; i++) {
+ KinesisRecord record = readOne();
+ assertNotNull(record, "read() returned null");
+ assertNotNull(record.getValue(), "record had no value");
+ assertTrue(record.getValue().length > 0, "record value was empty");
+ assertTrue(record.getKey().isPresent(), "record had no key (partition key)");
+ assertTrue(record.getProperties().containsKey(KinesisRecord.SEQUENCE_NUMBER),
+ "record missing SEQUENCE_NUMBER property; got " + record.getProperties());
+ log.info("Received Kinesis record: key={} value={} seq={}",
+ record.getKey().orElse(null),
+ new String(record.getValue(), StandardCharsets.UTF_8),
+ record.getProperties().get(KinesisRecord.SEQUENCE_NUMBER));
+ received++;
+ }
+ assertTrue(received >= EXPECTED_RECORDS,
+ "Expected at least " + EXPECTED_RECORDS + " records, got " + received);
+ }
Review Comment:
Fixed in 88f6eef — removed the redundant post-loop count assertion (the fixed-count for-loop already guarantees it).
##########
kinesis/src/test/java/org/apache/pulsar/io/kinesis/KinesisSourceIntegrationTest.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.io.kinesis;
+
+import static org.mockito.Mockito.mock;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertTrue;
+import java.net.URI;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.io.core.SourceContext;
+import org.awaitility.Awaitility;
+import org.testcontainers.containers.localstack.LocalStackContainer;
+import org.testcontainers.utility.DockerImageName;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisAsyncClient;
+import software.amazon.awssdk.services.kinesis.model.CreateStreamRequest;
+import software.amazon.awssdk.services.kinesis.model.DescribeStreamRequest;
+import software.amazon.awssdk.services.kinesis.model.PutRecordRequest;
+import software.amazon.awssdk.services.kinesis.model.StreamStatus;
+
+/**
+ * Integration test for {@link KinesisSource} that exercises the full path from a real Kinesis
+ * stream through the KCL (Kinesis Client Library) worker and into the source's record queue. A
+ * LocalStack container provides Kinesis, DynamoDB and CloudWatch (the KCL uses DynamoDB for its
+ * lease table and CloudWatch for metrics).
+ */
+@Slf4j
+public class KinesisSourceIntegrationTest {
+
+ private static final String REGION = "us-east-1";
+ private static final String ACCESS_KEY = "access";
+ private static final String SECRET_KEY = "secret";
+ private static final String STREAM_NAME = "pulsar-kinesis-source-it";
+
+ // KCL against LocalStack is slow to bootstrap (lease table creation + shard discovery), so the
+ // first record can take well over a minute to surface. Give each read a generous deadline while
+ // keeping the overall test bounded so an under-delivering source fails promptly rather than hangs.
+ private static final int READ_TIMEOUT_SECONDS = 240;
+ private static final int EXPECTED_RECORDS = 3;
+
+ public static final LocalStackContainer LOCAL_STACK_CONTAINER =
+ new LocalStackContainer(DockerImageName.parse("localstack/localstack:4.0.3"))
+ .withServices(
+ LocalStackContainer.Service.KINESIS,
+ LocalStackContainer.Service.DYNAMODB,
+ LocalStackContainer.Service.CLOUDWATCH)
+ .withEnv("KINESIS_PROVIDER", "kinesalite")
+ .withStartupTimeout(Duration.ofMinutes(5));
+
+ private KinesisAsyncClient client;
+ private KinesisSource source;
+ private ExecutorService readerExecutor;
+ private Thread writerThread;
+ private final AtomicBoolean keepWriting = new AtomicBoolean(false);
+
+ @BeforeClass(alwaysRun = true)
+ public void beforeClass() {
+ LOCAL_STACK_CONTAINER.start();
+ }
+
+ @AfterClass(alwaysRun = true)
+ public void afterClass() {
+ LOCAL_STACK_CONTAINER.stop();
+ }
+
+ @BeforeMethod
+ public void setup() throws Exception {
+ readerExecutor = Executors.newSingleThreadExecutor(r -> {
+ Thread t = new Thread(r, "kinesis-it-reader");
+ t.setDaemon(true);
+ return t;
+ });
+
+ client = createClient();
+ client.createStream(CreateStreamRequest.builder()
+ .streamName(STREAM_NAME)
+ .shardCount(1)
+ .build()).get();
+
+ Awaitility.await().atMost(60, TimeUnit.SECONDS).until(() ->
+ client.describeStream(DescribeStreamRequest.builder().streamName(STREAM_NAME).build())
+ .get().streamDescription().streamStatus() == StreamStatus.ACTIVE);
+ log.info("Created Kinesis stream {}", STREAM_NAME);
+
+ source = new KinesisSource();
+ }
+
+ @AfterMethod(alwaysRun = true)
+ public void cleanup() {
+ keepWriting.set(false);
+ if (writerThread != null) {
+ writerThread.interrupt();
+ }
+ if (readerExecutor != null) {
+ // A reader may still be blocked in read(), which never returns null.
+ readerExecutor.shutdownNow();
+ }
+ if (source != null) {
+ try {
+ source.close();
+ } catch (Exception e) {
+ log.warn("Failed to close source", e);
+ }
+ }
+ if (client != null) {
+ try {
+ client.deleteStream(builder -> builder.streamName(STREAM_NAME)).get();
+ } catch (Exception e) {
+ log.warn("Failed to delete stream", e);
+ }
+ client.close();
+ }
+ }
+
+ @Test(timeOut = 600_000)
+ public void testReadFromKinesisStream() throws Exception {
+ // Seed some records before the source starts; TRIM_HORIZON makes the worker replay them.
+ for (int i = 0; i < EXPECTED_RECORDS; i++) {
+ putRecord("seed-" + i);
+ }
+
+ // Keep writing after open as well so the worker has a steady supply regardless of exactly
+ // when its lease/shard bootstrap completes.
+ startBackgroundWriter();
+
+ SourceContext sourceContext = mock(SourceContext.class);
+ source.open(buildConfig(), sourceContext);
+
+ int received = 0;
+ for (int i = 0; i < EXPECTED_RECORDS; i++) {
+ KinesisRecord record = readOne();
+ assertNotNull(record, "read() returned null");
+ assertNotNull(record.getValue(), "record had no value");
+ assertTrue(record.getValue().length > 0, "record value was empty");
+ assertTrue(record.getKey().isPresent(), "record had no key (partition key)");
+ assertTrue(record.getProperties().containsKey(KinesisRecord.SEQUENCE_NUMBER),
+ "record missing SEQUENCE_NUMBER property; got " + record.getProperties());
+ log.info("Received Kinesis record: key={} value={} seq={}",
+ record.getKey().orElse(null),
+ new String(record.getValue(), StandardCharsets.UTF_8),
+ record.getProperties().get(KinesisRecord.SEQUENCE_NUMBER));
Review Comment:
Fixed in 88f6eef. The test now asserts the emitted payload and partition key carry what was written (value starts with "value-", key starts with "pk-"), not just that they are non-empty.
##########
kinesis/src/test/java/org/apache/pulsar/io/kinesis/KinesisSourceIntegrationTest.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.io.kinesis;
+
+import static org.mockito.Mockito.mock;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertTrue;
+import java.net.URI;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.io.core.SourceContext;
+import org.awaitility.Awaitility;
+import org.testcontainers.containers.localstack.LocalStackContainer;
+import org.testcontainers.utility.DockerImageName;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisAsyncClient;
+import software.amazon.awssdk.services.kinesis.model.CreateStreamRequest;
+import software.amazon.awssdk.services.kinesis.model.DescribeStreamRequest;
+import software.amazon.awssdk.services.kinesis.model.PutRecordRequest;
+import software.amazon.awssdk.services.kinesis.model.StreamStatus;
+
+/**
+ * Integration test for {@link KinesisSource} that exercises the full path from a real Kinesis
+ * stream through the KCL (Kinesis Client Library) worker and into the source's record queue. A
+ * LocalStack container provides Kinesis, DynamoDB and CloudWatch (the KCL uses DynamoDB for its
+ * lease table and CloudWatch for metrics).
+ */
+@Slf4j
+public class KinesisSourceIntegrationTest {
+
+ private static final String REGION = "us-east-1";
+ private static final String ACCESS_KEY = "access";
+ private static final String SECRET_KEY = "secret";
+ private static final String STREAM_NAME = "pulsar-kinesis-source-it";
+
+ // KCL against LocalStack is slow to bootstrap (lease table creation + shard discovery), so the
+ // first record can take well over a minute to surface. Give each read a generous deadline while
+ // keeping the overall test bounded so an under-delivering source fails promptly rather than hangs.
+ private static final int READ_TIMEOUT_SECONDS = 240;
+ private static final int EXPECTED_RECORDS = 3;
+
+ public static final LocalStackContainer LOCAL_STACK_CONTAINER =
+ new LocalStackContainer(DockerImageName.parse("localstack/localstack:4.0.3"))
+ .withServices(
+ LocalStackContainer.Service.KINESIS,
+ LocalStackContainer.Service.DYNAMODB,
+ LocalStackContainer.Service.CLOUDWATCH)
+ .withEnv("KINESIS_PROVIDER", "kinesalite")
+ .withStartupTimeout(Duration.ofMinutes(5));
+
+ private KinesisAsyncClient client;
+ private KinesisSource source;
+ private ExecutorService readerExecutor;
+ private Thread writerThread;
+ private final AtomicBoolean keepWriting = new AtomicBoolean(false);
+
+ @BeforeClass(alwaysRun = true)
+ public void beforeClass() {
+ LOCAL_STACK_CONTAINER.start();
+ }
+
+ @AfterClass(alwaysRun = true)
+ public void afterClass() {
+ LOCAL_STACK_CONTAINER.stop();
+ }
+
+ @BeforeMethod
+ public void setup() throws Exception {
+ readerExecutor = Executors.newSingleThreadExecutor(r -> {
+ Thread t = new Thread(r, "kinesis-it-reader");
+ t.setDaemon(true);
+ return t;
+ });
+
+ client = createClient();
+ client.createStream(CreateStreamRequest.builder()
+ .streamName(STREAM_NAME)
+ .shardCount(1)
+ .build()).get();
+
+ Awaitility.await().atMost(60, TimeUnit.SECONDS).until(() ->
+ client.describeStream(DescribeStreamRequest.builder().streamName(STREAM_NAME).build())
+ .get().streamDescription().streamStatus() == StreamStatus.ACTIVE);
+ log.info("Created Kinesis stream {}", STREAM_NAME);
+
+ source = new KinesisSource();
+ }
+
+ @AfterMethod(alwaysRun = true)
+ public void cleanup() {
+ keepWriting.set(false);
+ if (writerThread != null) {
+ writerThread.interrupt();
+ }
+ if (readerExecutor != null) {
+ // A reader may still be blocked in read(), which never returns null.
+ readerExecutor.shutdownNow();
+ }
+ if (source != null) {
+ try {
+ source.close();
+ } catch (Exception e) {
+ log.warn("Failed to close source", e);
+ }
+ }
+ if (client != null) {
+ try {
+ client.deleteStream(builder -> builder.streamName(STREAM_NAME)).get();
+ } catch (Exception e) {
+ log.warn("Failed to delete stream", e);
+ }
+ client.close();
+ }
+ }
+
+ @Test(timeOut = 600_000)
+ public void testReadFromKinesisStream() throws Exception {
+ // Seed some records before the source starts; TRIM_HORIZON makes the worker replay them.
+ for (int i = 0; i < EXPECTED_RECORDS; i++) {
+ putRecord("seed-" + i);
+ }
+
+ // Keep writing after open as well so the worker has a steady supply regardless of exactly
+ // when its lease/shard bootstrap completes.
+ startBackgroundWriter();
+
+ SourceContext sourceContext = mock(SourceContext.class);
+ source.open(buildConfig(), sourceContext);
+
+ int received = 0;
+ for (int i = 0; i < EXPECTED_RECORDS; i++) {
+ KinesisRecord record = readOne();
+ assertNotNull(record, "read() returned null");
+ assertNotNull(record.getValue(), "record had no value");
+ assertTrue(record.getValue().length > 0, "record value was empty");
+ assertTrue(record.getKey().isPresent(), "record had no key (partition key)");
+ assertTrue(record.getProperties().containsKey(KinesisRecord.SEQUENCE_NUMBER),
+ "record missing SEQUENCE_NUMBER property; got " + record.getProperties());
+ log.info("Received Kinesis record: key={} value={} seq={}",
+ record.getKey().orElse(null),
+ new String(record.getValue(), StandardCharsets.UTF_8),
+ record.getProperties().get(KinesisRecord.SEQUENCE_NUMBER));
+ received++;
+ }
+ assertTrue(received >= EXPECTED_RECORDS,
+ "Expected at least " + EXPECTED_RECORDS + " records, got " + received);
+ }
+
+ /**
+ * Reads a single record on a bounded worker thread, failing (rather than hanging) if none
+ * arrives in time. {@link KinesisSource#read()} blocks forever on an empty queue, so it must
+ * never be called on the test thread nor inside an Awaitility assertion.
+ */
+ private KinesisRecord readOne() throws Exception {
+ Future<KinesisRecord> future = readerExecutor.submit(() -> source.read());
+ try {
+ return future.get(READ_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+ } catch (TimeoutException e) {
+ future.cancel(true);
+ throw new AssertionError("Timed out after " + READ_TIMEOUT_SECONDS
+ + "s waiting for a record from the Kinesis stream. The source produced no "
+ + "record; see the KCL/worker logs above.", e);
+ }
+ }
+
+ private void startBackgroundWriter() {
+ keepWriting.set(true);
+ final AtomicInteger counter = new AtomicInteger();
+ writerThread = new Thread(() -> {
+ while (keepWriting.get()) {
+ try {
+ putRecord("live-" + counter.incrementAndGet());
+ Thread.sleep(1000);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ } catch (Exception e) {
+ log.warn("Background write failed", e);
+ }
+ }
+ }, "kinesis-it-writer");
+ writerThread.setDaemon(true);
+ writerThread.start();
+ }
+
+ private void putRecord(String id) throws Exception {
+ client.putRecord(PutRecordRequest.builder()
+ .streamName(STREAM_NAME)
+ .partitionKey("pk-" + id)
+ .data(SdkBytes.fromByteArray(("value-" + id).getBytes(StandardCharsets.UTF_8)))
+ .build()).get();
+ }
+
+ private Map<String, Object> buildConfig() {
+ Map<String, Object> config = new HashMap<>();
+ // The source config applies each endpoint via URI.create; LocalStack serves every service on
+ // one edge port, so the Kinesis endpoint URI works for the DynamoDB (lease table) and
+ // CloudWatch (metrics) clients too. All three must be set to reach LocalStack.
+ config.put("awsEndpoint", endpoint());
+ config.put("dynamoEndpoint", endpoint());
+ config.put("cloudwatchEndpoint", endpoint());
+ config.put("awsRegion", REGION);
+ config.put("awsKinesisStreamName", STREAM_NAME);
+ config.put("awsCredentialPluginParam",
+ "{\"accessKey\":\"" + ACCESS_KEY + "\",\"secretKey\":\"" + SECRET_KEY + "\"}");
+ // Replay from the start of the stream so the seeded records are delivered.
+ config.put("initialPositionInStream", "TRIM_HORIZON");
+ // Unique application name -> unique KCL lease table per run, avoiding stale leases.
+ config.put("applicationName", "pulsar-kinesis-it-" + System.currentTimeMillis());
+ // Enhanced fan-out (SubscribeToShard) is unreliable on LocalStack; use polling instead.
+ config.put("useEnhancedFanOut", false);
+ return config;
+ }
+
+ private KinesisAsyncClient createClient() {
+ return KinesisAsyncClient.builder()
+ .credentialsProvider(new AwsCredentialsProvider() {
+ @Override
+ public AwsCredentials resolveCredentials() {
+ return AwsBasicCredentials.create(ACCESS_KEY, SECRET_KEY);
+ }
+ })
+ .region(Region.US_EAST_1)
+ .endpointOverride(LOCAL_STACK_CONTAINER.getEndpointOverride(LocalStackContainer.Service.KINESIS))
+ .build();
Review Comment:
Fixed in 88f6eef — createClient() now uses Region.of(REGION) instead of hard-coding US_EAST_1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][test] Add integration tests for the Kinesis source connector [pulsar-connectors]
Posted by "david-streamlio (via GitHub)" <gi...@apache.org>.
david-streamlio merged PR #123:
URL: https://github.com/apache/pulsar-connectors/pull/123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][test] Add integration tests for the Kinesis source connector [pulsar-connectors]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #123:
URL: https://github.com/apache/pulsar-connectors/pull/123#discussion_r3606783858
##########
kinesis/src/test/java/org/apache/pulsar/io/kinesis/KinesisSourceIntegrationTest.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.io.kinesis;
+
+import static org.mockito.Mockito.mock;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertTrue;
+import java.net.URI;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.io.core.SourceContext;
+import org.awaitility.Awaitility;
+import org.testcontainers.containers.localstack.LocalStackContainer;
+import org.testcontainers.utility.DockerImageName;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisAsyncClient;
+import software.amazon.awssdk.services.kinesis.model.CreateStreamRequest;
+import software.amazon.awssdk.services.kinesis.model.DescribeStreamRequest;
+import software.amazon.awssdk.services.kinesis.model.PutRecordRequest;
+import software.amazon.awssdk.services.kinesis.model.StreamStatus;
+
+/**
+ * Integration test for {@link KinesisSource} that exercises the full path from a real Kinesis
+ * stream through the KCL (Kinesis Client Library) worker and into the source's record queue. A
+ * LocalStack container provides Kinesis, DynamoDB and CloudWatch (the KCL uses DynamoDB for its
+ * lease table and CloudWatch for metrics).
+ */
+@Slf4j
+public class KinesisSourceIntegrationTest {
+
+ private static final String REGION = "us-east-1";
+ private static final String ACCESS_KEY = "access";
+ private static final String SECRET_KEY = "secret";
+ private static final String STREAM_NAME = "pulsar-kinesis-source-it";
+
+ // KCL against LocalStack is slow to bootstrap (lease table creation + shard discovery), so the
+ // first record can take well over a minute to surface. Give each read a generous deadline while
+ // keeping the overall test bounded so an under-delivering source fails promptly rather than hangs.
+ private static final int READ_TIMEOUT_SECONDS = 240;
+ private static final int EXPECTED_RECORDS = 3;
+
+ public static final LocalStackContainer LOCAL_STACK_CONTAINER =
+ new LocalStackContainer(DockerImageName.parse("localstack/localstack:4.0.3"))
+ .withServices(
+ LocalStackContainer.Service.KINESIS,
+ LocalStackContainer.Service.DYNAMODB,
+ LocalStackContainer.Service.CLOUDWATCH)
+ .withEnv("KINESIS_PROVIDER", "kinesalite")
+ .withStartupTimeout(Duration.ofMinutes(5));
+
+ private KinesisAsyncClient client;
+ private KinesisSource source;
+ private ExecutorService readerExecutor;
+ private Thread writerThread;
+ private final AtomicBoolean keepWriting = new AtomicBoolean(false);
+
+ @BeforeClass(alwaysRun = true)
+ public void beforeClass() {
+ LOCAL_STACK_CONTAINER.start();
+ }
+
+ @AfterClass(alwaysRun = true)
+ public void afterClass() {
+ LOCAL_STACK_CONTAINER.stop();
+ }
+
+ @BeforeMethod
+ public void setup() throws Exception {
+ readerExecutor = Executors.newSingleThreadExecutor(r -> {
+ Thread t = new Thread(r, "kinesis-it-reader");
+ t.setDaemon(true);
+ return t;
+ });
+
+ client = createClient();
+ client.createStream(CreateStreamRequest.builder()
+ .streamName(STREAM_NAME)
+ .shardCount(1)
+ .build()).get();
+
+ Awaitility.await().atMost(60, TimeUnit.SECONDS).until(() ->
+ client.describeStream(DescribeStreamRequest.builder().streamName(STREAM_NAME).build())
+ .get().streamDescription().streamStatus() == StreamStatus.ACTIVE);
Review Comment:
The stream creation / activation check uses CompletableFuture.get() without a timeout. TestNG's @Test(timeOut=...) does not apply to @BeforeMethod, and Awaitility's atMost won't help if the supplier blocks in get(), so a stuck LocalStack/Kinesis call can hang the test suite indefinitely.
Use time-bounded get(...) calls for both createStream and describeStream.
##########
kinesis/src/test/java/org/apache/pulsar/io/kinesis/KinesisSourceIntegrationTest.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.io.kinesis;
+
+import static org.mockito.Mockito.mock;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertTrue;
+import java.net.URI;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.io.core.SourceContext;
+import org.awaitility.Awaitility;
+import org.testcontainers.containers.localstack.LocalStackContainer;
+import org.testcontainers.utility.DockerImageName;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisAsyncClient;
+import software.amazon.awssdk.services.kinesis.model.CreateStreamRequest;
+import software.amazon.awssdk.services.kinesis.model.DescribeStreamRequest;
+import software.amazon.awssdk.services.kinesis.model.PutRecordRequest;
+import software.amazon.awssdk.services.kinesis.model.StreamStatus;
+
+/**
+ * Integration test for {@link KinesisSource} that exercises the full path from a real Kinesis
+ * stream through the KCL (Kinesis Client Library) worker and into the source's record queue. A
+ * LocalStack container provides Kinesis, DynamoDB and CloudWatch (the KCL uses DynamoDB for its
+ * lease table and CloudWatch for metrics).
+ */
+@Slf4j
+public class KinesisSourceIntegrationTest {
+
+ private static final String REGION = "us-east-1";
+ private static final String ACCESS_KEY = "access";
+ private static final String SECRET_KEY = "secret";
+ private static final String STREAM_NAME = "pulsar-kinesis-source-it";
+
+ // KCL against LocalStack is slow to bootstrap (lease table creation + shard discovery), so the
+ // first record can take well over a minute to surface. Give each read a generous deadline while
+ // keeping the overall test bounded so an under-delivering source fails promptly rather than hangs.
+ private static final int READ_TIMEOUT_SECONDS = 240;
+ private static final int EXPECTED_RECORDS = 3;
+
+ public static final LocalStackContainer LOCAL_STACK_CONTAINER =
+ new LocalStackContainer(DockerImageName.parse("localstack/localstack:4.0.3"))
+ .withServices(
+ LocalStackContainer.Service.KINESIS,
+ LocalStackContainer.Service.DYNAMODB,
+ LocalStackContainer.Service.CLOUDWATCH)
+ .withEnv("KINESIS_PROVIDER", "kinesalite")
+ .withStartupTimeout(Duration.ofMinutes(5));
+
+ private KinesisAsyncClient client;
+ private KinesisSource source;
+ private ExecutorService readerExecutor;
+ private Thread writerThread;
+ private final AtomicBoolean keepWriting = new AtomicBoolean(false);
+
+ @BeforeClass(alwaysRun = true)
+ public void beforeClass() {
+ LOCAL_STACK_CONTAINER.start();
+ }
+
+ @AfterClass(alwaysRun = true)
+ public void afterClass() {
+ LOCAL_STACK_CONTAINER.stop();
+ }
+
+ @BeforeMethod
+ public void setup() throws Exception {
+ readerExecutor = Executors.newSingleThreadExecutor(r -> {
+ Thread t = new Thread(r, "kinesis-it-reader");
+ t.setDaemon(true);
+ return t;
+ });
+
+ client = createClient();
+ client.createStream(CreateStreamRequest.builder()
+ .streamName(STREAM_NAME)
+ .shardCount(1)
+ .build()).get();
+
+ Awaitility.await().atMost(60, TimeUnit.SECONDS).until(() ->
+ client.describeStream(DescribeStreamRequest.builder().streamName(STREAM_NAME).build())
+ .get().streamDescription().streamStatus() == StreamStatus.ACTIVE);
+ log.info("Created Kinesis stream {}", STREAM_NAME);
+
+ source = new KinesisSource();
+ }
+
+ @AfterMethod(alwaysRun = true)
+ public void cleanup() {
+ keepWriting.set(false);
+ if (writerThread != null) {
+ writerThread.interrupt();
+ }
+ if (readerExecutor != null) {
+ // A reader may still be blocked in read(), which never returns null.
+ readerExecutor.shutdownNow();
+ }
+ if (source != null) {
+ try {
+ source.close();
+ } catch (Exception e) {
+ log.warn("Failed to close source", e);
+ }
+ }
+ if (client != null) {
+ try {
+ client.deleteStream(builder -> builder.streamName(STREAM_NAME)).get();
+ } catch (Exception e) {
+ log.warn("Failed to delete stream", e);
+ }
+ client.close();
+ }
+ }
+
+ @Test(timeOut = 600_000)
+ public void testReadFromKinesisStream() throws Exception {
+ // Seed some records before the source starts; TRIM_HORIZON makes the worker replay them.
+ for (int i = 0; i < EXPECTED_RECORDS; i++) {
+ putRecord("seed-" + i);
+ }
+
+ // Keep writing after open as well so the worker has a steady supply regardless of exactly
+ // when its lease/shard bootstrap completes.
+ startBackgroundWriter();
+
+ SourceContext sourceContext = mock(SourceContext.class);
+ source.open(buildConfig(), sourceContext);
+
+ int received = 0;
+ for (int i = 0; i < EXPECTED_RECORDS; i++) {
+ KinesisRecord record = readOne();
+ assertNotNull(record, "read() returned null");
+ assertNotNull(record.getValue(), "record had no value");
+ assertTrue(record.getValue().length > 0, "record value was empty");
+ assertTrue(record.getKey().isPresent(), "record had no key (partition key)");
+ assertTrue(record.getProperties().containsKey(KinesisRecord.SEQUENCE_NUMBER),
+ "record missing SEQUENCE_NUMBER property; got " + record.getProperties());
+ log.info("Received Kinesis record: key={} value={} seq={}",
+ record.getKey().orElse(null),
+ new String(record.getValue(), StandardCharsets.UTF_8),
+ record.getProperties().get(KinesisRecord.SEQUENCE_NUMBER));
+ received++;
+ }
+ assertTrue(received >= EXPECTED_RECORDS,
+ "Expected at least " + EXPECTED_RECORDS + " records, got " + received);
+ }
Review Comment:
The final `assertTrue(received >= EXPECTED_RECORDS, ...)` is redundant because `received` is incremented once per iteration in a `for (int i = 0; i < EXPECTED_RECORDS; i++)` loop. If the loop completes, `received` is always exactly EXPECTED_RECORDS.
Dropping the counter + redundant assertion makes the test easier to read and avoids dead code paths.
##########
kinesis/src/test/java/org/apache/pulsar/io/kinesis/KinesisSourceIntegrationTest.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.io.kinesis;
+
+import static org.mockito.Mockito.mock;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertTrue;
+import java.net.URI;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.io.core.SourceContext;
+import org.awaitility.Awaitility;
+import org.testcontainers.containers.localstack.LocalStackContainer;
+import org.testcontainers.utility.DockerImageName;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisAsyncClient;
+import software.amazon.awssdk.services.kinesis.model.CreateStreamRequest;
+import software.amazon.awssdk.services.kinesis.model.DescribeStreamRequest;
+import software.amazon.awssdk.services.kinesis.model.PutRecordRequest;
+import software.amazon.awssdk.services.kinesis.model.StreamStatus;
+
+/**
+ * Integration test for {@link KinesisSource} that exercises the full path from a real Kinesis
+ * stream through the KCL (Kinesis Client Library) worker and into the source's record queue. A
+ * LocalStack container provides Kinesis, DynamoDB and CloudWatch (the KCL uses DynamoDB for its
+ * lease table and CloudWatch for metrics).
+ */
+@Slf4j
+public class KinesisSourceIntegrationTest {
+
+ private static final String REGION = "us-east-1";
+ private static final String ACCESS_KEY = "access";
+ private static final String SECRET_KEY = "secret";
+ private static final String STREAM_NAME = "pulsar-kinesis-source-it";
+
+ // KCL against LocalStack is slow to bootstrap (lease table creation + shard discovery), so the
+ // first record can take well over a minute to surface. Give each read a generous deadline while
+ // keeping the overall test bounded so an under-delivering source fails promptly rather than hangs.
+ private static final int READ_TIMEOUT_SECONDS = 240;
+ private static final int EXPECTED_RECORDS = 3;
+
+ public static final LocalStackContainer LOCAL_STACK_CONTAINER =
+ new LocalStackContainer(DockerImageName.parse("localstack/localstack:4.0.3"))
+ .withServices(
+ LocalStackContainer.Service.KINESIS,
+ LocalStackContainer.Service.DYNAMODB,
+ LocalStackContainer.Service.CLOUDWATCH)
+ .withEnv("KINESIS_PROVIDER", "kinesalite")
+ .withStartupTimeout(Duration.ofMinutes(5));
+
+ private KinesisAsyncClient client;
+ private KinesisSource source;
+ private ExecutorService readerExecutor;
+ private Thread writerThread;
+ private final AtomicBoolean keepWriting = new AtomicBoolean(false);
+
+ @BeforeClass(alwaysRun = true)
+ public void beforeClass() {
+ LOCAL_STACK_CONTAINER.start();
+ }
+
+ @AfterClass(alwaysRun = true)
+ public void afterClass() {
+ LOCAL_STACK_CONTAINER.stop();
+ }
+
+ @BeforeMethod
+ public void setup() throws Exception {
+ readerExecutor = Executors.newSingleThreadExecutor(r -> {
+ Thread t = new Thread(r, "kinesis-it-reader");
+ t.setDaemon(true);
+ return t;
+ });
+
+ client = createClient();
+ client.createStream(CreateStreamRequest.builder()
+ .streamName(STREAM_NAME)
+ .shardCount(1)
+ .build()).get();
+
+ Awaitility.await().atMost(60, TimeUnit.SECONDS).until(() ->
+ client.describeStream(DescribeStreamRequest.builder().streamName(STREAM_NAME).build())
+ .get().streamDescription().streamStatus() == StreamStatus.ACTIVE);
+ log.info("Created Kinesis stream {}", STREAM_NAME);
+
+ source = new KinesisSource();
+ }
+
+ @AfterMethod(alwaysRun = true)
+ public void cleanup() {
+ keepWriting.set(false);
+ if (writerThread != null) {
+ writerThread.interrupt();
+ }
+ if (readerExecutor != null) {
+ // A reader may still be blocked in read(), which never returns null.
+ readerExecutor.shutdownNow();
+ }
+ if (source != null) {
+ try {
+ source.close();
+ } catch (Exception e) {
+ log.warn("Failed to close source", e);
+ }
+ }
+ if (client != null) {
+ try {
+ client.deleteStream(builder -> builder.streamName(STREAM_NAME)).get();
+ } catch (Exception e) {
+ log.warn("Failed to delete stream", e);
+ }
+ client.close();
+ }
+ }
+
+ @Test(timeOut = 600_000)
+ public void testReadFromKinesisStream() throws Exception {
+ // Seed some records before the source starts; TRIM_HORIZON makes the worker replay them.
+ for (int i = 0; i < EXPECTED_RECORDS; i++) {
+ putRecord("seed-" + i);
+ }
+
+ // Keep writing after open as well so the worker has a steady supply regardless of exactly
+ // when its lease/shard bootstrap completes.
+ startBackgroundWriter();
+
+ SourceContext sourceContext = mock(SourceContext.class);
+ source.open(buildConfig(), sourceContext);
+
+ int received = 0;
+ for (int i = 0; i < EXPECTED_RECORDS; i++) {
+ KinesisRecord record = readOne();
+ assertNotNull(record, "read() returned null");
+ assertNotNull(record.getValue(), "record had no value");
+ assertTrue(record.getValue().length > 0, "record value was empty");
+ assertTrue(record.getKey().isPresent(), "record had no key (partition key)");
+ assertTrue(record.getProperties().containsKey(KinesisRecord.SEQUENCE_NUMBER),
+ "record missing SEQUENCE_NUMBER property; got " + record.getProperties());
+ log.info("Received Kinesis record: key={} value={} seq={}",
+ record.getKey().orElse(null),
+ new String(record.getValue(), StandardCharsets.UTF_8),
+ record.getProperties().get(KinesisRecord.SEQUENCE_NUMBER));
Review Comment:
The test currently only asserts that the key/value are present and non-empty, but it doesn't validate that the emitted key/value correspond to what was written to the stream (partition key `pk-<id>` and payload `value-<id>`). That leaves a lot of room for false positives (e.g., wrong deserialization or key mapping).
Consider asserting that the record's key/value follow the expected relationship, regardless of whether the record came from the seeded or background-writer traffic.
##########
kinesis/src/test/java/org/apache/pulsar/io/kinesis/KinesisSourceIntegrationTest.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.io.kinesis;
+
+import static org.mockito.Mockito.mock;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertTrue;
+import java.net.URI;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.io.core.SourceContext;
+import org.awaitility.Awaitility;
+import org.testcontainers.containers.localstack.LocalStackContainer;
+import org.testcontainers.utility.DockerImageName;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisAsyncClient;
+import software.amazon.awssdk.services.kinesis.model.CreateStreamRequest;
+import software.amazon.awssdk.services.kinesis.model.DescribeStreamRequest;
+import software.amazon.awssdk.services.kinesis.model.PutRecordRequest;
+import software.amazon.awssdk.services.kinesis.model.StreamStatus;
+
+/**
+ * Integration test for {@link KinesisSource} that exercises the full path from a real Kinesis
+ * stream through the KCL (Kinesis Client Library) worker and into the source's record queue. A
+ * LocalStack container provides Kinesis, DynamoDB and CloudWatch (the KCL uses DynamoDB for its
+ * lease table and CloudWatch for metrics).
+ */
+@Slf4j
+public class KinesisSourceIntegrationTest {
+
+ private static final String REGION = "us-east-1";
+ private static final String ACCESS_KEY = "access";
+ private static final String SECRET_KEY = "secret";
+ private static final String STREAM_NAME = "pulsar-kinesis-source-it";
+
+ // KCL against LocalStack is slow to bootstrap (lease table creation + shard discovery), so the
+ // first record can take well over a minute to surface. Give each read a generous deadline while
+ // keeping the overall test bounded so an under-delivering source fails promptly rather than hangs.
+ private static final int READ_TIMEOUT_SECONDS = 240;
+ private static final int EXPECTED_RECORDS = 3;
+
+ public static final LocalStackContainer LOCAL_STACK_CONTAINER =
+ new LocalStackContainer(DockerImageName.parse("localstack/localstack:4.0.3"))
+ .withServices(
+ LocalStackContainer.Service.KINESIS,
+ LocalStackContainer.Service.DYNAMODB,
+ LocalStackContainer.Service.CLOUDWATCH)
+ .withEnv("KINESIS_PROVIDER", "kinesalite")
+ .withStartupTimeout(Duration.ofMinutes(5));
+
+ private KinesisAsyncClient client;
+ private KinesisSource source;
+ private ExecutorService readerExecutor;
+ private Thread writerThread;
+ private final AtomicBoolean keepWriting = new AtomicBoolean(false);
+
+ @BeforeClass(alwaysRun = true)
+ public void beforeClass() {
+ LOCAL_STACK_CONTAINER.start();
+ }
+
+ @AfterClass(alwaysRun = true)
+ public void afterClass() {
+ LOCAL_STACK_CONTAINER.stop();
+ }
+
+ @BeforeMethod
+ public void setup() throws Exception {
+ readerExecutor = Executors.newSingleThreadExecutor(r -> {
+ Thread t = new Thread(r, "kinesis-it-reader");
+ t.setDaemon(true);
+ return t;
+ });
+
+ client = createClient();
+ client.createStream(CreateStreamRequest.builder()
+ .streamName(STREAM_NAME)
+ .shardCount(1)
+ .build()).get();
+
+ Awaitility.await().atMost(60, TimeUnit.SECONDS).until(() ->
+ client.describeStream(DescribeStreamRequest.builder().streamName(STREAM_NAME).build())
+ .get().streamDescription().streamStatus() == StreamStatus.ACTIVE);
+ log.info("Created Kinesis stream {}", STREAM_NAME);
+
+ source = new KinesisSource();
+ }
+
+ @AfterMethod(alwaysRun = true)
+ public void cleanup() {
+ keepWriting.set(false);
+ if (writerThread != null) {
+ writerThread.interrupt();
+ }
+ if (readerExecutor != null) {
+ // A reader may still be blocked in read(), which never returns null.
+ readerExecutor.shutdownNow();
+ }
+ if (source != null) {
+ try {
+ source.close();
+ } catch (Exception e) {
+ log.warn("Failed to close source", e);
+ }
+ }
+ if (client != null) {
+ try {
+ client.deleteStream(builder -> builder.streamName(STREAM_NAME)).get();
+ } catch (Exception e) {
+ log.warn("Failed to delete stream", e);
+ }
+ client.close();
+ }
+ }
+
+ @Test(timeOut = 600_000)
+ public void testReadFromKinesisStream() throws Exception {
+ // Seed some records before the source starts; TRIM_HORIZON makes the worker replay them.
+ for (int i = 0; i < EXPECTED_RECORDS; i++) {
+ putRecord("seed-" + i);
+ }
+
+ // Keep writing after open as well so the worker has a steady supply regardless of exactly
+ // when its lease/shard bootstrap completes.
+ startBackgroundWriter();
+
+ SourceContext sourceContext = mock(SourceContext.class);
+ source.open(buildConfig(), sourceContext);
+
+ int received = 0;
+ for (int i = 0; i < EXPECTED_RECORDS; i++) {
+ KinesisRecord record = readOne();
+ assertNotNull(record, "read() returned null");
+ assertNotNull(record.getValue(), "record had no value");
+ assertTrue(record.getValue().length > 0, "record value was empty");
+ assertTrue(record.getKey().isPresent(), "record had no key (partition key)");
+ assertTrue(record.getProperties().containsKey(KinesisRecord.SEQUENCE_NUMBER),
+ "record missing SEQUENCE_NUMBER property; got " + record.getProperties());
+ log.info("Received Kinesis record: key={} value={} seq={}",
+ record.getKey().orElse(null),
+ new String(record.getValue(), StandardCharsets.UTF_8),
+ record.getProperties().get(KinesisRecord.SEQUENCE_NUMBER));
+ received++;
+ }
+ assertTrue(received >= EXPECTED_RECORDS,
+ "Expected at least " + EXPECTED_RECORDS + " records, got " + received);
+ }
+
+ /**
+ * Reads a single record on a bounded worker thread, failing (rather than hanging) if none
+ * arrives in time. {@link KinesisSource#read()} blocks forever on an empty queue, so it must
+ * never be called on the test thread nor inside an Awaitility assertion.
+ */
+ private KinesisRecord readOne() throws Exception {
+ Future<KinesisRecord> future = readerExecutor.submit(() -> source.read());
+ try {
+ return future.get(READ_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+ } catch (TimeoutException e) {
+ future.cancel(true);
+ throw new AssertionError("Timed out after " + READ_TIMEOUT_SECONDS
+ + "s waiting for a record from the Kinesis stream. The source produced no "
+ + "record; see the KCL/worker logs above.", e);
+ }
+ }
+
+ private void startBackgroundWriter() {
+ keepWriting.set(true);
+ final AtomicInteger counter = new AtomicInteger();
+ writerThread = new Thread(() -> {
+ while (keepWriting.get()) {
+ try {
+ putRecord("live-" + counter.incrementAndGet());
+ Thread.sleep(1000);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ } catch (Exception e) {
+ log.warn("Background write failed", e);
+ }
+ }
+ }, "kinesis-it-writer");
+ writerThread.setDaemon(true);
+ writerThread.start();
+ }
+
+ private void putRecord(String id) throws Exception {
+ client.putRecord(PutRecordRequest.builder()
+ .streamName(STREAM_NAME)
+ .partitionKey("pk-" + id)
+ .data(SdkBytes.fromByteArray(("value-" + id).getBytes(StandardCharsets.UTF_8)))
+ .build()).get();
+ }
+
+ private Map<String, Object> buildConfig() {
+ Map<String, Object> config = new HashMap<>();
+ // The source config applies each endpoint via URI.create; LocalStack serves every service on
+ // one edge port, so the Kinesis endpoint URI works for the DynamoDB (lease table) and
+ // CloudWatch (metrics) clients too. All three must be set to reach LocalStack.
+ config.put("awsEndpoint", endpoint());
+ config.put("dynamoEndpoint", endpoint());
+ config.put("cloudwatchEndpoint", endpoint());
+ config.put("awsRegion", REGION);
+ config.put("awsKinesisStreamName", STREAM_NAME);
+ config.put("awsCredentialPluginParam",
+ "{\"accessKey\":\"" + ACCESS_KEY + "\",\"secretKey\":\"" + SECRET_KEY + "\"}");
+ // Replay from the start of the stream so the seeded records are delivered.
+ config.put("initialPositionInStream", "TRIM_HORIZON");
+ // Unique application name -> unique KCL lease table per run, avoiding stale leases.
+ config.put("applicationName", "pulsar-kinesis-it-" + System.currentTimeMillis());
+ // Enhanced fan-out (SubscribeToShard) is unreliable on LocalStack; use polling instead.
+ config.put("useEnhancedFanOut", false);
+ return config;
+ }
+
+ private KinesisAsyncClient createClient() {
+ return KinesisAsyncClient.builder()
+ .credentialsProvider(new AwsCredentialsProvider() {
+ @Override
+ public AwsCredentials resolveCredentials() {
+ return AwsBasicCredentials.create(ACCESS_KEY, SECRET_KEY);
+ }
+ })
+ .region(Region.US_EAST_1)
+ .endpointOverride(LOCAL_STACK_CONTAINER.getEndpointOverride(LocalStackContainer.Service.KINESIS))
+ .build();
Review Comment:
`REGION` is defined as a constant but `createClient()` hard-codes `Region.US_EAST_1`. Using the constant in both places avoids accidental divergence if the test region ever changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org